When I get a ticket that starts with "we think we've been breached," the first 48 hours matter more than anything else. Most teams waste that window making predictable mistakes that destroy evidence, leave backdoors open, or create legal exposure.
You need a plan before the breach happens, not after. Here are the nine mistakes I see repeatedly and what to do instead.
Mistake 1: Starting with password resets
People panic and reset every password immediately. This alerts the attacker that you know about the breach. If they have persistence mechanisms in place — and they usually do — they'll either destroy evidence or pivot to exfiltrate data faster.
What to do instead: Document everything first. Take snapshots of running processes, network connections, and open files before you touch anything. Your goal in hour one is visibility, not remediation.
# Capture current state before any changes
ps auxf > /var/log/breach-ps-$(date +%s).txt
netstat -tunap > /var/log/breach-netstat-$(date +%s).txt
lsof > /var/log/breach-lsof-$(date +%s).txt
ss -tulpn > /var/log/breach-ss-$(date +%s).txt
Password resets come later, after you've closed the entry point and removed persistence. Resetting credentials while backdoors are still active wastes time.
Mistake 2: Taking servers offline immediately
Someone shouts "pull the plug" and the entire production environment goes dark. Yes, you've contained the breach. You've also destroyed volatile evidence in RAM, killed your ability to trace the attacker's current activity, and potentially triggered data destruction scripts.
What to do instead: Isolate, don't shutdown. Block outbound traffic at the firewall level first, then inbound. This keeps systems running for forensic capture while preventing further damage.
# Block outbound first to stop data exfil
iptables -I OUTPUT -j DROP
iptables -I OUTPUT -d 127.0.0.1 -j ACCEPT
iptables -I OUTPUT -o lo -j ACCEPT
# Then block inbound except from forensic workstation
iptables -I INPUT -j DROP
iptables -I INPUT -s YOUR_FORENSIC_IP -j ACCEPT
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
Capture a memory dump before any reboot. Tools like LiME or avml can do this, but even dd if=/dev/mem is better than nothing if you have no other option.
Mistake 3: Assuming backups are clean
Restoring from backup feels like the fastest path to recovery. Except the breach might have started weeks or months ago, and your backups could contain the same malware, the same compromised accounts, or the same vulnerable configuration that let attackers in originally.
I've seen teams restore a compromised WordPress installation three times in a week because they kept using the same infected backup.
What to do instead: Test a backup restore in an isolated environment first. Scan it. Check file modification times against your breach timeline. If your attacker had access for 60 days and your backup retention is 30 days, every backup you have is suspect.
Build from a known-good state — original installation media, fresh OS image, vetted application code — rather than restoring blindly. Yes, it takes longer. It also works.
Mistake 4: Running AV scans on a live compromised system
You install or update antivirus software on the compromised server and run a full scan. Two problems: first, the attacker sees the scan running and can react. Second, modern rootkits can hide from AV scans running on the same kernel they've compromised.
What to do instead: Take filesystem snapshots or disk images and scan them from a separate clean system. If you must scan in place, at least boot from external media.
# Create read-only snapshot before scanning
lvcreate -L 10G -s -n root-snapshot /dev/vg0/root
mount -o ro /dev/vg0/root-snapshot /mnt/snapshot
Better yet, use YARA rules or custom scripts to hunt for specific indicators rather than relying on signature-based detection. Attackers know what AV products catch.
Mistake 5: Failing to preserve logs before they rotate
Most logging configs rotate or compress logs after a certain size or time period. If you don't copy them immediately, you'll lose the entries from the initial compromise. I've watched critical auth.log entries disappear during an investigation because someone forgot to disable logrotate.
What to do instead: Copy all logs to write-once storage the moment you suspect a breach. Check for evidence of log tampering — suddenly small auth.log files or gaps in timestamps are red flags.
# Copy logs preserving timestamps
cp -pr /var/log /forensics/logs-$(hostname)-$(date +%s)
# Disable log rotation during investigation
systemctl stop logrotate.timer
systemctl disable logrotate.timer
# Check for tampered logs
for log in /var/log/auth.log* /var/log/secure*; do
stat "$log" | grep -E "Modify|Change"
done
Also check for cleared command history, modified utmp/wtmp files, and any logs that are suspiciously small for the server's age.
What about legal and compliance requirements?
Here's where many technical people stumble. They focus entirely on the technical response and forget that breach notification laws have specific timelines. GDPR gives you 72 hours in many cases. State laws vary. Your contracts probably have notification clauses.
What to do instead: Loop in legal and compliance within the first four hours, not the first 40. They need time to assess notification requirements while you're still doing technical triage. Keep a timestamped log of every action you take — it'll matter later.
Designate one person to handle external communication. Everyone else stays heads-down on technical response. Mixed messages to customers, partners, or regulators create bigger problems than the breach itself.
Mistake 6: Treating all systems as equally important
When everything's on fire, people try to fix everything at once. You can't. Some systems matter more — database servers holding customer data, authentication servers, backup infrastructure, logging hosts. Others can stay broken for a day or two.
What to do instead: Triage ruthlessly. In the first six hours, focus only on systems that contain sensitive data or control access to other systems. Document the rest and come back to them.
Create a priority matrix:
- Active data exfiltration in progress
- Systems with customer PII or payment data
- Authentication and identity infrastructure
- Compromised admin accounts
- Everything else
If the marketing site is defaced but the database is intact, the database gets your attention first. Optics matter less than actual harm.
Mistake 7: Working without a communication plan
Three people are SSHed into the same server making changes. Nobody knows what anyone else is doing. Someone reboots a system another person was actively investigating. Commands collide. Evidence gets trampled.
What to do instead: Use a central coordination channel — Slack room, Discord server, even a shared tmux session with a dedicated log file. Every action gets announced before execution.
# Shared terminal logging
script -a /var/log/breach-response-team.log
# Now all commands are logged for the team
Assign roles: one person handles forensics, one handles containment, one handles documentation and timeline, one manages communication with stakeholders. Rotate if the incident runs past 12 hours. Exhausted people make mistakes.
Mistake 8: Trusting system binaries on a compromised host
You run ps, netstat, or ls to investigate, not realizing the attacker replaced them with modified versions that hide malicious processes. Rootkits do this routinely. You're looking at a filtered view of reality.
What to do instead: Bring your own known-good binaries on read-only media. Use statically compiled forensic tools or run everything from a live USB distribution.
# Use statically compiled busybox instead of system binaries
mount -o ro /dev/sdb1 /mnt/forensic-usb
/mnt/forensic-usb/busybox ps
/mnt/forensic-usb/busybox netstat -tulpn
/mnt/forensic-usb/busybox find / -mtime -7
Compare output from system binaries against your known-good tools. If they differ, you've found a rootkit.
Mistake 9: Declaring victory too early
The obvious backdoor is closed, the malware is deleted, passwords are reset. Everyone breathes easier and moves on. Then three weeks later the attacker is back because you missed a persistence mechanism.
In tickets I handled, the most common overlooked items were: cron jobs, systemd timers, modified .bashrc or .profile files, SSH authorized_keys, database stored procedures, and web application auto-update mechanisms that pulled from attacker-controlled repositories.
What to do instead: Hunt for persistence at multiple levels. Check every place code can execute automatically.
# Common persistence locations
find /etc/cron* /var/spool/cron -type f -ls
systemctl list-timers --all
grep -r "authorized_keys" /home /root
find /etc -name "*.d" -type d
for user in $(cut -f1 -d: /etc/passwd); do
crontab -u $user -l 2>/dev/null
done
Run a clean install in parallel and compare every config file, every installed package, every service. Difference analysis catches what checklists miss.
Keep enhanced monitoring active for at least 30 days post-incident. File integrity monitoring, full packet capture on critical segments, and aggressive log alerting. Assume the attacker will try to return.
What to check first
When you get the call, these five actions form your first hour:
Capture volatile data — memory, network connections, running processes. Isolate affected systems at the network level. Copy all logs to safe storage. Document your timeline with precise timestamps. Brief your legal and management team.
Skip the dramatic remediation. Build situational awareness first. The worst breach responses are fast, not thorough.