Most hardening guides tell you to disable root login and run updates. You already did that. This one covers the steps that actually separate a locked-down production server from one that passes a compliance scan but folds under a targeted attack.
I'm assuming you manage production Linux boxes—VPS, bare metal, or cloud instances—and you've handled the SSH key setup, firewall basics, and automatic patching. What follows are the configuration layers that absorb post-compromise attempts, limit lateral movement, and give you forensic data when something does break in.
Enforce mandatory access control at the kernel level
Discretionary access controls stop accidents. Mandatory access controls stop attackers who already have a shell.
SELinux and AppArmor both enforce policy at the kernel level, meaning even root cannot bypass them without explicit policy changes. Pick one and enforce it. SELinux is default on RHEL derivatives; AppArmor ships with Debian and Ubuntu. Don't run both.
For SELinux, confirm enforcing mode:
getenforce
If it returns Permissive or Disabled, edit /etc/selinux/config and set SELINUX=enforcing, then reboot. Permissive mode logs violations but doesn't block them—useful for testing, fatal for production.
Most policy violations come from custom applications or non-standard paths. When a service fails after enabling enforcement, check the audit log:
ausearch -m avc -ts recent
Don't disable SELinux to fix a violation. Generate a policy module with audit2allow or adjust file contexts with semanage fcontext and restorecon. The correct label fixes the root cause; disabling SELinux just hides it.
For AppArmor, list loaded profiles:
aa-status
Any profile in complain mode is logging but not blocking. Move it to enforce:
aa-enforce /etc/apparmor.d/usr.sbin.nginx
Write profiles for your own services. AppArmor uses path-based rules, which are simpler to write than SELinux type enforcement but less granular. For a custom binary in /opt/app/bin/server, create /etc/apparmor.d/opt.app.bin.server with read/write paths, capability grants, and network permissions it actually needs. Test in complain mode first.
Tighten kernel parameters for network stack defense
The default kernel network stack trusts everything. These sysctl settings add skepticism.
Edit /etc/sysctl.conf or drop a file in /etc/sysctl.d/ with these parameters:
# Ignore ICMP redirects (prevents MITM route injection)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
# Ignore source-routed packets
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
# Enable reverse path filtering (drop packets with impossible source addresses)
net.ipv4.conf.all.rp_filter = 1
# Log martian packets (packets with impossible addresses)
net.ipv4.conf.all.log_martians = 1
# Disable IP forwarding unless this is a router
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0
# Protect against SYN flood attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
# Disable IPv6 if you're not using it
net.ipv6.conf.all.disable_ipv6 = 1
Apply with:
sysctl -p
SYN cookies absorb most SYN floods without dropping legitimate connections. Reverse path filtering stops spoofed packets at the interface level. Both have negligible performance cost and block entire classes of attack traffic before it reaches your application.
One edge case: reverse path filtering breaks asymmetric routing. If your server has multiple upstream paths and return traffic uses a different interface than the inbound packet, you'll drop legitimate connections. In that scenario, set rp_filter = 2 (loose mode) instead of 1.
Configure auditd to capture privilege escalation attempts
The audit daemon logs kernel-level events. It's already installed on most distributions but rarely configured.
Key events to watch:
- Execution of privilege escalation binaries
- Changes to user accounts and authentication files
- Modification of audit rules themselves
- Network connections by sensitive services
Add rules to /etc/audit/rules.d/hardening.rules:
# Watch privilege escalation commands
-w /usr/bin/sudo -p x -k privilege_escalation
-w /usr/bin/su -p x -k privilege_escalation
-w /bin/su -p x -k privilege_escalation
# Monitor user and group changes
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/gshadow -p wa -k identity
# Watch authentication configuration
-w /etc/pam.d/ -p wa -k pam_changes
-w /etc/ssh/sshd_config -p wa -k sshd_config
# Monitor audit system itself
-w /etc/audit/ -p wa -k audit_changes
-w /sbin/auditctl -p x -k audit_tools
# Capture network service execution
-a always,exit -F arch=b64 -S socket -F a0=2 -k network_connect
Reload rules:
augenrules --load
Audit logs land in /var/log/audit/audit.log. Query with ausearch:
ausearch -k privilege_escalation -i
The -i flag interprets numeric IDs into usernames and syscall names. Without it, you're reading raw UID and syscall numbers.
One performance note: the socket rule logs every network connection and generates volume. On high-traffic servers, drop it or filter by specific UIDs with -F uid=1000.
Build defense-in-depth firewall chains
Basic iptables rules allow ports and block everything else. Advanced chains rate-limit, log anomalies, and drop attack patterns before they hit your application.
Create a custom chain for input validation:
iptables -N INPUT_VALIDATION
# Drop invalid packets
iptables -A INPUT_VALIDATION -m conntrack --ctstate INVALID -j DROP
# Drop new TCP connections without SYN
iptables -A INPUT_VALIDATION -p tcp ! --syn -m conntrack --ctstate NEW -j DROP
# Drop fragmented packets
iptables -A INPUT_VALIDATION -f -j DROP
# Drop XMAS packets
iptables -A INPUT_VALIDATION -p tcp --tcp-flags ALL ALL -j DROP
# Drop NULL packets
iptables -A INPUT_VALIDATION -p tcp --tcp-flags ALL NONE -j DROP
# Return to INPUT chain for further processing
iptables -A INPUT_VALIDATION -j RETURN
Create a rate-limit chain for SSH:
iptables -N SSH_RATELIMIT
# Allow established connections
iptables -A SSH_RATELIMIT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Rate limit new connections: 3 per minute with burst of 5
iptables -A SSH_RATELIMIT -m conntrack --ctstate NEW -m recent --set --name SSH
iptables -A SSH_RATELIMIT -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
# Accept connections that pass rate limit
iptables -A SSH_RATELIMIT -j ACCEPT
Wire both chains into INPUT:
iptables -A INPUT -j INPUT_VALIDATION
iptables -A INPUT -p tcp --dport 22 -j SSH_RATELIMIT
The validation chain drops malformed packets that waste CPU on protocol parsing. The rate-limit chain stops brute-force login attempts without blocking legitimate reconnects.
For web traffic on high-volume servers, connection tracking becomes a bottleneck. Bypass it for established HTTP connections:
iptables -t raw -A PREROUTING -p tcp --dport 80 -j CT --notrack
iptables -t raw -A PREROUTING -p tcp --dport 443 -j CT --notrack
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
This skips the conntrack table entirely for ports 80 and 443, reducing per-packet overhead at the cost of stateful inspection. Only do this if your application handles its own rate limiting and you're hitting conntrack table exhaustion.
Restrict resource limits to contain compromised processes
A shell with no limits can fork-bomb the system or fill the disk. Set hard limits in /etc/security/limits.conf:
# Limit processes per user
* hard nproc 1000
root hard nproc 2000
# Limit open files
* hard nofile 4096
root hard nofile 8192
# Limit max file size (in KB)
* hard fsize 2097152
# Limit CPU time (in minutes)
* hard cpu 60
# Disable core dumps for all non-root users
* hard core 0
These limits apply at login. Systemd services need separate controls in their unit files:
[Service]
LimitNPROC=100
LimitNOFILE=1024
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges prevents the service from gaining additional capabilities through setuid binaries. ProtectSystem mounts /usr, /boot, and /etc read-only. ProtectHome hides /home, /root, and /run/user entirely.
For containerized workloads, add a memory limit and disable new capabilities:
docker run --memory="512m" --cap-drop=ALL --cap-add=NET_BIND_SERVICE your-image
This blocks privilege escalation paths that rely on kernel capabilities like CAP_SYS_ADMIN or CAP_SYS_PTRACE.
Implement file integrity monitoring on critical paths
File integrity monitoring detects unauthorized changes to binaries, libraries, and configuration files. AIDE and Samhain both work; I've used AIDE on more production systems.
Install and initialize the database:
apt-get install aide # Debian/Ubuntu
yum install aide # RHEL/CentOS
aideinit
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
Edit /etc/aide/aide.conf to define what to monitor. Default rules watch /bin, /sbin, /usr/bin, /usr/sbin, and /etc. Add application-specific paths:
/opt/app/bin R+b+sha256
/var/www/html R+b+sha256
Run checks daily with a cron job:
0 2 * * * /usr/bin/aide --check | mail -s "AIDE Report" [email protected]
After legitimate changes (software updates, config edits), update the database:
aide --update
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
AIDE computes cryptographic hashes of every monitored file. If an attacker replaces a binary or injects code into a library, the next check flags it. Store the AIDE database on read-only media or a separate server to prevent tampering.
One operational note: the initial database build on large filesystems takes time and I/O. Run it during a maintenance window and exclude log directories and other high-churn paths.
Enable and enforce strict systemd sandboxing
Systemd's sandboxing directives confine services to minimal capabilities and filesystem access. Most distributions ship unit files with no sandboxing at all.
For any service you manage, add these directives to the [Service] section:
[Service]
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/yourapp /var/log/yourapp
NoNewPrivileges=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
Breakdown of what each directive blocks:
PrivateTmpgives the service a private/tmpand/var/tmpthat other processes cannot seeProtectSystem=strictmounts the entire filesystem read-only except paths inReadWritePathsPrivateDeviceshides physical devices; the service only sees a minimal/devProtectKernelTunablesblocks writes to/proc/sysand/sysProtectKernelModulesprevents loading or unloading kernel modulesMemoryDenyWriteExecutestops the service from creating writable and executable memory, blocking JIT compilation and most shellcode injectionRestrictAddressFamilieslimits network sockets to only IP and Unix domain socketsSystemCallFilterwhitelists safe syscalls and blocks dangerous ones likereboot,mount,ptrace
After editing the unit file, reload and restart:
systemctl daemon-reload
systemctl restart yourapp
If the service fails, check journalctl -xe for sandboxing violations. You might need to add specific syscalls to the filter or grant additional ReadWritePaths. Don't weaken the sandbox to fix a lazy application; fix the application.
How do I audit which syscalls a service actually uses?
Run the service under strace to capture syscalls:
strace -f -o /tmp/syscalls.log -e trace=all your-command
Then extract unique syscalls:
grep -oP '^[a-z_]+(?=\()' /tmp/syscalls.log | sort -u
Compare the list against systemd's syscall groups in the man page (man systemd.exec, search for SystemCallFilter). Most services fit into @system-service, @network-io, or @file-system.
What breaks when you enable MemoryDenyWriteExecute?
Languages with JIT compilers fail. Node.js, Java, and .NET Core all need writable and executable memory. Python and Ruby work fine because they interpret bytecode instead of compiling it.
For JIT-dependent services, drop MemoryDenyWriteExecute but keep the other sandboxing directives. You're still blocking most post-exploitation techniques.
Can I apply systemd sandboxing to legacy init scripts?
Convert the init script to a systemd unit. Write a [Service] section with Type=forking or Type=simple (depending on whether the script daemonizes), set ExecStart to the script path, and apply sandboxing directives.
For SysV init scripts, this is also faster to start and easier to monitor.
Lock down before the first packet arrives
Hardening is not a checklist you run once. It's a posture you enforce before deploying, then validate on every config change.
MAC policies, kernel parameters, and audit rules require a reboot or service restart to activate. Firewall chains take effect immediately but only stop traffic matching the rules—review logs to catch what you missed. File integrity monitoring only works if you update the baseline after legitimate changes.
What I see in support tickets: servers that pass automated scans but fail under real attacks. They have SELinux installed but set to permissive. Audit rules defined but the daemon disabled. Firewall chains with rate limits that never trigger because the threshold is too high.
Test each layer under load. Throw a SYN flood at your rate-limit rules. Try to read /etc/shadow from a confined service. Attempt to fork-bomb a resource-limited user.
The goal is not perfect security. It's defense in depth where each layer survives independent failures and gives you time to respond.