You wake up to a security advisory. A new CVE dropped overnight, rated critical, and your inbox is already filling with vendor bulletins. The clock is ticking, but patching production servers without a plan is how outages happen.
I've handled hundreds of these situations in hosting support. The teams that survive critical CVEs intact follow a clear triage workflow—they assess risk first, test patches in isolation, and roll out fixes in waves. The teams that panic-patch everything at once are the ones filing incident reports at 3 AM.
Here's the workflow I use when a high-risk vulnerability lands.
Step 1: Read the CVE and map your exposure
Start with the official CVE entry. You need three pieces of information before you touch anything: which software versions are affected, whether the vulnerability is remotely exploitable, and if a public exploit already exists.
Check the CPE (Common Platform Enumeration) strings in the CVE. They tell you exact package names and version ranges. If you're running Apache 2.4.49 and the CVE says 2.4.48 through 2.4.50 are vulnerable, you're in the blast radius.
Next, audit your infrastructure. For a typical Linux server stack, that means:
rpm -qa | grep -i <package_name> # RHEL/CentOS
dpkg -l | grep -i <package_name> # Debian/Ubuntu
For web applications, check your container images, dependencies, and third-party libraries. A vulnerability in a PHP library matters just as much as one in the kernel.
Document which servers are exposed. A spreadsheet works. So does a tagged inventory in your configuration management tool.
Step 2: Score the actual risk with CVSS
The base CVSS score in the advisory is a starting point, not gospel. A "critical" rating doesn't mean every installation is equally at risk.
Look at the attack vector. Network-exploitable CVEs that require no authentication are worse than local privilege escalation bugs that need an existing shell. If the vulnerability requires user interaction or a specific configuration you don't use, your effective risk drops.
I adjust scores based on:
- Attack complexity: Does exploitation require race conditions, memory layout luck, or chained bugs? High complexity buys you time.
- Privileges required: An unauthenticated remote exploit is a drop-everything emergency. A local root escalation on a single-tenant server where you trust all users is less urgent.
- Scope: Does the bug escape containers, VMs, or security boundaries? Scope-changing vulnerabilities get priority.
- Your actual attack surface: A vulnerable HTTP/2 module doesn't matter if you terminated TLS at the load balancer and the backend speaks HTTP/1.1.
Re-score using the CVSS calculator with your environmental metrics. That adjusted score drives your timeline.
Step 3: Check for active exploitation
A CVE with a public exploit in the wild gets bumped to the front of the queue, regardless of score.
Search GitHub, Exploit-DB, and Metasploit modules. If proof-of-concept code exists and it's trivial to run, assume attackers are already scanning for vulnerable hosts. I've seen mass exploitation start within hours of PoC publication for high-profile bugs.
Check your logs for indicators of compromise. For web server CVEs, look for unusual request patterns, unexpected user agents, or spikes in 4xx/5xx errors around the vulnerable endpoint.
grep -i "suspicious_pattern" /var/log/httpd/access_log
tail -1000 /var/log/secure | grep -i "failed\|invalid"
If you find evidence of exploitation before you finish patching, switch to incident response mode. Isolate the affected systems, collect forensics, and patch offline.
What if a patch isn't available yet?
Zero-day situations require compensating controls while you wait for the vendor.
For remotely exploitable bugs, firewall rules or WAF signatures can block attack traffic. If the CVE targets a specific URL path or requires a malformed request header, write a rule.
iptables -A INPUT -p tcp --dport 8080 -m string --string "exploit_string" --algo bm -j DROP
For application-level vulnerabilities, disable the affected feature if you can live without it. An unused Apache module that's vulnerable? Turn it off.
# /etc/httpd/conf.modules.d/00-base.conf
# LoadModule vulnerable_module modules/mod_vulnerable.so
Document every workaround. You'll need to remove them after patching or risk running unsupported configurations forever.
Step 4: Test the patch in a non-production environment
Never apply a critical patch directly to production, no matter how urgent. I've seen updates brick servers because of undocumented dependency conflicts or broken init scripts.
Spin up a staging clone that mirrors production as closely as possible—same OS version, same kernel, same installed packages. If your production web server runs CentOS 7 with a custom-compiled PHP, your test environment needs to match.
Apply the patch and verify basic functionality:
yum update <package_name> # or apt upgrade
systemctl restart <service>
systemctl status <service>
Run your application's health checks. Hit critical endpoints, check database connectivity, review error logs. If you have automated tests, run the full suite.
Pay attention to configuration file changes. Package managers sometimes overwrite customized configs during updates. Diff your edited files against the new defaults:
diff /etc/ssh/sshd_config /etc/ssh/sshd_config.rpmnew
If the staging test passes, document the exact update commands and any config adjustments. That's your playbook for production.
Step 5: Roll out patches in waves
Patching hundreds of servers simultaneously is how you turn a vulnerability into an availability incident. Stagger your rollout.
Start with a single production server or the least critical service. Apply the patch, monitor for 15-30 minutes, and confirm everything still works. If that server survives, move to the next tier.
A typical rollout looks like:
- One canary server (1-5% of capacity)
- Wait and monitor
- Remaining non-critical servers (dev, staging, internal tools)
- Primary production pool in batches of 10-20%
- Database servers and other stateful infrastructure last
For clustered applications behind load balancers, pull servers out of rotation one at a time, patch them, verify health checks pass, then return them to the pool.
# Remove from HAProxy backend
echo "disable server backend_name/server01" | socat stdio /var/run/haproxy.sock
# Patch, test, then re-enable
echo "enable server backend_name/server01" | socat stdio /var/run/haproxy.sock
Database servers need extra care. Take a backup before patching, verify replication lag is minimal, and have a rollback plan ready.
Step 6: Monitor for breakage and performance regression
Patches occasionally introduce new bugs or performance hits. Keep your monitoring stack open during and after rollout.
Watch:
- Error rates: Spikes in application errors or segfaults.
- Latency: Response time increases, especially for database queries or API calls.
- Resource usage: Memory leaks, CPU spikes, disk I/O changes.
- Connection counts: Drops in active sessions or connection pool exhaustion.
If something breaks, you need to know within minutes, not hours. Set temporary alert thresholds tighter than usual during the patching window.
Have your rollback procedure ready. For package-based patches, that usually means downgrading:
yum downgrade <package_name>-<old_version>
For source-compiled software, keep the old binaries around until you've confirmed the patch is stable.
Step 7: Verify remediation and document the event
Once the patch is deployed everywhere, confirm the vulnerability is actually closed.
Re-scan affected systems with a vulnerability scanner or manually verify the patched version:
rpm -q <package_name>
<software> --version
If you applied workarounds earlier, remove them now. Leftover firewall rules or disabled modules can cause confusion months later when someone troubleshoots an unrelated issue.
Document what you did:
- Which CVE you patched
- Which servers were affected
- Patch version applied
- Date and time of rollout
- Any issues encountered
This record proves compliance during audits and gives the next person on-call a reference when a similar CVE drops.
How fast should you actually patch?
Timeline depends on your risk score and exploit availability.
- Critical + active exploitation: Hours. Drop everything.
- Critical + public PoC, no exploitation yet: 24-48 hours.
- High severity, no public exploit: 7 days.
- Medium severity: 30 days during your normal patch cycle.
These are guidelines, not laws. A critical CVE in a service that's firewalled and only accessible to internal networks is less urgent than a high-severity bug in your public-facing web app.
Balance speed against stability. A hasty patch that takes down production is worse than a vulnerability that might never be exploited.
What separates a smooth patch cycle from chaos
The difference is preparation. Teams that handle CVEs well already have staging environments, rollback procedures, and an asset inventory before the advisory hits.
If you're building that foundation now, start small. Document your critical servers, set up one staging clone, and write down your service restart commands. The next CVE will be easier.
Speed matters when a CVE is being actively exploited, but most of the time, patching methodically and safely beats patching fast and broken.