![Critical CVE Patching Workflow [Solved in 4 Steps]](/images/blog/critical-cve-patching-workflow-solved-in-4-steps-2.jpg)
When a critical CVE drops for Apache, OpenSSL, or your kernel, you face two bad choices: patch immediately and risk breaking production, or wait to test and leave the door open. Most teams I've worked with swing between these extremes. A repeatable four-step workflow solves both problems.
Step 1: Triage and Assess Impact
Not every CVE that says "critical" actually threatens your stack. Start by reading the CVE description and checking three things: does the vulnerable component run on your servers, is the attack vector reachable from your network topology, and do you use the affected feature?
Check installed versions first:
# Debian/Ubuntu
dpkg -l | grep -i package-name
apt-cache policy package-name
# RHEL/CentOS/AlmaLinux
rpm -qa | grep -i package-name
yum info package-name
If the vulnerable version isn't present, document that and move on. If it is, map the attack surface. A remote code execution flaw in a library you compile but never expose to the network is still serious, but it's not the same as an unauthenticated RCE in your public web server.
Look at the CVSS score, but weight the impact and exploitability subscores more than the headline number. A 9.8 that requires local access and user interaction is lower priority than a 7.5 with network vector and no privileges required.
Set a patch deadline based on exposure. Public-facing services with active exploits need patches within hours. Internal services behind firewall rules can wait for your next maintenance window if you add temporary mitigations.
Step 2: Test in Non-Production
Skipping this step is how you turn a security patch into a four-hour outage at 2 AM. Spin up a staging environment that mirrors production package versions, configurations, and workload patterns as closely as you can.
Snapshot or clone the environment first:
# LXC container snapshot
lxc snapshot container-name pre-patch-snapshot
# KVM/libvirt snapshot
virsh snapshot-create-as vm-name pre-patch --disk-only
# AWS EC2 AMI
aws ec2 create-image --instance-id i-xxxxx --name "pre-patch-$(date +%F)"
Apply the patch using your distro's package manager:
# Debian/Ubuntu
apt update
apt install --only-upgrade package-name
# RHEL/CentOS/AlmaLinux
yum update package-name
Restart affected services and watch logs for the first five minutes. Then run your integration tests, health checks, and any smoke tests that touch the patched component. If you don't have automated tests, manually verify the critical user paths.
Load testing matters for patches that touch high-throughput services. I've seen OpenSSL updates cut HTTPS throughput by twenty percent because of cipher suite changes. Better to find that in staging.
Document what you tested and the results. If the patch causes a regression, you need to know exactly what broke so you can decide between rolling back, finding a workaround, or escalating to the vendor.
Step 3: Deploy with Rollback Plans
Schedule the production patch during a maintenance window if possible. If the CVE severity or active exploitation doesn't give you that luxury, deploy during your lowest traffic period and have your full on-call rotation ready.
Prepare rollback steps before you start. For package updates, that usually means pinning the old version or keeping the package file:
# Download current version before upgrading
apt download package-name
# Or use apt-cache to find the specific version
apt-cache madison package-name
On RHEL-based systems, yum and dnf keep old packages in cache by default, so you can downgrade:
yum downgrade package-name-old-version
Deploy to one server first if you run multiple instances behind a load balancer. Pull it out of rotation, patch it, verify it, then add it back and continue:
# Example with HAProxy
echo "disable server backend_name/server1" | socat stdio /var/run/haproxy.sock
# Apply patch and verify
echo "enable server backend_name/server1" | socat stdio /var/run/haproxy.sock
For single-server setups, the rollback plan is your snapshot from step two. Know the exact command to restore it before you patch.
Monitor error rates, response times, and resource utilization for at least fifteen minutes after patching each server. If something spikes, investigate immediately. Don't continue the rollout until you understand the cause.
Step 4: Verify the Fix
Patching the package doesn't guarantee the vulnerability is actually closed. Verify the CVE is remediated by checking the version number against the vendor advisory and testing the attack vector if feasible.
Confirm the new version:
# Check package version
dpkg -l | grep package-name # Debian/Ubuntu
rpm -q package-name # RHEL/CentOS
# Check binary version for services
nginx -v
openssl version
apache2 -v
If the CVE has a public proof-of-concept that's safe to run against your own infrastructure, use it. Many security researchers publish PoCs that check for the vulnerability without actually exploiting it. Running that against your patched server confirms the hole is closed.
For CVEs without safe PoCs, check logs and monitoring for any indicators of exploitation attempts. Compare pre-patch and post-patch behavior. A patched authentication bypass should now show failed login attempts where it previously showed none.
Update your asset inventory and vulnerability scanner. If you use Nessus, OpenVAS, or a commercial scanner, trigger a rescan of the patched systems to confirm they no longer flag for that CVE. This also gives you documentation for compliance audits.
What about kernel patches?
Kernel CVEs add one more step: you need a reboot. Test the patched kernel in staging first, but when you deploy to production, plan for the downtime or use live patching if your distro supports it.
For Ubuntu systems, Canonical's Livepatch service applies kernel security fixes without rebooting:
sudo snap install canonical-livepatch
sudo canonical-livepatch enable [token]
RHEL and derivatives offer kpatch for live kernel patching. The tooling requires some setup, but it's worth it for systems that can't tolerate reboot windows.
If you do reboot, verify the new kernel actually loaded:
uname -r
Compare that against the installed kernel package version. I've seen systems with the new kernel package installed but still running the old kernel because the bootloader configuration wasn't updated.
How do you handle patches that break things?
Rollback immediately using the plan from step three. Don't try to fix forward unless the fix is obvious and takes under five minutes.
After rolling back, you have three options. One: find a compensating control like a firewall rule or WAF signature that blocks the attack vector while you work on the patch issue. Two: coordinate with the vendor or package maintainer to get a fixed patch. Three: apply the patch outside business hours with a longer maintenance window that gives you time to troubleshoot.
Document the breakage in detail. Note the symptoms, the error messages, the affected functionality, and your configuration that might have triggered it. Send that to the vendor if you open a support case.
Temporary mitigations buy you time but they're not permanent solutions. Set a deadline to either get a working patch or accept the risk and document the decision.
Do you patch everything at once?
No. Patch in waves based on exposure and criticality. Public-facing web servers and VPN gateways first. Application servers and databases second. Internal admin tools and monitoring systems last.
This approach limits blast radius. If a patch causes issues, you catch it on the first wave before it hits your entire infrastructure.
For large server fleets, consider blue-green or canary deployments. Bring up new patched instances, shift a small percentage of traffic to them, verify everything works, then shift the rest.
What's the audit trail?
Log every step. When you triaged the CVE, when you tested it, what broke in testing, when you deployed to production, what you verified afterward. This matters for compliance frameworks like PCI-DSS that require documented evidence of timely patching.
Your ticketing system should track the CVE ID, affected systems, patch version applied, deployment timestamp, and verification results. If you use configuration management like Ansible or Puppet, the automation logs serve as additional evidence.
Keep the pre-patch snapshots for at least thirty days in case you need to investigate unexpected behavior that shows up later.
Check snapshots before patching
The workflow matters more than speed. Rushing a patch that breaks production turns a security problem into an availability problem, and you end up dealing with both. Triage accurately, test thoroughly, deploy carefully, verify completely. That's how you patch critical CVEs without extending the risk window or causing outages.
Every team I've seen adopt this four-step process has cut their patch-related incidents by more than half. The key is discipline: don't skip testing, don't deploy without rollback plans, and don't assume the patch worked without verification.