![Critical CVE Patching Workflow [Solved in 4 Steps]](/images/blog/critical-cve-patching-workflow-solved-in-4-steps.jpg)
A critical CVE drops at 3 PM on a Friday, and your manager wants the patch live before Monday. You have production servers running customer workloads, a staging environment that hasn't been touched in two weeks, and zero appetite for downtime. This workflow will get you through it.
I've patched everything from OpenSSL heartbleed-class bugs to obscure library CVEs that only affected one client. The four-step process below is what keeps production stable while closing the vulnerability window fast.
Step 1: Triage and Scope the Risk
Not every CVE with a scary score needs an emergency weekend deploy. Start by confirming three things: whether the vulnerable software is actually installed, whether your configuration exposes the attack vector, and whether an exploit exists in the wild.
Check your package versions first. On Debian-based systems:
dpkg -l | grep <package-name>
apt-cache policy <package-name>
On RHEL-based systems:
rpm -qa | grep <package-name>
yum info <package-name>
If the vulnerable version isn't installed, you're done. If it is, read the CVE description carefully. A remote code execution bug in an SSH daemon matters. An XSS in a web admin panel you disabled three years ago does not.
Document what you find in a shared ticket or wiki page: affected hosts, installed versions, services using the library, and whether those services are exposed to untrusted networks. I've seen teams waste hours patching internal-only services for a remote exploit that required internet access.
Check for Active Exploits
Search for proof-of-concept code on GitHub and exploit databases. If working exploits are public and your service is internet-facing, move fast. If the CVE is theoretical or requires local access you've already restricted, you have time to test properly.
Scan logs for signs of exploitation attempts. Unusual request patterns, failed auth spikes, or segfaults in service logs can indicate someone is already probing. Grep your access logs, error logs, and system journals for anything odd in the past 48 hours.
Step 2: Test the Patch in Staging
Never apply a security patch directly to production, even if vendors claim it's a drop-in fix. Patches introduce regressions. I've seen kernel updates break custom network drivers, library updates change API behavior, and "minor" security fixes cause services to refuse to start.
If your staging environment is stale, sync it first. Copy production configs, restore a recent database snapshot if relevant, and verify services start normally before you touch anything. A staging environment that doesn't mirror production will lie to you.
Apply and Validate
Install the patched package using your normal update workflow:
# Debian/Ubuntu
apt-get update
apt-get install --only-upgrade <package-name>
# RHEL/CentOS
yum update <package-name>
Restart affected services. Check three things: the service starts without errors, it handles normal requests correctly, and the CVE-specific attack vector no longer works. If the CVE included a test case or exploit script, run it against staging. Successful exploitation means the patch didn't apply or your config bypassed it.
Run your usual smoke tests—web endpoints, database queries, SSH access, whatever your application relies on. Automated tests are ideal, but manual checks work if you hit the critical paths. Spend 30 minutes here. It's faster than debugging production at midnight.
What If the Patch Breaks Something?
Rollback in staging and investigate. Check the package changelog for breaking changes, search vendor forums for known issues, and test with different config options. Sometimes a patched library expects a newer version of a dependency you haven't updated yet. Sometimes the patch is just broken, and you need to wait for a fixed release or apply a workaround.
If the patch is essential but incompatible, you have options: firewall rules to block the attack vector, upstream proxies to filter malicious requests, or temporary service isolation. None of these are as good as patching, but they buy time.
Step 3: Deploy to Production with a Rollback Plan
Staging passed. Now coordinate the rollout. Notify your team, your manager, and any on-call engineers about the maintenance window. If your environment supports blue-green or rolling deploys, use them. If not, plan for a brief service interruption and communicate it to users.
Deploy to one server first if you run multiple instances. Apply the update, restart the service, monitor it for five minutes, then proceed to the next host. Stagger the rollout so you always have known-good instances running. If your load balancer supports health checks, configure them to pull unhealthy nodes automatically.
# Example: update and restart Apache on one node
apt-get update && apt-get install --only-upgrade apache2
systemctl restart apache2
systemctl status apache2
curl -I http://localhost/
Watch logs in real time during the deploy. Open a terminal with tail -f /var/log/syslog or journalctl -fu <service-name> and keep it visible. Errors show up fast, and catching them in the first 60 seconds saves you from a full outage.
Keep a Rollback Path Open
Before you start, snapshot your VMs if you're in a virtualized environment or back up the old package version. On Debian-based systems, old .deb files live in /var/cache/apt/archives/ until you run apt-get clean. On RHEL-based systems, yum keeps them in /var/cache/yum/. You can reinstall the previous version with:
# Debian/Ubuntu
apt-get install <package-name>=<old-version>
# RHEL/CentOS
yum downgrade <package-name>-<old-version>
If the patched service crashes or behaves incorrectly, downgrade immediately and pull that host out of rotation. Better to run one vulnerable server behind a firewall rule than to have your entire stack down.
Step 4: Verify and Document
After all hosts are patched and services are stable, confirm the vulnerability is closed. Re-run the exploit test if you have one, or use a vulnerability scanner to verify the CVE no longer appears. Nessus, OpenVAS, and vendor-specific scanners will re-check known CVEs and confirm patch application.
Check your package versions one more time on every host:
for host in $(cat production-hosts.txt); do
ssh $host "dpkg -l | grep <package-name>"
done
If any host shows the old version, patch it now. Inconsistent patch states are how attackers pivot after initial compromise.
Update Your Documentation
Record what you did in your runbook or incident log: CVE number, affected hosts, patch version, deploy timestamp, and any issues you hit. Future you (or your replacement) will thank you when the next CVE in the same component drops and you need to know what worked last time.
Notify your security team or auditors if your org requires it. Compliance frameworks like PCI-DSS expect evidence of timely patching, and a timestamped ticket with before-and-after scans makes audits easier.
Automate What You Can, Test What You Automate
If you patch monthly or more often, invest in automation. Configuration management tools like Ansible, Puppet, or Chef can apply updates across hundreds of nodes in minutes. Automated testing frameworks can validate services after patching without manual checks.
But automation doesn't eliminate the need for staging or rollback plans. I've seen automated patch runs take down entire clusters because nobody tested the playbook after a config change. Run your automation in staging first, the same way you'd test a manual deploy.
Consider security-focused update streams like Debian Security or RHEL RHSA feeds. These repos publish only security fixes, reducing the risk of unrelated package updates sneaking in. Subscribe to vendor mailing lists so you hear about critical CVEs before they hit mainstream news.
Test First, Deploy Fast, Verify Always
Critical CVE patching is a balance between speed and stability. Triage to understand the real risk, test in an environment that mirrors production, deploy with a rollback plan, and verify the fix actually closes the hole. Skipping any of these steps is how small vulnerabilities turn into major incidents.
Document your process so it's repeatable, and automate the repetitive parts once you've proven the workflow. The next critical CVE is already being written—when it drops, you'll be ready.