When a critical vulnerability drops at 2 AM, you need a process that lets you move fast without pushing untested changes to production. I've seen too many midnight deployments turn into multi-hour outages because someone skipped staging or forgot to prepare a rollback path.
This workflow gives you five concrete steps that balance speed with safety. You'll know exactly what to patch first, how to test it, and when to push the button.
Step 1: Assess Severity and Scope
Not every CVE announcement means you need to wake the team. Start by answering three questions: is the vulnerability remotely exploitable, does it affect a service you expose to the internet, and is there already exploit code in the wild?
Check your inventory first. Run a quick scan to identify which systems run the affected software version:
# Example: find Apache versions across your fleet
for host in $(cat production_hosts.txt); do
ssh $host "httpd -v || apache2 -v" 2>/dev/null | grep "Server version"
done
Priority goes to internet-facing services with known exploits. A remote code execution bug in your public web server beats a local privilege escalation in a sandbox environment every time.
Document what you find in a shared ticket or runbook entry. Include the CVE identifier, affected package versions, and which hosts need patching. You'll reference this list in the next steps.
Step 2: Grab the Patch and Read the Changelog
Pull the update from your distribution's security repository, not random third-party sources. For RHEL-based systems, that's the -security repo. Debian and Ubuntu push critical fixes to *-security pockets.
# Check what's available without installing
yum list-security --security
# or
apt list --upgradable | grep -i security
Read the changelog before you install anything. Look for breaking changes, new dependencies, or configuration file format updates. I've seen OpenSSL patches that changed cipher defaults and broke legacy client connections. Five minutes reading release notes prevents hours of rollback firefighting.
Download the package to your staging environment and verify checksums if your distro publishes them. Store a local copy in case upstream mirrors go down mid-deployment.
Step 3: Test in Staging – Actually Test It
This is where most emergency patches fail. You can't just install the update and curl your homepage once. Run your full test suite if you have one. If you don't, at least exercise the patched service's core functions.
For web servers, hit authenticated endpoints, file uploads, and API routes. For databases, run a sample transaction and check replication lag. For mail servers, send and receive a test message through all configured domains.
# Example smoke test for Apache after patching
sudo systemctl restart httpd
curl -I https://staging.example.com
curl -X POST https://staging.example.com/api/test -d '{"key":"value"}'
tail -f /var/log/httpd/error_log
Watch your logs during the test. New warnings or permission errors often show up right after a security patch because the fix tightened file ownership or changed default umask values.
Timing matters too. If the patched service takes 30 seconds to start instead of 5, that's a problem you need to understand before production deployment.
Step 4: Prepare the Rollback Plan
Before you touch production, write down exactly how to undo the change. I keep a rollback.sh script in my incident directory for every patch deployment.
Snapshot your system state. On VMs, take a hypervisor snapshot. On bare metal, use your configuration management tool to pin the current package version and back up critical config files:
# Pin current version before upgrade
yum versionlock add httpd
# or
apt-mark hold apache2
# Backup configs
tar -czf /root/apache-backup-$(date +%F).tar.gz /etc/httpd
Document the rollback command:
# Rollback example
sudo yum downgrade httpd-2.4.53-1.el8 httpd-tools-2.4.53-1.el8
sudo systemctl restart httpd
Test the rollback procedure in staging. I once spent 40 minutes fighting a broken downgrade because the new package version had changed a database schema, and the old version refused to start. Knowing that before production saved the night.
Step 5: Deploy, Monitor, and Verify
Start with a canary host or the least-critical production node. Install the patch, restart the service, and watch it for 10-15 minutes. Check error rates, response times, and disk I/O.
# Apply the patch
sudo yum update httpd --security -y
sudo systemctl restart httpd
# Monitor immediately
journalctl -u httpd -f &
tail -f /var/log/httpd/error_log &
If the canary looks clean, roll out to the rest of your fleet. For clusters, do it in waves: patch and restart one node, let it stabilize, move to the next. Never patch all nodes simultaneously unless you enjoy explaining downtime.
Use monitoring tools to catch regressions. Set up temporary alerts for HTTP 500 rates, connection timeouts, or memory spikes. I keep a terminal with htop and iotop open during patch deployments just to get a feel for resource changes.
After the final host is patched, run a quick end-to-end test of your critical user paths. If you're running an e-commerce site, complete a checkout. If you're hosting APIs, hit your top three endpoints. Real traffic will find edge cases your staging tests missed.
When Do You Skip Staging?
Almost never. But if you're actively being exploited and the patch is a single-line code change with no dependencies, you can compress steps two and three into a 5-minute review and smoke test on one production node.
I've done this exactly twice in ten years: once for a Bash shellshock variant that was trivial to verify, and once for a kernel TCP stack bug that had a proof-of-concept hitting our logs in real time. Both times I had rollback scripts ready and a colleague watching monitoring dashboards.
What About Automated Patching?
Automated security updates work well for non-critical systems and workstations. For production servers, I use semi-automated workflows: the system downloads and stages patches, sends a notification, and waits for manual approval before applying them.
Tools like yum-cron or unattended-upgrades can handle this. Configure them to download only, not install:
# /etc/yum/yum-cron.conf
apply_updates = no
download_updates = yes
You still walk through the five steps, but step two is already done when you get the alert.
Keep Your Process Documented
The critical CVE patching process only works if you can execute it under pressure. Write a runbook that covers these five steps for your specific environment, including hostnames, repository URLs, and monitoring dashboard links.
Update it after every incident. The 15 minutes you spend documenting what went wrong—or what you wish you'd prepared—makes the next emergency patch deployment faster and safer. I keep mine in a Git repo with dated entries so I can see how our process evolved over time.
You won't eliminate risk entirely, but you will deploy patches faster and with more confidence than teams that skip staging or forget about rollback plans.