When a high-severity CVE drops, the clock starts ticking. You need a repeatable process that gets patches into production before attackers show up, but fast enough that you don't break everything in the rush.
I've seen teams scramble without a clear workflow—patching the wrong servers first, skipping staging tests, or forgetting to tell the on-call rotation what changed. The result is either prolonged exposure or self-inflicted outages. Neither is acceptable when you're defending real infrastructure.
This is the seven-step workflow I use for critical CVE patching. It balances speed with safety and keeps everyone on the same page from alert to deployment.
Step 1: Triage the CVE Within One Hour
You can't patch everything immediately, so the first job is figuring out whether this CVE actually affects your stack and how badly.
Start by checking three things:
- Affected software and versions. Does the vulnerable package run anywhere in your infrastructure? Check your asset inventory or run a quick scan across your server fleet.
- Attack complexity and prerequisites. Can this be exploited remotely without authentication? Or does it require local access and specific configurations?
- CVSS score and vendor advisory. A CVSS 9+ with active exploitation in the wild goes to the front of the queue. A score of 7 that requires authenticated access might wait for the next maintenance window.
In tickets I've handled, most panic came from CVEs that didn't actually apply to the running environment. Confirm exposure before you mobilize the team.
Document your triage decision in a shared location—a Slack channel, a ticket, or an internal wiki page. Note which systems are vulnerable and which aren't, along with the reasoning.
Step 2: Identify All Affected Systems
Once you know the CVE is real for your environment, map every server and service that needs patching.
For package-based vulnerabilities, use your configuration management tool or run a targeted scan:
# On Debian/Ubuntu systems
ssh user@server "dpkg -l | grep package-name"
# On RHEL/CentOS/AlmaLinux
ssh user@server "rpm -qa | grep package-name"
If you manage more than a handful of servers, automate this with Ansible, SaltStack, or a simple shell loop. The goal is a definitive list of hostnames or IP addresses, prioritized by criticality.
Put internet-facing servers at the top. Authentication gateways, API endpoints, and anything handling customer data come next. Internal dev boxes can wait.
Don't forget containers and VM images. A patched host is useless if your Docker base image still ships the vulnerable library.
Step 3: Pull the Patch and Read the Changelog
Before you deploy anything, verify that the patch is available in your distribution's repositories and read the release notes.
For most Linux distributions, security updates appear within hours of disclosure. Check your vendor's security mailing list or RSS feed:
# Check available updates on Debian/Ubuntu
apt update && apt list --upgradable | grep package-name
# Check on RHEL-based systems
yum check-update package-name
Read the changelog carefully. Sometimes a security patch introduces breaking changes—deprecated config options, altered default behavior, or new dependencies. You need to know this before pushing to production.
If the patch isn't in your repos yet, decide whether to wait, build from source, or apply a vendor-provided hotfix. Each option has trade-offs; building from source gives you control but adds maintenance burden.
Step 4: Test in Staging First
Never apply a critical patch directly to production, no matter how urgent the CVE.
Deploy the update to a staging environment that mirrors production as closely as possible—same OS version, same software stack, same configurations. Then run your standard test suite:
- Service starts cleanly and binds to the expected ports
- Application endpoints respond correctly
- Logs show no new errors or warnings
- Performance metrics stay within normal ranges
For web applications, hit the main user flows manually. For APIs, run a smoke test against the key endpoints. For system services like SSH or DNS, verify basic functionality with real queries.
If staging doesn't exist or doesn't match production closely enough, patch a single non-critical production server and monitor it for at least 30 minutes before rolling out further. This is your canary.
Step 5: Deploy with a Rollback Plan
Once staging tests pass, you're ready for production. But you need a rollback strategy before you touch the first server.
Take a snapshot or backup of the system state before patching. On cloud infrastructure, snapshot the root volume. On bare metal, document the current package version so you can downgrade cleanly:
# Note the current version
dpkg -l | grep package-name > /root/pre-patch-version.txt
# After patching, if you need to roll back
apt install package-name=old-version
For configuration management tools, tag your pre-patch state in version control. If the patch breaks something, you can revert the playbook and re-run.
Deploy in waves. Patch a small subset of servers first—maybe one or two per service tier—and monitor for 15-30 minutes. Watch error rates, response times, and system logs. If everything looks clean, proceed to the next batch.
Stagger the rollout so you never patch all instances of a service simultaneously. This limits blast radius if something goes wrong.
Step 6: Validate the Fix
After deployment, confirm that the vulnerability is actually closed.
Check the installed package version:
# Debian/Ubuntu
dpkg -l | grep package-name
# RHEL/CentOS
rpm -qa | grep package-name
If a proof-of-concept exploit exists for the CVE, consider running it against a patched system in a controlled environment to verify the fix. This isn't always practical, but when it is, it's the most direct confirmation you can get.
For web-facing services, use a vulnerability scanner to re-test the endpoint. Many scanners can target specific CVEs and report whether the server is still vulnerable.
Log the validation results. You'll need this documentation for compliance audits and internal post-mortems.
Step 7: Communicate the Change
Patching doesn't end when the package updates. You need to tell the rest of your team what happened and when.
Send a brief summary to your internal operations channel:
- Which CVE was patched
- Which systems were updated and when
- Any service restarts or brief downtime
- Where to find rollback instructions if needed
For customer-facing services, decide whether external communication is required. If the vulnerability was exploited in the wild or involved customer data, transparency matters. A short status page update is usually enough.
Update your asset inventory and runbooks to reflect the new package versions. The next person on-call shouldn't have to guess what's running where.
What if the Patch Breaks Production?
Sometimes a security patch introduces a regression that only surfaces under production load. When that happens, you have two options: roll back or patch forward.
Roll back if the breakage is severe—service crashes, data corruption, or total loss of functionality. Use the snapshots or version pins you prepared in step five. Notify the team immediately and document the failure mode so vendors can investigate.
Patch forward if the issue is minor and a workaround exists. Maybe a config tweak or a small code change can mitigate the regression while keeping the security fix in place. This is preferable when rolling back would re-expose you to active exploitation.
In either case, escalate to your vendor or the upstream project. They need to know that the patch caused problems in real-world deployments.
Automate What You Can, But Keep Humans in the Loop
For non-critical updates, unattended-upgrades or similar automation is fine. But for high-severity CVEs with potential for service impact, I prefer a semi-automated workflow.
Use automation to:
- Detect which systems are vulnerable
- Pull patches from repos and stage them
- Run pre-deployment tests
- Generate rollback snapshots
Keep humans responsible for:
- Triaging the CVE's actual impact
- Approving the production deployment window
- Monitoring the rollout in real-time
- Deciding whether to roll back
Fully automated patching is tempting, but when a bad update hits production at 3 AM and no one's watching, the damage can spiral before alarms even fire.
How quickly should I deploy a critical CVE patch?
For internet-facing systems with active exploitation, aim to deploy within 24 hours of vendor patch availability. For internal or less-exposed systems, 48-72 hours is reasonable. The key is having a triage process that identifies true urgency so you're not treating every CVE like a fire drill.
What if my distribution hasn't released a patch yet?
Check if the vendor has issued a hotfix or workaround advisory. Sometimes you can mitigate the vulnerability through configuration changes—disabling a feature, restricting access, or applying a firewall rule—until the official patch arrives. If the risk is high and no mitigation exists, consider building from upstream source or switching to a distribution with faster security response.
Should I patch development and staging environments first?
Yes, always. Staging should mirror production and get patched before production does. Development environments can follow the same timeline or slightly lag, but don't let them drift too far. A vulnerable dev box can become a foothold for lateral movement.
How do I handle patching for containers?
Rebuild your container images with the updated base layer or package, then push the new image to your registry. Roll out the updated containers using your orchestration tool's deployment strategy—rolling updates for Kubernetes, blue-green or canary deploys for Docker Swarm or ECS. Don't try to patch running containers directly; they're meant to be immutable.
What if a patch requires a reboot?
Schedule the reboot during a maintenance window if possible, but for critical CVEs with active exploitation, the reboot can't wait. Use rolling reboots across your server pool to maintain service availability, or put the service into maintenance mode briefly if redundancy isn't available. Document the downtime window and notify stakeholders.
Move Fast, Test First, Document Everything
Critical CVE patching is one of the few areas in operations where speed and caution must coexist. You can't afford to dawdle when attackers are scanning the internet for vulnerable systems, but you also can't afford to blindly push updates that take down your own infrastructure.
The workflow I've outlined here—triage, identify, pull, test, deploy, validate, communicate—gives you a repeatable framework that balances both. Adapt the timing and tooling to fit your environment, but keep the structure intact. When the next high-severity CVE lands in your inbox, you'll know exactly what to do first.