A critical CVE drops at 3 AM and your phone lights up. You've got production servers exposed and maybe a few hours before exploit code hits the wild. What you do in the next twenty minutes determines whether you patch cleanly or spend the weekend in a war room.
I've handled enough emergency patches to know the difference between a smooth rollout and a self-inflicted outage. The teams that survive these events follow a workflow. It's not glamorous, but it works.
Step 1: Triage the CVE Against Your Stack
Not every critical CVE affects you. Read the advisory first. Check the affected software, versions, and configurations against your actual inventory.
Ask yourself:
- Do we run the vulnerable software?
- Which versions are we on?
- Is the vulnerable feature enabled or exposed?
- What's the attack surface—public internet, internal network, authenticated users only?
In one incident I worked, a kernel CVE looked terrifying until we confirmed our production boxes ran a patched LTS release that already had the backported fix. Saved us a midnight maintenance window.
Keep an up-to-date asset inventory. A spreadsheet works. A CMDB is better. You can't triage what you can't see.
Severity vs. Urgency
CVSS scores matter, but context matters more. A CVSS 9.8 remote code execution bug in a service that only listens on localhost is less urgent than a CVSS 7.5 authentication bypass on your public-facing control panel.
Prioritize based on:
- Exposure (internet-facing beats internal)
- Exploit availability (proof-of-concept code in the wild moves it up)
- Data sensitivity (customer databases beat test environments)
- Compensating controls (WAF rules, firewall blocks)
Document your decision. When someone asks why you patched the web server before the mail relay, you'll have an answer.
Step 2: Identify the Fix and Assess Impact
Find the patch. Upstream maintainers usually ship updates within hours of disclosure for critical bugs. Check your distribution's security mailing list, the vendor's advisory page, or the project's GitHub releases.
For CentOS, RHEL, and Rocky Linux, security updates come through the normal repositories with an RHSA identifier. Ubuntu publishes USN advisories. Debian has DSA notices.
# Check for available security updates (RHEL/Rocky/Alma)
sudo yum updateinfo list security
# Ubuntu/Debian
sudo apt update
apt list --upgradable | grep -i security
Read the changelog. Understand what changed. A one-line bounds check is low risk. A rewrite of the authentication layer needs more scrutiny.
Check for breaking changes, deprecated options, or new dependencies. Some patches require configuration tweaks or service restarts that affect uptime.
When There's No Patch Yet
Sometimes the CVE drops before the fix does. Your options:
- Apply a vendor workaround (disable the vulnerable feature, add firewall rules)
- Backport the fix yourself if you have the engineering capacity
- Isolate affected systems behind additional controls
- Accept the risk with sign-off from whoever owns that decision
I've seen teams disable HTTP/2 on a web server to work around a protocol parsing bug until a proper patch landed. Not ideal, but better than staying fully exposed.
Step 3: Test in Staging First
Never apply a critical patch directly to production. I don't care how urgent it feels.
Spin up a staging environment that mirrors production as closely as possible—same OS version, same application stack, same configuration files. If your staging box runs different software versions or configs, your test results are worthless.
Apply the patch:
# RHEL-based
sudo yum update package-name
# Debian-based
sudo apt install --only-upgrade package-name
Restart the affected service and verify it comes back up:
sudo systemctl restart httpd
sudo systemctl status httpd
Run your smoke tests. Hit the application's key endpoints. Log in, perform a transaction, check that background jobs still process. Look at the logs for errors or warnings that weren't there before.
How long you spend testing depends on the severity and your downtime tolerance. A kernel patch that requires a reboot? Test for an hour minimum, check dmesg, run a load test. A library update for a non-critical service? Fifteen minutes might be enough.
Rollback Plan Before You Patch
Decide how you'll undo the change if production breaks. Options:
- Snapshot the VM or container before patching (fastest for virtual infrastructure)
- Use your package manager's rollback feature
- Keep the old package file locally for quick downgrade
- Have a known-good backup and a tested restore procedure
For critical services, I script the rollback steps and keep them in a runbook. When you're troubleshooting at 4 AM, you don't want to be Googling downgrade syntax.
# Downgrade example (yum)
sudo yum downgrade package-name-old-version
# Debian/Ubuntu (if you have the .deb)
sudo dpkg -i /var/cache/apt/archives/package-name_old-version.deb
sudo apt-mark hold package-name # prevent auto-upgrade
Step 4: Schedule the Maintenance Window
Pick a time that balances urgency against user impact. For a wormable RCE with public exploits, you patch now and apologize later. For most critical CVEs, a few hours of planning won't kill you.
Communicate the window to your team and your users. Send an email, post a status page update, ping the relevant Slack channels. Include:
- What you're patching and why
- Expected downtime (be honest—pad your estimate)
- Start and end time
- Who to contact if something breaks
Some hosting providers I've worked with keep a pre-written template for emergency maintenance notifications. Fill in the blanks and hit send.
Stagger the Rollout for Large Fleets
If you manage dozens or hundreds of servers, don't patch them all at once. Start with a small canary group—maybe 5% of your fleet. Monitor for issues. If everything looks clean after thirty minutes, continue in waves.
This limits your blast radius. A bad patch that breaks one server is fixable. The same patch breaking fifty servers simultaneously is a resume-generating event.
Step 5: Apply the Patch to Production
Take a final snapshot or backup right before you start. Verify it completed successfully.
Apply the update:
sudo yum update package-name -y
Restart services as needed. Some updates require a full reboot (kernel, glibc, systemd). Others just need the affected daemon restarted.
# Check if reboot required (Ubuntu/Debian)
test -f /var/run/reboot-required && echo "Reboot needed"
# RHEL/Rocky - check needs-restarting
sudo needs-restarting -r
Watch the logs in real time during the restart:
sudo tail -f /var/log/messages
sudo journalctl -u httpd -f
Bring up monitoring dashboards. Check that application metrics, response times, and error rates stay normal. If you have automated health checks, watch them turn green.
Verification Checklist
Before you call it done:
- [ ] Service responds to requests
- [ ] No new errors in logs
- [ ] Key application functions work (login, checkout, form submission)
- [ ] Monitoring shows normal resource usage
- [ ] Vulnerability scanner confirms the CVE is patched
For web applications, I usually test from an external client—curl a few URLs, check the response codes and content. Make sure you're not just seeing a cached page or an error message.
curl -I https://yoursite.com/
curl -s https://yoursite.com/login | grep -i "error\|exception"
Step 6: Monitor Post-Deployment
The first hour after patching is when surprises happen. Memory leaks, race conditions, edge cases your staging environment didn't catch.
Keep an eye on:
- Error logs (application, system, web server)
- Resource usage (CPU spikes, memory growth, disk I/O)
- Application performance metrics
- User-reported issues (support tickets, social media)
I've seen patches that looked fine in testing but caused a slow memory leak in production under real load. Caught it because we watched graphs for an hour instead of walking away immediately.
If something breaks, execute your rollback plan. Don't try to fix it forward unless the issue is obvious and trivial. You can always re-patch after you understand what went wrong.
Document What Happened
Write down what you did, what broke (if anything), and what you learned. Include:
- CVE identifier and description
- Systems affected
- Patch version applied
- Downtime duration
- Any workarounds or config changes needed
- Lessons for next time
This becomes your institutional knowledge. When the next CVE hits, you'll have a reference for how long things actually took and what caught you off guard.
Step 7: Communicate the Resolution
Tell everyone the maintenance is complete. Update your status page, send the all-clear email, close the incident ticket.
If there was user-visible downtime, acknowledge it. If something broke and you had to roll back, explain what you're doing differently for the next attempt.
Transparency builds trust. I've watched teams lose customer confidence not because they had an outage, but because they went silent during it.
For internal stakeholders, summarize the incident:
- What CVE was addressed
- Why it was urgent
- What the outcome was
- Whether any follow-up work is needed
Some teams run a brief post-incident review for critical patches that didn't go smoothly. Five people, thirty minutes, focus on what to improve. No blame, just systems thinking.
What If the Patch Breaks Production?
You tested in staging, everything looked good, but production fell over anyway. It happens.
First: execute your rollback. Get the service stable. You can investigate after users aren't impacted.
Once you're back to baseline, figure out what went wrong. Common culprits I've seen:
- Staging didn't match production (different load, different data, different config)
- Race condition or concurrency bug that only shows under real traffic
- Interaction with another service or dependency that wasn't in staging
- Hardware or kernel difference between test and prod
Fix the issue in staging again. Re-test with production-like load if possible. Try the patch on a single production canary before rolling out fleet-wide.
If the patch itself is broken and there's no workaround, you might need to wait for the vendor to issue a fixed version. In the meantime, apply compensating controls—WAF rules, firewall restrictions, disable the vulnerable feature.
Can You Skip Testing?
In my experience? Only if the alternative is worse.
If you have active exploitation happening right now, or if reliable intelligence says exploit code will be public in an hour, the risk calculation changes. A potential patch regression is better than a confirmed ongoing breach.
But most CVEs don't hit that threshold. Even high-severity vulnerabilities usually give you a few hours to test properly. Skipping the staging step is how you turn a security fire drill into a multi-hour outage.
What Matters Most in Emergency Patching
Speed matters, but blindly racing to patch causes outages. The teams that handle CVEs well have a documented workflow, maintain staging environments, and practice rollbacks before they need them.
Keep your asset inventory current. Test every patch, even urgent ones. Communicate with your users and your team. Have a rollback plan ready before you click apply.
When the next critical CVE lands, you'll move fast because you know exactly what to do.