A critical CVE hits the wire at 2 PM on a Tuesday. Your team has maybe twenty-four hours before automated scanners find every unpatched instance on the internet. I've walked through this drill enough times to know that speed matters, but so does not breaking production.
The workflow below is what security and operations teams actually use when the clock is running. It's not theory. It's the checklist you pull up when a CVSS 9+ vulnerability lands in your stack and you need to move fast without taking the site down.
Step 1: Confirm the threat is real
Not every "critical" CVE announcement deserves a 3 AM deploy. Read the advisory carefully. Check whether the vulnerable component is actually running in your environment and whether your configuration exposes the attack vector.
Ask three questions:
- Does the CVE affect a package version you're running?
- Is the vulnerable feature enabled and reachable from the internet?
- Does a working exploit exist in the wild, or is this still theoretical?
If the answer to all three is yes, you're patching tonight. If only the first two are true, you're still patching—just with slightly less panic. If none are true, document why you're not patching and move on.
In support work I've seen teams waste hours patching a CVE that didn't apply because they skipped the version check. Run rpm -qa | grep package-name or dpkg -l | grep package-name first. Know what you're running before you decide what to patch.
Step 2: Inventory every affected system
You can't patch what you can't see. Pull a list of every server, container, and VM running the vulnerable software. If you're using a config management tool like Ansible, Puppet, or Salt, query the inventory database. If you're not, this is going to hurt.
For bare-metal and VPS environments, an SSH loop works:
for host in $(cat servers.txt); do
ssh $host 'rpm -q openssl' 2>/dev/null
done | sort | uniq -c
Replace openssl with whatever package the CVE affects. The output shows which hosts are on which version. Save it—you'll need it to confirm the patch landed everywhere.
Containers add another layer. If you're running Docker or Kubernetes, check the base images. A vulnerable nginx:1.18 image might be running in thirty pods across three clusters. Use docker images or kubectl get pods --all-namespaces -o json and parse the image tags. Missing a container environment is how you end up with one unpatched instance that becomes the breach.
Step 3: Get the patch and test it somewhere safe
Download the patched package from your distro's security repo or the vendor directly. For RHEL, CentOS, Rocky, and AlmaLinux, security updates go through the usual yum update or dnf update path. For Debian and Ubuntu, apt update && apt upgrade pulls them. Always check the changelog to confirm the CVE is actually fixed in the version you're about to install.
Don't deploy directly to production. Spin up a test instance that mirrors production as closely as you can—same OS version, same application stack, same network config. Apply the patch there first. Check three things:
- The service restarts cleanly.
- The application still works (hit the login page, run a health check, query the API).
- Logs are clean—no new errors or warnings.
If you're patching something like OpenSSL or glibc that touches everything, test more than one service. A kernel patch means rebooting the test box and making sure it comes back up with all services running. I've seen kernel updates break custom drivers or third-party modules, so if you're running anything outside the base distro, test that too.
What if the patch breaks something?
It happens. The patch might introduce a regression, conflict with a custom build, or change behavior your application depends on. If the test instance won't start or the app throws errors, you have two options: find a workaround or apply a temporary mitigation while you fix the breakage.
Mitigation might mean firewall rules, disabling the vulnerable feature, or placing a WAF rule in front of the endpoint. It's not as good as patching, but it buys you time to troubleshoot without leaving the door open. Document whatever you do and set a deadline to revisit it.
Step 4: Plan the rollout
You're patching under pressure, but you still need a plan. Decide the order: patch dev and staging first, then production in waves. If you have load-balanced web servers, take one out of rotation, patch it, test it, then move to the next. That way if the patch does break something, you catch it before all your servers are down.
For database servers, the plan is different. If you're running a primary-replica setup, patch the replicas first. Verify replication still works. Then schedule a maintenance window to fail over and patch the primary. If you're on a single database server, you're scheduling downtime—warn your users and get approval from whoever needs to sign off.
Kernel patches require reboots, so coordinate with the application team. Most web apps can handle a rolling reboot if you've got more than one server. Single-server setups mean downtime, period. Plan for it and communicate it.
Step 5: Deploy and verify
Run the update on the first batch of servers. For a yum-based system:
sudo yum update package-name -y
For apt:
sudo apt-get install --only-upgrade package-name -y
After the package installs, restart the affected service. For Apache:
sudo systemctl restart httpd
For Nginx:
sudo systemctl restart nginx
For system libraries like OpenSSL, you might need to restart everything that links against it. The safest way is a reboot, but if uptime matters more, use lsof to find processes using the old library:
sudo lsof | grep libssl.so.1.1 | awk '{print $2}' | sort -u
Restart those PIDs one at a time.
Verify the patch landed by checking the installed version:
rpm -q package-name
or
dpkg -l | grep package-name
Then check that the service is running and responding. Hit the application URL, run a health check endpoint, or query the service directly. If you're patching SSH, open a second terminal session before you restart sshd—you don't want to lock yourself out.
Step 6: Monitor for breakage
After the deploy, watch logs and monitoring dashboards for at least thirty minutes. Look for error spikes, slow response times, or services that restart unexpectedly. If you're using Nagios, Zabbix, or any other monitoring stack, set up a temporary alert for the patched hosts so you get paged if something goes wrong.
In a load-balanced environment, watch the traffic split. If the patched server starts returning 5xx errors or timeouts, pull it back out of rotation and investigate. The rest of the pool keeps serving traffic while you figure out what broke.
For critical infrastructure like DNS servers or mail relays, test functionality directly after patching. Send a test email, run a DNS query with dig, or hit an internal API endpoint. Don't assume "service is up" means "service is working correctly."
Step 7: Document and report
Once the patch is deployed and verified across all systems, write down what you did. Include:
- Which CVE you patched
- Which systems were affected
- What version you patched from and to
- Any issues encountered and how you resolved them
- Verification steps and results
This isn't busywork. The next time a CVE drops, you'll want to know what you did last time. If an auditor or compliance team asks whether you patched a specific vulnerability, you'll have the documentation ready.
Send a summary to your manager and any stakeholders who need to know. Keep it short: "Patched CVE-YYYY-XXXXX across 47 web servers and 12 database replicas. No downtime. All systems verified and monitoring clean."
Emergency change control
Most organizations have a change control process. In a normal week, you submit a change request, wait for approval, then schedule the work. When a critical CVE drops, that process needs to move faster.
Your emergency change policy should allow you to patch first and document later, as long as you notify the right people and follow rollback procedures. If you don't have an emergency change policy, draft one now—before the next critical CVE lands. It should define what counts as an emergency, who can approve it, and how quickly you need to document the work after the fact.
I've worked in shops where even critical patches required a change board meeting. That's fine if the board meets daily, but if it only meets once a week, you've just given attackers a six-day head start. Push for a streamlined process that balances oversight with speed.
Rollback planning
Before you patch anything, know how to roll back. For package updates, your distro keeps old versions around:
sudo yum history undo last
or for apt:
sudo apt-get install package-name=old-version
For kernel updates, your bootloader keeps previous kernels. If the new kernel breaks something, reboot and select the old kernel from the GRUB menu. Then remove the broken kernel package so it doesn't try to boot into it again.
For container deployments, rollback means redeploying the previous image tag. Keep your CI/CD pipeline set up so you can roll forward or backward with one command. Test the rollback path during your initial test deploy—don't wait until production is on fire to find out it doesn't work.
Communicating with the team
When you're patching under pressure, keep the team in the loop. Use a dedicated Slack channel or a shared incident doc so everyone knows what's happening. Post updates as you hit each milestone: "Test deploy complete, moving to production wave 1." If something breaks, say so immediately.
Coordinate with the application team if the patch might affect their code. They might need to run their own tests or adjust configs. Coordinate with the network team if you're patching firewalls or load balancers. Surprises during an emergency patch are how small issues become outages.
What about third-party software?
If the CVE affects commercial software or a third-party application, you're waiting on the vendor for a patch. Check their security advisory page and support portal. Open a ticket if you don't see a patch yet—vendors prioritize customers who ask.
While you wait, apply mitigations. Firewall off the vulnerable service, disable the affected feature, or put the application behind a reverse proxy with filtering rules. It's not perfect, but it's better than leaving the vulnerability exposed.
Build the workflow before you need it
The time to figure out your CVE response process is not when a critical vulnerability drops on a Friday afternoon. Build the workflow now: document your inventory systems, set up test environments that mirror production, write the rollback procedures, and make sure your team knows who does what.
Run a tabletop exercise. Pick a past CVE and walk through the steps as if it just hit today. Find the gaps—missing tools, unclear ownership, process bottlenecks—and fix them before the next real emergency. The teams that patch fastest are the ones who practiced.
Speed matters in a CVE response, but so does accuracy. Patch the right systems, test before you deploy, and have a rollback plan ready. Do that and you'll close the vulnerability before the scanners find it.