A critical CVE hits the wire at 6 AM. Your phone lights up with alerts, Slack is blowing up, and you have production servers running the affected software. The next four hours will determine whether you contain the risk or spend the next week dealing with fallout.
I've walked through this scenario more times than I care to count. The difference between a smooth response and chaos comes down to having a repeatable process that your team can execute under pressure. Here's the four-step workflow that has kept our infrastructure secure when it mattered most.
Step 1: Triage and Impact Assessment
First question: does this CVE actually affect you? Not every published vulnerability matters to your specific environment.
Check the CVE description for affected versions, configurations, and prerequisites. If the vulnerability requires local access and all your services run in isolated containers with no shell access, your risk profile is different than if it's a remote code execution bug in an internet-facing daemon.
Make a quick inventory of exposed systems:
# Find all servers running the affected package
apt list --installed | grep package-name
# or for RPM systems
rpm -qa | grep package-name
For services, check what's listening:
netstat -tulpn | grep service-name
# or with ss
ss -tulpn | grep service-name
Priority goes to internet-facing services first, then internal infrastructure that handles sensitive data, then everything else. I keep a living document that maps services to servers so I can answer "where is this running?" in under two minutes.
Document your findings in a shared location. A simple spreadsheet works: server hostname, service version, exposure level (public/internal/isolated), patch priority (critical/high/medium). Your team needs to see the same picture you do.
Don't skip this step to "save time." Patching the wrong systems or missing a critical server because you rushed the inventory wastes more hours than a thorough triage ever will.
Step 2: Test in Non-Production First
I don't care how urgent the patch is. Test it before you touch production.
Spin up a staging environment that mirrors production as closely as possible. Same OS version, same service configuration, same dependencies. If you don't have a staging environment, now is not the time to learn why you should have built one, but you can at least test on a single non-critical production server before rolling it everywhere.
Install the patch:
# Debian/Ubuntu
apt update
apt install --only-upgrade package-name
# RHEL/CentOS/Rocky
yum update package-name
# or
dnf update package-name
Verify the service starts cleanly:
systemctl restart service-name
systemctl status service-name
journalctl -u service-name -n 50
Run through your smoke tests. Can users authenticate? Do API endpoints respond? Does the web application load? Check logs for unexpected errors or warnings.
If your service has a test suite, run it. If not, manually verify the critical path: the features that would cause an immediate incident if they broke.
Sometimes patches introduce regressions. I've seen security updates that fixed a CVE but broke backwards compatibility with older client libraries, changed default configuration behavior, or had dependencies on newer kernel versions. Finding these issues in staging means you can plan a mitigation or rollback strategy before users notice.
Step 3: Deploy with a Rollback Plan
You've confirmed the patch works in testing. Now deploy it in waves, not all at once.
Start with a canary: one production server or a small percentage of your fleet. Monitor it for 15-30 minutes. Check error rates, response times, and resource utilization. If everything looks normal, proceed to the next batch.
For critical infrastructure, I prefer manual deploys during a maintenance window. For less critical systems or when the vulnerability is being actively exploited in the wild, automated rolling updates make sense, but keep a human watching the metrics.
Always have a rollback plan:
# Before patching, note the current version
dpkg -l | grep package-name # Debian/Ubuntu
rpm -qa | grep package-name # RHEL/CentOS
# If rollback is needed
apt install package-name=old-version
# or
yum downgrade package-name-old-version
Some package managers let you hold back updates temporarily if you need to freeze a known-good version while you investigate an issue:
# Debian/Ubuntu
apt-mark hold package-name
# RHEL/CentOS (add to yum.conf)
exclude=package-name
Document what you deployed, when, and to which servers. A simple timestamped log in your runbook or ticketing system works fine. When the post-incident review happens, you'll want that timeline.
Step 4: Verify and Monitor
Patching is not done when the package manager exits successfully. Verification matters.
Confirm the new version is running:
# Check installed version
apt list --installed | grep package-name
# Verify the service is using the new binary
systemctl status service-name
ps aux | grep service-name
Some services load libraries dynamically. A package update might place new files on disk, but the running process still uses the old code until you restart it. For libraries like OpenSSL or glibc, you may need to restart dependent services or even reboot.
To find processes using deleted libraries (a sign they need restart):
sudo lsof | grep DEL | grep .so
Run a vulnerability scanner to confirm the CVE is no longer detected. If you're using something like Nessus, OpenVAS, or a cloud-native scanner, schedule a targeted scan against the patched systems. I've caught cases where a patch was applied but the configuration still enabled the vulnerable code path.
Monitor for at least 24 hours. Set up alerts for error rate spikes, performance degradation, or unusual traffic patterns. Check your logs more frequently than usual. The goal is to catch any problems introduced by the patch before they cascade into bigger issues.
If the CVE has a public proof-of-concept exploit, watch your WAF logs and intrusion detection alerts for attempts to use it. Attackers scan for unpatched systems within hours of a CVE disclosure, sometimes minutes.
What Slows Down Response Time
The biggest delay in my experience isn't technical. It's communication.
When a critical CVE drops, multiple teams need to coordinate: security, operations, application owners, and management. If everyone learns about the issue from different sources at different times, you spend the first hour just getting aligned on what needs to happen.
Set up a dedicated communication channel (a Slack channel, a conference bridge, whatever your team uses) at the start of the incident. Post updates there. Make decisions there. Document action items there.
The second delay comes from lack of automation. If you're manually SSHing into 50 servers to check versions and apply patches, you're wasting time. Configuration management tools (Ansible, Puppet, Chef, Salt) or even a well-written bash script with SSH keys can parallelize the work. I'm not saying you need a full CI/CD pipeline for every package update, but you should be able to push a patch to your entire fleet in minutes, not hours.
Missing or outdated inventory documentation is the third killer. If you don't know what's running where, you can't assess impact quickly. Keep your CMDB, asset inventory, or even a simple spreadsheet current. Automate discovery where possible.
Handling Zero-Day Situations
Sometimes a CVE drops and there's no patch yet. Or the patch exists but hasn't made it into your distribution's repositories.
Your options:
-
Apply workarounds. Many CVE announcements include configuration changes or firewall rules that mitigate the risk until a patch is available. Follow those exactly.
-
Compile from source. If the upstream project released a fix but your package maintainer hasn't, you can build and install manually. Document this carefully because you've now stepped outside your package manager and will need to track updates yourself.
-
Isolate the service. Firewall rules, network segmentation, or disabling the vulnerable feature can buy time. Dropping a service behind a VPN or restricting access to known IP ranges reduces your attack surface.
-
Shut it down. If the service is non-critical and the risk is high, turning it off is a valid choice. Uptime doesn't matter if you get compromised.
I had a case where a critical Apache vulnerability had no patch for our LTS distribution yet. We moved the affected sites behind Cloudflare with a WAF rule that blocked the exploit pattern, then waited 18 hours for the backported patch. Not ideal, but it kept us safe.
Getting Your Response Time Under an Hour
Speed comes from preparation, not from rushing when the alert fires.
Build the infrastructure before you need it: staging environments, automated deployment tools, monitoring dashboards, and communication templates. Document your runbooks. Practice the process with lower-severity patches so muscle memory kicks in when the stakes are high.
I keep a response checklist printed and posted near my desk. Even after dozens of CVE responses, having the steps in front of me means I don't skip verification or forget to update the incident log because I'm moving fast.
The four-step process—triage, test, deploy, verify—works because it's simple enough to remember under pressure and complete enough to catch the mistakes that turn a patch deployment into an outage. Adapt it to your environment, drill it with your team, and when the next critical CVE drops, you'll be ready.