Traffic doubled overnight and your single server started sweating. You need architecture that grows with demand, not against it. The seven patterns below are what I've seen work in production environments where uptime matters and load is unpredictable.
Pattern 1: Horizontal Load Balancing
Load balancing splits incoming requests across multiple identical servers. Instead of upgrading one machine's CPU and RAM, you add more machines.
The load balancer sits in front of your application tier. It receives all traffic and distributes requests using algorithms like round-robin, least connections, or IP hash. Round-robin cycles through servers in order. Least connections sends new requests to whichever server is handling the fewest active connections right now. IP hash keeps the same client talking to the same backend server, which helps with session persistence.
In my support queue, the pattern I see fail is running a load balancer without health checks. Configure active health checks that probe each backend every few seconds. If a server stops responding or returns errors, the load balancer removes it from rotation automatically. Your users never see the failure.
upstream app_cluster {
least_conn;
server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
}
server {
listen 80;
location / {
proxy_pass http://app_cluster;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
Session state is the trap here. If your application stores session data in memory on each server, users will lose their login when the load balancer sends their next request to a different backend. Move session storage to Redis or a shared database, or use sticky sessions as a short-term fix.
Pattern 2: Database Read Replicas
Most applications read data far more often than they write it. One primary database handles all writes. Multiple read replicas handle queries.
Replication happens asynchronously in most setups. The primary logs every change, and replicas replay those logs a few milliseconds behind. Your application sends INSERT, UPDATE, and DELETE statements to the primary. SELECT queries go to replicas.
Lag matters. If a user updates their profile and immediately views it, they might see stale data if the replica hasn't caught up yet. Two strategies handle this: send reads for recently-written data to the primary, or add a short sleep after writes before querying. Neither is perfect.
# Write to primary
primary_conn.execute(
"UPDATE users SET email = %s WHERE id = %s",
(new_email, user_id)
)
# Read from replica
replica_conn = get_replica_connection()
user = replica_conn.execute(
"SELECT * FROM users WHERE id = %s",
(user_id,)
).fetchone()
Connection pooling is non-negotiable. Opening a new database connection for every query kills performance. Configure your application to maintain a pool of open connections to both primary and replicas. Reuse them.
Pattern 3: Active-Passive Failover
Sometimes you can't split traffic across multiple nodes. Stateful services like databases need one source of truth.
Active-passive keeps a standby server warm. The active node handles all traffic. The passive node replicates data continuously but serves nothing. When the active node dies, the passive promotes itself and takes over.
Heartbeat monitoring makes this work. Both nodes ping each other every second. If the passive node stops receiving heartbeats, it assumes the active node is down and claims the shared virtual IP address. Traffic routes to the new active node within seconds.
I've debugged split-brain scenarios where network partitions made both nodes think they were active. They both started accepting writes and diverged. Fencing prevents this—if a node can't reach the other and can't reach a quorum of witness nodes, it shuts itself down rather than risking data corruption.
# Pacemaker cluster configuration example
crm configure primitive virtual_ip ocf:heartbeat:IPaddr2 \
params ip=192.168.1.100 cidr_netmask=24 \
op monitor interval=10s
crm configure primitive db_service ocf:heartbeat:mysql \
op monitor interval=30s
Pattern 4: Multi-Region Replication
Putting all your infrastructure in one data center means one fiber cut or power failure takes everything down. Multi-region replication spreads your system across geographic locations.
Each region runs a complete copy of your stack: load balancers, application servers, databases. A global load balancer routes users to the nearest region based on latency or geography. Data replicates across regions, either synchronously or asynchronously.
Synchronous replication waits for writes to reach multiple regions before acknowledging success. It's slow but guarantees zero data loss during regional failures. Asynchronous replication acknowledges writes immediately and syncs in the background. It's fast but you might lose seconds or minutes of data if a region fails before replication completes.
Cost adds up fast with this pattern. You're running duplicate infrastructure everywhere. Start with two regions, not five.
Handling conflicts
What happens when the same user modifies data in two regions simultaneously? Last-write-wins discards one update silently. Vector clocks or CRDTs let you detect conflicts and either merge changes automatically or flag them for manual resolution. Most applications pick last-write-wins and accept the rare data loss because building conflict resolution is expensive.
Pattern 5: Circuit Breaker for Dependencies
Your application probably calls external APIs: payment processors, email services, third-party data feeds. When those services slow down or fail, your application shouldn't wait forever and exhaust connection pools.
A circuit breaker wraps outbound calls. It tracks failure rates. If too many requests fail within a window, the breaker trips open and immediately rejects new calls without attempting them. After a timeout, it enters half-open state and allows one test request through. If that succeeds, it closes and resumes normal operation. If it fails, it opens again.
from pybreaker import CircuitBreaker
breaker = CircuitBreaker(fail_max=5, timeout_duration=60)
@breaker
def call_payment_api(order_id):
response = requests.post(
"https://payments.example.com/charge",
json={"order_id": order_id},
timeout=5
)
response.raise_for_status()
return response.json()
Fallback behavior keeps your application usable when the breaker is open. Queue the operation for retry, return cached data, or degrade gracefully by disabling the feature. An open circuit breaker is better than cascading failures that take down your entire stack.
Pattern 6: Autoscaling Based on Metrics
Manually adding servers when traffic climbs doesn't scale. Autoscaling watches metrics and adjusts capacity automatically.
CPU utilization is the common trigger. When average CPU across your fleet exceeds 70 percent for five minutes, launch additional instances. When it drops below 30 percent, terminate extras. Network throughput, memory usage, and application-specific metrics like queue depth also work as triggers.
Cooldown periods prevent thrashing. After scaling up, wait several minutes before scaling again. New instances take time to start and join the load balancer pool. Scaling down needs a longer cooldown because terminating instances is riskier than adding them.
auto_scaling_policy:
metric: cpu_utilization
target: 70
scale_up:
adjustment: 2
cooldown: 300
scale_down:
adjustment: -1
cooldown: 900
min_instances: 3
max_instances: 20
Pre-warming helps with predictable spikes. If you know traffic jumps every Monday at 9 AM, schedule extra capacity to launch at 8:45 AM. Reactive autoscaling alone leaves you under-provisioned for the first few minutes of a surge.
Pattern 7: Immutable Infrastructure with Blue-Green Deployments
Patching running servers introduces risk. Configuration drift makes environments inconsistent. Immutable infrastructure treats servers as disposable.
Build a new machine image with every code change. Launch a parallel environment with the new version. Test it. Route traffic over. Destroy the old environment. Nothing ever gets patched in place.
Blue-green deployment minimizes downtime. Blue is your current production environment. Green is the new version. Both run simultaneously. Your load balancer points to blue. When green passes health checks, switch the load balancer to green. If anything breaks, switch back to blue instantly. No rollback scripts or emergency patches needed.
# Build new AMI with Packer
packer build app-v2.json
# Launch green environment
terraform apply -var="app_version=v2" -var="environment=green"
# Health check green
curl https://green.example.com/health
# Switch traffic
aws elbv2 modify-listener --listener-arn $LISTENER \
--default-actions TargetGroupArn=$GREEN_TG
# Destroy blue after monitoring green
terraform destroy -var="environment=blue"
This pattern only works if your application is stateless or state lives in external data stores. Servers that accumulate local data can't be discarded safely.
How to choose patterns for your workload
Start with load balancing and read replicas. They solve the most common scaling bottlenecks with the least complexity.
Add autoscaling when traffic patterns are unpredictable or spiky. If load is steady, fixed capacity costs less.
Circuit breakers matter when you depend on flaky external services. Internal microservices talking to each other need them too.
Multi-region and blue-green are for mature systems where downtime has serious business impact. They're expensive and operationally complex. Don't start there.
Common questions
Do I need all seven patterns at once?
No. Most applications start with horizontal load balancing and database read replicas. Add other patterns as specific problems appear. Over-engineering early wastes time and money.
How do I test failover before production?
Chaos engineering tools let you kill random instances, inject network latency, or fill disks to verify your system recovers. Test in staging first. Start with small disruptions during low-traffic hours in production.
What if my database doesn't support replication?
Switch databases. Any production system needs replication. SQLite and single-instance setups don't belong in high-availability architectures.
Can I mix these patterns with serverless?
Yes. Serverless functions autoscale automatically but you still need circuit breakers for external calls, database replication for shared state, and multi-region deployments for geographic redundancy.
How much does multi-region cost compared to single-region?
Roughly double the infrastructure cost plus data transfer fees between regions. Cross-region bandwidth is expensive. Budget carefully and measure whether the availability gain justifies the expense.
Where to start tomorrow
Pick your biggest bottleneck right now. Single database choking on reads? Add replicas. One server handling all traffic? Put a load balancer in front of two or three instances.
Test failure scenarios in staging. Shut down a server manually and watch what happens. If users see errors, your architecture isn't ready.
Monitor everything. You can't fix what you can't measure. Track request latency, error rates, database query time, and server health. Set alerts before thresholds become outages.
These patterns aren't theoretical. They're how production systems stay up when traffic spikes and components fail. Build them in before you need them.