Running containers in production means dealing with orchestration, and Kubernetes remains the standard choice. I've walked through dozens of cluster setups for clients moving from Docker Compose or manual container management to a proper orchestration platform. The jump feels steep at first, but the process breaks down into five repeatable steps that work whether you're building a three-node cluster in a data center or a managed setup in the cloud.
This guide covers the full path from choosing your infrastructure to serving live traffic, with the configuration files and troubleshooting commands I actually use.
Step 1: Choose your control plane architecture
Your control plane runs the API server, scheduler, and controller manager. For production, you need at least three control plane nodes to survive a failure. A single control plane node is fine for development or staging, but one hardware fault takes down your entire cluster.
I run control plane nodes on separate machines from worker nodes. Some teams collapse everything onto the same hardware to save costs, but that creates scheduling conflicts and makes upgrades riskier. Keep them separate if you can.
The minimum viable production cluster looks like this: - Three control plane nodes (2 CPU, 4GB RAM each) - Three worker nodes (4 CPU, 8GB RAM each, scaled up based on workload) - An external etcd cluster or embedded etcd on control plane nodes
For etcd, embedded mode is simpler to manage. External etcd gives you more isolation if you need to snapshot and restore the cluster state frequently, but it adds three more machines to maintain.
Step 2: Install and bootstrap with kubeadm
Kubeadm handles the heavy lifting of certificate generation, component configuration, and node joining. Other tools exist—kubespray, k3s, managed services—but kubeadm gives you the most control over the result and works on any Linux distribution.
On your first control plane node, install the runtime and Kubernetes packages:
# Install containerd
apt-get update
apt-get install -y containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# Enable systemd cgroup driver
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl restart containerd
# Install kubeadm, kubelet, kubectl
apt-get install -y apt-transport-https ca-certificates curl
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' > /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
Now initialize the first control plane node:
kubeadm init --control-plane-endpoint="loadbalancer.example.com:6443" \
--upload-certs \
--pod-network-cidr=10.244.0.0/16
The --control-plane-endpoint should point to a load balancer in front of your control plane nodes. If you don't have one yet, use the IP of this first node and add a load balancer later—just know you'll need to update kubeconfigs and certificates when you do.
Save the join commands kubeadm prints. You'll need them to add control plane and worker nodes.
Copy the admin kubeconfig:
mkdir -p $HOME/.kube
cp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
Step 3: Configure pod networking
Kubernetes doesn't ship with a network plugin. Pods can't communicate across nodes until you install one. Calico and Cilium are the two I deploy most often.
Calico is straightforward and works everywhere:
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
Cilium gives you eBPF-based networking and better observability, but it requires a recent kernel:
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system
After applying the network plugin, watch the CoreDNS pods come up:
kubectl get pods -n kube-system -w
They should transition to Running within a minute. If they stay Pending, check that your pod network CIDR doesn't overlap with your node network.
What if pods stay in ContainerCreating?
Nine times out of ten, it's the CNI plugin not finishing its configuration. Check the node's CNI config:
ls /etc/cni/net.d/
You should see a 10-calico.conflist or similar file. If it's missing, the network plugin didn't install correctly. Check the calico-node or cilium-agent logs:
kubectl logs -n kube-system -l k8s-app=calico-node
Common issues include missing kernel modules (check lsmod | grep ip_tables) or firewall rules blocking VXLAN or BGP traffic.
Step 4: Add nodes and configure storage
Join additional control plane nodes with the command kubeadm printed earlier:
kubeadm join loadbalancer.example.com:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane --certificate-key <key>
Join worker nodes without the --control-plane flag:
kubeadm join loadbalancer.example.com:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash>
If you lost the join command, generate a new token:
kubeadm token create --print-join-command
Once nodes are joined, verify:
kubectl get nodes
All nodes should show Ready status.
For persistent storage, you need a StorageClass. If you're running on bare metal, use local-path-provisioner for simple cases or Longhorn for replicated block storage. On cloud providers, use the CSI driver for your platform.
Local-path example:
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Now PersistentVolumeClaims will automatically provision volumes on worker node disks.
Step 5: Deploy your first workload and expose it
Start with a simple deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-demo
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.25
ports:
- containerPort: 80
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
Apply it:
kubectl apply -f nginx-demo.yaml
kubectl get pods -l app=nginx
Expose the deployment with a Service:
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
On cloud platforms, type: LoadBalancer provisions an external load balancer automatically. On bare metal, you'll need MetalLB or an external load balancer.
For HTTP/HTTPS traffic, install an Ingress controller instead. NGINX Ingress is the most widely deployed:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.0/deploy/static/provider/baremetal/deploy.yaml
Create an Ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
spec:
ingressClassName: nginx
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
Point your DNS at the Ingress controller's external IP:
kubectl get svc -n ingress-nginx
How do I debug when a service is unreachable?
Start inside the cluster. Exec into a pod and try to reach the service by its ClusterIP:
kubectl run -it --rm debug --image=busybox --restart=Never -- sh
wget -O- http://nginx-service
If that works, the problem is in your Ingress or external routing. If it doesn't, check endpoints:
kubectl get endpoints nginx-service
Empty endpoints mean the Service selector doesn't match any pods. Check your labels:
kubectl get pods --show-labels
If endpoints exist but traffic doesn't reach the pods, check network policies and CNI logs.
Hardening and day-two operations
Once the cluster is running, lock it down. Disable anonymous auth and enable audit logging in /etc/kubernetes/manifests/kube-apiserver.yaml:
- --anonymous-auth=false
- --audit-log-path=/var/log/kubernetes/audit.log
- --audit-log-maxage=30
Restrict pod permissions with PodSecurityStandards. Create a namespace with restricted enforcement:
kubectl create namespace production
kubectl label namespace production pod-security.kubernetes.io/enforce=restricted
Set up monitoring with Prometheus and Grafana. The kube-prometheus-stack Helm chart bundles everything:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace
For backups, snapshot etcd regularly:
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
Store snapshots off-cluster. I've seen teams lose clusters and backups at the same time because they kept snapshots on the same disk.
Common questions
How many control plane nodes do I actually need?
Three is the minimum for high availability. Five gives you more fault tolerance but increases etcd write latency. Stick with three unless you're running a massive cluster.
Can I mix cloud and on-premises nodes?
Yes, but networking gets complex. You'll need a VPN or WireGuard mesh to let pods communicate across environments. Cluster API providers can help automate this.
What's the difference between a DaemonSet and a Deployment?
A DaemonSet runs one pod per node, usually for logging or monitoring agents. A Deployment runs a specific number of replicas scheduled anywhere in the cluster.
How do I upgrade the cluster?
Upgrade control plane nodes one at a time with kubeadm upgrade apply, then upgrade workers with kubeadm upgrade node. Always upgrade in minor version steps—don't skip versions.
Should I use Helm for everything?
Helm simplifies complex applications with lots of templating, but plain YAML is easier to read and debug. Use Helm for third-party charts, plain manifests for your own apps.
What to monitor after deployment
Watch these metrics daily in your Prometheus dashboards: node CPU and memory pressure, pod restart counts, persistent volume usage, and API server request latency. High restart counts usually mean your resource limits are too low or your app is crashing. API server latency spikes indicate etcd performance problems or too many watchers.
Set alerts for node NotReady transitions and pod evictions. Both signal capacity or stability issues that need immediate attention.
The first production cluster always feels overwhelming. Start small, add nodes as load grows, and keep your manifests in git so you can recreate the setup if disaster strikes.