Every production Kubernetes cluster I've worked with had at least one preventable security gap. The default install is built for convenience, not defense. Fixing that means configuring seven core controls before your first real workload goes live.
1. Enable Role-Based Access Control (RBAC)
RBAC decides who can touch what inside your cluster. Without it, any authenticated user can read secrets or delete deployments.
Most managed Kubernetes services enable RBAC by default now. If you're running your own cluster, check first:
kubectl api-versions | grep rbac
You should see rbac.authorization.k8s.io/v1. If not, edit your API server flags and add --authorization-mode=Node,RBAC.
Create a read-only role for developers who need visibility but shouldn't change production:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list"]
Bind it to a user or service account:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: production
subjects:
- kind: User
name: [email protected]
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Apply both:
kubectl apply -f role.yaml
kubectl apply -f rolebinding.yaml
Start with the principle of least privilege. Grant only the permissions each user or service actually needs. I've seen clusters where every developer had cluster-admin because "it was easier."
2. Enforce Pod Security Standards
Pod Security Standards replaced Pod Security Policies in Kubernetes 1.25. They stop pods from running as root, using the host network, or mounting sensitive host paths.
Three levels exist: Privileged (unrestricted), Baseline (minimal restrictions), and Restricted (hardened). Most production namespaces should run Restricted.
Label your namespace to enforce the standard:
kubectl label namespace production \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/audit=restricted \
pod-security.kubernetes.io/warn=restricted
Now try deploying a pod that runs as root:
apiVersion: v1
kind: Pod
metadata:
name: bad-pod
namespace: production
spec:
containers:
- name: app
image: nginx
securityContext:
runAsUser: 0
Kubernetes will reject it. The error message tells you exactly what violated the policy.
For workloads that need elevated permissions (monitoring agents, CNI pods), create a separate namespace with Baseline or Privileged enforcement. Keep those namespaces small and audited.
3. Configure Network Policies
By default, every pod can talk to every other pod. An attacker who compromises one container can pivot to your database or internal APIs.
Network policies work like firewall rules. They require a CNI plugin that supports them—Calico, Cilium, and Weave Net all do. The basic Kubenet plugin does not.
Check if your CNI supports network policies by creating a test policy and seeing if it takes effect.
Start with a default-deny rule for your production namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
That blocks everything. Now whitelist only the traffic you need. For example, allow your frontend pods to reach the backend API:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: production
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
Also allow DNS lookups for all pods:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
ports:
- protocol: UDP
port: 53
Test each policy change carefully. A misconfigured rule can break service discovery or external API calls.
4. Secure etcd with TLS and Encryption at Rest
Etcd stores every secret, config map, and cluster state object. If an attacker reads the etcd data files directly, they own your cluster.
Most managed Kubernetes platforms handle this for you. For self-hosted clusters, configure TLS between the API server and etcd, and enable encryption at rest.
Create an encryption config file:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
Generate the key:
head -c 32 /dev/urandom | base64
Save the config to /etc/kubernetes/encryption-config.yaml on each control plane node, then add this flag to your API server manifest:
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml
Restart the API server. Existing secrets are still stored in plaintext until you rewrite them:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
Rotate the encryption key every few months and keep old keys in the config temporarily so Kubernetes can decrypt secrets written with them.
5. Restrict Service Account Tokens
Every pod gets a service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token by default. If your app doesn't call the Kubernetes API, that token is unnecessary risk.
Disable automatic token mounting at the service account level:
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-sa
namespace: production
automountServiceAccountToken: false
Or disable it per pod:
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
serviceAccountName: app-sa
automountServiceAccountToken: false
containers:
- name: app
image: myapp:latest
For pods that do need API access, create a dedicated service account with minimal RBAC permissions. Never use the default service account for application workloads.
Bound service account tokens (introduced in Kubernetes 1.21) expire automatically and are scoped to a specific audience. Use them instead of long-lived tokens:
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
serviceAccountName: app-sa
containers:
- name: app
image: myapp:latest
volumeMounts:
- name: token
mountPath: /var/run/secrets/tokens
volumes:
- name: token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 3600
audience: my-api
6. Use External Secrets Management
Kubernetes Secrets are base64-encoded, not encrypted in the API by default (unless you enabled encryption at rest). They're also stored in Git if you commit your manifests.
Integrate an external secrets manager: AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, or Google Secret Manager. The External Secrets Operator syncs secrets from these systems into Kubernetes without storing credentials in your repo.
Install the operator:
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace
Configure a SecretStore that points to your secrets backend:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
Create an ExternalSecret resource that fetches a secret:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-password
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets
kind: SecretStore
target:
name: db-password
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: prod/db/password
The operator creates a Kubernetes Secret named db-password with the value fetched from AWS Secrets Manager. Your pods consume it like any other secret.
Rotate secrets in the external store. The operator will pick up changes automatically.
7. Enable Audit Logging
Audit logs record every API request: who did what, when, and whether it succeeded. They're your forensic trail after an incident.
Define an audit policy that logs important events but filters noise:
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
verbs: ["create", "update", "patch", "delete"]
resources:
- group: ""
resources: ["secrets", "configmaps"]
- level: Metadata
verbs: ["get", "list", "watch"]
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: ""
resources: ["endpoints", "services"]
Save it to /etc/kubernetes/audit-policy.yaml and add these flags to the API server:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100
Logs land in /var/log/kubernetes/audit.log in JSON format. Ship them to a SIEM or log aggregator—Elasticsearch, Splunk, or a managed service. Keeping logs only on the control plane node makes them easy to delete or tamper with.
Set up alerts for suspicious patterns: repeated authentication failures, secret reads by unexpected service accounts, or privilege escalation attempts.
What happens after you configure these
Your attack surface shrinks. An attacker who compromises a pod faces RBAC restrictions, network isolation, and no service account token to escalate with. Secrets aren't sitting in Git or etcd plaintext. Audit logs give you visibility.
These seven controls don't cover everything—image scanning, admission controllers, supply chain security, and runtime monitoring all matter too. But they form a baseline. Without them, you're running production workloads on a platform designed for local development.
Check your current cluster against this list. Most gaps take an hour or two to fix. The ones that take longer (external secrets, CNI migration) can be scoped into your next sprint.
Common questions
Do I need all seven configs for a development cluster?
No. Dev clusters can skip network policies and external secrets to reduce complexity. Still enable RBAC and pod security standards so developers test against realistic constraints.
Which CNI should I use for network policies?
Calico is the most widely deployed. Cilium offers advanced features like L7 filtering and eBPF performance. Weave Net is simpler but less actively maintained now. Pick based on your team's familiarity and feature needs.
Can I migrate to Pod Security Standards from Pod Security Policies?
Yes. Run both in audit mode first to see what would break. Fix non-compliant workloads, then switch to enforce mode. PSPs were removed in Kubernetes 1.25, so migration is required for newer clusters.
How often should I rotate encryption keys and secrets?
Rotate encryption keys quarterly. Application secrets depend on sensitivity—database passwords every 90 days, API tokens when team members leave or every six months, whichever comes first.
What if my app needs to run as root?
Rethink the requirement first. Most apps that think they need root actually just need specific capabilities like NET_BIND_SERVICE. If root is truly necessary, isolate that workload in a separate namespace with relaxed pod security and aggressive network policies.
Verify your baseline before adding workloads
Check RBAC with kubectl auth can-i --list --as=system:serviceaccount:production:default. Confirm network policies by testing blocked and allowed traffic between pods. Review audit logs daily until you're confident the signal-to-noise ratio is right.
Security isn't a checklist you complete once. These configs are your foundation. Build monitoring, incident response, and regular reviews on top of them.