Introduction

Kubernetes has become the de facto standard for container orchestration. However, running it in production requires careful planning and operational maturity.

Cluster Architecture

Control Plane High Availability

• Run 3+ control plane nodes across availability zones
• Use managed Kubernetes (EKS, AKS, GKE) when possible
• Separate etcd to dedicated nodes for large clusters

Node Pools

Node Pools:
├── system-pool (m5.large)
│   └── Core system components
├── general-pool (m5.xlarge)
│   └── Standard workloads
├── memory-pool (r5.2xlarge)
│   └── Memory-intensive apps
└── gpu-pool (p3.2xlarge)
    └── ML workloads

Resource Management

Requests and Limits

Always set both requests and limits:

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Pod Disruption Budgets

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Priority Classes

Ensure critical workloads survive resource pressure:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false

Networking

Network Policies

Implement zero-trust networking:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-policy
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - port: 8080

Observability

The Three Pillars

• Metrics: Prometheus + Grafana
• Logs: Fluent Bit → Elasticsearch/Loki
• Traces: OpenTelemetry → Jaeger/Tempo

Essential Dashboards

• Cluster resource utilization
• Node health and capacity
• Pod restart rates
• API server latency
• etcd performance

Security

Pod Security Standards

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted

Image Security

• Scan images for vulnerabilities (Trivy, Clair)
• Sign images (Cosign, Notary)
• Use admission controllers to enforce policies

Disaster Recovery

• Regular etcd backups
• GitOps for configuration recovery
• Multi-region deployment strategy
• Tested recovery procedures

Lessons Learned

1. Start with managed Kubernetes
2. Invest in observability from day one
3. Automate everything with GitOps
4. Practice failure scenarios regularly
5. Keep clusters reasonably sized

Conclusion

Production Kubernetes requires significant operational investment. Build your platform team's expertise gradually and automate relentlessly.