Amazon EKS operates hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Their recent architectural changes highlight which assumptions about Kubernetes management fail at scale and which practices effectively prevent outages.
Here's what changed and what your team should implement now.
Changes in EKS Architecture
EKS replaced etcd's Raft consensus mechanism with a custom journal system. This was not a minor tweak. Raft, the algorithm that maintains cluster state consistency across control plane nodes, becomes a bottleneck when processing thousands of state transitions per second. The journal system decouples write performance from consensus overhead.
Another significant change: EKS introduced scaling tiers for provisioned control planes (XL through 8XL) with defined performance boundaries. This formalizes the distinction between "high availability" and "can handle your workload."
Key Findings: Where Standard Kubernetes Breaks
Finding 1: Faults don't equal outages, but your monitoring probably treats them the same
EKS differentiates between component faults (like a single API server restart) and service outages (user requests failing). Your cluster might experience numerous faults daily without impacting users. The issue: most monitoring setups alert on faults, leading your team to ignore warnings until a real outage occurs.
EKS tracks two metrics separately: component health and request success rate. A component can be unhealthy while requests still succeed due to redundancy. This is crucial for AI and analytics workloads where resources are created and destroyed rapidly.
Finding 2: etcd write amplification kills performance at scale
Every Kubernetes state change writes to etcd. With Raft consensus, each write requires acknowledgment from a majority of nodes. At low transaction rates, this overhead is negligible. However, for batch jobs creating hundreds of pods per second, Raft's coordination overhead becomes significant.
The journal-based approach batches writes and handles consensus asynchronously. While you can't implement this in upstream Kubernetes, you can design your workloads around it. Batch pod creation instead of creating pods individually. Use StatefulSets with parallel pod management policies. Reduce unnecessary status updates.
Finding 3: Control plane capacity is not elastic
EKS's tiered scaling model (XL through 8XL) highlights a critical truth: you can't auto-scale a control plane like application pods. The API server, scheduler, and controller manager have fixed resource limits. Exceeding these limits results in queued or failed requests.
Most teams discover this during their first large-scale deployment. You've tested with 100 pods, production needs 10,000, and suddenly your kubectl commands timeout. The solution requires planning: right-size your control plane before the workload, not after.
Finding 4: AI workloads break traditional capacity planning
Training jobs and batch analytics create spiky, high-velocity resource demands. A single job might request 1,000 GPUs, run for six hours, then release everything. Traditional Kubernetes capacity planning assumes relatively stable workloads with gradual scaling.
EKS adapted by optimizing scheduler performance and reducing the time between resource request and pod binding. Your team faces the same challenge. If you're running ML workloads, measure your scheduler latency under load. Track the time from pod creation to running state. These metrics reveal whether your control plane can handle the velocity.
Finding 5: Multi-tenancy multiplies state management complexity
Running hundreds of thousands of clusters means EKS can't treat each cluster as unique. They need consistent patterns for upgrades, security patches, and configuration management. Your team probably runs fewer clusters, but the principle applies: every custom configuration is technical debt.
What This Means for Your Team
If you're running Kubernetes in production, you're managing a distributed database (etcd) that most of your team doesn't understand. The control plane is a single point of failure for your entire workload, yet it rarely gets the same attention as application performance.
The EKS findings matter because they identify where standard Kubernetes patterns break. You don't need to run at Amazon's scale to hit these limits. A mid-sized team running AI training jobs or high-frequency batch processing will encounter the same bottlenecks.
Action Items by Priority
Priority 1: Separate fault monitoring from outage alerts
Configure your monitoring to track API server availability separately from request success rate. Alert on request failures, log component faults. This prevents alert fatigue and ensures your team responds to actual service degradation.
Implementation: Set up two Prometheus queries. One tracks kube-apiserver availability (component health). The second tracks request success rate from your applications' perspective. Only the second should page.
Priority 2: Load test your control plane before production
Spin up a test cluster and simulate your peak workload. Create and delete pods at production velocity. Measure scheduler latency and API server response times. Identify your breaking point before users do.
Use a tool like kube-burner or write a simple script that creates pods in parallel. Target 10x your normal workload. If your control plane can't handle it, you need a larger tier or different workload patterns.
Priority 3: Audit your etcd write patterns
Review your controllers and operators. Are they updating resource status unnecessarily? Are you creating resources one at a time when you could batch? Each write has a cost.
Look for controllers with tight reconciliation loops. Check for status updates that happen every few seconds. These patterns work fine at small scale but become bottlenecks as you grow.
Priority 4: Right-size your control plane proactively
Don't wait until your cluster is struggling. If you're planning to double your workload, upgrade your control plane capacity first. The cost difference between tiers is minimal compared to the cost of an outage.
For managed Kubernetes services, check the provider's scaling tiers and their performance specifications. For self-managed clusters, monitor control plane resource utilization and scale before you hit 70% capacity.
Priority 5: Standardize cluster configurations
Reduce configuration drift across your clusters. Use GitOps or infrastructure-as-code to ensure consistency. Every custom setting is a potential failure point during upgrades or incident response.
Pick a cluster configuration baseline and document exceptions. When you need to deviate, require a written justification. This discipline pays off during the next security patch or version upgrade.
