Your Kubernetes controllers worked fine at 50 nodes. Now you're planning for 5,000, and you need to ensure they won't crash, leak resources, or create security gaps. This checklist walks through the operational requirements for scaling controllers in production environments where compliance frameworks like SOC 2 Type II and ISO 27001 demand demonstrable system reliability.
This applies to any custom or platform controller managing network policies, resource allocation, or security enforcement at scale. Each item has a clear pass/fail state.
Prerequisites
Before using this checklist, confirm:
- You have baseline memory and CPU metrics for your controllers under current load.
- You can reproduce your production topology in a staging environment.
- You have access to controller source code or configuration options for caching behavior.
- Your monitoring stack can track per-controller resource consumption and reconciliation latency.
Checklist Items
1. Controller caches only fields it actively uses
Examine your controller's watch configuration. A cached pod object can consume tens of kilobytes. If you're caching full pod specs but only reading labels and status.phase, you're wasting hundreds of megabytes across thousands of objects.
✓ Good looks like: Your controller uses field selectors or custom transformers to cache only the 3-5 fields it actually reads during reconciliation. Memory consumption per cached object is under 2KB.
2. Irreversible actions wait for cache synchronization
Your controller must verify its cache reflects current cluster state before executing any action that can't be rolled back automatically (security group assignments, external API calls, resource deletions).
✓ Good looks like: Controller code explicitly checks cache.WaitForCacheSync() returns true before any write operation. Integration tests verify the controller refuses to act on stale data.
3. Resource cleanup runs on a defined schedule
Controllers that assign external resources (network interfaces, security groups, cloud provider resources) need periodic scans to reclaim orphaned assignments. Without this, you accumulate stale mappings that create security exposure and cost overruns.
✓ Good looks like: A documented cleanup interval (hourly or daily depending on churn rate) where the controller scans for resources no longer associated with running workloads and releases them. Cleanup runs are logged with counts of reclaimed resources.
4. Controller memory limits account for worst-case cache size
Calculate: (maximum expected objects) × (bytes per cached object) × 1.5 safety margin. Set memory limits accordingly. Controllers that hit OOM under load create availability gaps that violate SOC 2 availability commitments.
✓ Good looks like: Memory limits are set based on actual measurement, not guesses. You've tested with synthetic load at 2× your current scale and the controller stayed within limits.
5. Reconciliation loops have explicit timeouts
Long-running reconciliation blocks the controller from processing other objects. At scale, this creates cascading delays where security policy changes take minutes to propagate instead of seconds.
✓ Good looks like: Each reconciliation operation has a timeout between 5-30 seconds depending on complexity. Timeouts are logged with object identifiers for debugging. You've measured p99 reconciliation latency under load.
6. Controller exposes metrics for cache size and reconciliation backlog
You can't manage what you don't measure. Your monitoring system needs real-time visibility into controller health before problems become incidents.
✓ Good looks like: Prometheus metrics (or equivalent) tracking: objects in cache, reconciliation queue depth, reconciliation duration (p50/p95/p99), and error rates. Alerts fire when queue depth exceeds thresholds or reconciliation latency spikes.
7. Rate limiting protects upstream APIs
Controllers that call cloud provider APIs, admission webhooks, or external services need rate limiting. Without it, scaling from 100 to 1,000 nodes can trigger API throttling that cascades into controller failures.
✓ Good looks like: Configured rate limits (QPS and burst) for all external API clients. Rate limit values are documented with the reasoning behind them. You've tested behavior when limits are hit.
8. Leader election configuration matches your RTO
If your controller runs multiple replicas for availability, leader election timeout determines how long you're down when the leader crashes. Default timeouts (15-30 seconds) might violate your recovery time objectives.
✓ Good looks like: Leader election lease duration and renewal deadline are explicitly configured based on your availability requirements. You've tested failover time matches your documented RTO.
9. Object finalizers clean up external state
When Kubernetes objects are deleted, your controller must release associated external resources (security groups, cloud provider resources). Finalizers ensure this happens even during rapid scaling down.
✓ Good looks like: All objects managed by your controller have finalizers that block deletion until external cleanup completes. You've tested that force-deleting namespaces doesn't leak resources.
10. Controller handles API server unavailability gracefully
Network partitions happen. Your controller needs a strategy for when it can't reach the Kubernetes API server that doesn't involve making security decisions based on stale state.
✓ Good looks like: Controller implements exponential backoff for API retries. It stops making external changes if it hasn't successfully synced with the API server within a defined threshold (typically 60-120 seconds). Behavior is documented and tested.
Common Mistakes
Assuming informers are always current: Controllers often check if pod.Status.Phase == "Running" without verifying the cache is synchronized. This creates race conditions where you act on objects that were already deleted.
Setting memory limits too low: Controllers hit OOM, restart, rebuild cache, hit OOM again. This creates a crash loop that looks like a memory leak but is actually undersized limits.
No cleanup for orphaned resources: Security groups and network interfaces accumulate until you hit cloud provider quotas. Then new pods can't start, and you're troubleshooting a production outage instead of running a scheduled cleanup job.
Treating all reconciliation failures the same: Transient API errors need retry with backoff. Validation errors need alerting and manual intervention. Conflating them creates noise that hides real problems.
Next Steps
Run this checklist against each custom controller in your environment. For platform controllers (AWS VPC Resource Controller, network policy controllers), verify your configuration enables these behaviors even if you can't modify the code.
Document your findings in your system architecture documentation required by ISO/IEC 27001:2022 Annex A.8.9 (configuration management). Schedule quarterly reviews as your cluster scales.
If you're running controllers at 1,000+ nodes, you need load testing infrastructure that simulates scale. Build it now, before your next major scaling event forces you to debug controller behavior in production.
