The Cloud Native Computing Foundation launched its Kubernetes AI conformance program in November 2025, and the timing is more significant than you might think. By the end of 2026, two-thirds of compute dedicated to AI will be for inference, not training. This shift changes how you'll provision resources, and it's happening now.
What Changed
The CNCF's conformance program standardizes how AI workloads run on Kubernetes across cloud providers. Major platforms have adopted these standards, allowing you to move inference workloads between AWS, Google Cloud, and Azure without rewriting deployment configs for each vendor's GPU allocation quirks.
The program focuses on Dynamic Resource Allocation (DRA), a standardized API that lets Kubernetes handle GPUs, TPUs, and other accelerators like CPU and memory. Before DRA, each cloud provider had different GPU scheduling, requiring vendor-specific YAML and locking you into that provider's ecosystem.
Key Findings
Inference will dominate your infrastructure budget. By the end of the decade, 93 gigawatts of compute power will be dedicated to inference. Unlike training, which occurs infrequently, inference runs thousands of times per second for every active user. Your infrastructure planning needs to shift from supporting quarterly training runs to scaling inference for production traffic spikes.
Vendor lock-in is easier to avoid. Conformance tests ensure a Kubernetes distribution handles AI workloads consistently. If both your current and backup providers pass conformance, you can execute a multi-cloud strategy. This flexibility is crucial when facing price increases or regional outages.
Resource allocation is predictable across platforms. With DRA, you define resource requirements once. A pod requesting two NVIDIA A100 GPUs will get two A100 GPUs whether it's on EKS, GKE, or AKS. This consistency eliminates debugging issues caused by different resource scheduling across clouds.
The shift to inference changes your security model. Training happens in controlled environments with known datasets. Inference occurs at the network edge, processing user input in real-time. You need runtime protection for models, input validation that doesn't add significant latency, and audit logs that handle millions of requests without overwhelming your SIEM budget.
What This Means for Your Team
Your current Kubernetes manifests likely don't account for inference-specific requirements. Training jobs are batch workloads scheduled during off-peak hours. Inference is a real-time service that needs to respond in milliseconds, scale horizontally during traffic spikes, and maintain consistent latency across availability zones.
If you're running inference workloads today, you've probably hard-coded GPU allocation for your specific cloud provider. This works until you need to shift 30% of your traffic to another region due to capacity limits. With conformant Kubernetes distributions, you change the cluster endpoint in your deployment pipeline. Without conformance, you rewrite resource requests and hope you catch all vendor-specific annotations.
The conformance program also strengthens your position in vendor negotiations. When a cloud provider knows you can move workloads to a competitor quickly, pricing discussions become more productive.
Action Items by Priority
Audit your current AI workloads to identify which are for training and which are for inference. Document the GPU type, memory requirements, and expected request volume for each inference workload. This baseline is essential for evaluating whether conformant Kubernetes can reduce your operational overhead.
Test DRA in a non-production cluster. Deploy one of your inference workloads using DRA resource requests on a conformant Kubernetes distribution and measure the difference in deployment complexity. If you're using vendor-specific device plugins or custom schedulers, this test will show how much YAML you can eliminate.
Map your inference scaling requirements to Kubernetes autoscaling. Inference workloads need horizontal pod autoscaling based on request latency, not just CPU utilization. Configure HPA with custom metrics from your model serving framework. If you're using TensorFlow Serving or TorchServe, you already have the metrics endpoints.
Build a multi-cloud failover plan for at least one critical inference service. Deploy a model that handles production traffic to two conformant Kubernetes clusters in different clouds and configure your load balancer to route traffic to both. Run this for a month and measure the operational overhead. If it's minimal, expand to more services.
Review your model security controls for inference-specific threats. Training pipeline security focuses on data poisoning and model theft. Inference security needs input validation, rate limiting per API key, and monitoring for adversarial inputs. Your WAF rules may not catch model-specific attacks.
