Kubernetes Cluster Sprawl: How to Find It and Keep It Under Control
Cluster sprawl is what happens when every team, environment, customer or experiment gets its own Kubernetes cluster and nobody removes them afterwards. Each decision made sense at the time. The total is a fleet that costs more than it should, runs on different versions and is hard to secure.
This post covers where the cost actually comes from, how to take inventory, and what to put in place so the number of clusters only grows when there is a real reason.
Where the cost comes from
A cluster is never just its workloads. Every cluster carries a fixed overhead:
- Control plane. EKS and GKE bill a management fee per cluster, and so do the paid AKS tiers. Check your provider's pricing page for current numbers.
- System add-ons. CNI, CoreDNS, kube-proxy, CSI drivers, ingress controller, cert-manager, metrics and log agents. They run in every cluster and reserve CPU and memory on every node.
- Minimum node count. Most clusters keep a few nodes running even when idle, often spread across zones.
- Network. Load balancers, NAT gateways and cross-zone traffic are billed per cluster or per VPC, not per workload.
- Engineering time. Each cluster needs upgrades, certificate rotation, add-on updates and access reviews. Ten clusters means ten upgrade projects every Kubernetes minor release.
The last item is the one that usually hurts most, and it never shows up on the cloud bill.
Typical causes
- A team asks for "its own cluster" for isolation that namespaces would have given.
- One cluster per service per environment, created from a template.
- CI pipelines or demos that create clusters and never delete them.
- Proof-of-concept clusters that quietly became production.
- No owner or expiry recorded anywhere, so nobody feels safe deleting anything.
Step 1: Inventory
Start with a list of every cluster in every account and region.
aws eks list-clusters --region eu-central-1
gcloud container clusters list
az aks list -o table
For each cluster, record owner, environment, purpose, Kubernetes version and creation date. If the cloud tags are missing, that is your first finding. Make tags like owner, env and expires mandatory from now on.
Step 2: Measure utilization
Compare three numbers per cluster: allocatable capacity, what pods request, and what pods actually use. With kube-state-metrics and cAdvisor metrics in Prometheus:
# share of allocatable CPU that is requested
sum(kube_pod_container_resource_requests{resource="cpu"})
/ sum(kube_node_status_allocatable{resource="cpu"})
# actual CPU usage in cores
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m]))
Low requests against allocatable means the nodes are too big or too many. High requests with low usage means requests are inflated. Both are fixable without touching cluster count, but they also tell you which clusters could be merged.
For cost per namespace or label, OpenCost (a CNCF project) and products built on it allocate node, storage and network cost to workloads. Without it, kubectl top nodes and kubectl describe node (the "Allocated resources" section) give a rough picture.
Also look for clusters with nothing meaningful running: only system namespaces, deployments scaled to zero, or no deploys in months.
Step 3: Decide what really needs a separate cluster
Separate clusters are justified when you need:
- A hard compliance or data boundary (for example, regulated data that must not share nodes or a control plane with other workloads).
- Different Kubernetes versions or cluster-wide configuration (CRD versions, admission policies, CNI).
- Separate failure domains: regions, or isolating production from everything else.
- Hostile multi-tenancy, where tenants run untrusted code.
Most other cases are covered by a shared cluster with per-team namespaces.
Step 4: Consolidate with namespaces
A namespace on its own is not isolation. Give each tenant namespace the same baseline:
apiVersion: v1
kind: Namespace
metadata:
name: team-payments
labels:
owner: payments
pod-security.kubernetes.io/enforce: restricted
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute
namespace: team-payments
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.memory: 60Gi
---
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: team-payments
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 128Mi
default:
memory: 256Mi
Add RBAC that scopes the team to its namespace, and a default-deny NetworkPolicy with explicit allow rules. Pod Security Admission labels block privileged pods. If a team needs its own CRDs or cluster-scoped resources, virtual clusters (for example vCluster) run a separate control plane inside a namespace of a shared host cluster.
Migrate one team at a time and delete the old cluster only after its workloads have run on the shared one for a while.
Step 5: Stop new sprawl
- Create clusters only through code. Terraform or Crossplane in a reviewed repository, with required tags. Restrict cloud IAM so only the pipeline role can create clusters.
- Give ephemeral environments a TTL. Prefer a namespace per pull request over a cluster per pull request. Label them and clean up on a schedule:
# delete namespaces labeled ephemeral=true that are older than 3 days (GNU date)
cutoff=$(date -u -d '-3 days' +%Y-%m-%dT%H:%M:%SZ)
kubectl get ns -l ephemeral=true -o json \
| jq -r --arg c "$cutoff" '.items[] | select(.metadata.creationTimestamp < $c) | .metadata.name' \
| xargs -r kubectl delete ns
- Manage add-ons for the whole fleet. A GitOps tool that targets many clusters (Argo CD ApplicationSets, Flux) keeps add-on versions the same everywhere and makes the per-cluster cost of upgrades smaller.
- Review the inventory regularly. Any cluster without an owner or past its expiry date goes on a deletion list.
Checklist
- Every cluster has an owner, environment and expiry tag.
- Requests, allocatable and real usage are tracked per cluster.
- New clusters need a documented reason: compliance, version, region or untrusted tenants.
- Tenants share clusters through namespaces with quotas, limits, RBAC, NetworkPolicy and Pod Security Admission.
- Clusters are created only from code by a pipeline role.
- Ephemeral environments are namespaces with automatic cleanup.
