Kubernetes cluster cost increase

A Kubernetes bill is a bundle: control plane, workers, disks, load balancers, NAT, logs and often GPU. Allocation comes after you know which of those moved. See Kubernetes cost allocation.

Symptoms

Cluster cost in Cost Explorer, Cost analysis or Billing reports is up. Node count or node SKU is higher than last week. kubectl get nodes shows new Ready nodes or a new GPU pool. Pending pods filled the cluster and Cluster Autoscaler or a managed autoprovisioner added nodes. LoadBalancer Services or unused PersistentVolumes remain after an app was “removed.”

Control-plane SKU changes (EKS cluster hours, AKS tier, GKE Autopilot vs Standard, ACK edition) can also move the bill with no pod change. Check that line separately.

Business impact

Worker nodes dominate most cluster bills. An autoscaler that cannot shed nodes, or a DaemonSet that pins a large pool, will keep paying until someone drains the pool. Leftover load balancers and disks keep billing after the Deployment is gone.

Deleting the cluster to stop the bill takes every namespace with it, including state you did not mean to drop.

Immediate checks

1. Name the cluster, region and the cloud account or subscription that pays for it.
2. Count nodes by pool and instance type. Note GPU and memory-optimized pools.
3. List Service objects of type LoadBalancer in every namespace.
4. List PersistentVolumeClaims and unbound PersistentVolumes.
5. Check autoscaler or node-autoprovisioning settings and recent scale events.
6. Check whether log, metrics or service-mesh agents were recently given a higher scrape rate.

Provider commands and console paths

AWS. EKS console, cluster and compute (node groups, Fargate profiles, EKS Auto Mode if you use it). Cost Explorer: filter by the cluster tag you actually set (eks:cluster-name only exists if you enabled the documented tagging).

aws eks describe-cluster --name CLUSTER
aws eks list-nodegroups --cluster-name CLUSTER
aws eks describe-nodegroup --cluster-name CLUSTER --nodegroup-name NODEGROUP
aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[?contains(Tags[?Key==`eks:cluster-name`].Value, `CLUSTER`)]'

Pricing units: Amazon EKS pricing.

Azure. AKS blade, Node pools. Cost analysis: filter the node resource group (MC_...) and the AKS resource.

az aks show --resource-group RG --name CLUSTER
az aks nodepool list --resource-group RG --cluster-name CLUSTER -o table

Tier and control-plane fees: AKS free and standard pricing tiers.

Google Cloud. Kubernetes Engine, cluster and Nodes. Billing reports: filter GKE and Compute Engine SKUs for the project.

gcloud container clusters describe CLUSTER --location=LOCATION
gcloud container node-pools list --cluster=CLUSTER --location=LOCATION

Pricing: GKE pricing (Standard vs Autopilot are different bills).

Alibaba Cloud. Container Service for Kubernetes console, cluster details and node pools. Billing Details, ACK plus ECS plus SLB plus disk.

aliyun cs GET /clusters/CLUSTER_ID

That call is DescribeClusterDetail. Node-pool list commands live on the same ACK API help index; use the console node-pool page if you do not have the list path handy.

In-cluster (any provider):

kubectl get nodes -o wide
kubectl get svc -A --field-selector spec.type=LoadBalancer
kubectl get pvc -A
kubectl get pv
kubectl describe nodes | grep -A5 'Allocated resources'

Interpretation

More nodes, same SKU: autoscaler, a new Deployment, or a stuck Pending queue.

Same node count, higher SKU: someone changed the pool machine type or moved to GPU.

Control-plane line only: tier change or an extra cluster, not pod growth.

Load-balancer or public-IP meters with no new nodes: leftover Services, ingress controllers or unused cloud load balancers.

Disk meters: snapshots, orphaned PVs, or a logging stack writing to large volumes.

Egress or NAT next to the cluster: image pulls and telemetry. Continue with the NAT or cross-region runbooks.

Safe mitigation

Scale a non-production node pool down after kubectl drain on the nodes you will remove. Set a lower max on the autoscaler once you know the Pending cause (resource requests that are too high are a common one).

Delete LoadBalancer Services you have confirmed unused, then watch the cloud load balancer disappear. Delete PVCs only when the data owner agrees.

Do not set production replicas to zero to save a night of cost.

If Autopilot, Standard, Fargate or a new edition is the driver, change the cluster mode only through the documented migration path for that product.

Risky actions to avoid

Do not delete the cluster, the node instance role, or the node resource group from the cloud console. Kubernetes-owned resources will not reconcile cleanly.

Do not remove Cluster Autoscaler or the managed equivalent while nodes are still scaling. You lose the mechanism that would shed them later.

Do not “rightsize” by editing requests to zero. That packs nodes and hides the next spike.

Do not buy a compute commitment against a node shape you have not explained.

Validation

kubectl get nodes matches the intended pool sizes. Cost Explorer or Cost analysis compute meters flatten the next complete day. Cloud load balancers and disks that you deleted no longer appear in the provider inventory.

If you changed autoscaler max, create a single test Deployment that would have overflowed, confirm it Pending or scales within the new cap, then delete it.

Prevention

1. Prefer one cluster per environment unless you already have namespace labels and a chargeback model. A shared cluster without labels cannot be explained on the invoice.
2. Require team / service labels on namespaces and use them in Kubernetes cost allocation.
3. Cap Cluster Autoscaler or node-pool maximums in version control. A missing max is an unbounded spend path.
4. Alert on node count and on LoadBalancer Service count, not only on the monthly invoice.
5. Review leftover PVCs in the same weekly pass as unused disks.
6. Read the pricing page for the cluster mode you actually run (EKS standard vs Auto Mode, AKS SKU, GKE Standard vs Autopilot, ACK). Prices change.