Autoscaling Failure Across Four Cloud Providers

Autoscaling failed when the group did not move toward the size the policy asked for, or when new instances never became healthy. A delayed scale-in is often configuration, not a broken autoscaler. Related: quota exhaustion if the activity text is a limit.

For Kubernetes node-group scaling, use this runbook on the node pool or Auto Scaling group, then Kubernetes cluster cost increase if the group scaled and the bill is the problem.

Failure classes

Treat these as different incidents. The first command is the same; the next action is not.

Class	What you see	What it is not
Policy did not request capacity	No new activity in the window; metric never crossed the threshold, or the policy is disabled / processes are suspended	A launch or quota failure
Request blocked by quota	Activity failed with a vCPU, instance, IP, or regional quota message	Regional stock of a SKU
Provider capacity shortage	AWS InsufficientInstanceCapacity / UnfulfillableCapacity; Azure allocation failure; Google Cloud ZONE_RESOURCE_POOL_EXHAUSTED; Alibaba activity text about insufficient inventory	Your policy math
Instance launch failure	Activity failed on image, launch template, IAM instance profile, subnet, or user data	Health checks after a successful launch
Launched but failed health checks	Instances exist; ELB/target, Azure health extension, MIG autohealing, or ESS health check marks them unhealthy	The autoscaler refusing to launch
Scale-in protection or cooldown	Desired stays high after load drops; AWS instance protection, Azure scale-in policy, GCP scaleInControl, ESS cooldown	A stuck scale-out

Cooldown, instance warmup, Azure flapping rules and GCP stabilization delay the next change on purpose. Copy the timestamps. If the last successful activity is still inside the documented window, wait or adjust the window; do not raise max.

Symptoms

Instance count stuck below desired while CPU or queue depth stays high. Scale-in never fires after load drops. New instances stay Pending, Unhealthy or fail the load-balancer check. The autoscaler reports Failed (AWS StatusCode), a failed Azure autoscale run, a MIG that cannot reach target size, or an ESS activity that did not succeed.

Business impact

Scale-out failure leaves the service short of capacity. Error rate and latency rise. Scale-in failure leaves extra instances running and shows up later as a cost spike. Repeated failed launches can consume quota and IP space without adding serving capacity.

Immediate checks

1. Name the Auto Scaling group, VM scale set, managed instance group or ESS group, plus region.
2. Record current min, desired, max and the metric the policy uses.
3. Read the last several scaling activities or autoscale runs. Copy the status text verbatim, including AWS StatusCode and Cause, Azure operation status, GCP error or autoscaler recommendation, and ESS activity status.
4. Check cooldown, warmup, health-check grace period, Azure scale-in flapping settings, GCP cool-down / scaleInControl, and instance protection flags.
5. Check whether a deploy, AMI change, or launch-template version landed in the same window.
6. Check load-balancer target health. Unhealthy new instances look like an autoscaling bug and are often an application or health-check bug.

Provider commands and console paths

Commands are per provider. Do not run an AWS CLI against Azure, Google Cloud or Alibaba Cloud.

AWS. EC2, Auto Scaling groups, Activity tab. AWS documents StatusCode values including Successful, Failed and Cancelled on describe-scaling-activities. Launch-failure causes are listed in instance launch failures.

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names GROUP
aws autoscaling describe-scaling-activities --auto-scaling-group-name GROUP
aws autoscaling describe-warm-pool --auto-scaling-group-name GROUP
aws elbv2 describe-target-health --target-group-arn ARN

Azure. Virtual machine scale set, Scaling / Autoscale. Activity log for failed allocations. Autoscale history is an Azure Monitor feature, not the AWS Activity tab.

az vmss show --resource-group RG --name SCALESET
az vmss list-instances --resource-group RG --name SCALESET -o table
az monitor autoscale list --resource-group RG
az monitor autoscale show --resource-group RG --name AUTOSCALE_NAME

See VM scale set autoscale and az monitor autoscale.

Google Cloud. Compute Engine, instance group, Monitoring / Autoscaler. Decision history and recommended size are documented in Understanding autoscaler decisions.

gcloud compute instance-groups managed describe GROUP --region=REGION
gcloud compute instance-groups managed list-instances GROUP --region=REGION
gcloud compute instance-groups managed describe-instance GROUP --instance=INSTANCE --region=REGION

Alibaba Cloud. Auto Scaling console, scaling group, Scaling Activities. Status names are ESS activity statuses, not AWS StatusCode values.

aliyun ess DescribeScalingActivities --RegionId cn-hangzhou --ScalingGroupId asg-xxxxxxxx

DescribeScalingActivities requires RegionId and either ScalingGroupId or activity IDs. History coverage is documented on that page (recent activities only).

Activity message decision table

Provider status names differ. Match the text you copied, then take the action for that class.

Activity or status text (qualified)	Likely cause	Next action
No activity in the incident window	Policy did not request capacity: metric below threshold, disabled rule, or AWS scaling processes suspended	Confirm the alarm or metric, resume processes, then wait for a new activity
Still inside cooldown, warmup, Azure flapping window, or GCP cool-down	Configured delay, not a failure	Wait out the window or change it in a change ticket
Quota, vCPU, InstanceLimitExceeded, Azure quota, GCP quota, Alibaba Quota Center	Request blocked by quota	Stop. Use quota exhaustion. Raising max will not launch
AWS InsufficientInstanceCapacity or UnfulfillableCapacity; Azure allocation failed; GCP ZONE_RESOURCE_POOL_EXHAUSTED; ESS insufficient inventory	Provider capacity shortage in that AZ or SKU	Change AZ, add a documented alternative instance family, or wait. This is not policy math
Image, launch template, IAM profile, subnet, ENI or IP exhaustion	Instance launch failure	Roll back the template or expand the subnet. Do not raise max
Instances InService / running but ELB, Azure probe, MIG autohealing or ESS health fails	Launch succeeded; health check or application failed	Fix the probe path or the app. More capacity will fail the same way
AWS instance scale-in protection, Azure instance protection, GCP scaleInControl, ESS cooldown after scale-out	Scale-in is blocked or delayed on purpose	Clear protection or wait. Do not treat this as a scale-out bug

Interpretation

InsufficientInstanceCapacity (AWS) or regional stock messages: the SKU is short in that AZ. Try another AZ or a documented alternative instance family.

Quota or vCPU limit in the activity text: stop here and use quota exhaustion. Raising max will not launch.

Subnet or ENI / IP exhaustion: the VPC or VNet ran out of addresses. Expand the subnet or reduce max.

Launch template, image or identity errors: the new version cannot boot or cannot register. Roll back to the last template that produced healthy instances.

Instances launch, fail ELB or the equivalent probe: the group is working. The application, listener or probe path is not.

Scale-in disabled, instance protection, or a metric that never returns to the low threshold: the group is doing what you configured.

Safe mitigation

Fix the cause in the status message. Restore the last known-good launch template or instance definition. If the SKU is unavailable, change AZ or family using a type the provider still offers in that region.

If quota is the blocker, request the increase and temporarily add capacity only in a region or family that still has headroom.

A manual set of desired capacity is acceptable as a short bridge after you know launches will succeed. Record who will set it back, and do not leave a CI job that overwrites desired on every apply.

Do not disable scale-in unless you are in a change window and someone will re-enable it.

Risky actions to avoid

Do not set max to a large number to “make it scale.” You can exhaust quota, IP space and budget without serving a single healthy instance.

Do not delete the scaling group or the load balancer to reset state.

Do not turn off health checks to force InService. You will send traffic to instances that cannot serve it.

Do not mix a second autoscaler (custom script plus managed policy) on the same group without an owner. They fight.

Do not hard-set desired capacity in every infrastructure-as-code apply while a target-tracking, step, scheduled or ESS policy is the runtime owner of desired. The next apply will undo the policy.

Infrastructure as code and desired capacity

Terraform, CloudFormation, Bicep, Deployment Manager or ROS may define initial min, desired and max when the group is created. After the group is live, the autoscaler changes desired as the metric moves. That is documented product behavior on AWS, Azure, Google Cloud and Alibaba Cloud.

Keep min, max, launch template and the policy in version control. Treat desired as a runtime value unless you are deliberately pinning size (for example a change freeze). If the module always writes desired, every apply fights the autoscaler and can look like a scale-in or scale-out failure.

Validation

A new activity should complete with Successful (AWS StatusCode), a succeeded Azure autoscale operation, a MIG at the autoscaler’s recommended size, or a successful ESS activity. Healthy target count should equal the intended desired.

Generate a small load or raise the metric in a test environment if you cannot wait for production traffic. Confirm scale-in after the metric drops, so you did not leave protection on.

Prevention

1. Alert on failed scaling activities (AWS StatusCode=Failed, failed Azure autoscale operations, MIG errors, failed ESS activities), not only on high CPU.
2. Inventory the vCPU, instance, IP and regional quotas the launch template needs before a launch or seasonal peak. Pair the review with quota exhaustion.
3. Run a scale-out drill after every image or launch-template change, in a non-production group first.
4. Document cooldown, warmup, health-check grace, Azure flapping settings, GCP cool-down / scaleInControl, and which instances have scale-in protection.
5. Keep min, max and the policy in version control. Do not let CI overwrite desired after the autoscaler owns it.
6. Watch request success and latency from outside the VPC as well as the load-balancer probe. External checks catch cases where instances are InService but users still fail. Vigiles is the sister site for that class of monitor; it does not replace the provider activity log.
7. One autoscaler per group. If a custom controller also sets desired, name the owner and the override window.