GPU and AI cost governance is a unit-cost model for training and inference, plus a process that treats idle accelerators as waste you can see in hours. List price per GPU-hour is not the unit. Training cost is GPU-hours times the effective rate, plus storage, plus egress to the dataset and checkpoint region. Inference cost is GPU-hours or token-path charges divided by successful requests. Reservations, Capacity Blocks and committed use lock both price and scarcity. They expire unused. Documentation dated 2026-08-30 is the SKU and product source. Generic EC2 rightsizing playbooks miss idle GPU and data-path meters.

The planning worksheet is the AI and GPU workload cost estimator. The practice name is What is AI FinOps?.

Assumptions

  • You can list running GPU or accelerator instances or node pools in at least one account or subscription.
  • Training and inference on self-managed accelerators are in scope. Managed token APIs are a separate meter, not forced into GPU-hours.
  • Reservations and Capacity Blocks are calendar commitments you already bought or are considering.
  • This is not a rightsizing playbook for general EC2 or VM families.

Inventory what is running, not what was requested

aws ec2 describe-instances \
  --filters Name=instance-state-name,Values=running \
            Name=instance-type,Values=p4*,p5*,g5*,g6*,trn1* \
  --query 'Reservations[].Instances[].{id:InstanceId,type:InstanceType,az:Placement.AvailabilityZone,launch:LaunchTime}' \
  --output table

Every row is a burning GPU-hour meter. LaunchTime older than the last completed job, with no serving traffic, is idle. Instance families differ by region. p5 in us-east-1 does not mean ap-southeast-1 has the same quota or price. AWS documents regional SKU availability in the Regions and Availability Zones material and the EC2 price list.

Azure: az vm list -d --query "[?contains(hardwareProfile.vmSize, 'NC') || contains(hardwareProfile.vmSize, 'ND')].[name,hardwareProfile.vmSize,location,powerState]" -o table. Google Cloud: gcloud compute instances list --filter='machineType~g2 OR machineType~a2 OR machineType~a3'.

On a node you already access, nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv showing 0 %, 0 MiB for hours is idle. That is a cost finding. It is also a capacity finding if another team is queued.

Unit cost that finance can reuse

Training job:

effective_gpu_hour_rate * gpu_count * wall_clock_hours + dataset_egress + checkpoint_storage

Use the effective rate after any reservation or spot discount you actually applied, not the public on-demand page if you are covered. Write the job ID next to the number. A monthly GPU bill with no job IDs is not AI FinOps.

Inference:

accelerator_hours / successful_requests or, for managed token APIs, billed_tokens / successful_requests. Include the queue time you keep GPUs warm to hit latency SLOs. Warm idle is a product choice. Hide it and the unit cost is fiction.

Managed model APIs (Amazon Bedrock, Azure OpenAI, Google generative APIs) are a different meter: tokens and provisioned throughput, not nvidia-smi. Do not force them into a GPU-hour model. Put them on a separate scorecard.

Reservations, scarcity and the landing zone

Capacity reservations and GPU Capacity Blocks buy a calendar slot. If the experiment slips, you still pay. Committed use discounts and reserved instances on GPU SKUs assume a utilization floor. Below that floor you wanted on-demand or a smaller pool.

Quotas for GPU families are regional and often start at zero. A DR or burst region without quota is not a standby. See cross-region disaster recovery and quota exhaustion.

Place training accounts or subscriptions in the landing-zone tree so SCPs can deny unexpected GPU families in a sandbox OU. Sandboxes with p5 access will find a way to use p5.

Kubernetes: GPU node pools need the same namespace allocation as CPU pools. GKE cost allocation labels help. EKS still needs Kubecost or equivalent. A GPU DaemonSet that reserves every device for a device plugin is platform cost.

Data path is part of the GPU bill

Checkpoints written to another region, training data pulled from a central bucket, and evaluation traffic out to the internet are egress. They show up as Data Transfer, not as p5.48xlarge. Plan them in egress cost planning.

Multi-AZ training frameworks that shuffle gradients across AZs add regional data-transfer lines. That can be correct for availability. It is still money.

Risks and limitations

Spot GPU interruption looks like a reliability event and a cost event. Checkpoint frequency is the control. Spot savings numbers that ignore wasted partial epochs are wrong.

Utilization from nvidia-smi is not tokens per second and is not model quality. Do not set a 80% GPU-utilization target as a proxy for good inference.

Provider price lists and quota pages move. Re-verify the SKU in the region you run. This site’s estimator is a formula, not a quote.

CloudArch Pro’s scoped work on this scorecard is AI FinOps consulting. General FinOps without the accelerator meter is FinOps consulting.

Official sources