GPU and AI workload cost governance

GPU and AI cost governance is a unit-cost model for training and inference, plus a process that treats idle accelerators as waste you can see in hours. List price per GPU-hour is not the unit. Training cost is GPU-hours times the effective rate, plus storage, plus egress to the dataset and checkpoint region. Inference cost is GPU-hours or token-path charges divided by successful requests. Reservations, Capacity Blocks and committed use lock both price and scarcity. They expire unused. Documentation dated 2026-08-30 is the SKU and product source. Generic EC2 rightsizing playbooks miss idle GPU and data-path meters.

The planning worksheet is the AI and GPU workload cost estimator. The practice name is What is AI FinOps?.

Assumptions

You can list running GPU or accelerator instances or node pools in at least one account or subscription.
Training and inference on self-managed accelerators are in scope. Managed token APIs are a separate meter. Keep them off the GPU-hour model.
Reservations and Capacity Blocks are calendar commitments you already bought or are considering.
General EC2 or VM family rightsizing is out of scope.

Inventory what is running, not what was requested

aws ec2 describe-instances \
  --filters Name=instance-state-name,Values=running \
            Name=instance-type,Values=p4*,p5*,g5*,g6*,trn1* \
  --query 'Reservations[].Instances[].{id:InstanceId,type:InstanceType,az:Placement.AvailabilityZone,launch:LaunchTime}' \
  --output table

Every row is a burning GPU-hour meter. LaunchTime older than the last completed job, with no serving traffic, is idle. Instance families differ by region. p5 in us-east-1 does not mean ap-southeast-1 has the same quota or price. AWS documents regional SKU availability in the Regions and Availability Zones material and the EC2 price list.

Azure: az vm list -d --query "[?contains(hardwareProfile.vmSize, 'NC') || contains(hardwareProfile.vmSize, 'ND')].[name,hardwareProfile.vmSize,location,powerState]" -o table. Google Cloud: gcloud compute instances list --filter='machineType~g2 OR machineType~a2 OR machineType~a3'.

On a node you already access, nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv showing 0 %, 0 MiB for hours is idle. That is a cost finding. It is also a capacity finding if another team is queued.

Unit cost that finance can reuse

Training job:

effective_gpu_hour_rate * gpu_count * wall_clock_hours + dataset_egress + checkpoint_storage

Use the effective rate after any reservation or spot discount you actually applied. If a commitment covers the usage, use that rate. Write the job ID next to the number. A monthly GPU bill with no job IDs is not AI FinOps.

Inference:

accelerator_hours / successful_requests or, for managed token APIs, billed_tokens / successful_requests. Include the queue time you keep GPUs warm to hit latency SLOs. Warm idle is a product choice. Hide it and the unit cost is fiction.

Managed model APIs (Amazon Bedrock, Azure OpenAI, Google generative APIs) are a different meter: tokens and provisioned throughput, not nvidia-smi. Do not force them into a GPU-hour model. Put them on a separate scorecard.

Reservations, scarcity and the landing zone

Capacity reservations and GPU Capacity Blocks buy a calendar slot. If the experiment slips, you still pay. Committed use discounts and reserved instances on GPU SKUs assume a utilization floor. Below that floor you wanted on-demand or a smaller pool.

Quotas for GPU families are regional and often start at zero. A DR or burst region without quota is not a standby. See cross-region disaster recovery and quota exhaustion.

Place training accounts or subscriptions in the landing-zone tree so SCPs can deny unexpected GPU families in a sandbox OU. Sandboxes with p5 access will find a way to use p5.

Kubernetes: GPU node pools need the same namespace allocation as CPU pools. GKE cost allocation labels help. EKS still needs Kubecost or equivalent. A GPU DaemonSet that reserves every device for a device plugin is platform cost.

Data path is part of the GPU bill

Checkpoints written to another region, training data pulled from a central bucket, and evaluation traffic out to the internet are egress. They show up as Data Transfer. The p5.48xlarge line is a different meter. Plan them in egress cost planning.

Multi-AZ training frameworks that shuffle gradients across AZs add regional data-transfer lines. That can be correct for availability. It is still money.

Risks and limitations

Spot GPU interruption looks like a reliability event and a cost event. Checkpoint frequency is the control. Spot savings numbers that ignore wasted partial epochs are wrong.

Utilization from nvidia-smi is not tokens per second and is not model quality. Do not set a 80% GPU-utilization target as a proxy for good inference.

Provider price lists and quota pages move. Re-verify the SKU in the region you run. This site’s estimator is a formula. The provider invoice is the quote.

CloudArch Pro’s scoped work on this scorecard is AI FinOps consulting. General FinOps without the accelerator meter is FinOps consulting.