What a Cross-Cloud Security Incident Changed in Our Kubernetes Rebuild

The response team found evidence that cloud identities and compute resources could no longer be trusted. The affected estate spanned AWS and Alibaba Cloud. Workloads were still running.

The first decision was to leave the source environment in place. The team preserved evidence, restricted access, and built replacement infrastructure beside it. Repairing the cluster where it stood would have mixed investigation with change, and deletion would have removed the record.

This article is anonymised. It does not identify the affected organisation, and it does not publish account identifiers, log paths, or attack commands. Ankit Mehta wrote it as an architecture lesson. The build steps for the replacement live in the ACK platform series.

What was known on the first day

Suspicious activity had shown up in cloud control planes and on hosts. The same response had to cover more than one provider, because a finding in one account did not describe the other.

Audit coverage was uneven. Some control-plane trails existed. Others were missing, short, or not searchable.

A gap is itself a finding. It names the hours you will never reconstruct.

Existing credentials could not be treated as clean. A key that still worked was a key an intruder might also hold. Long-lived access keys, shared administrator use, and kubeconfigs copied onto laptops all fell into that bucket until someone proved otherwise.

The source environment still served traffic. Turning it off would have been an outage, and it would have destroyed hosts, disks, and log buffers the investigation still needed. Immediate deletion was rejected for both reasons.

The picture on day one was small. Something untrusted had used cloud identity and compute. The estate was cross-cloud, the log set was incomplete, and the workloads had to keep running while a replacement was prepared.

Evidence, assumptions, and what stayed unknown

Three labels kept the write-up honest. A paragraph that mixes them will be quoted later as if every sentence had been proved.

Verified evidence is what logs, host inspection, cloud inventory, or a successful check showed. Suspicious cloud and host activity met that bar. So did the missing audit coverage, the decision to keep the source running, and the rebuild failures listed below.

Those failures were observed while the replacement was applied. They are facts about the rebuild. They are not a proved account of the original intrusion.

Working assumptions are explanations the team tested and could not prove. A long-lived key, a shared administrator credential, or a wide workload role could each explain untrusted compute. Where the trail was missing, those stayed assumptions.

The assumptions changed how access was restricted. They did not become a published root cause.

Unknowns are the questions the retained logs cannot answer. The original entry point is in that set. New logging does not fill it in. If a trail was off, or retention had already dropped the events, that history is gone.

This page does not call the incident data theft, ransomware, or lateral movement. Those words need evidence this record does not contain. The architecture changes below stand without them.

Why the replacement was built beside the source

A parallel rebuild keeps the source as an evidence store and as the system that is still serving users. The replacement is a new VPC, a new cluster, and new state. Nothing in the new plan is allowed to update or destroy the source.

Evidence stays put. Disks, network interfaces, security groups, and whatever logs remain are still there for the people reading them. A rebuild in place rewrites those objects while someone is trying to understand them.

Change risk stays on the new stack. A bad apply hits the replacement. The source keeps its current behaviour, including behaviour the team does not trust. That is why containment continues in parallel. A new cluster does not disable an old key.

Application migration waits until the platform is proved. The team can check nodes, add-ons, egress, and a clean plan before any production deployment moves. Database migration waits longer. Moving data on the same day as the first cluster apply stacks two cutovers.

The source databases kept serving until the replacement cluster had passed its own checks. Rollback, until cutover, is the source itself, for as long as it is still approved to run. Once traffic has moved, rollback is a planned return. That return path is written down first, which is the same discipline as a migration readiness review.

The new state does not import the source. Importing a security group, a disk, an elastic network interface, or a public address would copy an object the team could not vouch for. Unknown objects stay in the inventory. They stay out of the replacement state.

State is split on purpose. The network, the cluster, the node pools, and the remote-state stack are separate state objects. A plan for one of them cannot quietly edit another.

Create-only is the expected shape of each first plan: additions in the replacement, and no updates or deletes aimed at the source. A plan that wanted to change an existing cluster was a stop. The configuration was corrected, and the plan was run again.

Containment of the source continues while the new build proceeds. Keys get rotated or disabled. Human access and automation access get separated.

The people who can still change the source are a short named list. The diagram’s first steps keep running while the later steps start. A new cluster does not retire an old key.

Missing logs are part of the incident

A log that you enable on Tuesday records Tuesday. It does not reconstruct Monday. The response team hit that limit on more than one system.

Each signal answers a different question. Cloud activity logs, AWS CloudTrail and Alibaba Cloud ActionTrail, record API calls: who called, which API, and when, if the trail was on and the event is still inside retention. They are silent about a process inside a container.

Kubernetes audit logs record requests to the API server, such as who created a pod or read a secret. That record exists only when an audit policy was enabled and the sink still has the line. Turning the policy on starts a new file.

VPC flow logs record accepted or rejected connections as metadata: addresses, ports, and bytes. AWS documents them as flow logs. On Alibaba Cloud they land in Simple Log Service. They do not show HTTP paths, SQL, or file contents.

Host logs are the shell history, the process list, and the system journal on a node. They survive only if a shipper copied them, or the disk is still there. Rebuilding the node pool replaces that disk. Capture the host before you replace it.

Load balancer logs record requests that reached a listener with access logging on. Traffic that went straight to a node never appears there.

Application logs are whatever the process wrote. They often name a user and omit the cloud role that started the process. DNS logs record lookups after logging started. They do not show whether the following connection succeeded.

KMS audit events record which principal called encrypt or decrypt, when those management events were in the trail. They do not include plaintext. A key with no events in the trail cannot be audited after the fact.

Write down the oldest event you can still retrieve for each signal, on each cloud. Anything earlier is an unknown. A dashboard that started yesterday describes yesterday.

Identity was checked before the plan

Cross-cloud identity design is covered on the identity and security topic, in designing multi-cloud identity, and in IAM fundamentals. The incident rule was narrower. Nobody planned infrastructure with a credential that had not just been proved.

Before every plan, the caller identity was read and compared with the account named in the change window. The check was repeated after a laptop slept, a shell was reused, or a CI job started. A plan aimed at the wrong account is worse than no plan.

Human users in RAM and in IAM were listed with the policies attached to them. The useful questions were whether that user still needed a policy that could create compute, and whether anyone could say who last used it. Where the activity log could not answer the second question, access was reduced until it could.

Service roles are separate from human users. ACK expects roles such as AliyunCSManagedSecurityRole for secret encryption and AliyunECSDiskEncryptDefaultRole for disk encryption. Those names are in Alibaba Cloud’s public documentation. AdministratorAccess on the person running the plan does not create them.

The rebuild learned that from a failed apply, recorded in the failure table below. The preflight is GetRole for each required role, before apply.

Long-lived access keys were removed from the path that applied changes. Temporary credentials, issued for the change window, replaced them. A key in a shell profile outlasts the person who created it. AWS documents temporary credentials as the normal path for people and for workloads. The same limit belongs on Alibaba Cloud: a session that expires can be reasoned about.

Human access and CI access used different principals. A person debugging a node did not use the principal that applies OpenTofu. The CI principal was not a shared administrator password passed around a chat.

Who approved an apply was written down outside the tool. The change record names a person, a plan, and a time. The tool’s own log is a useful record. The approval is the change record.

Shared administrator credentials were removed from the response. If several people can act as the same user, the activity log cannot say which person ran the call. IAM access lockout covers the opposite case, a principal you can no longer use. This incident had the other problem: too much still worked, and too little of it was attributable.

Inventory before a new design

The team collected a read-only inventory before drawing the replacement. The list covered both clouds where the object type existed:

VPCs, subnets and vSwitches, Kubernetes clusters, databases, caches, load balancers, security groups, NAT gateways, elastic IPs, elastic network interfaces, disks, route tables, DNS records, KMS keys, and logging services.

The list was a denylist and a question list. Anything the team could not explain stayed out of the replacement. An incomplete denylist feels like a control and misses the object that was never written down.

A security group that nobody listed can still be attached to a new node if someone imports “the usual” group. The inventory has to be complete enough that “usual” is a named object, or it is refused.

Read-only matters. An inventory that creates, tags, or cleans up as it goes is an apply. Listing was allowed. Mutation waited until the identity check and the plan review were done.

The inventory also showed dependencies the application owners had not drawn. Which workload used which database. Which egress address a partner had allowlisted. Which DNS name pointed at which balancer.

That map is what made a later migration a sequence. Cloud migration readiness is the same list when you are moving on purpose. An incident removes the option to discover it during cutover weekend.

The replacement network

The replacement used a new VPC per cluster. The source VPC stayed the original VPC. Address space did not overlap, so a later route or a mistaken security-group rule could not stitch the two networks together.

The ranges below are examples for this article. They are not the ranges from the affected environment.

Example layout, for a document, not for a copy-paste into the incident account:

• Node address space 10.80.0.0/16, split across three zones
• Pod address space 10.81.0.0/16, used by Terway, not by the nodes
• A small management range for operator access paths
• A data range for databases and caches, with no default path to the NAT gateway
• An application entry range, used only if a reviewed load balancer is in scope

Three zones were required so a node pool could lose one zone and still schedule. A pool that is empty in one zone because the instance type has no stock is a single zone. Stock is checked per zone before apply. That check failed once during this rebuild. It is in the failure table.

Nodes received no public addresses. Outbound traffic that must reach the internet went through a NAT gateway and one stable egress address. Inbound destination NAT stayed off.

The stable address exists so a partner allowlist can name one address. It is not a reason to publish the nodes. The NAT path is also a capacity and a bill, which is why egress cost planning sits in the same review. The wider networking topic covers the patterns this incident refused to turn on by default.

The replacement VPC was not attached to Cloud Enterprise Network, and it was not peered to the source. Automatic attachment would have joined a network the team was keeping separate. A later hub is a separate review, closer to hub-and-spoke networking.

The Kubernetes API was private. The load balancer was internal. Operators reached it through a path they could name, such as a bastion in the management range. A public endpoint was not added to finish the week.

Account structure around the VPC is the subject of Alibaba Cloud architecture for APAC. This incident did not expand that scope. The new VPC was not attached to anything the inventory could not explain.

The ACK baseline, and which risk each setting covers

The replacement cluster was ACK Pro, managed, with a private API. Pro is the control-plane tier the team standardised on. Private means the API balancer has no public address. You can buy Pro and still publish the API. This rebuild kept the balancer internal.

Terway provides the pod network. Pods take addresses from the pod range, on elastic network interfaces, so that range cannot overlap the node range. NetworkPolicy is the pod allow list on that network. Kubernetes documents it separately from security groups. Security groups do not express namespace boundaries.

RRSA gives a pod a RAM role through a projected service account token. The incident had already shown that a key in a Secret or in an image was the wrong shape. The projected token still needs a narrow role. A token pointed at AdministratorAccess is a wide key with a newer delivery path.

Deletion protection stops a mistaken delete from being a one-line apply. A principal who can turn the protection off can still delete the cluster. The control raises the cost of an accident during a tense week.

System disks were encrypted. Kubernetes Secrets used a different KMS key. Disk encryption protects a stolen volume. Secret encryption protects Secret objects in the cluster store. A privileged process that is already allowed to read a secret can still read it. Encryption does not evict that process.

The two keys exist so the disk policy and the secret policy can be reviewed separately. AliyunCSManagedSecurityRole has to be allowed to use the secret key. AliyunECSDiskEncryptDefaultRole has to be allowed to use the disk key. A key that exists, and a key those roles can call, are different facts.

The cluster was created with the add-ons it needs on day one: Terway, the storage add-on, log collection, and ack-node-problem-detector. Managed node pools refuse some operations when that detector is not active. The order is cluster and add-ons, then pools.

System pools used subscription instances so control-plane add-ons were not preempted. The application pool used Spot capacity and an autoscaler, with a floor and a ceiling written down. Spot fits replicas that can drain. It is a poor match for the only copy of a single-replica system component. The pool design is in ACK node pools: subscription, Spot and autoscaling.

Nodes stayed without public IPs. SLS received control-plane logs and, once a flow log existed, VPC flow logs. A project and a logstore are not a searchable trail. The logstore needs an index, and the shipper has to be delivering lines. Node-problem events have their own retention, separate from the API server log.

Each setting fails in a different way. A private API does not encrypt disks. Disk encryption does not create flow logs. Flow logs do not rotate RAM keys. The secure ACK guide treats them as a sequence of checks.

What failed while the replacement was built

The failures below happened during the rebuild. They are anonymised. There are no request identifiers, resource identifiers, or account names. An incident does not exempt the account from quota, stock, or service-role rules. It only makes the clock louder.

The role and add-on rows cost the most calendar time, because the plan had already looked successful in review. A valid plan means the configuration parsed and the provider accepted the shape. It does not mean the account has authorised the service role, or that an add-on is running.

The correction in each row has the same shape. Fix the missing prerequisite, throw away the old plan, and plan again.

Quota and stock decide whether a half-created cluster appears beside a source you already do not trust. That half-created cluster is a new object you have to protect. Quota exhaustion is the general runbook. For ACK, read SLB usage and per-zone stock before the cluster apply.

The TableStore rows blocked shared work. The cluster API was a separate problem. Until the lock existed, one operator applied, and CI did not. The state guide covers that migration: OpenTofu remote state and private GitOps.

State is part of the security record

Local state was not enough once a second person, or a pipeline, could run a plan. A file on a laptop has no lock that another plan will honour. It can be copied, it can be stale, and it can be the only copy.

The person who holds that laptop is then a single point of failure during the incident. That is a poor moment to discover it.

The replacement used a private OSS bucket. Versioning was on, so a bad apply could be compared with the previous object. Server-side encryption used AES256. Public access was blocked. Those settings make the state file harder to lose and harder to publish by accident. They do not record API calls.

Locking used TableStore. The OpenTofu OSS backend expects a string primary key named LockID. A table with a different key type fails when the second plan tries to take the lock. One state object per stack kept a network plan from writing the cluster state.

Backups of the state object, with a checksum recorded beside the backup, gave the team a copy that did not depend on the old laptop.

Applies stayed manual at the approval boundary. CI could plan. A green job was not an apply. Jobs that could apply were serialised, so two applies could not start together and race the lock. Plan files and state files were not stored as CI build products. A state file in a job artifact is a copy of the infrastructure.

Remote state does not replace CloudTrail or ActionTrail. State says what OpenTofu last believed it had created. The activity log says who called the API. A console session, or a key used outside OpenTofu, will not appear as a line in the state file. The Kubernetes and containers topic links the state guide beside the cluster guide because the two records answer different questions.

What counted as working

“The cluster is up” was not the acceptance test. Migration waited until each line below was true.

The cluster reported running, and the API address was private. Nodes were Ready, including a node in more than one zone where the pool claimed to be multi-zone. Add-ons that the node pool and the log path depend on were active. Being named in the configuration was not enough.

Encrypted disks were checked on the instances. The module input was not treated as proof. No node had a public address. NAT and source NAT were tested with a real outbound connection, and the egress address matched the one a partner would allowlist.

The autoscaler was tested against the documented minimum and maximum. A minimum does not mean scale-down will ignore disruption budgets. That behaviour was tested. The notes record the result.

The final OpenTofu plan showed no changes. A plan with pending edits means the running objects and the configuration disagree. Shipping applications onto that disagreement adds drift on the day the platform needs to stay still.

Logging was three checks. The project and the logstore exist. Lines are arriving. A query returns those lines, which requires an index. An empty project, a store that accepts writes and refuses queries, and a query window with no retention are three different failures.

Argo CD ran as a ClusterIP service. It was not published on a load balancer so the people finishing the rebuild could click a URL. The acceptance test was that the Service type stayed private after the install.

A clean plan shows that the infrastructure configuration matches the last apply. It does not show that an application is healthy. Process health, database connectivity, and the user path through the application entry are separate checks. The platform team can build a place where those checks are possible. They cannot sign the application off from an empty plan.

What should have existed before the incident

The rebuild made these gaps expensive. They are cheaper as ordinary platform work. They are records and rehearsals.

A central activity trail on each cloud needs an owner, a retention number, and a query someone actually runs. “We have logs” and “we have ninety days” are different sentences. Flow logs on production VPCs have to exist before the week you need them. Enabling them during the incident keeps the next hour. It does not restore the last one.

Remote state, with a lock, has to exist before a second person applies. Temporary credentials should be the normal path, with a rotation date for anything that still has to be long-lived. A service-role preflight should fail the pipeline when GetRole fails, which is earlier than a cluster create error.

Quota and per-zone stock should be visible to the person who will apply. A CIDR registry should name every range in use, so the next VPC cannot overlap by accident. Every KMS key the cluster uses needs an owner, and a policy someone other than the creator has read.

The recovery notes should say who may rebuild, who may delete, and where evidence is stored. A restore that has been done is a different fact from a backup job that is green. The application map has to include databases, caches, egress allowlists, and DNS before anyone calls the move “later.”

Someone has to own status. A channel with no cadence means the absence of news becomes the status. Named roles matter: who leads, who changes production, who talks to leadership, and who keeps the notes. Cross-region disaster recovery uses that split for a region loss. The reliability topic is the wider set. An account compromise needs the same names, because the source environment may be the only copy you still have.

Tags are part of the same preparation. An untagged disk during an incident is a disk nobody will volunteer to own. Tagging and cost allocation is written for the bill. The same tags tell a response team which stack an object belongs to.

A cutover that keeps the source serving is an architecture decision: what still runs, what evidence exists, and who may retire the source. That is the review described on the cloud architecture consulting page.

What stayed in the architecture afterwards

The temporary rules that survived became the default for the replacement, and then the default for the next cluster.

Control planes stay private. Nodes stay without public addresses. Egress goes through NAT, and destination NAT stays off unless a specific listener has been reviewed. Pools span zones only where stock has been checked that day.

Disks are encrypted. Secret encryption uses its own key and its own role grant. State is one object per stack, locked in TableStore, in a private versioned bucket. A first apply is create-only. A plan that would edit the source is rejected.

Tags identify the stack. Log projects are created with indexes, and someone checks that a line has arrived. GitOps uses a private Service. The order is written down: identity, inventory, network, cluster and add-ons, pools, state backend, then applications. Validation is a list of observations.

That order is the ACK series in one line. When the source environment is untrusted, skipping a step is how the old objects leak into the new state.

Limits

The original entry point may remain unknown. Say that in the incident record. A confident story that the logs do not support will be the sentence everyone remembers, and it will be wrong.

Logging enabled after the incident does not restore the missing days. A new trail is still worth turning on. It records the next hour. The hour you already lost stays missing.

A new cluster runs the application you deploy to it. If that application trusts a header, stores a secret in an image, or runs a privileged container, the replacement runs that code. Private networking does not review IAM. A private API and a wide role can exist on the same cluster.

Encryption does not protect a process that is already allowed to decrypt. Infrastructure code will rebuild a mistake that is written in the module. Review the module. A green plan is not that review.

Security Center, or any posture product, can show misconfigurations. It does not preserve a disk, rotate a key, or decide that a database stays on the source for another week. Those decisions belong to the named roles.

No control on this page stops every attack. The list reduces the chance that the next response starts with no trail, one laptop, and a public API. That is the result the team can stand behind.

Before the next incident

Use this as a pre-incident check. The sections above remain the procedure.

• Each cloud account has an activity trail, and someone has run a query in the last month.
• Retention for that trail, for flow logs, and for Kubernetes audit is written down.
• Flow logs are on for production VPCs.
• OpenTofu state is remote, versioned, encrypted, private, and locked. A second plan blocks.
• Humans and CI use different principals. Credentials used for apply expire.
• Service roles the cluster will assume already exist.
• SLB quota and per-zone instance stock are visible before a cluster apply.
• Every VPC range is in a registry. New ranges are checked against it.
• Every KMS key used by the cluster has an owner and a policy the service role can use.
• The recovery notes name who may rebuild, who may delete, and where evidence is stored.
• A restore has been done. An application dependency map exists, including egress allowlists.
• Incident roles are named. The status channel has a cadence.
• The replacement pattern, if you needed it tomorrow, would be a new VPC, a private API, and no imports from the source.

FAQ

When should a team rebuild a Kubernetes cluster instead of repairing it?

Rebuild beside the source when identities or nodes can no longer be trusted, or when a repair would destroy evidence you still need. Keep the source serving if deleting it causes a worse outage than leaving it contained. Repair in place when the fault is a known configuration error, the credentials are still trustworthy, and the change keeps the record you need.

How do you investigate when audit logs are missing?

List the signals you still have, and the oldest event in each. Treat those as verified facts. Mark every earlier hour as unknown, and enable the missing trails so the next hour is kept.

Should a compromised VPC be reused?

No. Build the replacement in a new VPC with non-overlapping ranges. Leave the original VPC in the inventory until evidence collection and an explicit retirement approval are both done.

Does a private Kubernetes API make a cluster secure?

It stops anonymous clients on the internet from reaching the API. Nodes, roles, pod permissions, and disks are separate checks. A private API is one control on the list.

Why use separate OpenTofu state for each stack?

One state object lets one plan edit the network, the cluster, and the lock bucket together. Separate objects keep a bad plan smaller, and they make a lock mean this stack. Remote state still does not record API calls made outside OpenTofu.

Can Spot nodes run in a secure production cluster?

Yes, for application replicas that tolerate interruption, with a pool minimum and a disruption budget you have tested. System components that must stay up belong on subscription nodes. Spot does not change encryption, identity, or public addressing.

Which logs should exist before an incident?

Cloud activity logs, Kubernetes audit, VPC flow logs, load balancer access logs, shipped host logs, application logs that include the process identity, DNS logs, and KMS management events. For each one, write the retention and the name of the person who last proved a query works.

How do you migrate applications without changing the source environment?

Prove the replacement first: private API, Ready nodes, active add-ons, encrypted disks, no public node addresses, working NAT, a tested autoscaler, and an OpenTofu plan with no changes. Move stateless workloads next. Move databases in a later change with its own rollback. Leave the source VPC unimported until the named approval.