A critical security vulnerability allowing attackers to perform cross-account container takeover in Microsoft’s public cloud, dubbed “Azurescape”, has been uncovered by researchers.
The issue exists in Azure Container Instances (ACI), which is Microsoft’s container-as-a-service (CaaS) offering (which enables users to run cloud containers without having to deal with managing the underlying infrastructure). Part of the infrastructure hosting ACI consists of multitenant Kubernetes clusters, which can be compromised to allow cyberattackers to gain full control over other users’ containers, researchers from Palo Alto Networks’ Unit 42 team explained.
“A malicious Azure user could have exploited these issues to execute code on other users’ containers, steal customer secrets and images deployed to the platform, and possibly abuse ACI’s infrastructure for cryptomining,” they said in a Thursday posting.
Microsoft has rolled out a patch to ACI, but users should revoke any privileged credentials that were deployed to the platform before Aug. 31 to avoid compromise. They should also review access logs for any irregularities, Unit 42 recommended.
ACI Multitenant Architecture – Good Fences, Good Neighbors
In the multitenant architecture, each customer’s container is hosted in a Kubernetes pod on a dedicated, single-tenant node virtual machine (VM), according to the analysis, and the boundaries between customers are enforced by this node-per-tenant structure.
“Since practically anyone can deploy a container to the platform, ACI must ensure that malicious containers cannot disrupt, leak information, execute code or otherwise affect other customers’ containers,” explained researchers. “These are often called cross-account or cross-tenant attacks.”
The Azurescape version of such an attack has two prongs: First, malicious Azure customers/adversaries must escape their container; then, they must acquire a privileged Kubernetes service account token that can be used to take over the Kubernetes API server.
The API Server provides the frontend for a cluster’s shared state, through which all of the nodes interact, and it’s responsible for processing commands within each node by interacting with Kubelets. Each node has its own Kubelet, which is the primary “node agent” that handles all tasks for that specific node.
Thus, the end result of an API server compromise is “establishing complete control over the multitenant cluster and all customer containers running within it,” according to Unit 42.
Kubernetes Container Escape
The first step in the attack proved easier than expected. Unit 42 researchers found that Azure uses RunC, the industry standard container runtime, to power ACI. Unfortunately, this happens to be an ancient version of the technology (RunC v1.0.0-rc2), which was released on Oct. 1, 2016.
True to legacy software form, it harbors at least two container-escape bugs. Unit 42 researchers went on to use one of these (CVE-2019-5736) to break out of a test ACI container, gaining a reverse shell running as root on the underlying Kubernetes node.
“While we escaped our container, we were still within the tenant boundary – the node VM,” researchers explained. “CaaS platforms are designed to withstand sophisticated attackers who possess kernel vulnerabilities enabling privilege escalation and container breakout. A malicious container breaking out is a somewhat expected threat, tolerated through node-level isolation.”
Snagging a Privileged Kubernetes Account Token
Escaping this underlying node VM – the boundary with other customers – means finding a way to access the API server and force it to execute commands on neighboring nodes. The best way to do this is by obtaining a privileged access token for the API server, the researchers soon discovered.
Bridge pods handle execs in ACI. Source: Palo Alto Networks.
In hunting for a way to obtain such privileges, researchers observed that ACI supports executing node VM commands on uploaded containers via the “az container” exec command to a Kubelet, which is handled by a dedicated bridge pod rather than the API server itself (researchers said the implementation of the bridge pod is a fix for a previous vulnerability).
Curiously, researchers found that such exec requests arriving at the node included an authorization header carrying a Kubernetes service account token for this ‘bridge’ service account.
“Finding a token here was surprising,” according to the analysis. “Kubelets in the cluster were configured to allow anonymous access, so there was no need for requests to authenticate via a token. Perhaps this was a relic of an older implementation.”
Researchers then examined the token’s permissions, and found that it could be used to execute commands on any pod in the cluster – including the API server pod.
From there, exploitation was straightforward, they said – and they were quickly successful in using the token to open a shell on the API server container as a cluster administrator, with full control over the multitenant cluster and all customer containers within it.
“If you run Kubernetes, be careful to whom you send your service account tokens: Anyone who receives a token is free to use it and masquerade as its owner,” researchers noted. “Token thieves are likely to be very interested in the permissions of their stolen tokens.”
Azurescape Exploit Steps
In short, the Azurescape exploit worked like this, as demonstrated in this video:
- Deploy an image exploiting CVE-2019-5736 to ACI. The malicious image breaks out upon execution and establishes code execution on the underlying node;
- On the node, monitor traffic on the Kubelet port, port 10250, and wait for a request that includes a JWT token in the Authorization header;
- Issue “az container” exec command to run a command on the uploaded container. The bridge pod will now send an exec request to the Kubelet on the compromised node; and
- On the node, extract the bridge token from the request’s Authorization header and use it to open a shell on the API server.
The fix was simple: Microsoft merely ensured that the bridge pod no longer sends its service account token to nodes when issuing exec requests.
How to Prevent Cross-Container Attacks in the Cloud
Azurescape may be mitigated, but there are likely other avenues to achieving this kind of exploitation, Unit 42 researchers warned.
“Cross-account vulnerabilities are often described as a ‘nightmare’ scenario for the public cloud,” they concluded. “Azurescape is evidence that they’re more real than we’d like to think. Cloud providers invest heavily in securing their platforms, but it’s inevitable that unknown zero-day vulnerabilities would exist and put customers at risk.”
To protect one’s Kubernetes assets, users can implement the following best practices, the firm advised:
- Keep cluster infrastructure patched;
- Refrain from sending privileged service accounts tokens to anyone but the API server to prevent attackers from masquerading as the token owner;
- Enable the “BoundServiceAccountTokenVolume” feature: When a pod terminates, its token is no longer valid, minimizing the impact of token theft;
- Deploy policy enforcers to monitor and prevent suspicious activity within clusters, especially service accounts or nodes that query the SelfSubjectAccessReview or SelfSubjectRulesReview APIs for their permissions.
On the latter point: “We see adversaries actively abusing the SelfSubjectReview APIs to inspect the privileges of stolen Kubernetes credentials,” according to the analysis. “[We] recently observed the Siloscape malware leveraging these APIs to retrieve the permissions of the node it compromised, and then using them to determine whether to continue its campaign against the cluster.”
It’s time to evolve threat hunting into a pursuit of adversaries. JOIN Threatpost and Cybersixgill for Threat Hunting to Catch Adversaries, Not Just Stop Attacks and get a guided tour of the dark web and learn how to track threat actors before their next attack. REGISTER NOW for the LIVE discussion on Sept. 22 at 2 p.m. EST with Cybersixgill’s Sumukh Tendulkar and Edan Cohen, along with independent researcher and vCISO Chris Roberts and Threatpost host Becky Bracken.
