Microsoft has spotted a new, widespread, ongoing attack targeting Kubernetes clusters running Kubeflow instances, in order to plant malicious TensorFlow pods that are used to mine for cryptocurrency.
The Kubeflow open-source project is a popular framework for running machine learning (ML) tasks in Kubernetes, while TensorFlow is an end-to-end, open-source ML platform.
Given that the attack is still active, any new Kubernetes clusters that run Kubeflow could be compromised, according to Microsoft.
On Tuesday, Microsoft security researchers warned that toward the end of May, they saw a spike in deployments of TensorFlow pods on Kubernetes clusters – pods that are running legitimate TensorFlow images from the official Docker Hub account. But a closer look at the entry point of the pods revealed that their purpose is to mine cryptocurrency.
Yossi Weizman, senior security research software engineer at Microsoft’s Azure Security Center, said in a post on Tuesday that the “burst” of these malicious TensorFlow deployments was “simultaneous,” indicating that the attackers initially scanned the clusters, kept a list of potential targets, and then pulled the trigger on all of them at once.
Weizman explained that the attackers used two separate images: The first is the latest version of TensorFlow (tensorflow/tensorflow:latest) and the second is the latest version with GPU support (tensorflow/tensorflow:latest-gpu). The use of TensorFlow images in the cluster “makes a lot of sense,” Weizman said, given that “if the images in the cluster are monitored, usage of [a] legitimate image can prevent attackers from being discovered.”
Another reason why the attackers’ choice is understandable is that the TensorFlow image they chose is a convenient way to run GPU tasks using CUDA, which “allows the attacker to maximize the mining gains from the host,” he said. CUDA is a toolkit created by NVIDIA, used to develop, optimize and deploy GPU-accelerated apps.
Similar to Last Year’s Cryptomining Attack
The newly discovered attack is similar to a cryptocurrency mining attack that Microsoft reported last June. That earlier campaign also targeted Kubeflow workloads, exploiting misconfigured dashboards to launch a widespread XMRIG Monero-mining campaign. Fast-forward a year, and this recently discovered cryptomining pulls a similar move, using exposed Kubeflow interfaces for running cryptocurrency mining containers.
The latest campaign adds some tweaks: As Weizman described it, this time around, the attackers abused the access to the Kubeflow centralized dashboard in order to create a new pipeline.
As Weizman detailed in the post, Kubeflow Pipelines is a platform for deploying ML pipelines, based on Argo Workflow, which an open-source, container-native workflow engine for orchestrating parallel jobs on Kubernetes. Pipeline entails a series of steps, each one of them as an independent container, that together form a ML workflow. The image of the container that runs in each step is determined in the pipeline configuration, he said.
Kubeflow pipelines. Source: Microsoft
Access to the pipeline’s user interface is key in this attack: Once attackers gain access to that dashboard, they can create a new cluster in the pipeline. In this case, that means containers that run TensorFlow images that set up cryptocurrentcy mining.
All of the malicious pods were set up with the same pattern: “sequential-pipeline-{random pattern}”. That name is originated in the “generateName” field of the Argo Workflow object that’s used for creating the pipeline, Weizman said.
At least two pods were deployed on each cluster: One for CPU mining, and the other for GPU mining. The GPU container used the open-source Ethminer to mine Ethereum, while the CPU miner used the aforementioned open-source XMRIG Monero miner.
As part of the ongoing attack flow, the attackers are using a reconnaissance container – also run from a TensorFlow pod – to scoop up information about the environment, such as GPU and CPU details, in preparation for mining.
What to Do to Avoid a Cyberattack
Microsoft advised that those who run Kubeflow should make sure they’ve locked down the centralized dashboard so it’s not insecurely exposed to the internet. If Kubeflow has to be exposed to the internet, make sure it requires authentication.
Microsoft gave the example of Kubeflow being used to support OpenID Connect (OIDC) using Azure Active Directory for Azure deployments. In order to get all the pods running in the cluster in JSON format, run: kubectl get pods –all-namespaces -o json …
… and search for containers that run TensorFlow images. If they exist, inspect the entry point of those containers, Microsoft advised.
Cryptomining: Lead Weights That Bog Down the Cloud
Yaniv Bar-Dayan, CEO and co-founder at Vulcan Cyber, said that cryptomining is nothing to shrug off, given the hit it puts on cloud resources.
“While cryptomining campaigns can seem innocuous, they put excess strain on cloud resources, inflict additional cloud and utility costs on attacked parties, shorten the lifespan of IT devices and cause unnecessary business disruption,” he told Threatpost via email on Thursday.
These campaigns also highlight organizations’ risk of exposure, he noted – in other words, if an attacker can pull off a cryptomining attack, they can pull off even worse. “If an attacker can launch a cryptomining campaign on an organization’s infrastructure, it’s likely that they can launch ransomware as well or gain access to data, intellectual property, personnel files and other at-risk assets that can damage a business if breached,” he continued.
Bar-Dayan said that Vulcan Cyber recommends taking the appropriate steps for defense. For example, ensure proper configurations, and, echoing Microsoft, make sure that systems aren’t exposed to the open internet. Also, make sure the right identity and access controls are in place.
Unstoppable Kubernetes Growth Makes for Ever More Threats
Mark Bower, senior vice president at comforte AG, noted that as the use of Kubernetes spikes, so do the threats aimed at it. “Kubernetes growth is unstoppable, and while it provides massive agility benefits to enterprises for agile app delivery, there are two characteristics which make it an ideal attack target for exploitation,” he elaborated to Theatpost via email on Thursday.
“First, the ecosystem is both new and very complex,” he wrote. “Second, the workloads are often related to machine intelligence processing, demanding very large compute workloads which are not always predictable. Machine intelligence is about exploring unknowns, so a regular pattern of compute usage isn’t easy to determine to know what is normal and abnormal. As a result, it presents an ideal target for cryptomining to hide in among the likely thousands of micro services involved running on scaled compute.”
That’s not the worst of it, Bower said. What’s more concerning still is that for the most part, the data security capabilities built into Kubernetes “meet bare minimum standards – data at rest protection, and data in motion. There’s no persistent protection of data itself, for example, using industry accepted techniques like field-level tokenization. So if an ecosystem is compromised for cryptomining and compute exploitation, it’s only a matter of time before the sensitive data being processed by it succumbs to a more insidious attack. In the last 12 months Kubernetes vulnerabilities related to privilege escalation, firewall gaps, and remote code execution in Kubernetes tools certainly show it’s vulnerable.”
A glaring example of Kubernetes’ vulnerability cropped up just a few days ago when Siloscape, the first malware to target Windows containers, broke out of Kubernetes clusters to plant backdoors and raid nodes for credentials.
061021 14:00 UPDATE: Added input from Mark Bower.
Download our exclusive FREE Threatpost Insider eBook, “2021: The Evolution of Ransomware,” to help hone your cyber-defense strategies against this growing scourge. We go beyond the status quo to uncover what’s next for ransomware and the related emerging risks. Get the whole story and DOWNLOAD the eBook now – on us!
