A unique cyberattack campaign that targets Kubeflow, a machine-learning toolkit for Kubernetes, has affected large swathes of container clusters, according to Microsoft.
The Kubeflow open-source project is a popular framework for running machine-learning (ML) tasks in Kubernetes. According to an analysis this week, a suspicious Kubeflow image was seen deployed to thousands of clusters in April, all from a single public repository. Closer inspection showed that the image runs a common open-source cryptojacking malware that mines the Monero virtual currency, known as XMRIG.
“Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs [graphics processors],” explained Yossi Weizman, security research software engineer at Microsoft’s Azure Security Center, in a posting on Wednesday. “This fact makes Kubernetes clusters that are used for ML tasks a perfect target for cryptomining campaigns, which was the aim of this attack.”
In terms of how Kubeflow can be used as the entry point for this kind of attack, Weizman noted that Kubeflow can manage the various tasks required to put a ML model into action, such as training ML algorithms. For instance, according to its website, Kubeflow simplifies the many steps required to build and deploy an ML model, including “data loading, verification, splitting, processing, feature engineering, model training and verification, hyperparameter tuning and model serving.”
And, because Kubeflow is a containerized service, these various tasks run as containers in the Kubernetes cluster, and each can present a path for an attacker into the core Kubernetes architecture.
“Therefore, if attackers somehow get access to Kubeflow, they have multiple ways to run their malicious image in the cluster,” said Weizman. “The framework is divided into different namespaces [containers], which are a collection of Kubeflow services. Those namespaces are translated into Kubernetes namespaces in which the resources are deployed.”
Kubeflow functionality is made available over an API server that connects to a dashboard that users leverage to manage their tasks. Typically, the dashboard is only available via an internal Istio ingress gateway that sits on the edge of a cluster and routes traffic from the API server. This default configuration in turn makes it difficult for external attackers to gain a foothold. However, Weizman said that many users choose to make their dashboards accessible via the internet, to bolster ease of use.
“In some cases, users modify the setting of the Istio Service to Load-Balancer which exposes the Service (istio-ingressgateway in the namespace istio-system) to the internet,” he wrote. “By exposing the Service to the internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.”
In the case of the April cryptomining campaign, Microsoft observed attackers using an exposed Kubeflow dashboard for gaining initial access to the cluster. The malicious actors then set up their own container, using the default name of “anonymous,” which housed the malicious malware image.
“If attackers have access to the dashboard, they have multiple methods to deploy [this] backdoor container in the cluster,” according to Weizman.
For instance, the attackers could set up a malicious Jupyter notebook server (or use an existing, compromised notebook), then use the dashboard to deploy their malicious image from it.
“Kubeflow allows users to choose the image for the notebook server, including an option to specify a custom image,” the researcher explained. “This image doesn’t necessarily have to be a legitimate notebook image, thus attackers can run their own image using this feature.”
From there, the attackers moved laterally and deployed the malicious container throughout an infected cluster using the cluster’s mounted service account.
For protection from this kind of attack, if Kubeflow is deployed within a cluster, admins should take care to make sure that its dashboard isn’t exposed to the internet, Weizman noted. “Check the type of the Istio ingress service by the following command and make sure that it is not a load balancer with a public IP: ‘kubectl get service istio-ingressgateway -n istio-system,'” he said.
While this is the first known attack to use Kubeflow as an initial pathway into Kubernetes clusters, containerization technology is no stranger to cryptomining offensives. Kubernetes was at the center of another recent large-scale XMRIG campaign, according to Microsoft.
Also, in April, an organized, self-propagating cryptomining campaign was found targeting misconfigured open Docker Daemon API ports; while last October, more than 2,000 unsecured Docker Engine (Community Edition) hosts were found to be infected by a cyptojacking worm dubbed Graboid (so-named after the sandworms in the 1990 Kevin Bacon movie, Tremors).
