Serious Hypervisor Bug Fix Causes Unexpected Cloud Downtime

A number of cloud service providers like Amazon Web Services and Rackspace had to shut some systems down over the weekend to address a critical Xen security vulnerability.

The Xen Project published a security advisory yesterday about a critical vulnerability in its virtual machine and hypervisor systems that could expose public cloud servers to attacks capable of crashing host machines and even stealing small amounts of random data. The fix was made available under embargo to certain cloud service providers last week, leading to downtime as some of those providers performed emergency maintenance to resolve the vulnerability over the weekend.

The Xen Project develops popular hypervisor and virtual machine software used by a number of cl0ud service providers such as Amazon Web Services, Verizon Cloud, Rackspace and others. While the Xen Project has a policy of not commenting on its security advisories, their advisory board made an exception in this particular instance, in order to refute reports and comments that XSA-108 is in some way akin to the Bash vulnerability known as Shellshock. To be clear, there is no relation between the two bugs other than that both constitute critical security vulnerabilities.

“XSA-108 was caused by a bug in the emulation code used when running [hardware virtual machine or hardware-assisted virtualization] guests on x86 processors,” Xen Project’s advisory board explained this morning. “The bug allows an attacker with elevated guest OS privileges to crash the host or to read up to 3 KiB of random memory that might not be assigned to the guest. The memory could contain confidential information if it is assigned to a different guest or the hypervisor.”

They claim that the vulnerability does not apply to paravirtualization guests. The Xen Project has made a patch available for the bug. However, the Xen Project publishes their updates under an embargo. That embargo lifted only yesterday, so it’s not altogether clear who has or hasn’t patched their systems.

“Several cloud providers updated their servers, something that they decided was necessary in this case to best ensure their users were not put at risk,” the group claims. “Most likely smaller vendors have done the same. Product vendors and Linux distributions will make updates available to their users following the embargo date.”

The vulnerability was serious enough that both Amazon and Rackspace performed emergency maintenance, causing downtime for their customers.

Amazon gave their customers advance warning with a blog post on Sept. 26, saying that some users would be subject to system restarts in order to install an update before a Xen security announcement became public on Oct. 1.

Rackspace, on the other hand, decided to install the update without prior warning because they were concerned that name-dropping Xen could tip off attackers to a potentially serious vulnerability.

“We decided the lesser evil was to proceed immediately, at which time we notified you, and our partners in the Xen community, of the need for an urgent server reboot,” Rackspace CEO and president Taylor Rhodes explained in an apology letter. “Even then, to avoid alerting cyber criminals, we didn’t mention Xen as the reason for the reboot.”

Rhodes went on to call out “Another major cloud provider” for attributing their outages ahead of time to an update involving Xen hypervisors, saying this decision “put all users of the affected versions of that hypervisor at heightened risk.” Luckily, he said, there were no data compromises among Rackspace customers related to this vulnerability.

Suggested articles