Are the clouds in the sky rebooting?!

If you are like us at NopSec one of the companies that operators on Amazon AWS cloud, this past couple of days resembled a lot more a perilous path. A series of reboot of the entire Amazon cloud forced us and most AWS-based cloud providers to spend long hours in the office or remotely to make sure things were in order after the reboot.

“These updates must be completed by Oct. 1 before the issue is made public as part of an upcoming Xen Security Announcement (XSA),” according to the AWS blog. “Following security best practices, the details of this update are embargoed until then. The issue in that notice affects many Xen environments, and is not specific to AWS.”

Now that the AWS cloud update is complete (including Rackspace and IBM), today October 2nd AWS spilled the beans regarding the “unknown” vulnerability that forced their virtualization servers to be rebooted.

Amazon, Rackspace and IBM SoftLayer have all had to reboot their servers in the last several days to fix a flaw in Xen that was privately reported two weeks ago and only publicly disclosed on Oct. 1.

The flaw in question is detailed in Xen Security Advisory XSA-108 and is also identified as CVE-2014-7188. Technically speaking, the vulnerability is titled “Improper MSR range used for x2APIC emulation,” which is basically a memory-related issue. Model Specific Registers (MSRs) are control registers within an x86 chip, while x2APIC is Intel’s next-generation Advanced Programmable Interrupt Controller (APIC). “The MSR range specified for APIC use in the x2APIC access model spans 256 MSRs,” according to the Xen advisory. “Hypervisor code emulating read and write accesses to these MSRs erroneously covered 1024 MSRs.” The impact of the flaw is that an attacker could potentially crash the underlying host server and potentially read data from other virtual machines on the system. So, the problem for public cloud providers, for example, is that the flaw could have enabled an attacker to potentially get access to other resources and data on the cloud. Needless to say, that would have been catastrophic for any public cloud provider, especially the world’s largest.

The issue only affects hardware-assisted virtual machines (HVMs) and not para-virtualized (PV) virtual machines. HVMs leverage capabilities within silicon, including Intel’s VT-x and AMD-V.

The flaw was first reported two weeks ago to the open-source Xen Project by SUSE Linux employee Jan Beulich. In contrast with the Heartbleed vulnerability in April and the Shellshock vulnerability that was first reported Sept. 24, the open-source Xen project was able to keep details of the CVE-2014-7188 flaw private until the major public cloud providers could be patched.

The Xen Project has been run as a Linux Foundation Collaboration Project since 2013. Xen has had a detailed security response process in place since 2011 that has been incrementally updated many times to refine the process.

“If a vulnerability is not already public, we would like to notify significant distributors and operators of Xen so that they can prepare patched software in advance,” the Xen security response process document states. “This will help minimize the degree to which there are Xen users who are vulnerable but can’t get patches.”

Software vulnerabilities are an inevitable fact of modern applications. What the Xen project has managed to achieve is a way of properly managing the bug fixing process, without the hype and hysteria that is associated with zero-day bug disclosure. More importantly, by getting all the major cloud providers fixed before the flaw was publicly disclosed, the Xen Project likely saved the IT world from a major security nightmare.

The Xen Project has published a security advisory that could affect millions of virtualized servers running in Amazon’s cloud and other public hosting services. A flaw in the Xen hypervisor could allow a malicious fully virtualized server to read data about other virtualized systems running on the same physical hardware or the hypervisor hosting the virtual machine. The malicious system could also potentially crash the server hosting the virtual machines. A patch, which was privately disclosed last week under embargo, has been issued to correct the issue.