Stop me if you’ve heard this already: Some cloud providers—thus far IBM SoftLayer and Linode—have alerted customers about hurried-but-planned updates to their cloud infrastructure to come this week.
The culprit appears to be another vulnerability to the Xen hypervisor that many cloud providers rely on to pack lots of workloads onto shared computer servers.
Over the weekend, IBM(IBM) alerted customers of a “planned event” to fix a potential vulnerability affecting its Virtual Server Instances or VSIs. The fix or remediation will require that its hypervisor nodes be maintained and the VSIs that run on those nodes be restarted, according to the notice.
Affected cloud data centers will be updated during a six-hour window between 10 a.m. And 4 p.m. Eastern Standard Time on Tuesday, December 15. An IBM spokeswoman said the company performs global updates to protect clients from vulnerabilities identified on its virtual services. In this case, it alerted “a small number” of customers affected by this Xen issue.
Linode, a smaller cloud and hosting provider based in New Jersey, likewise alerted customers Sunday of needed maintenance.
In a status post Sunday, Linode referenced “several Xen Security Advisories” that require that its host servers be updated, which means fixed and rebooted. That has to happen before December 17 when the Xen project team disclose the updates publicly.
Coins2Day reached out to other cloud providers for comment and will update this story as needed.
A Rackspace spokeswoman said the company is not conducting reboots and no action is needed at this time but acknowledged that security issues evolve so that could change. The company’s support team will contact customers if there is a change, she noted via email.
The reason all of this may ring a bell is because in late September 2014, a Xen vulnerability forced public cloud providers—including Amazon Web Services (AMZN), IBM, and Rackspace (RAX)—to quickly alert customers about the need to reboot systems to keep hackers from exploiting security gaps. Then a few months later, the same process was repeated with the serious Venom bug.
Finding and fixing vulnerabilities is a delicate business. The goal is to fix the holes quickly and discretely, ideally without disruption to customers, before the flaws can be exploited by evil doers. The process is described in the Xen Security blog:
If a vulnerability is not already public, we would like to notify significant distributors and operators of Xen so that they can prepare patched software in advance. This will help minimize the degree to which there are Xen users who are vulnerable but can’t get patches.
If past is prelude, expect more cloud providers to start alerting customers of maintenance windows as well. Amazon uses its own highly customized versions of the Xen hypervisor. And Google Compute Engine uses KVM, another open-source hypervisor that is presumably unaffected by this flaw.
Google has said its “live migration” capabilities helps it perform fixes fluidly, while Microsoft Azure uses the company’s Hyper-V hypervisor.
Interestingly, while Amazon estimated that perhaps 10% of its Elastic Compute Cloud (EC2) customers were affected by reboots in the September 2014 fix flurry, it said that number was drastically pared to less than 0.1% during the Venom kerfuffle, showing that Amazon has also hit upon a better way to perform rolling updates. Whether that is another form of live migration or some hot patching capability is unclear.
As Coins2Day’s Robert Hackett explained at the time, the Venom flaw was particularly scary. In theory the virtual machines running applications in the cloud ensure that Customer A’s workload on a given virtual machine will not impact Customer B’s workload also running one the same system. It’s an efficient way to harness computing resources while also purportedly isolating them from each other.
But with Venom, or potentially other hypervisor flaws, a bad guy could conceivably move from one virtual machine into another at will. As Jason Geffner, CrowdStrike principal security researcher, told Coins2Day at the time: “This bug lets you escape a container and get into all other containers.”
That raises the specter of some hacker breaking into and perhaps taking or corrupting your data. Not a pretty picture.
Phew! You can see why tech providers want to act quickly and quietly to fix what’s ailing them.
This report will be updated as needed during the day.
For more from Barb, follow her on Twitter at @gigabarb, read her coverage at coins2day.com/barb-darrow or subscribe via this RSS feed.
Make sure to subscribe to Data Sheet, Coins2Day’s daily newsletter on the business of technology.
For more on the Venom vulnerability check out the Coins2Day video below:
This report was updated at 10:52 a.m. EST with comments from IBM and Rackspace and again at 10:00 a.m. EST on December 15 to note that Google Compute Engine relies on the KVM hypervisor, not Xen as previously stated.
