Inside some data center miles away, a portion of your cloud-hosted network may be running on the same system as someone else’s.
Normally, this isn’t a problem. So-called virtual machines—basically, computers simulated within other computers—prevent networks on the same machine from impacting one another. They’re an efficient way to manage large amounts of computing resources while, presumably, keeping them isolated and secure.
That’s not the whole story though, say researchers at the Irvine, Calif.-based security firm CrowdStrike. It turns out that an attacker can burst out of certain virtual machines and manipulate whatever’s running adjacently, thus shattering the notion that these vessels have hard and fast, protective boundaries.
“This destroys the isolation myth that you can have something run a virtual machine and have it be isolated from everything else,” says Jason Geffner, the senior security researcher at CrowdStrike who uncovered the flaw. “This bug lets you escape a container and get into all other containers.”
On Wednesday morning, Geffner’s team announced its discovery of a zero-day vulnerability, meaning a previously unknown computer bug, within a common virtual machine platform. The bug, dubbed Venom for “virtualized environment neglected operations manipulation,” affects a technology, known technically as a hypervisor, that controls and coordinates the virtual machines running on a system.
The vulnerability specifically affects the decade-old free and open source hypervisor called Quick Emulator (QEMU), which is used in a number of common virtualization products including Xen hypervisors, KVM (or “kernel-based virtual machine”), Oracle VM VirtualBox, and the native QEMU client. The popular products of EMC-owned VMWare (VMW) and Microsoft Hyper-V, on the other hand, are not affected.
Various security products also rely on the vulnerable technology to isolate and inspect malware—a potentially dangerous proposition given that certain virtual machines can, as now shown, leak.
“Even if you don’t use these services directly, chances are that accounts which store your personal data run these products,” Geffner says. In fact, CrowdStrike estimates that the bug could put thousands of organizations and millions of users at risk.
“With Venom, you’re able to break out of a virtual machine on a system and get access to other data on that system’s network,” Geffner says, adding that attackers can use it to “execute whatever code they like” by overwriting critical parts of a machine’s memory.
What does that mean exactly? To use a more familiar analogy: Picture an apartment building. That represents a cloud server, for our purposes. Now picture the apartments contained within that apartment building. These represent virtual machines. While different apartments may share resources such as water, electricity, heating, and gas—all managed, in this case, by a cloud infrastructure provider—all are locked and unable to access each other.
What Geffner has found, effectively, is a backdoor: a shared key that unlocks any apartment.
Dmitri Alperovitch, the chief technology officer and co-founder of CrowdStrike, says it’s a particularly bad bug. He compares Venom to other recently discovered vulnerabilities with high profiles such as “heartbleed” and “shellshock,” and says, “this is potentially much worse,” given just how much an attacker can compromise. Unlike those other bugs, he says, Venom tends to run on systems that have root level access, or heightened administrative privileges, which gives an attacker full access to an entire system, rather than just a single application.
“If the impact of heartbleed is akin to someone being able to walk up to your house and look through the window, and shellshock let’s them get inside the house and take out the TV,” Alperovitch says, “with venom someone can not only get inside your house and take the TV, they can also take your safe, steal your jewelry, and get into the neighbor’s house.”
Interestingly, Venom affects an almost entirely unused component of the QEMU hypervisor: its floppy disk controller. (In order to make virtual machines act enough like physical machines and have operating systems run on them, they require code that can speak to all parts of an actual system, even what may seem like outdated artifacts.) The vulnerability appears to have slipped through the cracks as no one was focusing on the security of that neglected area.
Dan Kaminsky, a security researcher and co-founder and chief scientists of the security firm White Ops, who was consulted about the bug during CrowdStrike’s period of responsible disclosure beginning in late April, plays it a little cooler. “These happen from time to time,” he says. “It’s not the first and it won’t be the last.”
Pressed on what similar vulnerabilities predate Venom, Kaminsky demurs. “None that can speak of publicly.” He pauses. Then qualifies his assessment by adding a superlative: “This is the most generic of these bugs that I’ve seen.”
“Everyone sort of absorbed this bug and no one thought to audit it,” he says. That’s why, he says, it’s important to hunt for bugs in non-obvious places. His advice for everyone right now? In the short-term, he says, “if your system doesn’t auto-patch, you need to go patch it today. If it needs to be rebooted, it needs to be rebooted today.”
In the longer term, he advises that people should, whenever possible, tell their cloud providers that they only want to share workflow with other people in their domain or company. Isolate your hardware to yourself. “If you have this sort of bug that can jump from their little piece of a server to your little piece of a server,” Kaminsky says, “the best way to avoid that is to not have anyone else on your server.”
“It costs more,” he says, “but you’re basically outbidding your attackers.”
Geffner says CrowdStrike has notified all the major software vendors that use this vulnerable QEMU code, and has worked with them to close their holes. He has not seen the bug exploited in the wild so far, though that may change now that the news has gone public.
“My hope and expectation is that the good guys are able to patch their systems before the bad guys get access to it,” Geffner says.