CloudFlare, a multibillion-dollar startup that runs a popular content delivery network used by more than 5.5 million sites, accidentally leaked customers’ sensitive information for months, the company said Thursday. The firm has since fixed the issues at the heart of the problem, CloudFlare said.
The leaked data included “private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings,” according to Tavis Ormandy, the Google (GOOGL) security researcher who spotted and reported the issue last week. “We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”
In a tweet posted Thursday, Ormandy pointed to sites including Uber, 1Password, FitBit, and OKCupid, as having spilled data. Indeed, even sites seemingly protected by HTTPS, a security measure designed to keep hackers and spies from snooping on Internet traffic, were affected.
Get Data Sheet, Coins2Day’s technology newsletter.
CloudFlare responded promptly to Ormandy’s notification early Saturday morning. Within hours, the security team disabled several new features to its service—for those inclined: email obfuscation, server-side excludes, and automatic HTTPS rewrites—that had caused the problem to surface.
It took a week, however, for the team to fully remedy the issue, CloudFlare said. Search engines such as Google (GOOG), Yahoo (YHOO), and Microsoft’s (MSFT) Bing had inadvertently stored leaked data as part of their web crawlers’ caches, and the CloudFlare team had to work with them to scrub these indexes.
The memory leakage issue, known technically as a buffer overrun, began in September when CloudFlare swapped a new bit of code (an HTML parser) into its system. The program itself didn’t contain the major flaw, according to CloudFlare, but rather its introduction caused a separate and earlier coding error to, for lack of a better term, go kablooey.
In a technical post-mortem of the incident, John Graham-Cumming, CloudFlare’s chief tech officer, detailed what went wrong. “The engineers working on the new HTML parser had been so worried about bugs affecting our service that they had spent hours verifying that it did not contain security problems,” he said.
“Unfortunately, it was the ancient piece of software that contained a latent security problem and that problem only showed up as we were in the process of migrating away from it,” he continued. He added that his team has since begun testing CloudFlare’s software for other potential problems.
According to Graham-Cumming’s post, the leakage problem reached a nadir between Feb. 13 and Feb. 18 when 0.00003% of every page request through its network potentially let private information slip. Responding to an inquiry on Y Combinator’s Hacker News forum, Graham-Cumming added his team found data leaked across 3,438 unique domains.
After reading the post on CloudFlare’s website, Ormandy commented that “It contains an excellent postmortem, but severely downplays the risk to customers.” Because downloading and caching content from the web is a common practice for so many different organizations, Ormandy said it is likely that other crawlers have collected the leaked data without realizing it.
Ormandy also drew a tongue-in-cheek comparison to the Heartbleed—a computer bug discovered in 2014 that also caused sensitive data to leak from HTTPS sessions—by referring to the CloudFlare bug as “CloudBleed.”
It remains to be seen whether CloudFlare, or any of CloudFlare’s customers, will advise or force people to change their passwords and authentication credentials, though multiple security professionals have recommended taking that precaution.