The Cloudflare content delivery network for months has been leaking customer data, everything from private messages to encryption keys and credentials belonging to users of some of the Internet’s biggest properties.
The vulnerability has been addressed, Cloudflare CTO John Graham-Cumming said, but not before sensitive data was exposed belonging to users of a number of web-based services including Uber, Fitbit, OK Cupid and others.
Google Project Zero researcher Tavis Ormandy privately disclosed the issue last Friday to Cloudflare, which said that three “minor” features were to blame and had since been turned off. The first of the features, Graham-Cumming said, was turned on last Sept. 22, but he said that the time of greatest potential impact started Feb. 13 and lasted until Ormandy’s disclosure last Saturday.
Ormandy said in a bug report posted to the Project Zero feed that he saw some unexpected data surface during an unrelated project. The data was uninitialized memory among valid data that he determined was coming from a Cloudflare reverse proxy.
“It looked like that if an html page hosted behind Cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like Heartbleed, but Cloudflare-specific and worse for reasons I’ll explain later),” Ormandy said in his report. “My working theory was that this was related to their ‘ScrapeShield’ feature which parses and obfuscates html – but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers.”
The issue has been informally called Cloudbleed given its similarities to Heartbleed, a major OpenSSL vulnerability in 2014 that also leaked sensitive information in memory.
Ormandy said it didn’t take long during an analysis of some live samples to see encryption keys, cookies, passwords, POST data and HTTPS requests for other Cloudflare-hosted sites among the data coming from other users.
Ormandy shared what he had found with Cloudflare and yesterday disclosed in a tweet that the service was leaking customer HTTPS sessions including those from Uber, Fitbit, 1Password, OKCupid and others.
Cloudflare have been leaking customer HTTPS sessions for months. Uber, 1Password, FitBit, OKCupid, etc. https://t.co/wjwE4M3Pbk
— Tavis Ormandy (@taviso) February 23, 2017
1Password quickly refuted that the Cloudflare bug affected its data, and said it designed 1Password to protect against incidents like this when TLS fails.
An Uber representative said the impact against its users was minimal.
“Very little Uber traffic actually goes through Cloudflare,” Uber told Threatpost. “Only a handful of tokens were involved and have since been changed. Passwords were not exposed.”
OKCupid also said it’s investigating.
“Cloudflare alerted us last night of their bug and we’ve been looking into its impact on OkCupid members. Our initial investigation has revealed minimal, if any, exposure,” an OKCupid representative told Threatpost. “If we determine that any of our users has been impacted we will promptly notify them and take action to protect them.”
Fitbit told Threatpost that affected users should consider changing their passwords.
“We are currently investigating the issue reported with Cloudflare’s service to understand how it impacts our users. We encourage anyone who believes they have an issue to notify our team at security@fitbit.com,” Fitbit told Threatpost. “Concerned users can change their account password, followed by logging out and in to the mobile application with the new password. We recommend that users avoid reusing passwords associated with their email address or any other accounts, as this practice leaves them more vulnerable to malicious behavior.”
None of the other implicated services have made public statements. Meanwhile, there is a tracker available on Github listing some 4.3 million sites potentially affected by Cloudbleed.
Cloudflare’s Graham-Cumming said that in some circumstances, the company’s edge servers ran past the end of a buffer and returned memory containing private information. He clarified that no customer SSL keys were leaked because SSL connections are terminated at an isolated NGINX instance.
Graham-Cumming blamed an HTML parser present in three features for the leakage. He said that between Feb. 13 and 18, 1 in 3.3 million HTTP requests resulted in memory leakage, 0.00003 percent of all requests.
Cloudflare said it replaced its Ragel HTML parser a year ago with a homemade parser called cf-html. The underlying bug, it said, was in the Ragel parser as well but was never triggered because of the way the NGINX buffers were used. The new parser, however, changed the buffering and caused the leakage. The three features using the parser: Automatic HTTP Rewrites (enabled Sept. 22), Server-Side Excludes (enabled Jan. 30), and Email Obfuscation (enabled Feb. 13) were globally disabled or patched upon learning of the bug.
“Once we knew that the bug was being caused by the activation of cf-html (but before we knew why) we disabled the three features that caused it to be used. Every feature Cloudflare ships has a corresponding feature flag, which we call a ‘global kill’. We activated the Email Obfuscation global kill 47 minutes after receiving details of the problem and the Automatic HTTPS Rewrites global kill 3h05m later,” Graham-Cumming said. “The Email Obfuscation feature had been changed on February 13 and was the primary cause of the leaked memory, thus disabling it quickly stopped almost all memory leaks.
“Within a few seconds, those features were disabled worldwide,” he said. “We confirmed we were not seeing memory leakage via test URIs and had Google double check that they saw the same thing.”
A lingering issue is that search engines have cached the leaked memory, and Cloudflare is working with Google and other providers to scrub those leaks from caches.
“We’ve been trying to help clean up cached pages inadvertently crawled at Google. This is just a Band-Aid, but we’re doing what we can. Cloudflare customers are going to need to decide if they need to rotate secrets and notify their users based on the facts we know,” Ormandy said on Sunday. “I don’t know if this issue was noticed and exploited, but I’m sure other crawlers have collected data and that users have saved or cached content and don’t realize what they have, etc. We’ve discovered (and purged) cached pages that contain private messages from well-known services, PII from major sites that use Cloudflare, and even plaintext API requests from a popular password manager that were sent over https (!!).”
