Remote code execution bugs are the gold nuggets of security research. They’re the ones that researchers stay up all night looking for and they’re the kind of vulnerabilities that often are worth big money, whether it’s from a vulnerability broker, a government agency or a bug bounty program. For Reginaldo Silva, when he came across a serious vulnerability in the OpenID module in Drupal, he wasn’t sure right away exactly what he had or how valuable it was, so he reported it and later received a $500 bounty from Google, which uses OpenID. Only later did he realize it might have a much broader impact, and that’s how he ended up with a much, much bigger bounty from Facebook.
Silva, a computer engineer from Brazil, said he was messing around with Drupal back in September 2012 and ended up discovering a problem with the way Drupal handled OpenID. The vulnerability was an XML external entity expansion bug that allowed an attacker to read any file on a filesystem and take some other malicious actions. He reported the bug and it went into the CVE system, but a few days later he started thinking about how widely used OpenID is. He tested some Google properties and found that AppEngine and Blogger were both vulnerable and got a $500 bounty for his trouble.
But Silva kept looking around and remembered that Facebook allowed OpenID logins, but couldn’t find a way to enter an arbitrary OpenID URL, so he figured the site wasn’t vulnerable.
“So for more than a year I thought Facebook was not vulnerable at all, until one day I was testing Facebook’s Forgot your password? functionality and saw a request to https://www.facebook.com/openid/receiver.php,” Silva wrote in a blog post explaining his find.
“That’s when I began to suspect that Facebook was indeed vulnerable to that same XXE I had found out more than a year ago. I had to work a lot to confirm this suspicion, though. Long story short, when you forget your password, one of the ways you can prove to Facebook that you own an @gmail.com account is to log into your Gmail and authorize Facebook to get your basic information (such as email and name). The way this works is you’re actually logging into Facebook using your Gmail account, and this login happens over OpenID. So far, so good, but this is where I got stuck. I knew that, for my bug to work, the OpenID Relying Party (RP – Facebook) has to make a Yadis discovery request to an OpenID Provider (OP) under the attacker’s control. Let’s say http://www.ubercomp.com/. Then my malicious OP will send a response with the rogue XML that will then be parsed by the RP, and the XXE attack will work.”
Silva continued working on the Facebook login and eventually found a way to trigger the bug. But he couldn’t find a way to read any files on the vulnerable system, until he realized he had a small bug in his own code. Once he fixed that, he was in he clear.
“That’s right, the response contained Facebook’s /etc/passwd. Now we were going somewhere. By then I knew I had found the keys to the kingdom. After all, having the ability to read (almost) any file and open arbitrary network connections through the point of view of the Facebook server, and which doesn’t go through any kind of proxy was surely something Facebook wanted to avoid at any cost,” he wrote.
Silva wanted to escalate the bug to a remote code execution vulnerability, though, and kept working on the problem. However, he also wanted to make sure he played by the rules of Facebook’s bug bounty program, so he reported the XXE flaw and asked for permission to continue working on elevating it to a RCE flaw. That initial report sent the Facebook security team into full-on quick response mode immediately.
“In November, we were reading through incoming bug reports and came across a claim we wanted to investigate right away: arbitrary file reads. The report was well written and included proof of concept code, so we were able to reproduce the issue easily. After running the proof of concept to verify the issue, we filed an urgent task—triggering notifications to our on-call employees,” the Facebook security team explained in its account of the incident.
The team implemented a short-term fix in one line of code immediately and then set about trying to figure out how to push it to all of its Web servers. Once that was handled, the team looked for any associated issues and tried to determine whether there was a better long-term patch.
“After debugging, we concluded that libxml_disable_entity_loader(true) was indeed the correct final fix. Because we want to leave the code in a better state than we found it (rewrite old code, write tests, etc), writing the long term fix is often the step in the lifecycle of a bug that takes the longest. We wanted this line to run before anything else, so we put it in the lowest level of the callstack in our request initialization code,” Facebook said.
While this was happening, Silva was at lunch thinking about what he would do to escalate the vulnerability to RCE. When he returned, he discovered that Facebook already had implemented its fix.
“Needless to say, I was very impressed and disappointed at the same time, but since I knew just how I would escalate that attack to a Remote Code Execution bug, I decided to tell the security team what I’d do to escalate my access and trust them to be honest when they tested to see if the attack I had in my mind worked or not. I’m glad I did that. After a few back and forth emails, the security team confirmed that my attack was sound and that I had indeed found a RCE affecting their servers,” Silva said.
The Facebook security team realized the severity of the flaw and was considering a major bounty for Silva. They settled on a formula that averaged the recommended bounties from several of the company’s program administrators and came up with the final figure: $33,500. That’s one of the higher bounties paid by any of the major vulnerability reward programs, outside of the special bounties that Google sometimes pays in its Pwnium contest or Microsoft pays for mitigation bypasses.
“Plus, and more importantly, I get to brag I broke into Facebook… Nice, huh?” Silva said.
