VANCOUVER–When Facebook announced last month that its corporate infrastructure had been compromised through a watering-hole attack against several of its employees, it was major news, both because of the attack itself and because the company had come out and owned up to it. The interesting thing, however, is that this was not the first major problem that the Facebook incident response team had handled. In fact it was the third one in less than a year.
The difference is that the previous two incidents–one involving a complete compromise of the Facebook network and the other involving an engineer getting owned via a zero day and the attackers inserting a backdoor into Facebook’s production code–were both red team exercises. They were conceived and executed by the company’s IR team as a way to prepare the security team for a real attack. As it happened, the real attack came not too long after the last drill and it looked a lot like that exercise, which had been codenamed “Loopback”.
On April 11, 2012, a Facebook security employee found a laptop behind a filing cabinet. The machine was plugged into the power outlet and had full WiMAX capabilities on it. The security team quickly noticed that the laptop was not a company issued one and realized that it was connected to the Facebook corporate network.
“We immediately tried to figure out whether they were bridging our network,” Chad Greene, manager of the Facebook CERT, said during a talk at the CanSecWest conference here.
The security team realized it had a major incident on its hands and went into response mode. The forensics specialists began looking through the machine and noticed that it was calling out to a command-and-control server and then, once they began looking firewall egress logs, and within a short period of time realized that this was not the only machine communicating with that C2 server. The laptop had been found around 9:30 a.m., and by 11 a.m., the product security team, the e-crime team and the internal information security team all were involved in the response, and the Facebook CERT group was coordinating the entire process.
“We realized this was a real incident and not just and employee who dropped a laptop behind a file cabinet and plugged it in,” Greene said. “They had domain admin-level accounts on the network. By 11:30 we realized that members of our response team were owned.One of those laptops was mine. They had backdoors installed and our log searches kept failing. The attacker was actively killing the searches and was on the server and had root access. Pretty much the entire domain was owned. There was backdoor and beacon activity across the board. It was kind of a mess at this point.”
But the worst was yet to come. A couple of hours later, the security team received an email from the attackers. The email contained several screenshots, one of which was of the system that Facebook uses to roll out source code. The message was clear: We own you.
The attackers apparently had been in the network for some time and were angered by Facebook’s decision in January 2012 to publish information on the crew behind the Koobface worm, which had caused trouble for Facebook users for a couple of years. In the email, the attackers demanded that Facebook officials to publicly admit that they had identified the wrong people for Koobface and then to grant the gang access to the network in order to propagate the worm.
When the extortion email arrived, the members of the response team began checking their bank accounts and personal webmail accounts to see whether they had been compromised, as well.
“We realized this was some retaliatory attack on us for doing that,” said Ryan McGeehan, director of incident response at Facebook, who spoke with Greene. “The next day we decided to completely rebuild the domain. We were going to pave it. This is by far the worst incident I’ve ever had to respond to. How are we going to burn down our Active Directory infrastructure? We were hired to prevent this, so are we going to be working here after this?”
Luckily, the answer was yes, because the incident wasn’t real. It was the first of two large-scale red team exercises that Facebook has conducted in the last year. Red team exercises are certainly not a new concept–they’ve been around in the military world for decades and carried over into network security. But few of them are conducted in the way that this one, known internally as “Vampire”, was. McGeehan’s team kept the ruse going for more than 24 hours and kept close tabs on the way that the various participants reacted, communicated and disagreed. The idea, of course, is to prepare the teams for a real-life incident.
“We assume we’re a target all the time. By the time we’re responding to the real thing, we know what it’s going to look like,” McGeehan said. “We know which communication channels are OK and which aren’t. There’s no tool for this. It’s something that’s got to be experienced. By the end of this incident, everyone should feel like this is the worst day you’ve ever been through.”
While that exercise was brutal, so was Loopback, which followed on Oct. 31, 2012. Not wanting to replicate the scenario from the last one, McGeehan and his team this time identified a likely attacker–China–and decided to impersonate its tactics. For this one, they recruited an internal engineer as an accomplice. They wanted to get a backdoor into Facebook’s production code, so they sent a spear-phishing email containing exploit code for a live zero-day vulnerability to the engineer. He dutifully clicked the link and his machine was promptly compromised. (McGeehan would not identify which product the vulnerability affected, nor how the Facebook team came into possession of it, but said that they disclosed it to the affected vendor before the Loopback exercise and used it before the patch was publicly available.)
“We got onto the developer’s system and then put a change into his PHP code and pushed it live,” McGeehan said. “That affects a billion users, but the backdoor was designed not to run.”
The Loopback attack is eerily similar to the actual compromise that affected Facebook earlier this year. In that incident, several Facebook employees were compromised after visiting a mobile developer site. The attack didn’t affect any user data, but it led the company to disclose some of the details of the incident and to warn other companies who also were affected by the same attack. Microsoft and Apple officials later admitted that some of their developers also were attacked on the same site. The response to that incident went according to plan, something that McGeehan says is a direct result of the Vampire and Loopback exercises.
“We’re very well prepared now and I attribute that to the drills,” he said. “I’m not sure it would have worked as well otherwise. It felt like the second time we were responding to it and we were all ready for it. It was a much more calm, smooth response. [The exercises were] an incredible net positive.”
