Late Tuesday morning, one of the engineers in CloudFlare’s San Francisco office saw a message on Twitter saying that the New York Times Web site was down. Minutes later, more messages appeared, as security researchers and others began looking into the situation and realized that someone may have compromised the site’s DNS records. Understanding the ramifications of that sort of attack, if that’s in fact what it was, Matthew Prince, CloudFlare’s CEO sent an email to Rajiv Pant, the CTO of the Times, saying that the company’s engineers would be available if Pant needed some help figuring out the situation. He did.
The attack, which initially appeared to have affected just the Times, and later some of Twitter’s domains, actually affected several other domains as well, and turned out to be the result of a compromise at an Australian registrar called MelbourneIT. Initially, technicians at the Times and MelbourneIT were having a difficult time getting control back of the company’s DNS data. The Syrian Electronic Army, which claimed responsibility for the attack, had changed the records to point to a server the group controls, redirecting users from the Times home page to the attackers’ server. So Pant and his team called the CloudFlare team to see if they could help.
“They called and asked if we could put them in touch with someone at VeriSign, because they couldn’t get control back through Melbourne,” said Prince. “We reached out to some contacts we had there, and we also got them on the phone with folks at OpenDNS and Google. We were just a facilitator. We knew Rajiv and we knew they were having trouble, so we tried to help.”
Engineers at CloudFlare, a cloud networking and security company, along with engineers from Google, the Times, OpenDNS, GoDaddy and other companies, spent much of the rest of the day on a videoconference, trying to work out the details of the attack, and more importantly, how they could recover from it. One of their first moves was to begin flushing the cached DNS records around the Web, a tactic that would remove the SEA-controlled data and point users back to the legitimate Times and Twitter sites. That’s a tall order, however, because the data is cached at a number of different levels around the Internet, and those caches expire at different times and are controlled by many different entities.
“There are all these different overlapping caches that make cleaning up a problem like this tough,” Prince said. “We got on the phone to get those records flushed. Google and OpenDNS were extremely responsive. The challenge when you have one of these DNS hacks is that even when you get the correct records reinstated in the authoritative registry, the bad records will end up being cached out at the edge of the network. That’s why a lot of the world may not be able to access the Times site.”
As the afternoon wore on, some users began reporting that they were again able to access the Times main site, while others were still seeing an error page or something else. The engineers knew this was a result of the caches around the Internet beginning to expire and the legitimate data being propagated back out to the various DNS providers. That was a good sign, but at the same time, Times technical staff were warning employees to be careful sending emails, as they were unsure whether the attackers had the ability to redirect or capture the company’s email traffic, something that could be possible with the kind of access the SEA had to MelbourneIT’s systems.
The Times had been burned by something similar already, when attackers from China had succeeded in compromising the New York Times network, planting malware and accessing reporters’ email. So caution was warranted.
“It’s still not clear whether the email was accessed, and that’s spooky,” Prince said. “You lose your email, especially if you’re dealing with sensitive emails, and you’re in trouble. They’re potentially capturing Web sessions, and that means potentially capturing cookies, and that’s bad news. If an organization goes through something like this, they have to reset all of that.”
By late Tuesday evening East Coast time, the Times site was accessible again for most users and the paper had published a short account of the attack, saying that it was the result of a compromise at MelbourneIT, the registrar that the Times, Twitter, the Huffington Post, and other victims of the attack used to register their domains. The attack on MelbourneIT appears to have been just a means to an end for the SEA. The attackers initially sent spear-phishing emails to employees of an unidentified reseller partner of MelbourneIT. When an employee of the reseller responded, the attackers were in, getting credentials to access the MelbourneIT system, and giving them the ability to modify the DNS records of the target sites.
“Staff of an overseas-based reseller unwittingly responded to a spear phishing attack which allowed attackers to access sensitive information, including usernames and passwords, which was used to access the reseller’s account on Melbourne IT systems. This resulted in unauthorized changes to the DNS records of two domain names associated with providing news related to the Syrian conflict,” MelbourneIT said in a statement.
As troublesome as the attack was, it easily could have been far worse. Had the attackers modified the time-to-live, or expiration time, of the DNS cache in the records of the Times and the other targets, the effects would have lasted much longer.
“The awful thing would be to compromise the DNS and immediately set the TTL to 72 hours or whatever the maximum is,” Prince said. “Even Google would have a hard time flushing that. There are certain choke points on the network, and those will continue to be the foci of attacks.”
Attackers know what those choke points and weak spots on the Internet are just as well as defenders do, and Prince said the attack on MelbourneIT and its customers is a clear illustration of that fact.
“This is a really bad hack,” he said. “I can’t think of a hack that scares me as much as this one.”
