Ransomware attacks against local governments and cities are repeatedly making headlines, with crippling results on city operations and budgets.
Last month, the Florida city of Riviera Beach paid hackers $600,000 after being hit by a ransomware attack that downed its computer systems for three weeks. In 2018, several Atlanta city systems were crippled after a ransomware attack extorted the municipality for $51,000. And The city of Baltimore is another recent victim of ransomware, which hit in May and halted some city services like water bills, permits and more, demanding a $76,000 ransom.
Why do cities appear to be a low hanging fruit when it comes to ransomware attacks? What hurdles do state and local governments face when securing their systems and responding to attacks?
In the first of a two-part series, Threatpost talks to Shawn Taylor, the senior systems engineer at Forescout who covers state and local governments across the country. Taylor was in the trenches during the infamous 2018 Atlanta ransomware attack and recounts what the experience taught him about remediation and recovery efforts when it comes to cyberattacks.
Click here for direct download of the podcast. Below is a lightly-edited transcript from the podcast.
Lindsey O’Donnell: Welcome to the Threatpost podcast. This is Lindsey O’Donnell with Threatpost here. And I’m speaking today with Shawn Taylor, the senior systems engineer at Forescout who covers state and local governments across the country. Sean, thanks so much for coming onto the show. How are you doing today?
Shawn Taylor:I’m doing great. Lindsey, thanks so much appreciate you having me on today.
LO: I wanted to talk about a topic that keeps coming up again, and again, not just this year, but over the past few years. And that is ransomware attacks, specifically targeting cities, towns and municipalities. So you know, starting last year, we saw the big Atlanta ransomware attack. And then, you know, we also saw Baltimore, and some other towns and cities as well, over the past few years. I feel like this is just an issue that’s continued on over the years. And, Shawn, I know that you mentioned you were kind of up front and center with the Atlanta ransomware attack too.
ST: I was, I was fortunate enough to be able to get Forescout on site, sort of parachuted in, in the immediate hours afterwards to help with the triage and sort of troubleshooting activities. Unfortunately for the city when they got hit last March, they had become incredibly debilitated. The ability of the city of Atlanta to provide basic fundamental services was significantly compromised. Basic operational capability from an IT perspective was lost, you know, so what Forescout was very uniquely able to do in, in helping them, was really trying to achieve some level of visibility, and really provide a system of record to help them. But, you know, that was one of, as you had pointed out, multiples that sort of keep coming up. And it is sort of like a broken record that these entities continue to go through, right. And I think what the city of Atlanta sort of suffered through was really sort of the collection of legacy infrastructures, legacy networks, combined with there’s probably some other misconfigured technologies or those kinds of things that ultimately, that when you’re dealing with ultimately, overworked, potentially understaffed organizations, the problem seems to sort of precipitate itself more and more so that these types of events are going to continue to occur. And as you had said, they have.
LO: Right, yeah, I mean, even just in the past month, we’ve seen the big headlines have been about to Florida cities that were targeted as well, which is Riviera Beach, and Lake City. And then, you know, a couple of other towns too. So it’s clear, this is ongoing. And I’m really interested, you being at the forefront of the aftermath, these types of attacks. Can you walk us through kind of what takes place from the get-go, starting with the initial injection for certain infection for certain attacks, all the way through remediation and recovery? How does that kind of play out once a ransomware attack occurs?
ST: Yeah, I love that question. Because it is something that I didn’t necessarily appreciate firsthand until I was able, I sort of was thrust into it being onsite in Atlanta. But there is a point of inception, where the breach occurs for one of the cities and I want to say it was it was Lake City, I’m not positive. But I know that I read an article recently where there was an IT administrator who was fired because he ended up clicking a link, right, so there was a phishing attack, where there was a link clicked, where an exploit occurred, an exploit was exploited, there was a vulnerability that was exploited. And ultimately, there were multiple steps in the process, the Emotet was installed, and then there was the Ryuk ransomware, was ultimately then installed and sort of propagates right. And each of these sort of take on different flavors, different shapes, but there is a vulnerability, there’s a weak link in the process. And whether it is a human, that has not gone through adequate cybersecurity training, or has weak moment, right, or just isn’t necessarily as well-articulated in looking at an email or looking at a document, right, that they ended up clicking a link right, that ultimately exploits that vulnerability that exists in that system.
Now, rewind a bit, there needs to be a vulnerability on that system, right? So that goes to fundamental core cyber security practices of an organization to ensure that one, first and foremost, you’ve got visibility to all of those connected devices in your landscape. What is connected, are they all domain members, should they be on the network in the first place, and ensure that whatever mechanisms you put in place to try and ensure that they are adequately patched, whether you’re leveraging what Microsoft provides, or you’ve brought in a third-party product, but ensure that those products are in fact doing their job and patching properly, to mitigate and remediate inner vulnerabilities, so that when that link is clicked, there is no – unless it’s ultimately a zero day … you know, those are going to be very hard to stop – but ensuring that you can keep that adversarial script from executing on that endpoint is the first step. So ensuring that you’ve got someone who’s trained, knows not to click a link and verify URL and hyperlink and all of those things, but then to make sure that the endpoints that that person’s on, as well as the endpoints around the enterprise are, in fact, properly patched. And you’re aware of them, right, because ultimately, what ends up happening, and what happened in the case of Atlanta, which was really interesting, when they came into the network, they penetrated the network and the ransomware used in Atlanta was different than what was used in Riviera Beach and Lake City, it was known as the SamSam ransomware, and it tended to use a specific port protocol combination of RDP, right, Windows Remote Desktop, to use it as a communication mechanism. So once they gained access, they were able to brute force, hijack credentials, gain access to the network. And then by harvesting credentials, using tools that exist, unfortunately, all over the dark web, they were able to leverage credentials, then they were able to elevate those privileges, and now starts to deploy that payload that they were then going to execute at a later date around the network. And there’s no way because they are going over standard ports and protocols, they’re not doing anything that would raise any alarms of any anomalous behavior, because they’re just traversing the network.
Similarly, the others, the Lake City and the Riviera Beach, are essentially doing the same types of things. They’re going system to system. And ultimately deploying and leveraging the vulnerabilities that exist and whether it’s the EternalBlue. Thank you, Edward Snowden, and the NSA, right? But they’re deploying those payloads out to be executed, and ultimately are going to kick off the ransomware. Right? And then once that occurs, then you know, you say, ‘Okay, let me figure out where patient zero is. Let me cauterize, let me stop the bleeding, minimize the east west, how much of my network has been compromised, right?’ So once it occurs, it’s all about ‘Okay, you know, stop the bleeding, try to identify, minimize the damage, and then try to understand what the compromised landscape is.’ Because systems could be touched, and not necessarily be encrypted, the payload could have been deployed but not executed. Right, those kinds of things. So they’re dirty, but they’re still usable. So obviously, the systems that have been encrypted, your data is gone, you’re going to have to rely upon backups at that point.
LO: Great. Yeah, that’s a really good point.
ST: So then you end up with a situation where you’re, you’re looking to try and understand, ‘Okay, now that we’ve been able to identify the extent of the damage the extent of the compromise, and across the landscape in the enterprise. How can I recover? Right?
Do I have adequate backups? Do I have off site backups?’ Because a lot of these, and you know, let’s think about ransomware attacks, the vast majority of the time they attack Windows systems, salvaging those Windows-based vulnerabilities. And oftentimes, you’ve got storage and backup and database mechanisms that are underlying applications and those types of things running on SQL server, running on Windows based environments, right. So your backups that are stored on Windows based environments are all of a sudden, unless they’re offline, in an off-site facility, all of a sudden have become compromised themselves, then when you go to recover, you don’t have a backup to recover to.
LO: It always kind of surprises me that, you know, a lot of these types of local governments don’t have backups in place or don’t have kind of these precautionary methods. Is that something that you’re seeing? Is that something that governments are becoming increasingly aware of?
ST: So that that’s an interesting question: Shortly after, probably midway through last year, I went and delivered a message on ransomware, to a consortium of state IT leaders across all of the state agencies for large state down in the south. And it was sort of a lessons learned, right, it was ‘this is my experience, as a vendor, as an OEM, but this is my experience in seeing it firsthand,’ right? Because we very rarely are provided those opportunities to be in the front lines, as you had said earlier, right to sort of be in the thick of it, and to see the value that you can provide. So I was able to sort of relay this message to the state agencies, CIOs and CISOs. And one of the things that came out of it, they said, as I had wrapped up, or it was a Q-and-A, they said, one thing that I noticed that you didn’t really mention in here, around lessons learned, is governance. And I said, ‘You know, I come from a very long history in the in the service management in the IT SIM space, and service desks and those types of things.’ And I said, ‘You know what, that’s a great question.’ And that was something that was totally out of sight, out of mind for me, because I was so much in the incident response activities, that I wasn’t thinking about the stuff that was having to take place to keep the lights on right behind the scenes. And one of the things that I really went back and sort of re-analyzed what I had been delivering to audiences was, you know, governance is a huge problem and a huge need. And not just disaster recovery plans and those kinds of things, but their reliance upon service desks. So the fact that you’ve got a disaster recovery plan and backups, or you don’t, that becomes hugely problematic, right? So there’s this sort of building block effect that takes place that you start to peel back. And if you don’t have a foundation of some of these underlying technologies and processes, your ability to recover is significantly impacted, if not totally negated. So when I talked to customers, when I talked to prospects I talked about definitely the need to ensure that there is a disaster recovery plan, that there are off site backups, you know, these kinds of things that are ultimately reliant upon ensuring that you’ve got an understanding of all your critical systems in your assets and your resources and all of that stuff. But yeah, the message of ensuring that there are backups, ensuring that there are off site backups and those kinds of things, very much becomes a topic that everybody says, ‘yep, we know we need it.’ And unfortunately, that was something that I think probably, if I remember correctly, both Riviera Beach and Lake City paid the ransom, correct?
LO: Yes, I think it was Lake City paid $500,000 and Riviera Beach paid $600,000.
ST: And the only reason in my mind that I could think that they paid those ransoms, was the fact that they did not have reliable backups that they could point to, right. They were like, ‘well, we had to because otherwise we would not have been able to provide services to our citizens.’
LO: And I wanted to talk briefly about the pay versus not pay issue. Because like you said, Lake City and Riviera Beach both paid and both were criticized by the security community for this decision. But then, but when you look at Atlanta, I think they ended up spending $2.6 million and Baltimore dished out $18.2 million. And I mean, what are the pros and cons of going one way or the other? And what kind of decisions go into paying versus not paying from the standpoint?
ST: So I think, certainly from the government’s perspective, from US-CERT and FBI and Secret Service, I think, pretty much their mentality is don’t pay the ransom. Now, you know, it kind of comes down to one of those, the ransom that that Atlanta was originally asked was 50 some odd thousand dollars in Bitcoin. But I think when at the time, they sort of looked at it, and it was a… so I think there’s multiple parts to the decision making process. Each of these to varying degrees, each of these entities have a have a cybersecurity insurance provider. The larger metropolitan areas have larger policies, I’m sure. And there are underwriters in those policies and whether it is, Deloitte, or an insurance underwriter, I think a lot of shots are called by them. I think Atlanta probably said, ‘look, we’ve got a legacy network and old legacy network. You know, we’re going to take advantage of this opportunity, we’re not gonna pay it. And we’re gonna take advantage of the opportunity. And we are going to, we’re going to build this sucker from the ground up, the right way.’ So if you hear Gary Brantley, the new CIO, and actually, Ty Hayes, the CTO and even the CISO, you hear each of them sort of talk about, you know, going forward, it’s there is it’s all about sort of the new and really sort of building out best from the word go. So I think part of it is attributable from a decision making process to the cybersecurity insurance provider, I think part of the decision making goes into, do we have, can we not pay the ransom? Can we just say you got back up? So we’re good, right? Because you here, I’ve heard less that have not paid then I paid. But then when you when you deal with the those that have they’ve got the backups, right, they’ve got the good hygiene, from an operational perspective, maybe there was a negligent events that occurred that a opened up the breach, but they could, clean the environment, they could recover to the cloud, or recover from the cloud, and all of a sudden, they’re up and running and no worse for the wear, and they could patch the hole relatively quickly. I think there’s, just there’s various schools of thought that the challenge that that the state and local government entities, unfortunately, face is if there wasn’t a market for it, the adversaries wouldn’t be trying to do it.
LO: Yeah. Right. It almost plays into what the bad actors want, and then also incentivizes other bad actors to do the same.
ST: Right. And honestly, one of the interesting things I have the luxury of being able to speak to, to regionalized industry professionals, security professionals, and was down in Florida recently and spoke to a group of folks down, down in Tallahassee, when I went to talk about sort of talking about the ransomware, there are things that are available, ransomware for hire, you could, you could go to the dark web, and request a hack for, you know, whatever, and you pay for it, somebody’s going to give it to you, right. So ultimately, it makes it so easy. And 10 or so years ago, there was a study done that looked at the maturity level of adversaries. And, maturity level of the bad actors from a hacking perspective, has declined, right, rather steadily. But their ability to penetrate and to hack into the networks, and all of those kinds of things, has continued to increase. And that’s because of the pervasiveness of what’s out there, and what’s on the dark web and all that stuff. Right. And it’s just one of those things, that the more the problems exists, the more people pay, the more people pay the more the problems going to exist. So it’s this evil cycle, it just continues to spiral.
LO: This concludes part one of a two-part series on ransomware threats for local governments. Stay tuned for part two next week.
Don’t miss our free live Threatpost webinar, “Streamlining Patch Management,” on Wed., July 24, at 2:00 p.m. EDT. Please join Threatpost editor Tom Spring and a panel of patch experts as they discuss the latest trends in Patch Management, how to find the right solution for your business and what the biggest challenges are when it comes to deploying a program. Register and Learn More
