Companies are lagging when it comes to keeping up with software security patches – causing them to fall into “security debt,” Chris Eng, chief research officer with Veracode said.
Today, challenges around patch management are being worsened by applications using third-party code and open source libraries, which often introduce another entire set of vulnerabilities, said Eng, speaking at the RSA Conference 2020 in San Francisco last week.
“What will happen is companies will get further and further behind on those on those open source version patches,” he said. “And the further you get behind, the harder it is to catch up.”
The upside, however, is that researchers are finding that DevOps and security can actually coexist very nicely, creating opportunities to vastly improve software security. “If you incorporate security in the right way, DevOps is actually a great opportunity to improve the way that you’re doing software security. And so I think that’s the big… takeaway,” said Eng.
Eng discusses the biggest patch management challenges in his full interview with Threatpost, below.
Below find a lightly edited transcript of the interview.
Lindsey O’Donnell Welch: This is Lindsey O’Donnell Welch with Threatpost, and I’m here today at RSA Conference, joined by Chris Eng with Veracode. Chris, thanks so much for joining us today. How’s your conference been?
Chris Eng: It’s been very busy, as usual.
LO: It’s always a little crazy, definitely. So, thank you so much for joining us today, I wanted to talk to you about a report that you had released back in October. And it was called the “State of Software Security.” And basically, it was breaking down some of the most prevalent vulnerabilities that were discovered in software, but then also looking at how those vulnerabilities were being addressed and patched and some of the challenges around patch management for companies. Just to start, there is a term that you had used in the report called “security debt” – can you tell us what that term means and how it relates to this report.
CE: Yeah, I’ll give you that and kind of a little bit of kind of context for the report. So security debt is kind of if you think about it, anything that you know about in your software, vulnerability wise, that you just select not to fix, that you’re going to fix later. Right? You can think of it like, like financial debt, like racking up, you know, dollars on a credit card, and then paying the minimum each month, right? You’ll eventually get to paying all that down over time, but you’ll pay a lot of interest, it’ll will cost you more, it’ll be more painful. And so we see the same thing with with security debt across the applications that we scan. The report is basically a consolidation of all of the customer scams that happen in our product, so tens of thousands of applications and millions of scans over a one year period. So we can kind of see exactly what’s happening out there, across different industries and so on and what patterns are actually happening.
LO: Yeah, that’s really useful information to have obviously. So what are some of the key factors that go into security debt that you guys were specifically looking at?
CE: Well, the picture is not the greatest in terms of what we’re seeing out there, we’re seeing over half of applications are actually accumulating more security debt over time. About 25 percent are staying flat, and then 25 percent are actually reducing them. And obviously, that’s what you want, you want to reduce it so that you get to a point where you’ve eliminated all the debt, and then as you go along, you’re detecting new vulnerabilities along the way, but you’re fixing them as they come up, right? Just, again, with the financial analogies, paying down what you spend each month. And a lot of companies are struggling with the security debt for these applications that they may have been building for many years, and just kind of pushing the security vulnerabilities off to the side. So what we really wanted to figure out was, what are the factors that that contribute to how well an organization gets after the security debt. So we looked at a number of different areas there to try and like, find some correlations.
LO: Yeah. Can you go deeper into that?
CE: Yeah, so – and some of these are going to seem obvious once you get once you talk about them – but it’s actually different to have the data behind it and kind of show that that’s the case. One was just scan frequency, right? So if you’re scanning your applications, you know, once a year, or once a month, like that’s not as good as once a day, or more often than that. And so, you know, we found the ones that are scanning more frequently, the most frequently, are fixing stuff about three times faster. And so the amount of security debt, it’s not growing as much.
We also found that scan cadence matters a lot. So if you imagine over the course of a year, you could be scanning on a steady cadence, like either once a day or once a week or once a year, but at some steady pace, probably built up through some automation and you’ve got some scripts to do whatever. And then you can imagine a different visual picture where you have this like flurry of scanning, right? Like a security sprint type activity, and then you do nothing for six months, and then you decide you’re going to pay attention to it again. So you do this flurry of activity, and then you do nothing. And so we call that “bursty.” And so when you when you compare, like when you take every application that, you know, in the whole data set, and you map it against that, steady versus bursty versus irregular, you find that the steady ones actually do decrease their security debt over time, and the other ones get worse.
So it’s not a problem that goes away immediately, it’s a lot to deal with, it takes a lot of work to fix these, these coding issues. But you know, there are there are these factors that can make it easier to to overcome.
LO: You also looked at vulnerabilities like cross site scripting, all the way up to authentication and misconfigurations. And so how did that relate to the concept of security debt, in terms of how long it was taking companies to patch certain vulnerabilities versus others?
CE: Yeah we had a theory that higher severity vulnerabilities – so ones that were just like, had a higher impact if exploited – would be fixed faster. That seems reasonable. Or that higher criticality applications like for the business that says, “Well, this applications like way more important business value wise than this other one,” we expected that those, you know, those would be fixed faster. And it turns out that, you may have a little bit of speed improvement on, let’s say, the higher severity versus the medium or low severity flaws, but not significantly. It wasn’t nearly as impactful as the scan frequency and the scan cadence, which was interesting.
LO: Yeah.
CE: You know, you would just expect that intuitively, to work that way. And it turns out it doesn’t. We looked at a number of factors that we thought might influence that, and it really didn’t. And over the course – this is the tenth time we’ve done this report – and, over the course of those years, we are continuing to see the same types of vulnerabilities crop up over and over again, right? They’re not disappearing. If you looked at an individual organization, I think you’d see a downward trend, but as companies are starting to scan more and more of their software that they have, and companies are starting to do security testing that maybe have never done it before, those companies that are improving, those companies that are new, are kind of averaging each other out, right? You still see, the cross site scripting, the SQL injections, and all the issues that we’ve known about for decades.
LO: Well, yeah, you mentioned that this is the tenth time done this report, so have you noticed any sort of shifts in terms of things getting better or getting worse or is it remaining the same?
CE: You see slight declines in certain categories, especially the ones that are well known, tied to breaches, SQL injection, things like that. But it’s still fairly prevalent. And again, that’s still partly education, partly some applications never having been tested before. And so they’ve got a lot of stuff piled up that that was never addressed. So as we get better with education, as we get better with, again, like better automation of scanning and incorporating it into the development lifecycle at all possible phases, shifting that left and actually, you know, fixing stuff before it gets so expensive to fix. I think we are going to see that get better.
LO: Yeah, I mean, can you talk a little bit about patch management in general, and the complexities and challenges that companies are facing every day?
CE: Specifically in the software space, what we see a lot of is just like open source use, right? So nobody’s building applications from scratch, right? They’re using open source libraries for a lot of that functionality. And so you may write 10 percent of the code yourself and you may be borrowing the other 90 percent for a new application. And so when you do that, that’s great. You don’t have to reinvent the wheel every time. But you also inherit a lot of the risk from those open source libraries.
And what typically happens is you’re developing a new product, you choose those libraries, you download them. And then whatever version those libraries were on at the time, that’s the ones that you stick with it. So as you can imagine, over time, vulnerabilities are discovered in those libraries. And so the security of those libraries gets worse and worse and worse over time. So the patch management issue with regard to software is well, how much risk are these libraries now introducing, and when is the right time to patch those right? If something is is announced, and there’s an exploit for it, I’m suddenly vulnerable today, when yesterday I was fine. And I didn’t make any changes to my software, right? I just, the ecosystem just got worse. So what will happen is companies will get further and further and further behind on those on those open source version patches. And the further you get behind, the harder it is to catch up, if you imagine going from version one to version two on something that’s a lot easier than going from version one to version eight, because things break along the way. So that’s the pattern you see is this inherited risk tends to grow over time. And that is essentially another form of security debt.
LO: Right. And that makes it even more complicated, right, because you do have kind of, it’s almost like out of your hands a little bit.
CE: It is, things can change underneath you without you actually doing anything. And that’s the sort of the part that’s hard to wrap your head around, you don’t control the risk entirely. You hope that something gets patched around the same time that a vulnerability is announced. But sometimes you’re just left wide open. And you have to figure out a way to kind of code around that. So it’s nearly we’re paying a lot of attention to, the next version of the report that we’re actually working on now. We’re trying to drill in a lot more on the the open source and the third party stuff and and try and find some interesting tidbits that will hopefully tell us a little bit more about what’s happening there.
LO: Were there any other key takeaways that you wanted to highlight from this previous report especially looking into 2020?
CE: I think, really, the takeaway for us is, you know, there’s been a lot of tension, I think, between DevOps and security in the past, there’s a notion that, well, DevOps is trying to move so quickly, and how can they possibly do that? Because where will the security happen? Right? And so some security professionals that haven’t really kind of caught onto how DevOps is working are a little bit afraid of what that’s going to do to the safety of their software.
At the same time, you want the developers to be able to create business value and like code and solve these problems confidently and put the software out there in a secure manner. So what we’re finding especially with that scan cadence and scan frequency thing is that DevOps and security can actually coexist very nicely. And in fact, the practices that DevOps brings to software development actually are beneficial for security as well, that regularity that automation, the feedback loops. And so if you incorporate security in the right way, DevOps is actually a great opportunity to improve the way that you’re doing software security. And so I think that’s the big, the big takeaway.
LO: Definitely something to keep an eye on. Well, Chris, thank you so much for joining us today at RSA to talk about your report and what you’re seeing in terms of vulnerabilities and how they’re being addressed.
CE: My pleasure, nice talking to you.
LO: Great. Thank you.
