WASHINGTON–There has been a big push in recent years in the security community toward metrics, and measurements of all types have become a hot topic in certain corners of the industry. But measurement for measurement’s sake is useless-and perhaps even counterproductive–if the security team in an organization doesn’t define its goals and parameters ahead of time, experts say.
Security professionals have been measuring things such as vulnerabilities in a given application and the time it takes to fix flaws for years. Those things are easily quantifiable and it’s fairly simple to define the value in doing so. But there’s likely more value in finding ways to measure things such as the cost of fixing a vulnerability at various stages of the software development lifecycle and the cost of a data breach relative the cost of fixing a flaw before a breach occurs, said Chris Wysopal, CTO of Veracode, in a talk at the AppSec DC conference here Friday.
“Evaluating your spend on this is something that’s really hard to do,” he said. “You want to be headed toward mapping the cost of fixing vulnerabilities up front to the cost of a data breach.”
Data breaches have become an epidemic in the last couple of years, with laptop thefts, internal end external attacks and good old-fashioned human error combining to put an untold amount of sensitive corporate and consumer data at risk.Security experts and economists have been looking at what the hard and soft costs of these breaches are to the companies involved, but there still is quite a bit of disagreement on this front.
And many experts have said that it’s much more efficient and cost-effective to fix vulnerabilities early on in the development process, long before the application gets to a final code review or live testing. But, Wysopal said, this needs to be measured in each organization individually and used as background for executive decisions on development and deployment practices.
“You need to have goals for your application security metrics. You should measure progress and provide quantifiable information that you can roll up to the executive level in your organization to sipport decision-making,” Wysopal said. “You need a repeatable way assess improvement in the organization. For each vulnerability you find, it’s important to list things like the CVE number, severity on some practical scale and the likelihood of exploitation. If you do this, you’ll begin to build up data sets that you can roll up the organization.”
One of the interesting trends in corporate environments right now, Wysopal said, is that many security executives don’t really want to be the best at security; they just want to be better than most of their peers. It costs too much to be the best, and the difference in the return on the investments required to be better than average or near the top just isn’t justifiable.
“Security executives really just want to be more secure than their peers. They want to make sure they’re better than the average, because then no one can come to them and say they’re doing a bad job,” he said. “They think that if you’re way up in the top quartile, you’re probably spending too much on something. That’s not necessarily the way security purists think, but it’s the way a lot of security executives think.”
Wysopal also pointed out one of the ugly truths of the security metrics movement: A lot of people have no desire to be measured.
“Metrics can show you if you’re doing a good or bad job and that’s one of the reasons people don’t want to be measured,” he said.
That’s the bad news. The good news is, that despite those objections, more and more organizations are finding the value of measurement, creating data they can then use to improve, he said.
