Patch management is a thankless job. Data shows, despite best efforts, that 80 percent of enterprise applications have at least one unpatched vulnerability in them, according research by Veracode.
It is not for lack of trying that vulnerabilities persist. Last year 16,500 vulnerabilities were reported, making patching each one nearly an impossible task for any one company. Perhaps it shouldn’t be a surprise that Windows patching times appear to be moving in the wrong direction. According to Edgescan, the average of 63 days to patch in 2017 to 81 days in 2018.
What these statistics reveal is a process suffering under the weight of a shifting IT ecosystem that has ballooned to include a flood of bug submissions from a new crop of bounty programs, scrutinizing the growing army of chip-enabled gear.
The good news is that there’s hope. Patching experts say there are concrete steps companies can take to avoid patch fatigue. Identifying the problems is an important first step.
Click to Enlarge (courtesy Veracode)
Building a Better Patch Process
“One of the biggest patching challenges is first identifying everything that needs to be patched,” said Chris Goettl, director of product management and security for Ivanti. “The challenge becomes keeping a handle on how big a company’s universe of devices is — and knowing what has been patched and what still needs to be.”
Add in vulnerabilities related to software dependencies, such as the third-party code underlying software, and patching becomes even more arduous. Code repositories, open-source projects and small vendors poorly communicate bugs in their often complex library dependencies, he said.
Stagefright, Devil’s Ivy and the Zip Slip flaw are just of few examples of vulnerabilities affecting thousands of open-source projects. And last month, VLC developer VideoLAN alerted customers of a “high-risk” bug tied to a third-party component called MKV demuxer — a component responsible for multiplexing digital and analog files. The bug could allow an adversary to craft a malicious .MKV video file that could be used in an attack to gain control of the victim’s PC, according to VideoLAN.
“We assume a given developer is going to provide patches for their code. Obviously, they are going to fix any of vulnerabilities. But almost every product these days is based on other third-party components. There is Apache Struts, Microsoft .NET core and Java development kits. It’s very important that the components are updated as well,” said Todd Schell, senior product manager, security, for Ivanti.
These are vulnerabilities that are compiled into the code and aren’t something found by regular IT staff, Schell said. Developers, not just IT and security operations staff, need to be aware – pushing companies to adopt a DevOps approach to security.
A Race to Zero
Goettl also noted that there is a race by hackers and researchers alike to shrink the time between a zero-day bug discovery and the publishing of a proof-of-concept (PoC) exploit of the flaw.
In the case of the “wormable” vulnerability known as BlueKeep (CVE-2019-0708), Microsoft patched the bug on May 14, and by May 22 a proof-of-concept (PoC) exploit of the flaw was demonstrated. Other PoCs followed in the subsequent months. On Thursday, Palo Alto Networks published three additional ways to exploit system that have not been patched for BlueKeep.
PoCs offer security professionals vital and needed clues on how to identify a threat and mitigate against it. But, if patching isn’t part of the PoC discovery and fix, it could lead to catastrophic results – think EternalBlue and WannaCry.
As of July, the number of systems that remain exposed and unpatched to BlueKeep is close to 800,000, according to BitSight.
Making Sense of CVSS Scores
Tyler Reguly, manager of security, researcher and development for Tripwire, points out that the sheer number of patches that face IT and security teams leads to patch fatigue. The Common Vulnerability Scoring System (CVSS) is an industry standard for assessing the severity of security vulnerabilities and is meant to help with that; CVSS scores assign severity scores to vulnerabilities with the intent of giving security professionals the ability to prioritize responses.
However, oftentimes the severity score is higher than the threat really warrants; and sometimes it’s the other way around, Reguly said.
“What you end up with is a lot of people looking at these CVSS scores, confused, asking themselves ‘what should I patch, when should I patch it,'” he said.
Matthew Howell, senior director of product for Flashpoint, recommends that when it comes to patch prioritization, IT needs to evaluate the threat variables of a bug as they relate to one’s specific network. In a recent blog post, he recommends that security teams ask themselves:
- How likely the vulnerability is to be exploited in the wild?
- If the vulnerability is exploited in the wild, how likely it is to impact your organization?
- If the vulnerability is exploited at your organization, what impact it is likely to have?
The CVSS score assigned to a vulnerability reflects severity, not risk, Howell wrote.
Automating the Patching Process
Reguly says one of the best antidotes to fend off patching fatigue is automating as much of the patching process as possible. “You really need to architect to automate. That includes patching software, but also the processes as well,” he said.
That includes configuring orchestration tools such as Puppet to help automate the patching process. “If possible, some patch software supports APIs [and] can be used to tie in through an API for patch management and configuration management,” Reguly said.
He said automation also includes patch validation and making sure patches and security updates are implemented properly. Lastly, he recommends determining if a software vendor’s service-level agreement might also be leveraged to help with the patching process.
Driving a Stake in the Heart of a Check-Box Security Mentality
Jimmy Graham, senior director of vulnerability management, for Qualys, calls the process vulnerability management, not just patch management. He warns against focusing exclusively on patch management to the exclusion of the bigger security picture.
He promotes the idea of vulnerability management lifecycles that start with asset inventory, collecting vulnerability information locally, prioritizing vulnerabilities and then moving to patch management.
“Patch management is a cyclical process where you are patching systems based on the fact a patch was released by a vendor. Vulnerability management is there to make sure that patching was effective as well as the configuration management,” Graham said.
Best Practices
Best practices, according to Graham, include creating a patching cadence based on vendor releases. IT should also identify and prioritize patches based on the company’s unique infrastructure, and then test and deploy the patches themselves. Follow-up is just as important, he said, such as documenting the process for each technology patched — from operating systems to databases.
He also recommends patching and updating “golden images” (master software images used by IT for mass software deployment) at least quarterly. “That way all new systems are remediated before they go into production,” he said.
The goal is to significantly reduce or eliminate one-off patching.”If you’ve ever tried to figure out what version of Flash to deploy, then you aren’t doing it right,” Graham said.
The DevSecOps Approach
Still others advocate taking a DevSecOps approach to security — i.e., a blending of software development, IT operations and security practices into a streamlined system lifecycle.
“There is a strong correlation between how many times an organization scans and how quickly they address their vulnerabilities,” Veracode said in its State of Software Security report. “DevOps, or agile-driven development teams, are scanning more often, and as a result, they are making incremental [security] improvements every time they test.”
Veracode asserts that once organizations hit 300 or more internal scans per year, companies are seeing the “fix velocity going into overdrive.” Therefore, DevSecOps in theory could be vital when it comes to spotting bugs early in the development process – making vulnerabilities easier, faster and less expensive it is to fix, security experts say .
There’s a caveat to this though: The idea presupposes an unlikely harmonious relationship between developers and security teams. Often development teams are under pressure to deliver feature-rich applications on near impossible deadlines. Security teams, on the other hand, are under growing pressure to safeguard data.
Patching Landscape Moves to Cloud
Things are changing in the patching landscape as more compute moves to the cloud. Applications such as Salesforce and Office 365 represent a growing number cloud-based solutions running on AWS, Google Compute and Microsoft Azure. As the move to cloud services grows, cloud providers share the responsibility for security with their customers, Schell said.
The Cloud Native Computing Foundation, the organization behind popular container project Kubernetes, has had to tackle a number of vulnerabilities in its platform. One bug found last month could allow an adversary to access, modify or delete computing and storage resources configured across a cluster. Also last month there was an Azure update addressing a Kubernetes component that had a known vulnerability.
The shift from on-premise to off-premise (cloud) computing is a game changer for security teams, and it forces them to forfeit some control to platform providers, Schell said.
“It’s fully upon the shoulders of the [cloud] service provider to make sure the application is running with the most recent performance enhancements and security updates. And there is not much we can do as users,” he said.
(A portion of this article is adopted from a previous Threatpost webinar “Streamlining Patch Management“. In this 60-minute presentation, available on-demand, experts Todd Schell, senior product manager, security, for Ivanti; Tyler Reguly, manager of security R&D, for Tripwire; and Jimmy Graham, senior directory, vulnerability management, for Qualys, are joined by Threatpost editor Tom Spring to discuss streamlining patch management and offer patching advice, tips and tricks.)
