Patch management has become, in the words of one bleary-eyed IT guy, “just freaking ridiculous.”
Here’s a look at what this IT guy, whose primary role is managing risk at a medium-sized business, was up against in the last two weeks:
- May 22 — Novell released updates for GroupWise 7 and 8 to address multiple vulnerabilities in GroupWise WebAccess and Internet Agents.
- May 26 — Microsoft releases Service Pack 2 for Windows Vista and Windows Server 2008
- May 27 — RIM ships patch multiple vulnerabilities in the PDF distiller of the BlackBerry Attachment Service.
- May 28 — Microsoft releases pre-patch advisory to warn of zero-day attacks against DirectShow, via Windows Media Player
- May 29– VMware fixes multiple vulnerabilities VMware Workstation, Player, ACE, Server, Fusion, ESX, and ESXi.
- June 2 — Apple releases QuickTime 7.6.2 (10 documented vulnerabilities)
- June 2 — Apple releases iTunes 8.2 (1 vulnerability)
- June 8 — Apple releases Safari 4.0 (52 vulnerabilities)
- June 9 — Microsoft releases June security bulletin (10 bulletins, 31 vulnerabilities)
- June 9 — Adobe releases security update for Adobe Reader and Acrobat (13 documented vulnerabilities)
- June 10: Google updates Chrome to fix WebKit vulnerabilities.
- June 10: FreeBSD releases a fix for a stack-based buffer-overflow that presents code execution risk.
And that’s just the tip of the iceberg. According to data from the National Vulnerability Database, there were 515 vulnerabilities patched between May and June 2009. That covers software products from a roster of big-name vendors not listed above — like Cisco, Sun, Avaya, Mozilla, Red Hat, HP, Drupal, etc. etc.
“It’s nonstop man, just nonstop,” says James Juarez, a security manager at a New York-based financial institution. “There isn’t a day that we aren’t sifting through bulletins and firing up our patch deployment process.”
Yesterday’s Patch Tuesday was especially intense. Microsoft’s 10 bulletins (covering a record monthly count of 31 vulnerabilities in Windows, Internet Explorer and Microsoft Office) were released just hours before Adobe’s update for Reader/Acrobat (patching 13 highly-critical vulnerabilities).
But Juarez would have it no other way. “Hey, at least we knew it was coming and we were able to plan for it.”
Juarez said Adobe’s decision to piggyback on Microsoft’s Patch Tuesday was like a breath of fresh air and he’s calling on other vendors — particularly those with widely-deployed desktop products — to follow suit.
“I’d really like to see a Patch Week when everyone releases at the same time. It might sound chaotic but the more I think about it, it’s better than what we have right now. I could be talking to you right now and boom, we could have a big Firefox update to deal with. To me, a Patch Week is less chaotic.”
It’s a thought being whispered in many corridors.
Andrew Storms, director of security operations at nCircle, has long grumbled about the insanity of vendors dropping patches without advance notification or without a publicly known schedule.
“A Patch Week would be interesting,” Storms said, just 24 hours after dealing with the deluge of updates from Adobe and Microsoft. “It might be helpful but, at what point does it become too much?”
“On Monday and Tuesday this week, I can tell you that a lot of hours were spent dealing with patches. Could security teams manage a third or fourth day in a row doing that? That’s a different question. It would be interesting to see the reaction to something like that,” Storms added.
Storms, like Juarez and countless others in the patch-management trenches, says it’s the scheduling and pre-notification that make all the difference.
“Whether it’s two or three vendors releasing on the same day, you know in advance what’s coming and you know how to grab resources and manage everything.”
“When you get forced into fire-fighting mode, it becomes impossible,” Storms added. “You basically have to take resources from current projects to deal with a surprise update from Apple or Mozilla or any vendor who drops an update without any notice.”
Since Microsoft moved to the predictable second-Tuesday-of-the-month release schedule, Oracle and Adobe (quarterly) have followed suit. Cisco releases software patches every Tuesday and IOS updates twice a year.
Adobe has said it will always release its fixes on the same day as Microsoft and since Oracle’s Critical Patch Updates are always in the middle of the month, those will almost always coincide with Microsoft’s Patch Tuesday.
But there’s a laundry list of vendors — especially those in the open-source world — that simply drop patches at any time and without any notice, forcing IT guys into fire-fighting mode on a daily basis.
“At this point in the game, these vendors are mature enough with their secure development lifecycles that they know a patch is going to come out even a week ahead of time. At the very least, should be giving you a few days notice,” Storms said. “More importantly, they should all be shipping patches on a monthly or quarterly schedule.”
Some IT risk managers I spoke to for this article are not so sure a dedicated patch day or patch week is practical. The idea of wading through advisories, testing and deploying updates from more than three vendors was dismissed as too cumbersome.
Yet, when I asked about the last two weeks (see the list above), it’s clear that enterprise IT administrators are doing exactly that — putting out daily fires because there’s no coordination of pre-notification from anyone.
I’m beginning to think the idea of an industry patch day — or patch week — is much more practical than the status quo.
