Google’s recent announcements that the company is doubling some of the rewards in its Chromium Vulnerability Reward Program and will also be committing up to $2 million for another round of the Pwnium contest in a couple of months brought a round of cheers from the security research community. The Google rewards programs have been quite successful in drawing submissions from researchers, as have similar programs from Mozilla, Facebook, Barracuda, PayPal and others, but the question around all of these programs is whether they actually succeed in making software, and by extension, the Web, safer.
This is a question that has been kicking around the security industry for a long time, both among researchers and vendors. The uncertainty is part of the reason that some of the larger software vendors in the world–namely Microsoft and Apple–have not initiated their own reward programs. If there was a way to prove definitively that paying external researchers to find and report vulnerabilities made those applications demonstrably more secure, then every vendor in the industry would be on board and the rewards would be a lot higher than they are now.
But the reality is a lot fuzzier than that. Vendors can look at the vulnerabilities that come in through their reward programs and make an educated guess as to whether they’d have been found otherwise, through internal methods or maybe via a normal vulnerability report from a customer or researcher. They also have the ability to look at the kinds of vulnerabilities that come in, where they’re located in the application, how difficult it is to develop a working exploit for them and whether they point out some underlying weaknesses in the code that hadn’t been noticed previously. All of that can be taken into account and the security team at a given vendor can then say, “OK, this app is more secure because of the submissions we got through the reward program.”
But that doesn’t amount to hard data that can be shoved into a slide deck and showed to executives. It’s just the considered opinion of the security folks and developers and engineers involved.
Earlier this year, Google officials, looking back at the progress of their reward programs to date, said that there was no question the Chromium and Web vulnerability bug bounty offerings had made a difference in user security.
“Google has gotten better and stronger as a result of this work. We get more bug reports, which means we get more bug fixes, which means a safer experience for our users,” Adam Mein of Google said at the time.
And that’s a key point: Google is getting more bug reports as a result of the reward program. Researchers at Mozilla said the same thing last year about the volume of bug reports in Mozilla Web properties. More bug reports means more patches and that means fewer vulnerabilities for the attackers. Not zero, but fewer, and that’s the goal. In fact, Google officials said this week they’ve seen a significant decrease in the number of submissions to its Chromium program.
“This signals to us that bugs are becoming harder to find, as the efforts of the wider community have made Chromium significantly stronger,” Chris Evans of Google said.
Microsoft officials have said they have no plans to institute a vulnerability reward program, saying that they believe there are better ways to spend money and work with researchers.
“We are always looking at the best ways to work with the researcher community,” Katie Moussouris, a senior security strategist at Microsoft, said in a story about bug bounties last year. “In terms of a per-vulnerability bug bounty program, the analysis from us is that not the best way to invest in the security of our product.”
Microsoft has gone a different way, with the creation of its BlueHat Prize contest, which offered $200,000 for the most innovative defensive technology.
Google’s program is among the more high-profile reward systems for researchers, and the company has spent a lot money on it. To date, Google has paid out more than $1 million in rewards through its various programs, including the Pwnium contest and Web and Chromium rewards. That’s a lot of money, but is it enough to really make a difference?
For researchers, there’s a specific calculus that determines whether the rewards offered are adequate, and it hinges on whether they think they can make more money doing something else with their time.
“I only wish bug bounties gave more money. Google is the only company which seems to be going in the right direction in that regard. Bug bounties are important because, if nothing else, it shows that the company takes bugs seriously. As for how [much] payout is ‘enough’, it is a complicated formula,” said Charlie Miller, a security researcher and principal research consultant at Accuvant.
“Each researcher, for each product needs to make a guess at how long it will take to find a bug in that product. Then they need to figure out how much money they could make during that time doing something else, for example consulting. This is going to vary depending on the individual researcher and the economy of where they live. For example, I’d guess it would take me a week or two to find a bug or two in Chrome. Well, in that time I can make a lot more money doing consulting than I would get for submitting a bug to Google. But I have a nice job at a good consulting company. If I was a help desk worker in the Ukraine, the answer might be different.”
So the success of a given reward program depends not so much on the vendor as on the researcher population. If the rewards are high enough to entice talented researchers to spend hours or days on a particular bug, then the program has a chance to be effective.
Facebook officials also consider their bug bounty program to be a success. It’s perhaps not as well-known as Google’s or Mozilla’s, but the Facebook program has gathered submissions from around the world and resulted in some big changes to facebook security.
“Facebook Security’s bug bounty program has been hugely successful so far and we’ve gotten great feedback from our active researchers. To date, we’ve paid out over $300,000 to 131 researchers and have one researcher coming on board as an intern this summer. Bounties are a great iteration on our responsible disclosure policy (which we’ve had for years),” Ryan McGeehan of Facebook said in a post on Quora. “We’ve made several site wide improvements based on input through bug bounty while overall being cost effective and fair to researchers. We’ve been able to pay far over our minimum bounty on a pretty regular basis, and in many cases it makes more and more sense to increase our investment in what has turned into a global community of researchers who are making contributions.”
What’s interesting about the discussion of bug bounties and their effectiveness is the distinction that some vendors draw between external researchers such as Miller, and the consultants that many companies bring in during development to look for vulnerabilities. There are a number of firms that specialize in that kind of work and they’re paid quite well for it. Microsoft and many other vendors use external security consultants to help with this task, but stop short when it comes to offering bug bounties.
What’s the difference, one might ask, between consultants and researchers? Titles. And their place in the development lifecycle. The consultants come in typically during development and researchers don’t get a crack at the software until after it’s released. For some vendors, that’s a big difference.
“Vendors create the bugs, hire engineers to find bugs, hire consultants to find bugs, so why not pay independent researchers when they do the same? The only way I can think of to motivate most researchers beyond big payouts is to make them mad or claim a product is unbreakable, and between the two approaches, I think this is a much better approach!” Miller said.
Bugs are bugs, regardless of who finds them. And software vendors often say that it’s far cheaper to find and fix them during development than after release.
But what’s the cost to customers of the vulnerabilities that are never found until an attacker stumbles upon them?
This article was updated on Aug. 17 to add comments from Facebook.
