Barely a week goes by that woeful statistics on the proliferation of cyber crime don’t make headlines in the trade and mainstream press. But a new study by Microsoft Research finds that many of those surveys are so rife with catastrophic statistical errors as to make their conclusions almost useless.
Cyber crime surveys are “so compromised and biased that no faith whatever can be placed in their findings,” said the paper’s authors, Microsoft researchers Cormac Herley and Dinei Florêncio.
A number of issues that negatively impact cybercrime surveys, among them: finding survey populations that represent the general public, the relative rarity of cybercrime incidents, especially when balanced of against the huge volume of legitimate online activity, and the effect of so-called “outlier” incidents that can create catastrophic statistical errors, Microsoft said.
Herley is no stranger to controversy. His research for Microsoft has frequently poked holes in conventional wisdom, whether about the profitability of cyber crime, the economies of targeted attacks and what makes a password “strong.” Measuring cyber crime is akin to measuring other relatively rare phenomenon, such as wealth and fame, the Microsoft researchers found. Such measurements are highly susceptible to the impact of outliers – respondents whose experience is well outside the norm.
“For a phenomenon that affects 5% of people, 95% of the population will have nothing useful to say: their answers contribute nothing to the estimate,” the researchers write. “Cyber-crime losses … are rare phenomena that are also extremely concentrated. That is, only a few percent of people suffer from ID theft. Even among those that do suffer from it the losses are extremely concentrated. The fragility of the sample distorts the results, Herley and Florêncio found.
Recent months have brought a slew of high profile reports on cyber crime – not all of them consistent. The 2010 Verizon Data Breach Investigation Report (DBIR), for example, found that reports of stolen records plunged by 97% in 2010, despite an increase in the number of incidents.
