Security Metrics Go Prime Time at Metricon 6

The metrics movement that has been slowly but surely infiltrating the security community in the last few years has had its own annual gathering–Metricon–for some time now. It’s been a small, quasi-academic conference since its inception, but now Metricon’s organizers are branching out a bit, bringing in some speakers and content that are outside the norm and appealing to a broader audience.

The metrics movement that has been slowly but surely infiltrating the security community in the last few years has had its own annual gathering–Metricon–for some time now. It’s been a small, quasi-academic conference since its inception, but now Metricon’s organizers are branching out a bit, bringing in some speakers and content that are outside the norm and appealing to a broader audience.

By design or happenstance, the security metrics community has mostly been an insular one, with a small group of enthusiastic participants debating the value of various methods for measuring the effectiveness of security controls or programs. There was disagreement, there was innovation, there was little consensus. And it was good.

It got even better when Metricon hit the security world upside its head in 2006, putting the combined brain power of Andy Jaquith, Dan Geer, Adam Shostack, Gunnar Peterson, Betsy Nichols and Pete Lindstrom to work on the idea of a conference that would help define and shape the metrics conversation. Jaquith, who actually wrote the book on security metrics in 2007, Geer, Shostack and the rest worked hard to make the idea of metrics appealing and palatable to the rest of the community, and they’ve succeeded to a large degree.

Now, the challenge is making metrics–and talks about the topic–entertaining and interesting enough to get rank and file security folks involved in the effort and looking at what the outcomes of their security programs are and how they can be improved, rather than how many alerts a given IDS threw off in the past 24 hours.

Metricon 6, which is Aug. 9 in San Francisco in conjunction with the USENIX Security Symposium, is the first attempt to make that happen.

“It’s more of an actual conference where you’ll talk to former CSOs and people like me and Jake Kouns and Wendy Nather and have access to real data from real research, rather than what’s happened in the past sometimes with people talking about IDS data for correlation that no one is ever going to use,” said Alex Hutton, the organizer of this year’s conference.

Among the speakers and topics slated for next month’s conference are a talk by Kouns, the founder of the Open Security Foundation, on cyberinsurance, a presentation on the effects of insider threats from US-CERT and a talk from Nather of the 451 Group on risk.

“Everybody believes that security is way too important to describe it using adjectives and adverbs. The obvious thing you want to do is give objective numbers. Even subjective numbers would help in some cases,” Hutton said. “The goal this year is to say, let’s move beyond the color scale, management by Crayola color. Let’s really talk about how we can use metrics in a way that matters, and not just as an academic pursuit.

“The opportunity this year is to talk to people who have skinned their knees in this field and are some of the best risk analysts out there,” Hutton said. “People who know where metrics fit and where they don’t fit.”

Suggested articles


  • Anonymous on

    Don't be fooled.  I went to one of these in San Jose and it was a core group of academic types bickering over minutiae.  Dan Geer and AndrewJ are awesome, but the other core people were in it to tear each other down, insult speakers, and do other non-productive stuff.  Save your money and time for something more worthwhile.  Do avail yourself of any presentation slides they post on their site.  Some of the guest speakers content from outside academia is worth a read.


  • Alex on


    I felt the same EXACT way.  Which is the only reason I decided to help out this year, to change the format this time.  I see my job is to make it different, very different, to make it helpful rather than spiteful.

  • Russell C Thomas on

    I've been to most of the Metricon and Mini-metricon conferences since 1.5, and I take issue with the comments by "Anonymous" that these conferences involve "a core group of academic types bickering over minutiae"... or that people "tear each other down, insult speakers...".  I also take issue with the author's assertion that Metricon or is "insular" or "quasi-academic".

    Some people may feel that it's insular because you need to get apply to join the mailing list  (mostly to keep the signal/noise ratio high and avoid blatant sales pitches) and that you had to apply to attend Mini-metricon (held w/ RSA Conf. every year), due to limited capacity.

    The logic behind these community practices is the belief that we needed to bring together people who were *specialists* in metrics, who had spent considerable time and effort working on the fundamental unsolved problems, in order to break the intellectual, professional, and cultural log jams.

    Because of this approach, and the state of knowledge in security metrics, we will inevitably have many diverse views and approaches, including sharp disagreements on method and philosophy.  And to it's credit, the community has never been shy about voicing that disagreement.  Occasionally it has gotten bitter and personal (I have been on the receiving end of this), but the vast majority of interactions and debate has been focused on substance, which is what we need.

    Various people have been tossing around the label "academic" as a thinly veiled pejorative --  implying "impractical, irrelevant, detached from reality", and  so on.  A hidden implication of using this label is that the community would be better off if only it were guided by practitioners in industry and consulting.

    I think this is misguided on both counts.  First, and Metricon have not been dominated or over-run by academics, as some might believe from these statements.  In both cases, there are a *few* academics involved (professors, researchers, students), and other people who may be academically oriented, but any review of attendee and presenter credentials would show that these folks are a distinct minority.  Some people may notice them more because they are used to communities that are 100% practitioners.

    In contrast, go to the Workshop on Economics of Information Security and Privacy (WEIS), and you'll find that 90% of the presenters are academic, and about 75% of attendees are academic.

    The second implication is even more misguided.  Security metrics has unsolved *theoretical* and *empirical* research problems.  Practitioners don't solve research problems.  What we need is *MORE* engagement and collaboration between academic researchers and practitioners, not less.  But so far there have only been a few examples of success that can lead by example.  If we can organize collaborations that yield important results, then the resistance will melt away.

    Finally, regarding "bickering over minutiae", another way to describe this is "critical examination of methods, assumptions, and hypothesis testing".  Sometimes this can get very detailed and very technical (in the sense of mathematical analysis, probability/statistics, or simulation, etc.).  While some people may view it as "minutiae", such critical examination is *ESSENTIAL* if we hope to arrive at robust results.  Otherwise we risk falling victim to hand-waiving, fluff, incompetent methods, or fakery.


    In sum, I believe that Metricon and is one of the most healthy and productive (mixed) communities I've ever been involved in.   Anyone who is serious about solving the metrics problems should be involved. 


  • Betsy on

    You go Alex.  Kudo's to you.  This year's program looks awesome.

  • Anonymous on

    This series of events definitely hit its high mark with the publication of Andrew Jaquith's book, and has slid downhill since.  I wish good luck to the people trying to breathe new life into this, but it might be a better use of your money and time to look elsewhere for a conference.  Just buy Andrew Jaquith's Security Metrics book and leave it at that until this conference has proven itself worthwhile. 

  • Rich S on

    As a practitioner (as opposed to bickering academic) that has attend and presented at metricon, and knows the organizer, I can guarentee any non-productive bantering will be disallowed.   If you manage large scale security operations and need to measure the effectiveness of your investments - then participation in this event and the ongoing narratives is strongly encouraged.

  • Anonymous on

    They must be short on attendees to be publicly be soliciting people to come like this.  I think  when the event came up short in the past, they would post in places like Dailydave to increase the amount of paid attendance to help cover the cost of the event, meet whatever room guarantee they had made to the hosting hotel, etc.  Then you listened to the core half dozen or so problem people in the group let loose with the unpleasant and hostility, with the only pleasant relief being Dan Geer, Andrew Jaquith, and the practitioners who spoke from various large organizations.

    Again, I wish the event much success, but in the form that Andrew and DanG envisioned, not what it actually devolved in to.



  • The Awesome AndrewJ on

    @Top poster: The San Jose conference (Metricon 3) was not one of the better ones, I'll grant you.

    The community is a diverse mix of 800+ people with different backgrounds. Some are businesspeople, some are consultants, some are academics. Some are more inclined to discuss how to model security; others want to measure things and see what the data tells them ("modelers" versus "measurers"). That diversity is a strength.

    Because of that diversity, the conference planners -- who change every year -- have experimented with a lot of different formats. Long versus short, moderated versus not, half-day versus full, etc. The ones that have been more successful, I think, are the ones where the Chair takes a more active role in soliciting fresh and and interesting presentations, keeping the presenters concise and on time, and making sure there is plenty of time for discussion with the group. It's hard to pull off all three goals. Last year, I chaired. We did 10 talks plus a "rump session," but to fit it all in, I had to forcibly gong people off the stage after 30 minutes. That kept things interesting: the event got a 62% Net Promoter Score, which is great as these things go. But a few people wanted more discussion time, which just shows that you can't please everyone.

    I'm excited about what Alex, Mike are the team are doing with the format this year. It's a little different than in years past, but that's good. I like that they've eliminated the invitation-only restriction; it should be bring in a whole new set of participants. I have full confidence in Alex and Mike. Alex delivered the top-rated talk at last year's Metricon (4.58 on a 5 scale), and Mike invented the Security B-Sides conference. Maybe you've heard of it?

    It's going to be a fun one.

Subscribe to our newsletter, Threatpost Today!

Get the latest breaking news delivered daily to your inbox.