I’m a dirty vendor. That
may not be the best way to start a serious dialogue about security product
effectiveness, but I hate to read a post on security theory by some insincere
tie-wearing wonk only to discover afterwards that he or she is Lord High Poobah
of Marketing at “Scaring You For Profit, Inc.”
So I’ll just tell you who and what I am up front. I may have to wear the
tie, but I don’t have to be that guy.
Continue at your own risk.
The company I work for, Mandiant, provides incident response and computer forensics services to the private and public sectors. When I talk to customers, its often about managing the
constraints of their budgets while deploying new technologies to address
emerging threats. On top of established, often mandated legacy spends (think:
firewalls and AV) information security professionals are presented with a wide
array of technologies to address the changing landscape of miscreants,
professional criminals, and state-sponsored actors that threaten information
assets.
The challenge for them (and us): implementing effective security
measures in the face of budget constraints and externally-driven mandates. In my experience, this conversation usually features
customers lamenting the cost of ineffective but “required” technologies and the
expense of investing in new, future-facing solutions, often from small and
specialized vendors.What I don’t hear –
or see -isis information security practitioners evaluating the effectiveness of
the technologies they buy.
When is the last time you really studied the
effectiveness of your security technologies and compared them against their
cost? “Security metrics are hard” is a
weak excuse – there are a number of common sense approaches you can apply:
- Effectiveness
versus advertised value of the product: Product value claims vary widely by vendor. Some are outright preposterous (“prevent the
next attack before it starts”). Others
are more realistic, but perhaps only attainable in a limited context within
your environment. You need to identify
what return you’re actually getting. For example, anti-virus products
might be evaluated by looking at the number of hours spent by IT help desk,
security, and incident response teams resolving malware infections (I’m not
even talking real breach here – just cleaning up the common detritus that can
affect any computing environment). Don’t
have the data? Someone out there has done research you can use in a discussion. To be
clear, I’m not saying ‘antivirus is worthless’; but I am saying all security
technologies have shortcomings and frequently don’t deliver on advertised
value. You can benefit by understanding
that delta and use it to effectively manage your vendors into realistic pricing
expectations.
- Effectiveness
versus applied value of the product in your environment: Products may have many value
propositions that are actually true. However, constraints in your environment,
your staff, or the realities of space and time may make some of that value impossible
to realize. For example, network intrusion detection systems (NIDS) are
difficult to use for proactive detection without significant additional
investments in second order analysis, which costs you dollars and time. Theoretically
they have proactive detection value. In practice, that value isn’t
realized in most environments. However, NIDS are one of the most important
tools I can think of when investigating a breach. Used in that fashion they
have high impact, even without significant additional data aggregation,
analysis, and correlation capabilities.
While one advertised value may not be attainable for you, be sure to
assess the secondary value, identifying its utility and worth in your security
program. In this case I might attempt to
measure the number of incidents against the number of times NIDS were used as
part of the investigation.
Thinking about advertised versus applied value is a clean and
logical way to assess the true worth of a security technology. Vendors set product
and service pricing based on the value we think you should get (and be willing
to pay for). It is then incumbent on
you, and the marketplace at-large, to figure out what the actual value is and
then assess vendor performance based on your willingness to spend.
Of course, information security has another driver that
distorts the clean, logical flow of market fundamentals: compliance. I won’t
bother with an anti-compliance screed here – you can find plenty of others more eloquent on
the topic.
Compliance has its place, but it has less and less to do with managing
risks posed by real threats.
Compliance-only technology spends should have the same shape
in your budget: lowest total cost of ownership to get a ‘check in the go
box.’ Why? Because you have limited resources. You need
to preserve budget for things that actually matter, namely: countering threats
and mitigating risks to your information assets. If compliance forces your hand to invest in
something that has little or no risk mitigation value you should be a very
thrifty shopper indeed.
I’m not saying compliance spends have no risk mitigation
value. I’m saying they don’t automatically have value. You need to assess the advertised value of
the compliance-oriented control and associated technology versus its applied
value in your environment, and invest accordingly. For a control technology with reasonable
value (I’ll refer back to NIDS – not for detection, but for response): buy
solid technology your team can actually use, as well as the training for them
to use it well. For a meaningless
checkbox technology that doesn’t lower your risk (the guilty shall go unnamed):
spend as little as possible and blow the dust off when the auditors come
calling.
In addition to assessing product effectiveness and
identifying true value, you need to manage your vendors. We are simple creatures responding to market
demand. Invest your time and money in the security products that matter by
evaluating their worth to you. Share your assessment of effectiveness versus
advertised and applied value with your vendors.
Bring them up at the sales meeting and offer a blunt appraisal of your
return on investment. If you are buying compliance technologies that have
little to no value for addressing current risks your enterprise is facing, be
honest with your vendor. Let them know
how you view their contribution. Put them on notice that to win business beyond
the compliance need, they must demonstrate actual value in the face of modern
threats.
This really boils down to being an educated buyer with
realistic expectations about the computer science behind security challenges. ‘Protect
me from everything’ isn’t a realistic expectation. Conversely, ‘pay me like I
protect you from everything’ isn’t a realistic vendor expectation.
Dave Merkel is Vice President of
Products and Threat Management Services at MANDIANT.
