BOSTON – Downstream is where you live today as a security person. If Gene Kim has his way, you’ll be inline soon enough.
Kim’s keynote today at Source Boston 2013 took listeners on a deep dive of the integration of development and IT operations and helped map out how organizations may be able to wedge security into the conversation and help security practitioners escape a system that pre-ordains failure—one they are for the most part powerless to avoid today.
Kim has spent more than a decade studying high-performing operations teams in a variety of industries inside and outside of IT. Those which are successful, are so with a combination of rigor and discipline, and pay more than lip service into the integration of security into application or process development. To put it in Star Trek terms, as Kim did, developers embody Mr. Spock in that they sit closely to the boss and think too hard about problems, while operations are more like Mr. Scott, engineers who pull levers and knobs, and yell a lot in an emergency. Security? They’re the token security guard who wears the red uniform and usually ends up as the casualty in every episode.
“We need to span the boundary between the two,” Kim said of development and operations. “We need to increase the flow of work in the proper direction and not pass defects downstream.”
Kim relayed an example of how Twitter injects static analysis into the development lifecycle every time a developer hits save on a project. If there’s an issue, they’ll get an email informing them of a vulnerability and how to remediate it. When the problem is fixed, the developer will get a “thank you” email.
“Security is done not at the end of a project when you add costs, but they do it inline,” Kim said. “In my opinion, this is the way all information security is going to be done 10 years from now. Not in batches and not at the end of a project.”
Kim said companies are collectively spending $2.6 trillion annually on IT failures, ranging from downtime, to data loss and more. Adding $2.6 trillion to the economy would radically change things, he said.
“Creating a culture and process that pre-ordains failure, for security downstream, this affects lives,” he said.
Kim assured attendees too that this kind of rigor isn’t reserved for rock star companies such as Google or high-end financial services companies, or Netflix. He’s seen success stories with retailers, higher education institutions and in many other industries. Learning from the big guys, however, never hurts.
Netflix, for example, was the only company running Amazon Web Services instances not to endure any downtime during a 2011 outage, Kim said. That’s because they made a decision never to rely on AWS for availability, he said, pointing to a decision to introduce chaos into its DevOps environment. The Chaos Monkey tool built by Netflix randomly kills processes in production all the time, forcing developers and operations to work together with security and learn how to defeat failure.
“They got really good at having code and an environment that survives failure,” Kim said. “The goal is to break things before they get into productions. Find misconfigurations, enforce HTTPs, add static code analysis to their automated integration and testing; they did all these things.”
Ultimately, organizations must evolve toward a culture that accepts risk and learns from failures. Google, for example forces its developers to manage their own code for six months before its passed on for approval and ultimately production.
“If an application is fragile, there is a hand-back mechanism where it goes back to the developer,” Kim said. “It’s a way for developers and operations to hold each other accountable.”
That accountability also includes feedback loops that include DevOps and security so that all are involved in incident escalation and mutual understanding of respective issues.
“The outcome is that defects are fixed faster,” Kim said. “If you do it for one issue, you should be able to replicate it throughout an organization. You have better communication and cooperation.”
