Few organizations experience the scale of Web-based application security challenges that Netflix engineers deal with on a regular basis. Sometimes the response to a threat requires a homespun tool that, more often than not, ends up being released to open source.
“Our assumption is that we predominantly see open source as a way for us to have more players on the field,” said Scott Behrens, senior application security engineer at Netflix. “We are in an information-sharing age, and from a security perspective, it’s really powerful to get more companies working in this space to solve problems together. That makes us all more effective in the long run.”
Since April, four tools built in-house at Netflix have been made available for use and review that solve a unique blend of application security problems facing not only Netflix, but other large organizations developing critical web applications.
“We want to encourage that,” Behrens said, adding that engineers put an open call at the end of each project soliciting ideas, contributions and success stories that can be rolled into the software. “We know for a fact that getting an outside perspective from others in the security industry is important because they can look at these tools and come up with ways to make them work even better.”
This week Behrens and colleague Andy Hoernecke, also a senior application security engineer at Netflix, released a pair of tools that work hand-in-hand searching malicious sites, discussion forums and social media for hints of impending attack, and then taking snapshots of those sites, saving admins from having to physically visit the site with a browser.
The tools, called Scumblr and Sketchy, come with a set of libraries for major sites such as Google, Facebook and Twitter, as well as a mechanism for building custom searches. Scumblr and Sketchy come on the heels of a late-June release of Security Monkey, a tool developed by another Netflix engineer, Patrick Kelley, that can be used to monitor and analyze Amazon Web Services configurations.
Scumblr and Sketchy ran in production at Netflix in February and March and once it ran internally for a while, Behrens and Hoernecke saw the benefit of releasing them to open source.
“A lot of the issues organizations deal with is chatter; hacker groups talking about attacks on Twitter or forums,” Behrens said. “We wanted a way to monitor the general feel for security around Netflix services. We wanted to track when things could potentially go wrong and have a plan or a set of proactive responses ready in place. Those initiatives drove the development of Scumblr.”
Scumblr is a plug-in, and Behrens said it’s easy to write custom plug-ins for other sites that aren’t in the library. Sketchy integrates with Scumblr and offsets having to visit a malicious site and potentially infecting a domain with a drive-by download or Java exploit. Sketchy, a headless browser built with PhantomJS and zero support for Flash or Silverlight, will take a real-time snapshot of what’s happening on the sketchy site and an analyst will be able to understand what it’s doing without having to visit the site. Both are set up in Amazon Web Services environments isolated from one another, because as Behrens said, if Sketchy crashes or is infected, it would not be able to act as a pivot point for an attacker.
Hoernecke said he expects similar levels of feedback and contributions to these two tools as have already been contributed to Security Monkey.
“We all have a unique perspective and come from different places in our careers,” Hoernecke said. “We all think that we’re not helping security by keeping tools like these a secret. There’s value in sharing, and allowing the community to make suggestions and improvements and help others improve their security posture at the same time.”
Behrens and Hoernecke in April announced during the SourceBoston conference that they were releasing a side project to open source called Ensnare. Ensnare is a tool that detects attacks against Web applications and reacts to those attacks by sending responses back to the offending browser that range from error messages to security alerts to long delays hoping to send the attacker away to another target.
“There is plenty on the drawing board. Our long-term strategy is to continue to open source security tools,” Hoernecke said. “The goal is to continue to keep giving back to the community.”
