Here’s an edited version of the full interview:
Dennis Fisher: Okay, welcome to the Digital Underground podcast. This is sort of a special edition. I’ve got Brad Arkin, the director of product security and privacy from Adobe as a guest, and we also have Ryan Naraine making his debut on this podcast. He’s been – we have our other podcasts that we do together, but he wanted to get in on this one with Brad, so we’re gonna talk about the Adobe stuff, the vulnerabilities that have been going on in the news in the last week or so. So, Brad, thanks for joining us. What have you been up to?
Brad Arkin: Oh, it’s been a busy week, here, as we, you know, head into the holidays, but it – it’s good. We’ve been able to get on top of some of these things that have come in, so hopefully we’ll – we’ll get things added as soon as we can for our users.
Dennis Fisher: Yeah, so let’s talk about the vulnerabilities. I think they were first disclosed, at least publicly, on Monday. When did you guys find out about them and – and how were they disclosed to Adobe?
Brad Arkin: So, the – so, there’s one bug that is in Adobe Reader and Acrobat. And we got several reports from different partners that we work with all within a couple of minutes of each other Monday afternoon, Pacific Time. And so we opened these files and did maybe two minutes of triage work to verify that it was an attack that worked against the latest fully patched version of Reader and Acrobat. And then, based on the number of contacts that we’d received on this, we figured this was something that, you know, people were seeing and that it was likely to get some coverage in the media soon. And so, we posted our PSIRT log as soon as that information came in, just saying that this appeared to be real, we were looking into it, and that we’d have more information later.
And so, from there, the team worked here in North America, and then in our remote offices overseas, on the problem through the night and we got the advisory out on Tuesday. And that advisory contained the CVE numbers, some details about what some users can do in order to mitigate the problem before a patch is available, and then we updated the advisory later on Tuesday with the ship schedule, which is January 12th, 2010. And so, these mitigations, just real quick, in case people haven’t seen the advisory –
By putting that into the black list, then any PDF document that it attempts to call that, that call will be denied. And so, it’ll deny valid calls as well as malicious calls that try to corrupt the call in order to create a crash. And this is something individual users can do, and also administrators for managed desktop environments can also do this using group policy objects to roll-out the change as a registry key.
Ryan Naraine: This docmedia.netplayer API, what kind of disruptions it could potentially cause to people that actually use that one legitimately?
RN: Right. Just to back up for a bit. Did you already have this vulnerability in the queue to be fixed before these public attacks? Or was this something brand new?
Dennis Fisher: So, that just triggers a crash in the Adobe application?
Brad Arkin: Yes, and the process would crash. But, it doesn’t appear to be exploitable in our testing.
Dennis Fisher: How much did the information that this was already being exploited play into how you guys reacted to the vulnerability?
Brad Arkin: We get a lot of e-mails every day that come into PSIRT-at-adobe.com, which is the – the way that most things get reported to us. And so, we’ve got different levels of priority that we handle things. Whenever something comes in advertised as being an exploit in the wild, then it gets our full attention. And depending on who reports it,we get a lot of contacts from anonymous people that are set-up, one-time-use G-mail accounts or something.
Those things have a wide variety of quality. You know, some of the time they’ll say that it’s a bug in Reader, and they send us a Word doc. Or, you know, things like that, and just doesn’t mean anything to us. But, the contacts that we got on Monday were from partners that we had established relationships with, very high credibility, and so this was pretty much our, you know, top tier priority response. And, you know, within a few minutes we were able to verify that it appeared to be an exploitable crash against the fully patched version. We pulled the alarm and everyone moved into our top flight response for these types of things.
Dennis Fisher: How much do you guys monitor “underground” to look for early signs of activity and attacks? I know that Microsoft and some of the other vendors do, and you were on the other side of this a little bit when you were at @Stake, you know, kind of the offensive side of things. You know how this stuff works. So, how much do you guys monitor the development of these exploits, how much the attacks are – are being seen in the wild, and that kind of thing?
Brad Arkin: So, our focus, historically, has been on getting a copy of a reproducible bug, and then we shift all of our focus into remediating the bug, and then communicating mitigations and that sort of thing. Maybe a year ago the answer would’ve been that we largely turn our backs on the details about new exploit techniques or, you know, anything fancier – escalation of attack levels, because from our perspective it’s the same bug, and so we just wanna get that bug fixed. And that’s where all our attention goes. Nowadays, some of the things that we’ve changed over the past 12 months is that we’re really putting a lot more energy into understanding how attacks evolve and how they scale out, and so, you know, the very first attack using this vulnerability might’ve been against some high value individual or company, but now we’re seeing it more widespread, and it goes from – it goes from very targeted attacks to more widespread. And that’s useful for us in how we communicate to the users and the kinds of mitigations that we can design-in at a more strategic level.
And so, the PSIRT team, which is Product Security Incident Response Team here at Adobe, which is one of the teams that I manage, those guys look after not only the communication with people who report vulnerabilities to us and getting patches out and that sort of thing, but they’re also more tapped into what we see as trends that are happening out there in the real world attacks.
And then, also hooked into the kind of places where you would pick-up the chatter about what’s going on. And so, all the different mailing lists and forums and things like that.
Dennis Fisher: Okay, so you’re definitely paying more attention to that these days, because it seems like Adobe and some of these other third-party applications are becoming more and more of a focus for the attackers, you know, I think because Microsoft and the operating system has – they’ve done a better job of locking that down in recent years, so it seems like the third-party apps have become much more of a high-value target for these guys.
Brad Arkin: Yeah, and you know, when you’re looking at it from the attacker’s perspective, the install base is – is a big attractive metric to look at. And with Adobe Reader and Flash Player, these are two applications that are installed on a lot more machines than Windows is, for instance. And so, that’s something that paints a bigger bull’s eye. And so, that’s something that’s not gonna change. You know, we’ve got this ubiquitous software, and the responsibility is on us in order to do the things that we can do in order to help protect our users.
Ryan Naraine: It’s well established that you guys are now wearing the target, and I’ve got a couple of questions. What’s the size of your security response staff? Do you feel, internally there, that you have enough resources? You’re delaying this fix until January 12th because, as you explained on a separate blog entry that, if you stopped everything to actually look at this it would affect the regular [quarterly] patch schedule. Which gives me the sense that there may be a resource issue there at Adobe, or is that not fair?
Brad Arkin: Well, so there’s – there’s a couple different things here. The incident response teams handle the security details of figuring out, “Is it a bug? Okay, it is, what’s the impact? And that team – we’ve got a core team that works for me here as part of the secure software engineering team. And then, the major product teams also have incident response personnel that work directly as part of the product teams.
Ryan Naraine: Right, so once you get it triaged it has to be kicked over there and then you’re basically at the mercy of those guys getting a fix ready for testing?
Brad Arkin: Well, not exactly. The incident response guys are there to – ’cause they’ve got the special skills for how to handle malicious samples and that sort of thing. Now, once we get a fix into the code-base, then we need to roll that fix into the actual update that a user can install and deploy on their machine out in the real world. And that process is not security specific. It’s just a question of, “How do you take, you know, a line change in the code and then turn that into an – an installer?” When you’re dealing with software that gets deployed on hundreds of millions of machines, the threshold is a lot different. And so, for us, if you make a one-line change in the code, then you need to roll that into all the different versions and flavors of Reader and Acrobat that are getting deployed on all the different platforms that we support.
And it turns out that something like 29 different binaries get produced from that process. And so, you have Adobe Reader for Windows Vista, Windows 7, Windows XP, Windows Server, all these different flavors, and then Mac, different Linux flavors. And out of those 29 binaries, then you multiply that across 80 languages, or however many are supported depending on the exact details of the – the platform and the software versions. And then, you need to make sure that in the process of producing all of those different flavors of the actual installer, that we didn’t introduce any new problems, and that we didn’t introduce anything that might break something else down the line. And so, all of this, it’s not a matter of just, you know, hitting the “compile” button and then an hour later you’re done. Once we get a build that looks like it addresses all our needs, then we need to run it through all the paces to make sure this thing is gonna deploy correctly, it’s gonna deploy only the change we’re looking at and not change anything else or change configuration settings or anything else that people care about.
That is the process that takes a long time. So, we can get a malicious sample, triage it, figure out in the code where the bug is and then get a fix into that code-base within, you know, maybe 12, 18, 24 hours. Then, we do a lot of testing to make sure that the code change we made didn’t break anything else. And then, also, we look around the code to see if there’s anything else that makes us uncomfortable that we should tighten-up. And – and all of that gets done pretty quick. And then, it’s the build process from there on out and getting everything tested and making sure it’s gonna work. Because, if we were to roll-out a patch that had some tiny little bug that impacted less than one percent of users, you’re still talking about millions of machines worldwide are gonna have these problems.
Ryan Naraine: But, you’re saying, “If I stop to just deal with this one, it affects everything else that’s already in the queue.” Is that a difficult decision for you to make, though, to say, “Well, listen, this is zero-day, it’s being attacked. Maybe the attacks are limited –and which brings me to another question, are you leaving the door open to change the schedule and go emergency if this things escalates at some point, say, in the next week or so?
Brad Arkin: So, to answer your first question first, then I’ll – I’ll talk about the – the escalation stuff. For us, we’ve got two different code branches that we work on. One of them is sort of the next release that’s scheduled, and we’re just grinding towards getting that out the door. And – and that thing – ’cause we’re coordinating the input of hundreds of people, it’s – it’s a program management thing of beauty how we get everything to line-up in order to get that out on the right day. And then, we’ve got an emergency branch that’s queued-up and ready to go in case we wanna put in a one-off fix and get that out quickly. And so, we do a lot of work to get that queued-up and ready, just in case. And then, when you look at some of the bugs that we patched in May and July, we were able to use that branch to get it out very quickly.
And so, this is an important input that we took into consideration. The other thing is that deploying that mitigation is a lot cheaper and easier for administrators and for users than downloading and installing a brand new update for Reader. And we’re looking at people – when we think of our customers, we have many, you know, home users, like, people like my dad, that are just managing, you know, one machine, maybe, like, you know, the laptop and the workstation; maybe two machines. And then, you’ve got administrators that are looking after managed desktop environments with 300,000 machines or more. And so, the cost of rolling a patch out to your personal laptop is, you know, a few minutes of annoyance. The cost of rolling that out to 300,000 machines is – is quite significant. And so –
Ryan Naraine: Yeah, but I still gotta roll-out the mitigations to 300,000 machines. I still have to do something twice.
Brad Arkin: And – and that’s a lot cheaper – much, much cheaper, the mitigation is. And you know, that’s a – a group policy object setting. It doesn’t require the same kind of work and testing that they have to go through for a – a full roll-out. And so, for us, the – the math was, we’ve got a cheap and viable mitigation that works effectively against the problem, and then we get to question of, if we do an out-of-cycle patch, that means we’re gonna delay January. If we delay January, that means all the things that are scheduled to get fixed in January are gonna get delayed. Maybe moved to February. We didn’t do all the details about what that schedule would be, but it would be a delay. And then, we’d be incurring that roll-out cost twice in quick succession for our user-base, and we talked with some of our users just to – to do a quick sample of what their take would be. And the question was, “You got the mitigation, and you’ve got the choice of having, you know, two patch roll-outs within a few weeks of each other.”
Or, we can do it once in January, which means you don’t have that full patch that you can install for an extra few days, but you’re gonna get something that is one time, and then hopefully you won’t see another one for another quarter.
The other thing to take into account is where we are in the calendar. And so, if we did everything we could and got a patch out, you know, and maybe that would hit December 29th, December 30th, who’s gonna be at work to deploy that? And where are all those machines that they need to deploy to? They’re gonna be in a bag sitting by the door, you know, ’cause people took it home for the – for the holidays. And so, it’s another introduction of complexity.
This is a tough situation. We’re starting out from a negative in that there’s already a vulnerability. And so, we’re not gonna make anyone completely happy with our decisions, and we know that. So, our goal is to do what we can do to protect as many users as quickly as possible, and – and then just work as hard as we can to get that done. And so, that’s the calculus that we went through coming up with this, and it’s definitely, you know, a hard decision. We put a lot of effort into trying to do the best job we could, but in the end that was the – the – the option that we selected.
Brad Arkin: We put out a notice on Monday and then we didn’t have mitigations and all the details until the advisory came out on Tuesday. I can walk through a little bit about how we make the decision about what we’re gonna publicly talk about, and – and then the information that we include in there. And so, whenever a report comes in, the first thing that we evaluate is what the priority is. And so, for us, it’s a credible source that reported it to us, and in the first 30 seconds that we looked at it, it looked like it was a real issue. And then, based on the fact that we got three reports, we figured this is something that more than a few people are seeing. And so, we decided to post immediately with a notice saying, “From our perspective, this is real. We’re looking into it.”
And in this case, because we knew it was gonna go public, we wanted people to know that we knew about it, we were aware of it, and we were working on it, and it looked real. In other cases, if someone reports something to us and it’s a situation where it looks like it was a targeted attack, there’s a good chance this one organization was the only entity in the world who’s seen this malicious PDF, then we might take a day or two in order to make sure that we really understand it before we do a full advisory in our first publication, which has all the details, all the information, all the platforms. And these are case-by-case decisions we make. We’ve got a lot of sort of written criteria to help us guide the decisions that we’re gonna make, but at the end of the day, what we’re trying to do is get the information out that can help people and make sure that that information has very high assurance levels, because lots of organizations and individuals take action based on what we put out there. And if we were ever wrong or had an error, then it – it – it’s a really painful thing for us to unwind. So, that’s a little more background on – on, you know, how we decide what to publish and when it goes out.
Dennis Fisher: How do you guys sort of deal with the difference between your enormous home user and consumer user-base, and the enterprise user-base? Because, you know, the enterprise guys are probably, you know, reading stories that me or Ryan write, or you know, other people, they’re aware of this problem and they know that a patch might be coming. But, you know, the millions of users, like your dad, my dad, unless – if they don’t have the automatic update features turned on, how do they know that this stuff is happening if they’re not paying close attention, which you know, most of them are not?
Brad Arkin: It’s a real challenge for us, and it’s something that, when we sit down and evaluate, you know, what – we’ve got sort of two needles that – that we measure very carefully. One of them is real-world attack activity, and then the other one is the fear level. And so, it’s the perception of real world attack activity, whether or not it’s real. And – and our goal is to drive both of those down into a healthy range where as few people as possible are being attacked and people have a good understanding of where the real threats are. There’s a lot of communication involved with that, and then a lot of technical things that we can do as well.
And when we sat down and we designed the new update that we have put out in a beta pilot for Reader, the design for that is something that we thought very carefully about how can we protect the most users, particularly at the consumer or individual level, because these are the folks that don’t have managed environments where someone does it for them. And so, for this new updater that – we shipped it in October. The January release is gonna be the first time that we use it for the beta users. We’re gonna learn from that, and then hopefully, if all that goes well, then we’re gonna roll it out as the new updater for all users in the next release. And what that does is it gives people an automatic download in the background and install the updates with no user interaction option. And then, it also – for people that want more control, they can download and then give them the choice to install it, or it can just notify – or you can turn it off completely. And so, by giving users these options, you know, people who have a well managed environment and they’ve got good reason for why they don’t wanna install an update…
We wanna be able to support that. But, for most of the people who, you know, they don’t wanna be bothered by it, but at the same time they need to be protected, that’s why we’ve got that automatic in-the-background update. And that’s because no amount of communication or any kind of education and outreach that we do is gonna reach every corner of the world. And so, figuring out ways that we can just do it for them in the background without bothering the user, that was a big thing that went into the design of the new updater for Reader.
Dennis Fisher: Okay. I know that Microsoft tracks pretty closely how many people use the Windows Update mechanism. Do you guys keep a handle on that as well?
Brad Arkin: We’ve got a lot of numbers internally. We’ve got a lot of metrics, but you – you get weird situations where a single download by an administrator, who then rolls it out to his company. That shows-up as one download in our numbers. And so, even though we have lots of numbers, we don’t share them publically, because there’s so much interpretation required before you go from the number to, you know, our best guess of what’s really happening. And so, we’re always watching the dials trying to learn from that and figuring out how to improve the – the patch update process.
Dennis Fisher: Okay. Do you feel like the number of people that are doing that automatically is going up?
Brad Arkin: Well, you know, with the new updater, which is in beta right now, that’s something that we’re hoping to – to make it the norm, where just, you know, everybody has it set up that way. But, right now, people don’t wanna get interrupted when they’re working inside a document, or you know, there may be something that disrupts the process so it doesn’t finish completely. There’s always these little errors that can get introduced in the update process, and so with the new updater we tried to incorporate ways to address as many of those as possible. And it’s not just at a technology level, but also at the human level, and the unit – what do you call it? – the human interface layer. So, making sure that if people are clicking “no thanks” when the update’s offered today, let’s give them a way to get the update installed without bothering them, or without disrupting what they’re doing.
Ryan Naraine: Are you guys doing any sort of extra auditing around API calls? Have you seen any sort of trend around bad guys looking at API calls to find vulnerabilities? Would you consider banning this particular API, or any API? ‘Cause, you know, Microsoft has done this. They’ve got this exhaustive list of banned APIs…Is that part of your process?
Ryan Naraine: You’ve got – you got the word “privacy” in your title. Flash cookies, virtually impossible to delete. There’s a lot of publishers and advertisers who are actually doing this respawning thing where they’ll actually re-instate traditional cookies using, you know, Adobe Flash cookies as an option. Are you making any sort of move towards giving the end-users some sort of controls about how Flash cookies are stored and how to delete them? What’s happening in that area?
Brad Arkin: Yeah, so, the terminology that we use [for Flash cookies] are: Flash player local shared objects, because they behave quite differently from browser cookies. But, the local shared object is something that – what we find is that there’s a lot of great uses for that where the developer will store data locally, it’ll improve network performance, it’ll improve the user experience where they can queue stuff up immediately rather than having to wait for network latency. But, then we’ve see there’s some confusion about how to manage the local shared object, and then also there’s things that subvert the user’s intention where, you know, we’ve seen things like this respawning that you talked about. And so, our goals are to make it as easy as possible for the user to exercise whatever it is they’re intending to do. And it’s actually not any harder managing local shared objects through Flash Player in terms of just, if you measure the number of clicks required. It’s just, it’s less familiar to users, and so people know how to go to their browser file menu and click on, you know, “clear cookie cash.”
But, doing those same clicks for Flash Player is something that people aren’t as familiar with, and we for a long time have been trying to work with the web browser vendors for them to open-up the API, so that when the user clicks “clear browser cookies,” it’ll also clear the Flash Player local shared objects. But, the browsers don’t expose those APIs today. And so, that’s something that we’ve been working with those guys, because if they can make that open up that API ability, then we can hook into that as Flash Player, so that when the user clicks “clear” it’ll clear Flash Player as well as the browser cookies.
Dennis Fisher: Is there a good reason why they browser vendors are not doing that now, Brad, aside from they just don’t want to?
Brad Arkin: Yeah, I don’t know. Every browser has their own way of evaluating priorities, and so you know, it’s more work, and so maybe they’ve – they’ve set priorities in other areas higher. I can’t explain, you know, how they prioritize their system, but for us that’s something that, if you want that one-click experience through your web browser, the web browser has to make that available for the plug-in. Our goal is very much to make it as easy and as intuitive as possible for the users to manage local shared objects. And so, there’s a lot of study going on right now around the user interface and the integration at the browser level of how we can best support that.
Ryan Naraine: Lastly, are we gonna get an Adobe Reader one day that’s kind of sandbox, like Google Chrome? That’s the consensus among researchers that that’s where we have to go. This is just – it’s just crazy to be relying on mitigations and patches…