Dennis Fisher: Welcome to
the Digital Underground podcast. This is the first episode in what’s going to
be a series of podcasts with CSOs from states around the country. We’re going
to be discussing the unique challenges of running an InfoSec program in the
public sector and what lessons enterprise security staffs can learn from their
counterparts in government. So my guest today is Bob Maley, the chief
information security officer of the commonwealth of Pennsylvania. So Bob,
welcome to the podcast.
Bob Maley: Thanks, glad to be here.
Dennis Fisher: Alright, so
you and I had a discussion a couple of years ago and you were the first CISO of
Pennsylvania when you took the job in 2005 I think and you were sort of
starting from scratch. They really didn’t have an information security program,
at least sort of a united one at that time. So what was the state of things
when you took the job in ’05?
Bob Maley: Well back then given that most states
typically are very distributed with their IT as well as their security
and
under the governor’s administration here there’s around 47 agencies,
boards,
and commissions and at that time most of those folks, the larger ones
were
doing things independently. There was no coordination and several
years, I
guess it was about a year and a half before I took the job that the
commonwealth had been victim to several virus outbreaks where they
experienced
some significant outages and at that time the office of information
technology
decided to launch a lot of technical controls and consolidation of
those types
of controls like centralizing antivirus and they had put some things in
place
that they were ready to take a look at but there really was nothing
there, for
instance enterprise-wide intrusion detection and prevention in a host
space,
incident monitoring, incident reporting.
So there were a lot of things that had been talked about but nothing
had
been done. There was no policy really. There was an older policy that
was written
about three or four years ago that really had become irrelevant. So it
was kind of like a clean slate. I was able to come in and really start
a
program from scratch designed after where I felt we should be taking
the
commonwealth and it’s worked out pretty good.
Dennis Fisher: So what were
your top priorities when you came in?
Did you have specific mandates from the higher ups as to what they
wanted you to focus on?
Bob Maley: No, there were no mandates at all. Obviously other than the number one mandate
and from the very beginning I put this into the security division’s mantra, our
reason for being, and that’s protecting citizen data and whatever we had to do
to make that happen is what we did.
Dennis Fisher: Okay so what
were the things that you focused on, the first maybe three or four things when
you got in saying you know what, we need to tackle this, this, and this?
Bob Maley: Well, like I said, there had been those
technical programs that were under way and I had the luxury I think of being
here as a consultant prior to taking the role and I was managing five, or I was
the project manager for five of those technical controls. So the first thing obviously was those things
we were rolling out, those things we were beginning, we had to ensure that a)
they were appropriate b) that they actually would produce the results that were
intended and c) we had to make sure they were rolled out effectively on time
and we did those under budget as well.
So that was the number one priority, to get those technical controls in
place. The second thing that we
undertook was looking at the security policy that we had and we had a policy
that was written several years before and was really written with an
operational slant. At the time there
really weren’t any connections between the needs of business. Even though we are government we are in
business for the benefit of our citizens, services to our citizens, and our
agencies all have specific business needs and that’s a point that kind of got
missed. So we had to go through that
process and totally rewrite what was there, introduce new policy for areas
where it was lacking, and then an ongoing process of making sure the policies
that we were doing were kept up to date.
Dennis Fisher: Okay and
you
mentioned the sort of unique challenges of being in the government. It
is a business as you said. You’re not trying to waste money aside from
what some people might think. But how
much pressure is there in terms of budget because obviously the
economic
situation in the country is not great these days and everybody is sort
of
feeling the pinch. So how much has the economic climate affected what
you’re
able to do internally?
Bob Maley: Well I think not a lot simply because we’ve
developed our programs around the fact that budgets are tight even before the
economic climate hit us. We’re very,
very cognizant of the security in the private sector. It’s tough enough to get security elevated to
a position where senior management, the stakeholders look at it as an
investment in the bottom line of business as opposed to an expense and the
worst security programs obviously are looked at by management as an expense.
Dennis Fisher: Yes.
Bob Maley: I was able to develop a marketing strategy
that I think we overcame that early on.
We did a lot of things that were more or less no budget that really
produced significant results that allowed me to go and show that we do want to
do this project and yes this project is going to have an initial cost to it but
here are the ongoing benefits. Here how
the citizens of Pennsylvania are going to get value out of what we do and by
doing that I think we’ve put ourselves in a situation that when we do go for
funding that I don’t have that significant challenge of well we don’t even want
to talk to you, you’re not even allowed at the table because you’re just an
expense.
Dennis Fisher: Right.
Bob Maley: So I don’t have that challenge. Now I do have the challenge that every other
business unit has that we are competing for the same amount of funds and
sometimes it is difficult but when we do our requests for funding, I don’t use
the fear, uncertainty, and doubt factors.
Dennis Fisher: Yes, yeah.
Bob Maley: We actually produce a very comprehensive business
case analysis that shows what’s the benefit and this may sound strange coming
from government but what’s the benefit to the bottom line and our bottom line
here is we’re not a profit obviously. We
collect taxes. we do all those things
that the government does that we all hate but we still have to do them and so
we have to make sure that our business units, that they can accomplish those
goals that have been set out for them by the governor, set out for them by the
legislature, whatever goals they happen to have and we just make sure we can
enable them to do those things securely without impeding their business
processes and it’s really resounded well throughout here and it’s put me in a
position of getting to implement programs is not as difficult as one might
thing.
Dennis Fisher: You made an
interesting point, this thing about actually developing a business case for the
programs that you want to implement in the coming year and I feel like maybe
that’s a mistake that some CSOs and even other security managers in
private organizations make the mistake
of maybe not doing that because they haven’t had to in the past but they just
kind of walk in there with a list of the current threats and how much more
malware there is this year than last year and saying it’s only going to get
worse next year so I need more money next year.
Bob Maley: Right and it doesn’t work. It doesn’t fly because in private sector
right now, all the vendors that we’re dealing with, everybody’s tight. Nobody’s flush with cash.
Dennis Fisher: Yeah.
Bob Maley: And there are definitely competing folks
within the organization. So if you’re
going to go in and try to sell these things by that fear, uncertainty, and
doubt, unless you’ve really had a serious event that hit home, most people are
going to go well it hasn’t happened and okay you told us that last year and the
year before and nothing’s happened so why should we waste money on a program
that we don’t perceive as we need?
Dennis Fisher: Yeah and you can keep saying well it could happen.
Bob Maley: Right.
Dennis Fisher: But that’s not going to get you too far.
Bob Maley: Right and I learned that lesson a long time
ago when I was a cop that home security, basic home security is locking your
front door. It’s like why should we lock
our door? Nobody’s ever bothered us here
and it’s because nothing’s ever happened and when something bad happens in the
neighborhood, well then it hits home and it wakes people up. The problem we have here is our neighborhood,
it’s the entire commonwealth and so we have been fairly good at preventing
serious problems. We’ve had them in the
past obviously like there’s been publically announced back in 2007 data
breaches. But we’ve put in some
significant programs in place that have reduced those to almost nonexistent and
that’s great but that doesn’t mean that something’s going to happen
tomorrow. I can’t say that. So I can’t say I need more money because it
may happen tomorrow.
Dennis Fisher: Right.
Bob Maley: So what we’ve done is we’ve actually shown
through one of our programs that we have we are very proactive in looking for
problems that we might have before someone else finds them and I imagine the
private sector may have similar challenges to what we have is that when IT grew
up here in the commonwealth it wasn’t centralized. Every person that had a need that knew how to
program in BASIC or when we moved on to Visual Basic or all the different types
of things, Microsoft Access, everybody started doing their own programs. Oh you know I do web development, here I can
stand up an application and this will really help us do that and I think a lot
of those types of things, the legacy things that are out there are serious
holes.
Dennis Fisher: Yeah.
Bob Maley: So we’ve done very proactive and we still do
this on a regular basis. We go out
randomly. We do pin testing. We try to hack
our own systems and in 2008 we found holes that would have exposed 400,000
records of confidential information and through our forensic follow-up and
investigation we found we were the first ones to find those, that we had to
beat the bad guys to our own problems and close the door and the cost avoidance
of data breach notifications was significant.
Using some old numbers at $90 a record, we showed that our programs are
returning a result of $38 million in cost avoidance to the commonwealth.
Dennis Fisher: Not to mention the bad publicity avoidance which is always nice.
Bob Maley: Yes. Yeah, we have to stay out of the press.
Dennis Fisher: Yes, that
should be a goal every year I would imagine and you found that through pin
testing you said.?
Bob Maley: Mhm.
Dennis Fisher: It must have
been a good feeling to once you went back and tracked through it to be able to
figure out hey okay we were the first ones to find it. We can’t find any evidence that anybody else
has gone through this vulnerability aside from us.
Bob Maley: Yeah, extremely.
Dennis Fisher: Okay.
Bob Maley: One of the good things though is since we
are the government we can keep our logs forever and the follow-up forensic
investigation may be time consuming but to be able to go back to the first date
that an application was deployed and investigate all the logs, all the access,
all the things we looked at to determine whether we were the first ones or not
is very good to be able to demonstrate.
We did find this. We were the
first ones in.
Dennis Fisher: Nice. So it sounds to me that you fairly early on
were able to win some friends high up in the organization there to get on the
side of your security programs.
Bob Maley: Yeah, that’s been since day one. I understood when I had come into the role
how the environment works and without that senior management buy in to what
we’re doing it really was just the temporary type process. If I wanted to develop a program that was
successful, that was high yield recognized, that I could go into those
environments and speak to the senior level people, that they took what we were
talking about seriously, and that was from day one that I had to do that and
have been very good at doing it.
Dennis Fisher: Yeah, I feel
like that’s another mistake that a lot of guys in other organizations make is
that it may be through no fault of their own because they’re busy doing their
jobs a lot of the time especially if it’s a small organization and they’re doing
operational things as well as the management and planning and all of that sort
of thing. They don’t necessarily have
the time to sit in on big meetings and kind of do a little politicking to win
friends for their department and get support for the programs that they might
need down the road.
Bob Maley: It’s the changing role. I’ve seen NASCIO I think did a paper on
it. I’ve seen it in many magazines, anything
that’s based around CISO or CSO, the changing role of the CISO and it is
significant because a lot of folks came to that position through their
technical competency and their technical expertise and while that’s good, being
able to connect to your business people and understanding the business of your
company or in our case the government.
For instance our state police, they have a very different business model
than the department of public welfare and they have different needs.
Dennis Fisher: Sure.
Bob Maley: When you understand that instead of trying
to come down as well like I’m in charge, I’m at the top, and you guys all do
what I say, when you go in and you understand what their needs are before you
do any of that and you incorporate their business needs into your programs, you
do that a couple times and it gets really easy.
Dennis Fisher: And it also
I would guess pays dividends even if they’re not immediate. It’s got to pay dividends down the road when
you’ve got a pet project or something that you think is important and maybe
it’s not getting the support it needs.
You can sort of go to these guys and say listen this is really important. It may not look like it now but trust me.
Bob Maley: Absolutely.
Dennis Fisher: Yeah.
Bob Maley: Credibility, it builds your credibility.
Dennis Fisher: Yeah. How much user education have you had to do in
your time there? Is that a big initiative
inside the commonwealth?
Bob Maley: Well it’s huge in my book. It’s difficult to do given the size of our
environment and the disparity of where employees work. We were able to get a policy that requires
very commonwealth employee and contractor to undergo annual security awareness
training. We do other types of
events. We’re pretty involved with the Multi-State
ISAC so every October we do our annual cyber security awareness month. In the past we attend conferences with tables
so we can get our message out not just to commonwealth employs but local
government as well throughout the state and it is key because the change in the
threat landscape that we’ve been seeing, not just recently (I know a lot of
magazines are talking about lot now) but we’ve seen it for about a year and a
half, almost two years now is that the vendors out there with technical
controls are very, very good. There are
a lot of very good things that we can put in place and they work. But the bad guys, well, when they’re stopped
one way they’re always, well let’s find out what’s our next avenue, how do we
make a compromise happen and if we’ve really got ourselves protected, we have
that hard crunchy exterior, we have all the multi-layers of defense so if they
get through one hole we catch them. We
have all those things and it works really great. But the one layer that is most vulnerable and
is very difficult to secure, it’s the human layer.
Dennis Fisher: Sure.
Bob Maley: And we have some here that our
vulnerability, attacks against that layer have increased significantly and so
we’re making as much effort as we can for the security education, the security
awareness. We do updates now every time
I get a chance to go and do a presentation someplace for any group of
stakeholders. I’ll do that to try to
educate the folks about staying safe on the internet and the basics and we
can’t say this enough, not to click on a link that you get in an email. I know for security pros it sounds so, well,
duh, but most folks out there aren’t security pros and they don’t understand
and it’s that prurient interest that’s, “Oh, I bet this will be cool,” and then
usually they always have that oh no factor after they’ve clicked and it’s too
late and the damage is done.
Dennis Fisher: Right, yes.
Bob Maley: So that security awareness, the education of
our end users is always going to be a high priority and were going to always
look at whatever we need to do to make it better, to reach more people and
reach them more often.
Dennis Fisher: Did you encounter
any resistance from the end-users when you started the user education program?
Bob Maley: Yes.
Dennis Fisher: And how did
you get by that?
Bob Maley: There was an executive order requiring employees
as part of their employee training.
Dennis Fisher: That helps,
yes.
Bob Maley: That helps, yes, it’s a big stick.
Dennis Fisher: What about
high up? Did the managers or the people
running these disparate agencies say listen we’re busy collecting tolls or
doing whatever it is. We don’t really
have a lot of time for our guys to go through this?
Bob Maley: I’m sure there were those types of comments
made but we’ve been pretty successful with it.
I think people are starting to understand that the assets that we do
have, although we don’t have stock value, we don’t have those types of things,
we have significant assets and confidential information about citizens and I
think senior management recognizes that and takes it seriously and of course
obviously it’s not100%. There’s always
somebody who’s not going to be on board but we have really done well. We’ve had people buy into the understanding
that we are the keepers of a very significant asset and we need to treat it as
such. So it’s gone over pretty easy.
Dennis Fisher: Okay,
that’s
good. You mentioned earlier on in the
podcast that you guys did have that data breach a couple years ago and
you’ve
put in some things to ensure that doesn’t happen again. What kinds of
things did you do? Did you go down the road of encryption and
those sorts of things?
Bob Maley: Yes, every mobile system in the commonwealth
is required to be a ____ disk encryption.
We’ve enterprise-wide intrusion prevention and intrusion detection on
every host. We have a SIM in place that
we’re collecting about a billion events
a month that get correlated, allowing us to look at trends, look at the
areas that may be problematic. Already
we’ve used that to identify some SQL injection attacks that were underway that
reduced a potential breach of 40,000 records down to 2, so those types of
things. All the different programs that
are in place and, again, we’re continuing.
We’re moving forward as well.
We’re currently implementing a third party email encryption so that all
commonwealth employees that need to exchange information outside of the
commonwealth they can be secure and be encrypted very easily. It allows us to put in the compliance tools
as well that doesn’t count on an employee thinking about it. It understands and it sees this should be
encrypted, so that type of a tool. We
don’t have everything in place but those things are coming out now as well. So
we’ve just tried to stay current with where our threats are and it’s been
successful. We’ve reduced those information problems and we’ve been proactive
in discovering those that may hit us down the road as well.
Dennis Fisher: Yeah, yeah
well it’s good to see that people are certainly utilizing the tools that are
out there, especially on the mobile devices which, you know, I feel have gotten
a little overlooked a little bit. With
the amount of data that they can carry now and the fact that everybody is
sending corporate email with attachments and that sort of thing from them and
have been for a couple of years now, that’s a major sort of weak spot in a lot
of networks I feel like.
Bob Maley: Yeah because the rest, it’s pretty
secure. Those areas like you were
talking about, previously nobody even thought about them but now they are the
avenues that the bad guys are going to try to figure a way to exploit.
Dennis Fisher: Yeah,
alright well listen, Bob, I appreciate your time. I’m going to let you get back and help
protect the citizens and I really appreciate it and it was good talking to you
again. Hopefully we can do it again
sometime.
Bob Maley: It was good talking to you as well.
Dennis Fisher: Alright,
take care Bob.
Bob Maley: You bet.
Have a great day.
Dennis Fisher: Thanks.
