Call in to speak with the host
The electric grid is one of man's oldest and largest machines, and while it's beginning to show it's age in some ways, it's also being modernized with sensors and advanced communications technologies at a rapid pace. In the wake of Stuxnet and more recently the powerful Shamoon attacks on energy companies in the Middle East, governments around the world seem to be waking up in unison to the scale of cyber threats facing their electric sectors, and the other critical national infrastructure systems that depend on it. This conversation will discuss the current state of security at a typical electric utility, some of the of new and emerging rules in place to govern security processes at utilities and beyond, and as Smart Meters have come to symbolize the Smart Grid moderation effort for many, some thoughts on consumer privacy and how it's being protected.
In this podcast Caleb will be joined by Andy Bochman. As the Energy Security Leader at IBM, Andy works all aspects of cyber security, privacy and compliance facing US and international utilities. He's the founder and editor of the influential Smart Grid Security and DOD Energy blogs, and a frequent speaker, writer and adviser on topics at the intersection of grid modernization, privacy and cyber security.
Broadcasting from the IBM Massachusetts Laboratory, this is Information Security Radio where we're talking about IT Security by design. I'm your host, Caleb Barlow. Welcome to the show.
Everyday 2.5 quintillion bites of data created shared by more than 5 billion into connected devices and the techno code are becoming more sophisticated. In an ever connected world with escalated security needs, IBM has the expertise to secure a smarter planet. Protecting more than 6 million endpoints across 4000 global client, we managed more than 13 billion security event everyday. We offer one of the world's broadest most advanced and most kindly integrated enterprise security portfolio that it's supported by our own security research organizations from big data to mobile into cloud environment. We provide advanced security intelligence for your organizations. This is IBM Security. Are you protected?
This podcast is a proud member of the tech podcast network. If it is tech, it is here. Listen to other great tech podcast at www.techpodcasts.com.
Hi! This is Caleb Barlow and today we're gonna be talking about Smart Grid Security. A reminder as well that tomorrow, we will be launching the X-Force threat research report and at last, we will be doing a live podcast tomorrow at 11 a.m. Eastern to walk through with the security research and the detail of what they found over the last year. So today, I am joined with Andy Bochman who is the IBM Energy Security Lead and we're gonna talk about Smart Grid Security. Welcome Andy.
Oh! Hey Caleb. It's pleasure to be here.
So before we dive into kind of all the cyber stuff, let's clarify what is the grid -- What makes the grid? Who runs it? What are we talking about here?
Okay sure. Well, I'll tell you what -- I'll try to lay it out somewhat particularly for generalists. It's the people that use electricity which is pretty much everybody but that happened really had reason to figure out how it's all put together. Usually, we say its a couple main pieces. There's generation which are the large power plants whether they're run on coal or nuclear or increasingly natural gas. There's the transmission part of the grid which is your high voltage line so that it take that power that has been generated and help move it across tenths or hundredths of miles. Then when you are getting closer to the point of views, you switched over at substations and transform and trade it with transformers, take the voltage down a few notches and that's called distribution and distribution runs all the way from the transmission high voltage part to the point of views whether that's commercial and industrial called CNI or whether it's residential. So those are your main pieces and to pinpoint it out that the grid is probably the largest machine, the North American grid largest machine that man has device. It is also very old, parts of it had been around for many, many decades and it's not in common to find an equipment still in the field that is working that 70 years old. Do you think about that and in terms who runs it? I will just say, a combination of federal authorities coming out of the department of energy in the United States FERC and NERC, you can look up those acronyms if you want to. They have responsibility for the first two parts that I mentioned, generation and transmission. Distribution is managed by the States utility commissioners and they are responsible for making sure that things are secured. All those groups have responsibility for helping ensure the security of the grid.
So you know when I think about the grid, I have got to think this is in a state of change. I mean the -- all of these older systems that you've talking about you know some things that may be 60 to 70 years old, those were obviously getting replaced. I'm assuming they are getting replaced to the things that are more automated. Also people are generating their own electricity nowadays. You know my parents have a house that basically generates all its own juice and sends it back onto the grid. Is this complexity? Is this starting to change the security posture of the grid as well?
Yeah. That's fantastic. It's good to hear about your parents in that situation and they're becoming less of minority and more and more people are doing that even here in Massachusetts where we doing it all.
No -- that my dad runs around and turned off light for all the time and try to stay it at zero but that's a whole other discussion for another day.
Well he is clearly got with the program for sure. I say two things, one is that sure obsolescence itself and plausibly a run to sale is what's kept the light of all the equipment in the field. If it doesn't fail yet then clearly, it is not broken and so that stuff keeps going up until it's time to replace it. When it is time to replace it guess what, 70-year-old pipes don't exist anymore. They all have a modern analogue so the utility doesn't really have much of a choice to that point except for it's time to replace it. Besides obsolescence, there's also Mother Nature at work and one of the great grid modernization programs of latest than hurricane Sandy. You have some old stuffs that you might have had and either replaced, but you never had business case for or suddenly when it completely disappears out of the place to the area, now you have your business case where you can modernize. The second driver for modernizing though is the fact that in order to fulfill the needs of the whole changing industry, all the different other sectors that are supported by the electrical grid utilities themselves has had to change and that's the reasons like you said with your parents distributed energy generation especially in places like Southern California is really adding a whole new type of power to the grid one that is much more how come we have say this unpredictable, it's one way to say it. The traditional operators are used to very predictable forms of electricity and I have a lot of variation and now there's huge spikes and not just in demand but in our ability to produce the energy. The old grid is not capable of handling that type of uncertainty so we're making it newer with new technologies. I can explain more about those.
Okay so we've got you know newer technologies rolling in because of obsolescence. We've got you know kind of the storm business case where things get up data and even in more rapid timeframe. We've got you know people generating their own juice. All of this means these things are ultimately running on various types of systems. What's the significance here from a proper security point of view?
Well, in the old days and I would refer to the old days and the new days properly a couple of times during this talk. In the old days, the grid and the equipment that made, managed and moved the electricity was very much a mechanical process in some ways and now they gets to telecom right? But telecom has long since modernized and now the electrical grid is catching up. To make the Smart Grid which includes but certainly is not limited to just the more public phase of it in the form or smart meters. Utilities are introducing modern communication systems that now give them two-way flow of communications with the assets held in the field all the way down to the smart meter and not just two-way flow data, but two-flow of control signals. So they can- the grid itself is becoming more intelligent and can react more rapidly, more flexibility and get some resilience. However, one of the things you have get when you start adding two-way flow communications and control signals and increasingly interconnecting systems that used to be protected in large part by isolation is that you are introducing a whole many new pathways in for bad guys to find a way in and to be able to reach equipment that previously was they could have never touch without physical access.
So talk to me a little bit about your typical North American utility you know and I don't even know what's the boundaries of the utility is anymore right? I lived literally almost line up site from a nuclear power plant and the sign of the door seems to change about once or twice a year depending on who's bothered and how they moved and you know and then the juice comes from somebody else. I mean how are these things structure today and what they're strengths and weaknesses?
Okay sure. It's depending on how you're counting and who's counting. There are about three thousand or four thousand electric utilities in the United States and they arranged in size from the largest once they're called investor own utilities or IOUs may run and have old structures that are very similar to any of the other companies. You are familiar with in the fortune 500 per say. All the way down to small rural coops or cooperatives where the IT department educated who is running a snow cone stand next to two winter by in the field and his other job is to keep things running on the IT side of that coop and everything between those two extreme examples. In terms of strengths and the cyber security. One thing I had have to say is as much as utilities get critique from the press from folks on capital hill, the very fact that on -- security I'm saying, the very fact that there has been no major cyber security incidents yet where say the city was black out for a substantial amount of time because of the a cyber attack where massive amounts of data logs have really jeopardized trust in the whole the system of people just approval what is adequate cyber security means that the lack of those massive incident so far to me anyway says that utility has been doing enough right up until this point in time.
It doesn't mean if nobody came tomorrow but your point is enough up to this point.
Yeah, yeah just like the definition of what is adequate security right? It's not -- you are not gonna make a fortune with cyber security. You're trying to mitigate potential damages in the future. The things have been adequate but our case to those utilities is by starting of on that somewhat friendly gestures and saying whatever you have been doing up until now clearly has been by any definition has been adequate. However, a couple of things in the world of change one is you guys has been changing the world through the Smart Grid projects and modernization products we've been talking about. That a new technology to unable great new function that everybody is happy for and that's still working progress but a large part you have been doing that by interconnecting systems that were previously protected by being suffered and isolated. The second thing in the world that changed is that the attackers have substantially grown up since say the last 5 or 10 years when we first considered this issue and now they are more numerous, more motivated, financially motivated and armed with some really serious tools and we've say I would say anyway that that means -- it may be time for a refresh in the way you look at, how you organized to delivery cyber security capabilities and manage and set policy. On behalf of your own company and your investors and also on behalf of your customers who essentially are not just mom-and-pop in their house but all the bit companies and government organizations that made our country well.
Well, I mean the challengers this is a moving target right? So, let's roll down on kind of SCADA industrial control systems that are using these companies the operational technology side of their business what's the cyber security risk to them?
Okay. In the past, the cyber security risk to the -- and I'd like to bracket all of these types of machines and the cyber systems that support them as operational technology to differentiate from information technology or IT. In the past OT systems, which basically or what make a utility an electric utility right? These things were protected by the fact that they work on network because they work very much network. If they were network the used very obscure protocols that were unfamiliar to your hackers coming in from the CCP IT world and so they essentially were a very hard to find target and if you could even find them they were very hard to reach. Because of modernization like for it's like the Smart Grid those systems are being connected on standardized IT more -- were there's a little bit more and more like IT and in fact in some cases they are being connected to the IT system without proper segmentation and some cases like the new search engine shoban were reveal they were connected directly to the internet and that's not at all a good thing. Some times that's on purpose for convenience for main and say some times it's by accident and people operating don't even know that there is a way into the system from the internet but you know who does know, anybody that is aware that showed in there and noticed to put in a few right search words and see which one of the things are attached and one last detail because they were designed for an our security was not a consideration. The types of protections that we take for granted on IT system simply aren't there. You don't find inscription, you don't find logging often and in terms of access control and passwords very often they're still set to the default password and those default passwords by the way are published on the internet in case you ever want to know what they were so clearly we are handed not to saw but clearly where need for a refresh in the part of the world.
So part of this is the operational side using best practices and understanding what those evolving that practices are but some of this is also a supply change issue right? I mean you've got to have -- during the utility you've got to be on the parts that use encryption that can update perhaps or is it can be on the internet without being attacking can be updated. So you care a comment on it?
Yeah sure. This is I think one of the best cases of the chicken and the egg syndrome. Well, that's been covering the space, the electric factor and cyber security now for somewhere between seven or eight years. I asked utilities why is it that you continue -- let's say, let's accept the fact that the stuff is already filled and has been there for years or decades that's in place and we're trying to look at remediation to help protected now, but let's agree to do more no harm in the future right? So if you are doing a source collection or RP for new OT side equipment, why not put in new updated modern security requirements and what the utilities often will say and there are exceptions with this by the way, but what many utilities will say is if we do that there won't be any products because none of the products have these types of capabilities so that I will usually say them okay hold that back now stay there and I will walk over to just figured it by the way. I looked over to an OT supplier and say, the utilities want to have more securing OT equipment but they say that it does not exist yet. Otherwise, they would stratify it so what do you have to say about that? And they say, look we would love to build those capabilities into the products and sure it has passed time. However, there is just not enough demand for it. No enough utilities are asking for it. So until there -- if and until that becomes a proper business case to make those changes, the product managers hands are basically tied. And then I have made that walk back and forth now several times between the two of them and guess what, slowly and surely this is the value of persistence. Utilities are starting to put more and more security requirements into those facts and slowly and surely some of the suppliers are starting to add those capabilities. These can take a little while and some people who wanted to have them faster but it is happening.
I have also heard a lot about NERC CIPs. You have mentioned NERC earlier in your response to kind of __16:59__ the grid. What is this?
The CIP stand for Critical Infrastructure Protection standards. They came out of the energy protection -- Federal Energy Act excuse me or Federal Power Act of 2005 with follow on legislation in 2007 and 2008. It's when we decided at a federal level that the current state of cyber security practice at the electric utilities is governed by DOE, FERC and NERC was inadequate and we are going start giving them some mandatory prescriptive security rules for them to follow. Over the years, they have evolved across different versions and the new functionality gets decided in a process that's made up of folks from NERC as well as representatives from the electric utilities and from industry. We decided what additional capabilities can be added. I think from the state of the art of the security practice in 2013 if you look at financial services, companies or Telco for example. The NERC CIPs requirements are fairly rudimentary and they often get critiqued as being inadequate to completely secure the grid. First of all, they only cover the federal part which as I said earlier is the larger generation assets and the larger and more important transmission assets. So they leave a big swap at the grid, the distribution part which is arguably the biggest part wholly -- I won't say unprotected but wholly ungoverned. They don't cover that. They don't have the right jurisdiction.
No. Their policy or guidance documents that are particularly helpful or important when people are thinking about this?
Yeah. Well, if you are talking about the NERC CIPs. They are the actual language of those regulations. We are on version three now and migrating rapidly to version four. The federal government is now telling -- or excuse me -- NERC is now telling a utility which assets more exclusively come under the jurisdiction of the program and another version is coming. So this kind of drives the utilities a little bit crazy because as the standards evolve, they are constantly having to change their program and for the larger utilities now NERC CIPs compliance generally takes a fulltime staffing of dozens and dozens of employees causing them many millions of dollars a year. As I said its mandatory and what that means is there are substantial finds involved for -- when you have an audit and you are showing to be out of compliance, the law says up to a million dollars a day. So that's what put the utilities into motion.
Yeah, that will get your attention. Now, what about you know, we have talked about North America a lot, Andy. What about the rest of the world? You know what's going in Europe, Asia, the emerging world, what are they doing about it?
Okay. Well I tell you what...
Are they buying all the new stuff or they are taking you know old recycled stuff and applying it or they are dealing with security here?
As part of the previous question you know, what regulations are happening I stuck to NERC. But I had also you know to segue to the next question, a couple of documents are really helpful for security with large outside of the NERC CIP regime. One is called the NISTIR, N-I-S-T-I-R, that's NIST Interagency Report 7628 guidelines for securing Smart Grid. The whole world generally sees that in the 600 or so pages that comprise it as a very helpful document. Its high level enough that it doesn't prescribes you know individual technologies that you have to use, but it is a great road map for how to do this across all the different parts from generation all the way through to consumption. There are other documents that had been coming out last year primarily about how to gauge your maturity model of cyber security in the utility, like DOEs, ES the Cyber Security Capability Maturity Model. Very helpful document. It's being increasingly embraced by utilities. The rest of the world has the advantage of being not necessarily a fast follower on security, but sort of a mid level follower on security. They have poured over the NERC CIPs. They have heard the good parts and the bad parts of them. Similar with the NISTIR 7628, the DOE documents and others and they are getting to pull from those the best parts that seem to fit they regions specific needs and leave the more troublesome parts behind. And we are seeing legislation and industry groups in motion all across Europe right now, in Australia, in certain parts of Asia and certainly Canada is -- which leads more with privacy than security has been on top of this stuff for a while. So yeah, the whole world is in motion. Your question about new technology, I would say that one of the best examples for that is Africa where IBM is increasingly active.
And we are -- there just was no old technology replaced. There was nothing, it was a blank slay. So we were able to deploy the most modern communication technologies and some interesting Telco stuff that's happening there that makes it very affordable and on those backbones, we will ride the Smart Grids of the future in places like Africa and other __22:15__ parts of the world.
Well, sometimes it's just either when you have got nothing to reflect.
Sure. And that's probably true in China too for the most part. You are getting a lot of brand new equipment, modernized equipment going in for the first time not having to wait until the old piece dies or is blown away in a storm.
So how are these utilities organized from a security government standpoint. I mean would we -- will we find a __22:35__ in the utility? Would it be a C-level position? Should it be a C-level position?
Okay. Okay. First of all, let me process this remark with the fact that some utilities are really on the ball and you know who you are. Your exemplar is that myself and other people point to and say these guys understand the nature of the problem and they are out in front. Okay, so please don't take this when I now switch to my more general comment which is... Hold on. A lot of utilities...
So five from those of you have to act together to know who you are. Let's talk about everybody else.
Please stand down. A lot of utilities are organized for cyber security as if it's the 1990s and we are transitioning and we are still in AOL dial-up land and we are having our first virus protections on our PCs in the IT part of the house. So you will find the most senior security parts and __23:30__ securing their title lives in the IT side and they are usually one, two or three levels below the CIO. This does not give them much of visibility across the whole enterprise. It surely doesn't give them much authority to develop or to __23:46__ or to enforce security policy that really matters now based on all the things that we talked about. We would recommend that based on the current thread landscape and the new attack landscape that attack surface that Smart Grid modernizations have produced. Their utilities reconsider the way that they are organized and consider appointing a head of security at the executive level getting that position out of the boiler room deep in the IT doing patch management and anti virus and getting them into a more strategic position in the company that has __24:28__ over both IT and OT and has their respect of both all the people, the middle upper managers and those departments. And also because of that placement is someone who frequently wines and dines with rest of the __24:42__ and the board of directors. So those folks kept much more abreast of their current security posture and what's going on.
I mean the point being security needs to be occasionally on the CEO docket to get an update of what are the matrix? How is it going? You know what are the things that we need to be working on, right?
Yep. That's right. And in some utilities it certainly is. But in the vast majority I would have to say in early 2013, that... Here is an anecdote. From last year, DHS brought together the CEOs of the top 60 or 70 US utilities and briefed them on the threads who they are facing them. And it was a real serious session. A lot of them got quite concerned more agitated about security then they had every been in the past according to some reports. Well, one CEO at that session leaned over and said to a friend of mine , you know what I am really worried about this stuff but I cant even tell you the name of my security guy. I have no idea who that is. And to me that speaks volumes about the problems with the current organizational structure of utilities that such as strategic concern now concerned to the utility themselves and their customers and concern at the national critical infrastructure level. That situation shouldn't be allowed to stand for very much longer.
So let me switch topic on your bit here, Andy. Smart meters and privacy. You know there is some certain groups that inform fighting for the right to __26:11__ at these new leaders for a privacy reason. What's the issue?
What people will say about smart meters and if you Google it, I am not encouraging you to. But would you Google smart meters and privacy? You would find a lot of provocative information. People making well produced videos suggesting that the United States will become a police state because every single motion you take in your house will now be recorded by the federal government and obviously I told the federal government what I want to do is now when you are using your electric toothbrush, your toaster, your whirlpool or your __26:49__ and things like that and so people are concerned that they are loosing their privacy. They I usually discuss that to try to calm people down is, if you are living in the United States or the modern world in 2013, you are using PCs and smartphones, etc. you have already whether you meant to or not given up so much of your privacy its already so easy for third parties I want to say to be able to tell almost everything about who you are, what you do and what you care about. That smart meter on your house which shows usually on an hourly basis once its up and running how much electricity your using. Its an incremental change to what people can tell about you but its not a night and day change. Its not a watershed in the epic of privacy.
Well you know, I love your analytics to the mobile phones right, because you know I have spent a lot of time to put __27:48__ and one of my favorite things is like you know the flashlight out. Everybody use this in the restaurant.
So, that the finding when you install the vast majority of flashlight application on your phone and after this install it comes up and as for permission then it says flashlight need to access your contacts and GPS location. What other for, right? But of course what do we often to do is do yup, yup, yes, yes and you are up in running it. Who knows what that information being use for?
But one of the things I like to do in a, done this a couple of time, you walk into presentation and you say hey I am gonna act like a __28:26__ can you give me all your contacts, please. Effectively what you are going is __28:33__ so that's a great point.
Hey, so any closing comments on where you think these factors had within the future.
I think that we are going to see progress made in the electric utilities so again I will argue that we will se progress move more rapid phase if somebody organizational structure and governing issues are resolved soon that will facilitate of a lot more positive change. But whether that happens or not in the given utility, I think we are going to see new technologies coming on that help them better secure themselves. One point that gets lose in conversations like this sometimes is when I say that the increase thing interconnection systems that were protect by isolation is great and a bigger attack service and therefore it's a bigger security test another point about the Smart Grid whether its mark meters or things that about to reach a level for security intelligence and wide areas situational awareness is that this technology is give as much better potential to understand what's going on from a security point of view using technologist provided by companies like IBM and others we can jump on immerging security issues but simply because we can feel and are aware of them in a ways that we weren't before and before it's kind of like everything there is some costs some security costs and risk edition to the smart __29:59__ but we're also gaining a lot and the more we learn to use that new technologies to be more introspective and have a better understanding of what going on. The faster and the better will see these guys building up in a security posture.
Alright, well that's Andy Bochman the IBM energy security lead and should also mention that if you'd like to learn a little bit more about what Andy's up to. He is a prolific blogger both on the Smart Grid security blog which you can find easily just __30:31__ as well as the DOD energy blog. Also I like to remind you that we've got an upcoming show tomorrow the live release of the explorers rep report which whether you are in the energy sector or just about any other industry is very much and much read and will be talking live with the folks that put that report together. Lastly, if you enjoyed listening to Andy and you are iTunes users, with rate the show add us a comment tells us how great Andy is. If there is something else you'd like to see that's always helpful to kinda get that feedback from all of you. So thanks again for listening.
This content was provided for informational purposes only. The opinions and the insights discussed others that the preventer and guest and do not necessarily represent those of the IBM corporation. Nothing content in this material or the products discussed is intended to nor also have the effect of creating any warranties or representations from IBM or its suppliers for altering the terms and conditions of any agreements you have with IBM. The information present is not intended to imply that any actions taken by you will result in any specific results or benefits and it should not be relied on in making a purchasing decision. IBM does not warrant that any systems, products or services are immune from or will make you enterprise immune from malicious or illegal contact of any party or product, plans, directions, and intent are subject to change or withdrawal without notice. References to IBM products, programs or servicers do not imply that they will be available in all countries in which IBM operates. IBM, the IBM logo and other IBM products and services are trademarks of the international business machine corporations in the United States other countries abroad. Other company, products or services main maybe trademarks or service marks of others.
It's good to talk.