Site icon Easy Prey Podcast

Top 5 Supply Chain Risks with Kevin Kumpf

Easy Prey
“When you look at the entire spectrum of organizations that have been attacked, it’s about their downstream suppliers.” - Kevin Kumpf Click To Tweet

As businesses embrace digital transformation and rely on interconnected networks for their operations, the vulnerabilities within the digital supply chain become increasingly apparent. From data breaches to third party vulnerabilities, the threats are multi-faceted and ever-evolving.

Today’s guest is Kevin Kumpf. Kevin has more than 20 years of IT Security and Compliance experience including over 10 years of cybersecurity, governance, and critical infrastructure experience working in the energy, medical manufacturing, transportation, and fedramp realms. 

“You have to know what data is being exchanged.” - Kevin Kumpf Click To Tweet

Show Notes:

“You trust a process and then when something breaks, how do you find out about it?” - Kevin Kumpf Click To Tweet

Thanks for joining us on Easy Prey. Be sure to subscribe to our podcast on iTunes and leave a nice review. 

Links and Resources:

Transcript:

Kevin, thank you so much for coming on the Easy Prey Podcast today.

Absolutely. Pleasure to be here, Chris.

Can you give myself and the audience a little bit of background about who you are and what you do?

My name is Kevin Kumpf. I'm the Chief OT Strategist here at Cyolo. What I do is help the company understand the people, process, technology of our customers, and work to build our technology and our platform to address their needs, not the other way around where we build technology and say, “Here's what you want to solve for and here's your needs.” My goal is to keep those customers front and focused and mindful in our process.

How did you get into the field?

It was on a, “Hey, we have a job here in cybersecurity.” I knew nothing about it. I wanted to change after owning my own organization and having staff globally for several years. I said, “Sure, this is interesting.” I thought it was going to be more of the death-by-Excel, audit compliance checking thing. It turned out to be running IDS, IPS, and firewalls for one of the global banks out there, building that infrastructure and started from there.

Is that something like when you heard the job opportunity, you were like, “Oh, this sounds really exciting; I totally want to do this”? Or was it, “Well, I guess I better do this if I want to stay employed”?

It was the latter. They had to call me three or four times and said, “Look, this is really fun,” and I kept thinking to myself, fun back in the day was taking apart your own computers and building your own serial cables and stuff. It wasn't, “We're going to give you a pager and put you on call 24/7.” I don't know many that have really gotten the cyber security from my area that said, “I wanted to do this for the fun.” It was, “You're here because you have to” too many times now.

For you and I, when we were coming up, there were no cyber security degrees. Security was barely even a thought, let alone cyber security.

Even going back further, even going through college, you had to be a mathematician. Everything was all math because it was an applied science. I look now and say that most of the people that I deal with couldn't even figure out where the calculator is on their phone to bring up to add math to do gratuity on a dinner bill. It totally changed the realm.

I think one thing that we have an advantage over many coming out today is we learned by the hard knocks. We didn't learn by taking the certification and saying, “I'm here; I have knowledge.”

The practical experience of, “I tried to do this and it didn't work,” or, “I did do this and it did work.”

Yeah. A part of where I think my role comes into the organization is being able to look at people and say, “I've been in your shoes. I've been on your side of the fence. I've been on the seesaw of a major multinational utility company on the operational technology side. I've dealt with major manufacturing companies building their infrastructure program where day one they said, ‘Hey, by the way, we have malware over in this robotic cell zone, but don't worry, we're not going to let it out’ as their solution.”

We're just going to keep the bad stuff over in the room over there; we're not going to talk about Bruno.

Exactly.

One of the questions I try to ask my cyber security guests as we begin the episodes of—have you ever been either personally or professionally a target, a victim, of a cybersecurity incident, and/or a scam?

First off, let's be fully transparent. I've had all my credit card information stolen just like everybody else out there. I think it's a cottage industry and a badge of honor if you haven't at this point, but everybody has. But, yes.

When we actually bought our house back a bunch of years ago, going back in the old forge, actually somebody forged my signature, went to the bank, and actually put our home up for sale, and we had a realtor showing up at the door because they had signed the paper trying to sell our house.

That was just because I had actually written an article about real estate and the cybersecurity around the process. Somebody decided to say, “Hey, let's see if your mortgage company will fall for it,” and they did. So victim by choice, I guess.

That is always the challenge of being in this space and being vocal about something. If you stick your head up, sometimes you get bonked by the mallet.

Very much so.

Today, we're talking a bit about supply chain hacks, supply chain issues. Not supply chain in terms of boxes on boats necessarily, getting from one side of the world to the other, but on our technical infrastructures. Let's explain what is supply chain.

You brought up the point about it not being the boxes on the boats and things like that, but that is directly correlated to this because when they attack a third party, a vendor, a supplier, generally it's for something that they're providing you as a service or a component that really aligns to it.

When I look at supply chain, I don't want to say it's just third-party risk for many people. They view it as that. But when you look at the entire spectrum of the organizations that have been attacked out there—there are thousands upon thousands these days, as you know—it's really about their downstream suppliers, and there are attacks on them.

I have a good example that I'll get to at a later point in the discussion. Part of the question is, where do we not realize that the supply chain actually exists?

I think it comes down to ownership. That's probably one of the biggest areas. If you look at third-party risk, control over your third parties, or your suppliers overall, it isn't something you want to get into many times because you're assuming their risk. You're assuming if you tell them what to do and something happens, they can point back to you just like you said, “He told me to do this.”

The downstream challenge, though, with this is that you tie your system so tightly together today, you’re not going to the cloud where all your data is shared between all of your suppliers. When that data is compromised for one, it has a ripple effect to the other one. Now you have access to the information to credentials many times, or do enough information to cause a disruption or ransomware in the environment.

As we discussed, there are multiple categories of this. If we're talking data, there are the data risks, there are system risks, there are operational risks, things physically stop working, or people get control of stuff. How do you even start to figure out how to deal with it?

I tell people one word: baseline. If you look at what a baseline is in the industry, I don't care if it's your own people, process technology, or somebody else's, you have to know what your eyes and nose are, where your interaction points are, what systems talk to what systems, and what data is being exchanged.

Sadly, if we look at the cloud, many things are now in the cloud, or third parties aren't even third parties. They're the person that says, “I do this,” but they're on behalf of somebody else that white labels service for them that does it on their behalf. Knowing your baseline of who is responsible, whose, what they call, “throat to choke,” is key, but you have to know that. If not, you'll never assess or posture properly.

How do you do that assessment in terms of how do I reduce? I know you wouldn't want to talk about risk per se in who's responsible for the risk, but as an entity, I want to reduce my risk. What are the concepts that I need to be looking at in how my systems connect, who I deal with, and how I deal with them?

You do want to take and understand where the components, data, or other information fits into your business process. If you look at your primary business, let's say I'm out there doing manufacturing, doing the production, or transportation of even data information, whatever it is, the key point is that knowing how critical that information is not only to your business at that moment in time, but if it went away tomorrow.

I tell people this very clearly. Nobody worries about what they put on their cell phone today; they throw everything on it until your cell phone won't put up in the morning and then panic sets in because you don't know what's on your phone. You think that, “Well, I'll just recover it through the network.” They've put it off to say, “I'm trusting somebody else to be there for me.”

You have to treat yourself as the network. You have to turn around and say, “Look, if my phone falls in the lake tomorrow, what would I lose, how do I recover it, and how do I make it so that I can function tomorrow just to walk in the store and pick up that device and say, ‘This is my new phone’”?

In theory, that is the supply chain, because the phone to you is an area where all this data intermingles and comes together for you. But you're not in control because you've given them control.

I like the example of for people who are not in this space, your cell phone is a perfect example of a supply chain. It's stuff that is important to us, but we don't have control over any of it.

Exactly. You say, how does that differentiate in gradients? If you were traveling today and you had an airline ticket, you expect your ticket to be on your phone. You expect the Wi-Fi to work when you're at the airport. Now people have turned around and said, “You know what? I know the Wi-Fi is not going to work. I have to screenshot this to make sure it's on my phone.”

If you look at your own email account, it's like, I can go to my computer and grab this, and I can do that. It's different levels. Other things that are out there, I can guarantee that my wife has accounts that she can only get into with a PIN that she knows into the bank to do a credit card, doesn't get a statement, and expects the bill to be paid because they notify her. If that notification doesn't come in for people today, that is truly supply chain. It's an ecosystem of the whole process working together and what happens when it breaks or a component is compromised.

The challenge in this space of not knowing, either when something has been compromised or when something has even totally failed until you're trying to back-trace it. This other thing that's very public has now failed, and you're trying to figure out why.

Exactly. This is that intersection point of—let’s take the logical level of firewalls. You put a firewall between your network and your partner's network. Everybody knows the firewalls are just mirrors of each other, because the ports open on one side that's on the other. Where's your real security coming out of that? You've got them there as a checkbox.

That data going back and forth, there are many times that people don't even know what's being transferred when, where, and how. I'll give you a real-world example of when I was back in the banking industry. We had an employee that was going to retire. He was a little disgruntled with the bank. This was back when they had mainframes. They still do, but the mainframe is 100 jobs scheduled to run every single week like clockwork. The code hadn't been touched in 15–20 years.

He went in and modified the code so that the day that he was leaving, it went out, issued basically blank checks to senior leadership, and gave all the lower employees different pay amounts. The checks came out, and nobody checked the system because every week it ran, it stuffed the envelopes, and out the door they went. Nobody knew how to put the script back because he was the one who wrote the code for it, didn't know where it was buried, and the JC in other languages that were controlling the process.

Oh my goodness. That must've been a nightmare to figure out.

It was. The funny part about it was at that point, he was retired. He had actually already moved down to the South Pacific and set this to go basically two weeks after he left. This is where you trust a process. When something breaks, how do you find out about it?

Yeah, I have my own weird story. It seems a little similar to that. I run whatismyipaddress.com, and I get a support ticket from someone saying, “Why is your website using my browser to mine cryptocurrency?” At the time, there were popular plugins that you could put on your website to use your visitors' computers to earn a little bit of cryptocurrency in theory, in exchange, instead of doing advertising on the site. I was like, “I'm definitely not doing that. He must be mistaken. I'm sure his computer is compromised and it's not me.”

I looked at the site. I looked at the HTML code on my back end. There's no cryptocurrency miner here; I don't have anything to do with this. I thought, “Well, let me just run a trace.” I visited the website, run a trace. All of a sudden, yup, there's a cryptocurrency miner running. I'm like, nothing on my side has been compromised.

All my files are right, there's no cryptocurrency in here. I started to do the fun web developer process of stuff. “Let me just start disabling third-party products one at a time and see if this thing goes away.” Lo and behold, there was a third-party vendor that I had a script on. I don't know what it was at this point. I don't remember now what the specific script was, but lo and behold, I disabled their script and the cryptocurrency miner went away.

I know that they're not in the cryptocurrency space, so I disabled the script, emailed them and said, “Hey, something's going on.” They say, “No, no, no. All of our stuff is fine; we've checked.” I'm like, “Well, let me put it back on and visit my website.” Yup, it's happening. They go back and they just keep going down their supply chain and their vendor's supply chain. Someone somewhere along the line got compromised, and the bad actor inserted the cryptocurrency miner in a script that has just slowly rolled up through multiple vendors coming out to the public.

I encountered something very similar. I was hired to help an organization do their multi-cloud environment and really build their process for FedRAMP and things. The first day that I was there, I said, “Can I just sit here and watch your environmental parameters, variables, and system performance? I've been told earlier that when basically an instance gets up to 85%, we kick off another instance.”

I kept watching, and it was two instances out there at 97%. I called over the team lead and said, “Why are these two at 97%?” He goes, “Because they're busier.” I exactly heard the sake of, “What do you mean?” I said, “I thought your process was for 85%.” They said, “It is.” I said, “But those are 97%.” He goes, “I get that.” I said, “This isn't Spinal Tap because it goes to 11 as opposed to 10. This is you're deviating from your baseline.” Those were crypto miners.

He didn't even know they were out there being used. He turns around and goes, “Well, we'll just kill them off.” I said, “OK. Is it in your code, because they were building everything with terraform and stuff continually.” It turned out that the code release that they put out two months earlier that nobody ever took down the old ones, basically terraformed crypto mining into them.

Again, it's back to knowing your code that you start out from. That's, I think, part of the biggest problem we have as an industry. We go back and keep grabbing open source or other code again and again and again. We back up redundant images. You go ahead and clean up a breach. They said, “We'll go back to a copy six months ago.” It's like, you re-infect yourself.

That's, I think, part of the biggest problem we have as an industry. We go back and keep grabbing open source or other code again and again and again. -Kevin Kumpf Click To Tweet

You go back and reintroduce the, “Hey, clean slate,” with the door wide open.

Exactly, so it's not shocking to hear your story.

I’m discouraged, disappointed, disillusioned, all sorts of D words here. How does someone actually solve the problem or deal with the problem if it's not mine? Even if my partner's doing the right job, their partner's partner’s doing the right job. It’s 16 partners away, and it all rolls up into what goes on to my website in my example. How do we deal with this aside from not trusting anybody and rewriting everything ourselves?

Everybody wants to go out there and do a third-party risk assessment. But how many times have you done those and came back and said, “We shouldn't trust these people,” and the entity says, “But they're the only supplier for this, and we already have a contract for this much money, so can we pretend we didn't have that done or it won't happen to us? Trust me?”

The other part is contractually. If you're farming out a service, you want to know where that service is. I tell people if you really want a long-term job in this industry from the cybersecurity perspective, get a contract review. Because the things you don't ask in a contract that are assumed and apply implied are the things that kill you.

What are the things that we're not asking that we should be asking?

What do they do in their environments? Do they share their data? Do they use any third parties to do any of these resources’ outsourcing or anything else? Who owns the code that they're creating for us or the applications that are out there? What is their system redundancy plan if something happens in their environment? Where's your data being shared?

I was talking about this the other day, the new world of IoT, which I love to call Internet of Threats, not Things. If you look at these devices now, this is a prime example of third-party data going out there that nobody knows about, and yet there are places out there like the big cloud providers that are now starting to collect it, get diagnostics out of it, and start to sell that data. It's an area I call digital rights management.

Now you're going to see even more data out there. Is it a supply chain for me and you? No. Where does the supply chain come in? If I've got all my lights in a facility that are now being managed remotely by an application and a provider for energy efficiency, I've had customers say, “The lights in my facility went off. What happened today?”

It wasn't, well, somebody turned off the switch. Somebody actually breached that third party because that supply of light energy to you is a supply chain. You didn't know where the management system was or what the data was doing. Good luck having a contract for it.

I'm laughing because I'm thinking about just all the things in our life that are connected to something. Whether we are an end user, a small mom-and-pop business, a medium-sized business, or a Fortune 500 company, I can't imagine how many devices, pieces of hardware, chunks of code, software applications, are sitting places that people just don't think that they are or don't even realize even exist.

They are. In fact, there was a recent news article about a home appliance where a man out, and I believe it was California, started looking at what was using so much bandwidth. It was one of his home appliances that was phoning home.

I told the manufacturer what was going on with it. I'm like, “Is there anything in your agreement that says no? What's your option? Take your dishwasher or appliance back?” It's almost implied now that you are feeding the global supply chain of data, irrespective of whether you want to or not, and yet asking what that data would be is impossible.

It's almost implied now that you are feeding the global supply chain of data, irrespective of whether you want to or not, and yet asking what that data would be is impossible. -Kevin Kumpf Click To Tweet

The person at the local big box store doesn't know what data your refrigerator is sending back to the manufacturer. I sometimes even wonder if the manufacturer even knows what data is being sent back.

No. If you look at the components that come in with these environments to build these appliances, they're not built by LG, Whirlpool, or anybody else. They are common components that are sourced from one supplier or two.

We've had this in the power grid where even if you want to talk about true supply chain risk, if you're talking about environmental components for RTUs, for PLCs, for other things, key components in all of our industries, where does that firmware chip come from? One source or two. Where are those sources? China or other places that are unfriendly nations to us. How do you know what's in those chips?

That's a scary thought.

Until one day, maybe it's a date down the road here that somebody says, just like my old friend that stopped printing that change the checks, “Hey, today's leap year in this year, whatever day, we're going to turn you all off.”

Before we go down the massive global shutdown post-apocalyptic discussion—I don't think that's where we want to go today—what are the practical things that we should be doing?

It's really working with your third parties to understand how important their data, their information, their components are to your process, building that bridge across, that yes, you need the legal in the contract to say, “You must do this,” but really coming together and saying, “Look, we trust you because you're a partner of us, but things are going to happen. If something does happen between us, how do we work as a team to figure out how to resolve this and to get us up back quickly, to get you restored quickly, and to put some faith around things for leadership management, elected officials, or anybody else based on the size of this that we can handle this and restore with confidence?”

That's where the teams really come in. It's when you have this, they're a third party, they’re arm’s length, “We don't trust you” mindset. That's where a lot of the problems begin. It's closed doors. We have to open doors to each other, I want to say we've done technologically, but as you can see, technologically, we've opened many of the doors uncontrolled. Let's find it on the human side to start controlling it.

Do you see this as a need for governmental regulatory compliance? Or should the industries self-regulate and get out ahead of it before the government comes along and says, “This is the way you're going to do it”?

If you look at government regulation in its purest form, it's always after the horse has left the barn. If you look at it in another purist form where you talk about, “Thou shall do this, thou must do this,” it's always the interpretation of what does “thou” mean, what does “shall” mean, or how does this apply?

You're seeing so many organizations out there today throwing out legislation, requirements, edicts. Then you read the fine print. It's like, “Well, we need three-plus years to get this implemented, and then we'll have our first audit six months later.” That is not going to do it, but it's the only way that organizations out there spend the dollars because you can't get an organization that has things that need to be done to keep the lights on saying, “I'm going to take money for this and worry about the boogeyman.”

Just to add to that, when you look at it from a political perspective, even though we're passing regulatory things and such out there, one of the key components is if you're a local politician in your hometown, Chris, which is going to get more votes for you? Filling a pothole that the people drive through, complain about, and feel every day, or putting money to cyber security that they can't see, they can't feel, and they assume that everything's good anyways, and if not, it'll come back up like magic? Where do you get the votes?

It's always challenging because you have people potentially making decisions about technology and concepts they don't understand. You hope that the cybersecurity experts are involved in writing the legislature, but we know that politics like to throw their little pieces into these things.

Yup. I remember my first cybersecurity meeting, where I was to be the cybersecurity lead on a project. I walked into the bank meeting with the two banks that I was working with. The first person said, “You must be Kevin.” I said, “Yup.” He goes, “Where do I sign?” I said, “What?” He goes, “Sign my acceptance and of risk so you can get out of here.”

Chris, that's not far off. How many meetings did you go into, say, “Where do I assume the risk? Where do I sign off, because I don't want to deal with security to slow down my project”? “You're going to ask too many questions.”

Ask questions, ask questions, ask questions.

Yup. The key point is if you look at the supply chain, it's about asking questions. It's not about the regulatory that's going to fix it tomorrow. As I said, it's about partnering, peering, asking questions, and being able to accept the answers you get and realize that the world's not going to stop for cyber security whether we want it to or not. But then how do you build that defense in depth around it and compensating control where you can look at your baseline and say, “What's acceptable risk?”

You're baselining the risk and you're baselining, like you're talking about before, the servers aren't going to run over 85%. My normal bandwidth isn't going to be this. These are the red flags that something needs to be looked at.

Yup. If you look at the physical side of the supply chain, we've gone to so much just-in-time manufacturing things out there. I was working for a manufacturing company and they put all their money and security and things into their biggest facilities. We had wheels that needed to go onto some farm implement equipment. One of their suppliers that provided O rings for a fitting for these vehicles got breached. They could not guarantee that the material and the process that they made were quality at that point, so they literally had to shut down and wait for 40-some odd days for parts to come in.

In the meantime, they've got these multi-million dollar pieces of equipment coming off the line that they can't even put the tires and stuff on, so they had to build up another building to hold these vehicles just because the 5¢ widget down here wasn't important to them, but yet all the security went into the big building down here of putting out the product.

I've always found the physical supply chain really interesting, particularly throughout the pandemic. We realized that just-in-time is often just not in time.

No, it's not. If you want to truly look at the supply chain, it's really as we've talked about an ecosystem here. I'll give you an example. I was doing some work for a poultry company. Somebody had posted an image out there of one of their processed machines out on the Internet. Somebody came in, took that image, doctored it a bit, and sent it through an email to management saying, “I've been into your systems. I changed this parameter. Pay me this money, or I won't tell you what I've done to your systems.”

They had to immediately go into shutdown mode, recall all the product because they weren't sure what had been done to it out there. Was it properly cooked? Anything else? Now, the Food and Drug Administration, the Department of Agriculture, all these types of entities get involved, because now it's critical infrastructure.

If you look at the supply chain out, it's like, “All right, well, we're not going to get products to the stores.” Look at the supply chain you're impacting down the other way. You got all the chicken that's showing up now to be slaughtered for this. You've got the trucks that are bringing it in.

Now this food can't be used. What are they going to do with it? Do they dispose of it? What about the people that are raising the chickens? They've got their next one to come up to you on this day here; just that trickle down to the real supply of where it comes from. The ground up is really where your supply chain is, be it data or product.

I like the thought of not just looking at your supply chain backwards, but looking at it forwards in terms of, “OK, if I have issues, what happens after me in the process” and not just assume I'm the end-all, be-all that all stops with me. “Hey, if I'm down, it just affects me,” but realizing, “Hey….”

I think if we're thinking of I'm an electricity delivery system, I understand that my job is to make sure I get electricity downstream. But if we don't view ourselves as critical, then people will get their product or service from someone other than me, but that may not always be the case.

Yeah. I forget the exact instance, but there's a recent news article about electricity that said that suppliers cannot be held liable. I want to think it was related to ERCOT in the winter storm in Texas, where it said suppliers cannot be held liable for not providing enough energy because it's an on-demand type service. It's not viewed as a must-have service.

Clearly, that approach and mindset is because you're saying you rely on me, but if I have a problem, don't rely on me because I'm not legally challenged and capable to say, “Yes, it's my fault.”

It's a very messy system out there in our lives.

Very much so.

As we wrap up here, any parting advice, resources that people should be looking at, recommended reading?

As I said, when we started this, there was a recent article out here. I've given you the info on it if you want to publish it towards the end, but just some numbers that I think people need to understand. If you look at zero days for the supply chain overall, 72% above the previous record with 3205 attacks this year. But if you look at the other side of the coin, they said that, “Hey, the number of individuals breached went down. We went down to 353 million and some odd from 425 million.”

You're getting to the point where it's like, “Well, it's half-a-billion people, that, to me, is scary when we can take a positive and say, “Well, it's not going up to half a billion. It's going down.” That's scary.

I seem to remember a report saying that while there are more zero days out there, less of them are being exploited. Is that your experience? “OK, yes, there's a zero-day vulnerability on this piece of hardware, but you can't get to it unless you have physical access to it.” Even though that zero-day exists, it isn't really an imminent threat unless someone has physical access to the device.

That is true. That's in the space that we work in, where it's really compensating controls. That's really annoying, because I say you physically can't get there from the outside world, but now you're seeing people. In fact, an email or something like that that infects the laptop internally, that thing goes and talks to the device. Unless you're a complete standalone, isolated computer or technical component that there's nothing I/O-related, in and out, that compensating control can only go so far.

What we are seeing more of in the exploits is a deeper understanding of what the core technologies they use are, the protocols, the ports, the services, and it's easier for us now to tune up firewalls and things like that or other technology to say, “I'm going to look for this.” The problem is if you don't know your baseline, you don't know what that difference is.

If you don't know what to look for, you don't know what's normal, you're not going to find what's abnormal.

Exactly. That's where it comes back to knowing your baseline. You're going to find a zero day. It's going to be announced. We saw this with SolarWinds and others where it's like, “Hey, we patched it.” Then, “Hey, we thought we patched it. We really thought we patched it.” It was all because they didn't understand their baseline of what was truly going on in the product.

Everyone just go back and look at every line of code on everything that you've ever touched.

Absolutely, yeah. That's how I look at it and say, you know what? Fred Flintstone and those guys had it right. Just carry around your own slate. When you're done with it, turn it into rubble.

I think that's a great place to wrap it up for today. Kevin, if people want to find you online, where can they find you?

I'm on LinkedIn and a couple of the publishing sites out there for a couple of the other media sources out there, Chris. They'll be on my LinkedIn profile or at Cyolo, lecturing the circuit and doing things like that.

Awesome. Thank you so much for coming on the podcast today.

Very welcome, Chris. Thank you.

Exit mobile version