Convincing Deepfakes

“People want to believe that something is real, and if you want to believe it, then you ignore the signs that something is a little off.” - Tom Cross Share on X

A familiar voice on the phone or a recognizable face on a video call used to offer some reassurance that you knew who you were dealing with. AI has changed that. Voice cloning, face swaps, and real-time video impersonation now allow scammers to convincingly pose as executives, job candidates, romantic interests, or even family members. Understanding how these attacks work and where they may be headed is a central part of Tom Cross’s work as Head of Threat Research at GetReal Security.

Tom has spent more than 30 years studying cybersecurity threats, software vulnerabilities, and the methods attackers use to exploit technology. Before joining GetReal Security, he held leadership roles at IBM X-Force, Lancope, and Drawbridge Networks, and he has shared his research at major security conferences including Black Hat and DEF CON. His current work focuses on deepfake-enabled social engineering and the development of tools that can detect digital impersonation, including signs that are often too subtle for a person to recognize.

In this episode, we learn how little audio is needed to clone someone’s voice, why live video is no longer reliable proof of identity, and how deepfakes are being used in romance scams, investment fraud, identity theft, and remote hiring schemes. We also talk about AI agents that can carry on persuasive conversations, adjust their behavior based on a victim’s reactions, and repeat those tactics on a massive scale. The discussion offers a revealing look at why familiar advice for spotting fakes is quickly becoming outdated and what individuals and organizations will need to do differently as the technology improves.

“An audio deepfake is pretty easy to pull off. If you’ve got maybe 20 or 30 seconds of someone speaking, you essentially end up training a little AI model to replicate their voice.” - Tom Cross Share on X

Show Notes:

• [01:05] Tom shares how his early work in vulnerability research led to a career studying sophisticated cybersecurity threats.
• [03:32] Running a bulletin board system as a teenager helped spark an enduring interest in hacking and computer security.
• [06:11] A look back at early online communities, text-based games, Fidonet, and the pre-internet era.
• [09:36] The conversation shifts to deepfake research and the technology being developed to detect manipulated media.
• [11:29] Deepfakes are divided into audio, visual, file-based, and real-time forms, each creating different risks.
• [15:20] Modern voice clones can fool both people and biometric authentication systems, making specialized detection increasingly important.
• [18:30] Virtual backgrounds and other subtle processing artifacts may reveal manipulation even when nothing looks obviously wrong.
• [20:31] Deepfake detection has become another cybersecurity arms race as attackers continually improve their methods.
• [22:30] Romance scams, investment fraud, remote job schemes, and identity theft are among the growing uses of deepfake technology.
• [25:26] Scammers succeed by exploiting desires, expectations, and the human tendency to rationalize warning signs.
• [27:56] Large-scale phishing and business email compromise attacks only need a small percentage of targets to respond.
• [29:22] Criminal compounds in Southeast Asia use trafficked workers and deepfake tools to conduct scams on a massive scale.
• [30:27] North Korean remote workers may use stolen identities and shared deepfake personas to secure jobs at American companies.
• [33:45] Purpose-built criminal software combines face swapping with appearance-enhancing features designed for romance scams.
• [35:05] Common visual tests for identifying deepfakes are becoming unreliable as the technology advances.
• [39:24] Emotional investment makes detection even harder because people often explain away signs that something is wrong.
• [40:47] Building authentication into the internet could help people determine whether digital content is genuine.
• [42:13] AI agents may soon conduct automated scams that respond naturally, apply pressure, and adjust to a victim’s behavior.
• [45:14] Automation could allow criminals to target hundreds of thousands of people while making impersonation increasingly convincing.
• [46:22] Machine learning may create self-improving scams that test countless variations and concentrate on the tactics that work.
• [49:18] AI lowers the technical barrier to cybercrime by helping people create tools they could not build on their own.
• [51:06] Phones and messaging platforms may eventually require stronger controls as automated calls and texts become more common.
• [53:09] Digital signatures, watermarks, and verified content could help distinguish malicious deepfakes from authorized uses.

“Criminals are looking for things that they can do over and over again. They find what works repeatedly, and then they build processes around it.” - Tom Cross Share on X

Thanks for joining us on Easy Prey. Be sure to subscribe to our podcast on iTunes and leave a nice review. 

Links and Resources:

• Podcast Web Page
• Facebook Page
• whatismyipaddress.com
• Easy Prey on Instagram
• Easy Prey on Twitter
• Easy Prey on LinkedIn
• Easy Prey on YouTube
• Easy Prey on Pinterest
• GetReal Security
• Tom Cross – LinkedIn

Transcript:

Tom, thank you so much for coming on the podcast today.

Thanks for having me on.

Looking forward to our conversation. Can you give myself and the audience a little background about who you are and what you do?

My name's Tom Cross, and I'm a threat and vulnerability researcher. My career started maybe 25 years ago. I had a high interest in internet security issues and ended up working on security vulnerability research. Understanding exactly how flaws in software are exploited to break into computer systems, I was part of a pretty well-respected research organization called X-Force Research.

We did a lot of work on vulnerabilities and vulnerability exploitation. Eventually, we were part of a company that got acquired by IBM, so we were part of IBM for a while. Subsequent to that, I've been involved in a few different startup companies—a company called Lancope that was a computer security software vendor; a company called Drawbridge Networks, another computer security software vendor. A lot of that kind of activity.

Really trying to figure out how to build technologies that can protect organizations against security threats. I talk a lot about conferences, and there's a couple of different areas that I often speak about: understanding threats, understanding vulnerabilities, how do you build mental models that enable you to think about threats and really complicated, sophisticated attack scenarios, particularly around how nation states are breaking into computer networks. How do you contend with that as someone who's responsible for defending them?

Recently, maybe about a year ago, I got this new job at GetReal Security. I'm the head of threat research at GetReal. What we do at GetReal, we specialize in building software to detect deepfakes. In a video conference call like this one, there could be a face swap where someone's not showing you their real face. There could be manipulated audio where someone's voices, they're trying to sound like somebody else.

We build technology that detects that. What I do here is I help us understand the threat actors that are using this stuff, how they're using it. That helps guide the way that we develop our protections so that we're aligned to the threat.

I like that. How did you get involved? What got you originally involved in cybersecurity or in security in general?

When I was a teenager, I ran a bulletin board system back in the early 1990s.

What was the name of it? Maybe we knew each other.

It changed names a few times. For a while, it was called the Ninja's Domicile. It was in the 615 area code. It was a fun place to hang out and talk with my friends. I only had one phone line. I think at the end, maybe I had a second phone line. We played games with each other. There was a lot of discussion going on about hacking and the like.

There were all these text files that people were sharing. I found this stuff really fascinating. I went to Georgia Tech. I'm a Georgia Tech Comp E graduate. I always found the computer security facets of computer engineering to be really interesting. I think what it boils down to is that there is just sort of what I would characterize as a hacker mindset. When I say hacker, I don't necessarily mean it in a purely criminal sense.

There are all kinds of differences. We talk about white hat hackers and black hat hackers. We're the Old West or whatever. People use these skills in different ways. What unites it all is just sort of a different way of looking at things and that perspective that enables you to see. The way I describe it is you're looking at technology and you're trying to see it for what it really is as opposed to what it's meant to be. People expect computers to work a particular way. That's what they're meant to be.

If you see what they really are, sometimes that reveals ways to get them to do things that they're not supposed to be able to do. I've always found that fascinating. I've tried to use that ability for good by doing things like vulnerability research where I'm trying to figure out ways that I can get software to do stuff that it's not supposed to do. Then we work to fix those problems and make sure that people are not taking advantage of them to do things that are evil.

That's fun. We have a similar upbringing timeframe of BBSs and Commodore 64s.

What was your BBS called?

It was called the Citadel. I think it was a 714 area code.

Were you running the Citadel BBS software?

No. What was it?

I was running remote access for a while and then I switched to this thing called Oblivion that had a bunch of really cool artwork built into it.

I wish I could remember the platform. I actually made a game for it.

That's cool.

I'm horrible that I can't remember the name of the BBS.

I had a bunch of games on my BBS. There was this thing called The Pit. It's all text. It's text. There's no graphics, but you're in this Roman Colosseum scenario and you're fighting these monsters. You're an X and you're moving it around. There was a Tetris. I had a Tetris. There was this girl that used to call into the BBS. She would call in at 300 baud, which was a really slow baud rate in the early 90s, and then play Tetris.

The Tetris pieces fell really slowly down the screen because her band went through that. She would rack up this. She dominated the Tetris rankings. There was a game called Solar Realms where you're building this intergalactic empire and you're conquering these planets and you're teaching trade. Other people call into the BBS have their own empire and you're trying to trade or fight them. Those things were super fun. That was a great era.

There was a unique BBS software called Citadel that structured the space as if it was a physical space. You would come into a room and then you would move from that room to other rooms within the building and you got to design your building as the guy that was running the thing. That was a particularly cool BBS software. That was an interesting era for sure.

I remember running, oh gosh, it was part of FidoNet. I don't know.

Yes, I remember.

It wasn't just messages on your board. It was now messages on other people's BBSs.

That's right. You could call into your local BBS and have conversations with people all over the world. The way FidoNet worked is at night, your computer would call other computers and share these messages. We were able to relay them all the way around the world without having to make a bunch of long-distance phone calls. At that time, long-distance calls were really expensive.

We were building this computer network in this world where you couldn't reach it. Nowadays, you can call all the way across the country and it costs nothing. It's the same thing as calling down the street. In those days, geography was really limited in terms of the network. We built a protocol that enabled us to have a global network despite those charges.

It was fun. At one point, I was trying to be clever and say I wanted to get some of the feeds from Europe sooner than other people did. I started speaking with the BBS in France. I did that until I got the first phone bill. I said, “Well, I'm definitely not doing that ever again.” It got expensive real fast.

All the old days, the pre-internet days, that was the internet before the internet.

Your work now in deepfakes, what got you switched over from maybe more traditional security to dealing with deepfakes?

I have always been interested in technology and being at the leading edge. This is an entirely new space. It's really fascinating, I think, from a technical perspective. The group of people that I get to work with are just really interesting. Our company was founded by this guy, Hany Farid, who's a Berkeley professor. He spent his career on the topic of image forensics. How do you tell that a picture has been Photoshopped?

If you're going to go into court and you're going to use this picture as evidence and somebody's going to argue, “Hey, that picture was photoshopped. It's not real.” He built his career on what are the techniques that you can use to determine that one way or the other. We've assembled a team of really excellent researchers with different backgrounds, image forensics being one of them, but also audio and video processing, stream processing, topics like steganography, which is how you can hide information inside of multimedia content, and then also machine learning.

We're training these AI models, and we're using the most sophisticated stuff we can get our hands on. We're also operating at the bleeding edge of AI technology here. It's just a tremendously interesting set of things to learn about. It's also an increasingly concerning threat. We're seeing this technology advance. Like I said, every few months, there's new capabilities that come out. Maybe it would help if I talk a little bit about deepfakes and the different kinds of deepfakes.

Yeah, let's talk about what are the different types of deepfakes.

We categorize them as either being audio or visual. Sometimes when people hear deepfakes, they're thinking about manipulated pictures or they're thinking about things they see on the Internet and videos that are on the Internet, and that's definitely part of it. But we're also seeing deepfakes used in real time, like in calls like this one or on telephone calls.

There's file-based deepfakes and real-time deepfakes. There's video deepfakes and then there's just audio deepfakes. An audio deepfake is pretty easy to pull off. If you've got maybe 20 or 30 seconds of someone speaking, and those of us who go out and speak publicly, it's pretty easy to get that for us, right? You essentially end up training a little AI model to replicate your voice. And it's really easy.

It's open source software. You can download it, put it on your computer, train up your model, and you can make the thing say anything you want using that person's voice. And we see those kinds of things wired up to the phone so that they can place phone calls. We see them wired up to AI, like LLM models that can have conversations with people in a totally automated way.

We also see them used sort of in real time where I'm speaking and the computer is translating it into someone else's voice and then using it to target the, I'll say, victim in this case, right? There are lots of legitimate uses of this technology, by the way. People are doing all kinds of creative things with it. But unfortunately, it is something that the criminals are also taking advantage of.

In real time, there's a couple of things that we see. One of them is face swapping, where you can run a program and actually put someone else's face on your face, and now you look like them. You can interact and talk and everything… Share on X

And then on the video side, there are these fully AI-generated videos that people create, but it takes a lot of processing power to make those, so usually those don't work in real time. In real time, there's a couple of things that we see. One of them is face swapping, where you can run a program and actually put someone else's face on your face, and now you look like them. You can interact and talk and everything you do, that face is going to do.

That technology is actually quite sophisticated at this point. And again, there's lots of different applications that do it. Many of them are free, open source. You can download them, run them on your computer, and create fake video and feed it into a call like this one. There's this new technology that we've seen, which is this sort of real-time, fully AI-generated deepfakes, or AI-generated avatars.

In March, for example, we saw the first example of a system where you can inject video into a call like this. It’s fully AI-generated, so it could be any person, and it will look exactly like them, and it will do anything that you do. If you're in front of the camera, like I am, and you're moving around and talking, this thing will do exactly what you do, using that person's persona. Those kinds of tools, as they become more commonplace, I think will open up more and more opportunities for bad guys to do things. This is just going to continue to get more sophisticated and look expensive as the years go by.

With the audio deepfakes, how good is it now? I assume that, very easily, the layman could be fooled. Is it from a forensic position? Is it hard to figure out whether it's real or not?

Let me look at this from three different lenses. Yes, the layman could be fooled, but anybody could be fooled. A human being is not going to be able to distinguish these voices from the real voice that they are simulating. I will give you a counter example just to illustrate that a little bit. The CEO of Wiz; Wiz is a cybersecurity company. Somebody took his voice from a public presentation that he did, made a deep fake of it and was calling employees trying to get their password.

It didn't work. But the reason it didn't work is that this guy had a different sort of vocal effect when he was presenting publicly than he does when he's speaking personal. Because all these people had sort of interacted with this individual, they knew something was off about his tone. But that's a sort of characteristic of the person that you'd have to understand in order to defeat. The voice is, it sounds just like his real voice.

I said three things. The second thing is biometrics. A biometric authenticator says that's this person's voice or that's this person's face. Our research shows that our deep fake faces and voices pass biometric authenticators. We've tried the open source ones. We've tried closed source commercial ones. We can always pass biometric authenticators with deep fakes. We actually were planning to publish something on that soon.

The third thing is, can we detect these deep fakes with—so you mentioned forensically. The answer is yes, we can detect them, but we have to train specialized models that are designed to detect very subtle traces of processing that appear in either in the audio signal or the video signal that tell us that a deep fake has taken place, and they're so subtle that you really can't describe them to a human.

It's weird. That's why we're using AI to detect this stuff. It's the kind of thing that's so subtle that only an AI can detect it. One of the analogies I give here is, you know, we're making great advances with the use of AI and medical diagnosis. AI can look at a radiological image of someone's lungs and see cancer that, like, a doctor couldn't see or see it before the doctor could see it.

That's a tremendously powerful diagnostic tool, but it's literally that the AI can sometimes identify patterns that are subtle enough that it's very difficult for a human to, like, mentally model them, if that makes sense.

There's something so subtle about the video that no person would realize it was, it’s there.

Yeah, I mean, even the medical scientist in a lab can't show you something visual that is the thing that is being detected. But it's various sort of processing artifacts that are evidence of something that isn't real. And I'll give you a simple example of something that is, like, cognizable. One of the things that's very challenging is telling the difference between a real background and a virtual background.

Often, one of the pieces of advice that we give people that are worried about being scammed through these calls is to make sure that, you know, ask the person to turn off a virtual background. Show me the room you're in. Because if you're in a call center with a bunch of other people, then you might not be who you're claiming to be.

If your camera is looking at a real background, which this is, it's getting photons and, like, randomly, it will get a different, you know, slightly different color, slightly different number of photons each frame. There's going to be this noise that's present and it's very subtle. But if it's a virtual background, it's being rendered by the computer.

It's not of actual film. And so frame to frame, there's no noise. There's no difference between the pictures. And so we can write software that can identify that distinction and tell us, “Hey, this is a virtual background, or this is a real background.” That's the kind of thing that we're building capabilities to detect.

Using your example, which is that the image quality was too good or too consistent.

Right. It's not real.

Yeah. I suppose that someone hearing this will say, “Well, why not just make the deep fake software introduce noise? But I guess if you don't really even know, like, it's so far down the chain of things that we've overly simplified it.

No, I mean, we get in these sort of like arms races in cybersecurity, right, where somebody will have a technique they're using to do stuff. Then we develop a security technique that protects against it. In turn, mature their attack technique or the attacks change and hopefully over time, we're reducing the overall problem. We continue to see increasing sophistication because there's such a desire out there to commit fraud, frankly.

We continue to see increasing sophistication because there's such a desire out there to commit fraud, frankly. -Tom Cross Share on X

So far, I'll say two things. The first thing is that the kind of AI that most people who are making these tools, like I said, they're not making the tools in order to enable criminals. They've got some legitimate use case or desire for these tools. Just people want to talk to their AI agents so that AI agents can talk and they can also, like, listen to human speech and translate it.

These things exist. That they don't necessarily they need to fool a human. They need to sound real to a human, they need to look real to a human, but they don't need to fool my AI model. There isn't necessarily a desire amongst the people making this technology to, like, to figure out not how to not have any artifacts. And in fact, there is an effort by a number of different companies to put what we call digital watermarks in some of the content that they produce via AI so that you can electronically tell that a video is AI generated.

There is an effort by a number of different companies to put what we call digital watermarks in some of the content that they produce via AI so that you can electronically tell that a video is AI generated. -Tom Cross Share on X

It'll have a signature in it that tells you that. And I think that's a responsible thing. I would like to see more companies that produce some of this technology, like introduce those kinds of watermarks. What's happening sometimes is that there are criminal groups who get access to the software, and like I said, some of the software is open source so you can download it off the Internet.

It was built by an academic team. And then they embed it into tools that they have purposely built for committing crimes. They're not creating maybe the deep fake technology, but they're making software that the deep fake technology is embedded into that provides a bunch of features that criminals want.

Yeah, that's scary. How are the different ways that some of the video deep fake is utilized? Not necessarily, kind of not like, “Hey, they're using it for romance games,” which they are. But are there, like, different styles of, like, what are the intentions to do this with the software?

Well, so let me lay out, like, a couple things. So first of all, there are attacks against consumers. Yes, romance scams and also investment scams; there are attacks against companies. We see, like, candidate fraud where people are in particular, North Korea likes to get themselves jobs at US companies from remote.

They've placed thousands of people at US companies and they're collecting that money and funneling it back into their weapons development programs. Sometimes they use deep fakes so that, you know, they can disguise their identity or show up as, maybe there's a couple different people that are doing it, but they want to show one person they want the employer to think they hired one person, right?

When you go make a bank account online, you’ve got to do this KYC process, right, where you hold up your ID, you take a selfie. They're using deep fake technology to bypass those KYC checks so that they can get KYC-validated bank… Share on X

We see that kind of stuff. We also see a lot of attacks against consumer-facing financial services. That's really the third category. When you go make a bank account online, you’ve got to do this KYC process, right, where you hold up your ID, you take a selfie. They're using deep fake technology to bypass those KYC checks so that they can get KYC-validated bank accounts.

I've seen people either—I’ve seen people selling those on the internet for, like, $180, $200 a pop.

They're probably selling them to people engaged in large-scale money laundering. And then they're also, sometimes, people engaged in identity theft will use that as part of the process of establishing bank accounts under someone else's identity. And they actually work to build that person's credit up over time. They can take out a big loan in that person's name. There's this wide swath of ways that deep fakes are being used and sometimes they're used to impersonate a specific person.

I was saying in the KYC scam I'm trying to animate the identity on this ID that I have so that the bank software thinks that I'm really this person, right? Sometimes there are social engineering scams involved, like the Wiz example, they involve a real person and showing up and presenting that person, sometimes they're used to disguise identity, like in the North Korean case, they may not be trying to impersonate a particular person, they're just trying to hide who they are, maybe change their nationality.

Sometimes, like with romance scams, they're trying to use this software to become attractive, right, and sort of maximize their attractiveness. Something that, like, you and I, so we had a short conversation before we started recording and I want to come back to something that we were talking about then, which is the fact that, like, criminals, like, exploit people's desires, right? People, you're much more likely to fall for; some people fall for investment scams.

In fact, that's the if you look at the FBI statistics, that's the most lucrative type of crime that's being committed over the internet, like something like $8 billion was stolen last year through investment scams. And it's because we all want to make more money. Everybody does, right? This criminal starts communicating with you and they convince you that this is an opportunity where you could really make a lot more money than you have.

And you want to believe that that's real. And so if you want to believe it, then you ignore this, you know, the face is a little off. -Tom Cross Share on X

And you want to believe that that's real. And so if you want to believe it, then you ignore this, you know, the face is a little off. Well, you just look past that stuff because you're emotionally invested in the idea that this is really happening to you and that the romance games are the same, like Keanu Reeves is really in love with me. I want to believe that, right? The weird fact that he never meets me in person and he seems to always need money, like, those things, like, I don't just rationalize them away because I want this to be real.

And I think that that is like this fundamental thing that these people exploit in us that they they try to take advantage of our desires. The reality is that everybody in the world is susceptible to this on some level. I think we all have some circumstance where if the scam perfectly aligns with our life circumstances, the chance for us falling victim to it are higher than they would otherwise be.

If I'm never expecting something to come through customs, if I get an email about, “Hey, your package is stuck in customs.” “Well, no, it's not. I didn't order anything.” But if I did order something from China, and I'm expecting it to show up this week, I'm expecting to get an email from customs, and a scam email comes in from customs, the chance for me to go, “Oh, well, this is what I was expecting.” This was like you said; this is what I was hoping for. This was what I was looking for all of a sudden. Why would I question it? It's exactly what I was expecting.

On the one side, right, these scammers are putting out just a massive volume of stuff; they're casting a very wide net. And I'm going to come back to that in a minute. To your point, most of it doesn't land. But that doesn't matter, right? Like, if one out of 1,000 lands for them, they get a cook, it's really inexpensive for them to, like, just fire stuff off.

Working with organization companies that have been victims of business email compromise. So if you look at that, those FBI statistics, that's the second largest or most lucrative scam after investment scams. That's where the threat actor has contacted a company and said, “Hi, I'm your supplier, but my bank account information has changed. The next time you pay me, pay this different bank account.”

Every once in a while, you get a phishing email that's exactly aligned with what's going on in your life right now when you click on it. -Tom Cross Share on X

Usually, often the victims of this, like, they have good processes. But it just so happened that that week, like, the CFO was on vacation or something, right? This is happening all the time. But, like, every once in a while, there’s, like, an opening. It works. To your point, every once in a while, you get a phishing email that's exactly aligned with what's going on in your life right now when you click on it.

There you are. That's definitely a thing. With respect to some of this deep fake crime, I don't know if you’re—I'm sure you're familiar with this, it's the scale. The evil of some of the things that's going on are just, like, sort of horrifying. I think, like, there are these compounds of being constructed in places like Myanmar, where tens of thousands of people are essentially, like, being held against their will and forced to engage in the scam.

They're treated violently; it's a way of motivating them. They're out on the internet, and they're, again, looking for, they're hitting tens of thousands of people attempting to find somebody that's going to fall for it, right? They literally have a gun to their head. And just the scale of it and the brutality of it, it's just awful. These are the people who are using some of these deep fake tools to contact people and present an attractive identity for them to interact with and to talk to them into spending money.

I think the interesting thing you're telling me is when you have multiple threat actors—we’ll use that phrase—all pretending to be the same individual.

Right. One of the things that I mentioned before is North Korea is trying to get, you know, people candidates placed at US companies because of remote work. Before COVID, the amount of remote work happening where, like, the person is 100% remote and never comes into the office was fairly small but now it’s, like, 10% of the US workforce.

There's lots of these roles where, like, this person, you will never meet this person ever in person, and that's created this opening that they're taking advantage of where they're coming in and applying for these jobs. They work in cells. They work in little groups that are separated from each other, and their objective is just to make money.

These people are all living together in some dormitory somewhere, and they're all getting paid like a US salary. Their expenses are very low and the profit goes back into North Korea's intelligence collection activity. They want to present. They want to convince this company that they're hiring, a person they often have stolen an American's identity, right? They've created a fake LinkedIn profile for them, they got a fake profile photo, and they're applying their jobs.

When the recruiter hires them, there may be more than one person in the back supporting this job, right, that there may be a person who's really good at getting and interviewing, who’s, like, the person that they start talking to. But then that guy's job is to just go get more jobs. Then there are other people who actually do the work, right.

They may be working together because each one of them is supporting like four or five different full-time jobs, right, and they're trying to convince all these companies that they're there, they're a single person, and they're… Share on X

They may be working together because each one of them is supporting like four or five different full-time jobs, right, and they're trying to convince all these companies that they're there, they're a single person, and they're working full time for them. Using a deep fake face, they can show up in a work meeting. It's always the same guy. It's always the same guy, but it's not really the same guy in the back end.

That's just sinister and crazy. You have to, I think you have to give a little bit of credit of like the ingenuity behind that is pretty amazing.

They are systematizing, and this is something that happens, you know, something that happens with….In my profession, like, I talked about vulnerability, so we think a lot about what are the weaknesses in the system where are the holes. But the fact that a hole or weakness or vulnerability exists does not necessarily mean that a bunch of criminals are going to start exploiting it or targeting it.

The criminals are looking for things that they can do over and over again that work repeatedly, and they build these processes. -Tom Cross Share on X

The criminals are looking for things that they can do over and over again that work repeatedly, and they build these processes. And unfortunately, North Korea has figured out this process that works: they can get employed. They make money at it. They're doing it over and over again, and even though the FBI has focused a lot of energy on this over the past few years there, they remain undeterred, and they're continuing to do it. In fact, the other thing that happens is this trickle-down effect where people who know that they see that and they see that it's working for them, they say, “Well, I want to do that.”

Some of these, when I was talking about we got the opportunity to dissect this criminal deep fake tool called how it can AI. It's a Chinese-language deep fake tool. It's primarily used by romance scammers, and so it has face-swapping abilities, and then it's got all these libraries for beautification: you can make your eyes bigger, cheeks rosier. You can kind of almost apply virtual makeup to yourself and modify yourself in order to make yourself more attractive.

These guys are selling the software to criminals. They're almost, like, out there saying, “Hey, criminal, here's a scam you can run. Buy our software, and it's a thing that you can do repeatedly that will produce money for you.” And the Bitcoin wallets that those guys use that they've received $4 million over the past.

I was gonna ask you how much they sell for, how much have they made. So the answer is they've made millions from it.

They're charging thousands of dollars for the software, and they've made millions of dollars, so they must have lots of customers.

That's crazy. What are some of the techniques that people talk about that the—not what your company is doing techniques for identifying deep fakes? I used to be, when I started the podcast, I used to tell people, “Look, jump on a Zoom, jump on a FaceTime call, because they're not going to do it if they're not the real person.” And then it was like, “Oh, well, I could do it, but I'm at the airport and it gets laggy, and also let me just turn it all off.” What are some of the latest techniques that may work at the time of this recording but in six months won't work?

Oh, well, so let me say one thing, which is there's always a pressure for you to make an exception to your policies that happens, and it comes with some amount of guilt, right, so, you know, “I can't turn my camera on today. There's something going on in my house, or my mom's here because she's not feeling well,” or one of the things that we talk about with respect to DPRK is make sure that, like, you send the laptop when you onboard an employee to their address that they gave you that's their address.

And often, what you'll hear is, “Oh, you know, so my mom got hurt and she's in Arizona, and so I’ve got to go to Arizona for a few weeks so could you ship the laptop there.” And you're like, you feel bad. It's like, “OK.” What you're doing is you're shipping the laptop to, like, what's called a facilitator that a whole bunch of their laptops are set up with, right, and they don't have access to the address they gave you that's an identity they've stolen. Some of the things you're talking about are good things, like if you're asking somebody to do something, like turn off their virtual background and show me the room they're in, and they refuse or they can't turn their camera on.

These are important signs that it makes sense to be cognizant of them. I think we actually did a survey of a bunch of IT professionals and we asked them, like, do you think you could detect the deep fake? It's really interesting because the survey results showed that all these people thought that their organizations were being targeted by them, but also thought that they would have no problem detecting them.

There's this sort of like the people don't understand how fast this technology is moving. And so they may have had exposure to something at some point, and they don't know that, like, six months has gone by, and that thing is totally obsolete right now. It used to be if you did AI-generated video, like, the people would have six fingers, and that's not the case anymore. Like, they're incredibly compelling at this point, and so much has changed even in the past 12 months.

What people used to say with face swaps is, like, put your hand in front of your face, but that they respond fine to that. They respond to changes in the lighting condition so you know if you hold your phone up and move it around, you know the lighting will change and the deep fake will respond to that. We call this occlusion, where you have something that is, like blocking, part of the face.

Sometimes, if you have, like, multiple occlusions, so what I did to prove to you that I was, like, I have my hand like this where all my fingers are spread out, and then I move that across, that’s kind of a condition that's very difficult for the current generation of tools to deal with, but I would say that I talked about the real-time AI avatar thing that was released in March. That thing would have no problem with that, right?

Even these sort of visual things, where you're looking for stuff at the edges. A year from now, none of the software is going to do that anymore. I think we were talking about this before, in the interest of the people building these tools to make the tools as realistic as possible from the perspective of a human. The things a human could look for, they're going to go away if they have not already and we can reference academic studies that show that people can't tell the difference.

If you create a controlled study situation, they can't tell. And the other thing that I want to add on to that layer cake is what I said before, which is that if you are emotionally invested in believing in this thing, you found the job candidate you were looking for. It's been so hard to find the right guy, right? And I'll tell you, when the FBI goes in and, like, tells people, like, you are employing a North Korean spy, many of them don't want to fire that person because they're a good-performing employee.

Yes, like that is sometimes the reaction for real. Keanu Reeves is in love with you and you want to believe that. Well, if his face is a little off, it's like, there must be something wrong with the connection. You'll tell, you don't rationalize, that there's a reasonable explanation for anything that you see that's out of the ordinary.

I have a great explanation of why Keanu Reeves doesn't look quite the same. Because everywhere else you see him, he has makeup on, he's been in front of a makeup artist, and they've done post production on his face to make him look younger, older, whatever. And now you're seeing the real Keanu Reeves. This is what he really looks like.

Yeah, yeah. What we need to do, and what we—I basically am speaking for my professional community—is we need to sort of embed into the internet capability to determine, what is real and what is not real. And that's a long project, and there's many facets to it because we're talking about scams. There’s also, like, disinformation as well that's being spread out there, you know, that just for the purpose of manipulating people's understanding of the world, right?

There's many, many facets to this, but I think I want my computer to protect my understanding of the universe. I wanted to help. I'll go, but Douglas Engelbart, you know who that is. He was the inventor of the mouse and created a lot of our modern user interface concepts when he was at Stanford University in the 1960s. He's one of like fathers of, really, the internet and modern computing.

And he talked about his whole framework was that computers are supposed to make us more intelligent, they're supposed to augment human intelligence. I don't know if that's happening. You know what I mean? I think computers might be making us dumber right now. I think as computer scientists, like, we’ve really got to think about that. We’ve got to think about how our computers can do things to support to make us smarter and make sure that we know what's real and what's not real.

As we start coming in for landing here, how do you see this technology from, let's say, the adversarial, not people who are, “Hey, we're intentionally going to put watermarks and things,” but let's look from an adversarial position. Where do you see the technology going over the next year or two?

It's automation through agents. We did an experiment in our lab where we took, I was talking about we took an internet soft phone. We connected it up to a voice deep fake that was controlled by an LLM and we tried a bunch of different ones. We tried local ones that we could run on our computers and we also tried like commercial ones. We taught it to engage in social engineering. It would call you up and it would tell you it was an IRS agent and it would say that you have some sort of overdue taxes and you're risking arrest if you don't address the issue and, you know, you need to pay up.

What is sort of really shocking about it, and again, like, there's this future shock thing happening. We've all called at this point into some customer service center that has some AI that answers and it's always stupid, and it’s, like, just operator. You just want it to send you through to a human. That thing is not going to help you. The sophistication of where, you know, this technology actually is is much greater than that. This thing was very sort of emotionally responsive.

If you were compliant and agreed to buy the gift cards, it would be nice with you, and if you started sort of saying, “I think this is a scam,” it got very aggressive, would really react to your behavior. Because people haven't experienced that, very few people have been called on the phone by an AI that interacted with them conversationally and adapted to their sort of business, but you can build stuff like that and it's not like we spent a lot of money on, “This was just a little project we wired up,” right?

The one thing is, we were also looking at the guardrails that some of these companies have. Some of the more reputable companies had good, reasonable guardrails that we would get busted by them after a few hours. Some of them did not really have guardrails and we could use some commercial models, sort of indefinitely for this kind of activity. And then of course the models that we can download and run on our own computer, there's no guardrails.

The power of this technology that we're seeing today, it's going to double in 12 to 18 months. -Tom Cross Share on X

The power of this technology that we're seeing today, it's going to double in 12 to 18 months. Moore's law is a continuing process here right now. Whatever you're seeing right now, that's sort of exclusive or on the bleeding edge, it's going to be commonplace a year later. We expect to see criminals, like, engaging in, like, more and more automated scams like this. It's just really cheap for them to scale it up. They're obviously motivated to do so, like, again I talked about what's going on in Myanmar.

I think that once these guys are going to use these computer systems and they're going to scale it up to hundreds of thousands of targets a minute and it's going to get really annoying. Actually, I think we're going to get inundated with the stuff, because it's just so easy to produce. I think that's what we're on the on the cusp of, and also, like I said, the convincing this to a person of these things is going to get very, very good over time.

Anybody's identity could be fully animated talking to you using their voice, moving around in whatever setting you want in real time and responding to you. What I mean, video and audio, those things are right on the edge of those things being available. You can imagine what the next few years are going to look like as far as this is concerned.

The impersonation aspect is scary and the AI aspect is scary on the machine learning of the AI. I'm going to test 10,000 different variations of my first sentence that I give you the phone, and just start going, “OK, well, here's the 1,000 that didn't work. Let me focus on the 2,000 that didn't work to get to the next step.” Just the machine learning of that humans just don't have the ability to scale and understand why it worked, machine learning is just going to go, “I'm just going to try all sorts of crazy stuff and find the ones that work the best over time.”

Absolutely. Right. It's going to become like this self-evolving threat thing.

And then it's going to learn, “Oh well, people in this demographic behave this way, so I'm going to modify it this way. People in this part of the world, you know, threatening someone, calling someone in Cambodia threatening the IRS doesn't work because, well, they're not in the United States.”

It's like we are on the cusp of this situation where computers are improving themselves, and there's lots of ways in which that's good. In the context of, like, the computer being a threat actor. It's bad, but absolutely all the things you're describing are fairly straightforward, It's going to run a bunch of experiments. It's going to see what worked and it's going to double down on the things that are effective in practice, and it's very easy to see how it would orchestrate that and how it would mature what it was doing over time.

Scary. And it could be really subtle things. I know that in the past, Google had run thousands and tens of thousands of small variations in the color of the click link.

Sure, just see what resulted in the highest.

Yeah, no one visibly knew that anything was happening, but they're like, “Hey, we're just going to engineer this thing. If we can get 1% better click-through rates, then we make that much more money,” like the threat actors are going to use the same.

I hope that the average threat actors, sort of intelligence and scientific method is not quite at the level of the people running Google. But unfortunately, yes, I do think that they will have these things, improving themselves.

Yeah, because you can very easily ask AI currently to develop a test platform to make this better over time. I don't have to be a scientist. I don't have to understand the technology.

One of the other facets of AI-driven threat besides the automation and scale is the lowering bar. We talked about criminals having means, motive and opportunity. The problem is that you might not have the means. OK, there may be an opportunity there and you might have the motive, but you just don't have time, space, or energy to develop an actual criminal enterprise.

A part of it is that you don't have to write software or you don't know anybody that does. I think it's incredible that you can, like, get Claude to write software. It opens up a lot of opportunities for people to do creative things, so I think, in general, it's going to be great. Obviously, if you have malicious intent, it is now much easier for you to produce something that kind of works. Obviously, a trained engineer can do more.

It definitely lowers the bar, to getting access to tools that could be used to form for malice. A lot of these AI companies are trying to put guardrails in their models so that the model won't do certain things. There are other models that you can run locally that will do whatever you want to, don't have the guardrails, and again, over time, the sophistication of those, like on laptop models, is going to improve.

We can see where this is going, so yeah, you lower the bar for means, and there's going to be more people engaged in this activity because they have the motive and they have the opportunity.

Once people could run their own email servers, they started spamming. Right.

The internet is still full of spam. Unfortunately, email just—you really have to think about that. Unfortunately, dystopian example like the internet is full of spam. The email really isn't as useful—I mean, I often, like, do direct messaging in other platforms rather than using email, and in part it's because there isn't spam in those direct messaging platforms because you have to approve everybody you're communicating with.

That model of just open, “Here's my email address, anyone can send it an email” ended up becoming a model that, although we still use it, it’s just not as effective for person-to-person communication as we'd hoped, and it's because of all the spam. That's an interesting question to ponder. It's like as this agentic stuff becomes available, are we going to have the same problem with our phones? I'm already annoyed about the amount of text messages and other things that I get on my phone. Is it going to become unreasonable to have a phone number that anybody can call? Do I have to have a phone social network where I have made friends with people before they're allowed to call me, right?

I think that already exists on most of the mobile phone platforms. There’s a setting on my phone that says, “Send everybody who's not in my contact book straight to voicemail.” Therefore, my phone never rings because my friends don't call. I look up and I'm like, “Oh, there's eight new voicemails from people I don't know.”

I think you've early-adopted this capability, and I think maybe that more of us are going to end up needing to use it, right? I mean, it's interesting to consider whether there are other kinds of changes in the way that our civilization is going to work that need to happen and adaptation to the situation that's developing with these agents.

You know what? People will go back, will be motivated to go back to face-to-face interactions with people.

I think the other thing is that we need to, you know, to certify content. Some of the things, some of the work that we do involves, like, you know, files on the internet, like videos on the internet. Is this really the Prime Minister of India or whatever? We can often identify that they're deep fakes. But the other way around, like, if you're a celebrity or a politician, and you're putting content on the internet, maybe you should be authenticating that content in some way, so that we can validate that it's real.

One of the things that is so, again, there are a number of responsible AI companies and one of the things that some of those companies are doing is they're embedding what's called digital watermarks in the videos that they're producing so that we can tell that they're fake. All these efforts that are going on, all sort of hopefully will meet in the middle and make an internet where, you know, we have a better understanding of like, what's real, what's fake, but authorized.

Talking about India. In India, politicians use deep fakes officially to communicate with their constituencies because of the very large number of languages. A politician might use deep fake to generate, like, 20 versions of, like, a speech that they send out to all the different language communities within the area they represent. It's like there are legitimate, authorized deep fakes that are being used in a political context.

Which results in inclusion and understanding.

It's like there are—I focus so much on the threat actor use of these technologies, like the benefits of them can be lost in the shuffle but there’s—it's interesting to think that that there are a bunch of benefits to these things, but it's going to be very hard to tell the difference between an official, authorized deep fake from this politician and unauthorized deep fake from this politician unless we have some other layer there, which authenticates the content that really does represent their point of view, right?

I mean, it seems like the more natural progression, rather than an arms race to debunk video is digitally signed—this is legit.

Yeah. Who's doing it today right and, like, how is it good? How much pain are we going to have to feel before people start doing it at scale? And then our computers. Explain those things to people, like yeah, I know what a digital watermark is, but normal consumers, like, how can you represent the truth of things to them in a way that that they would understand, but also the threat actors can't manipulate? That's a hard problem from a user experience standpoint.

When you're in zone, there's a big blue checkmark.

We actually do that. Our product will give you—it’s a green checkmark that says that we think this person's real, or at least we have no sign that this person isn't real is probably a more precise way of putting it. It may be that we need that sort of thing throughout the internet in the long term.

That's probably the result of where things are going to end up going. We've gone long, and I appreciate your hanging in there with me. If people want to find you online, where can they find you.

I mean, the easiest way to, I guess, I'm often on LinkedIn. My name is Tom Cross, and I work for GetReal Security. If you want to follow what I'm talking about professionally, that's the right place to find me. And you can also visit our blog. I wrote a blog post about how to [inaudible] an AI, and it's a little technical, but it kind of talks about this is that Chinese-language romance scam tool that I was describing.

And sort of talks about how we got a hold of it, and how it works and shows you some screenshots and what we think you know, what's going on with, so that's kind of an interesting post if someone wants to dig deeper.

Cool. We'll make sure to link all of those resources in the show notes. Super appreciate you coming on, Tom, and appreciate the insight that you bring to this.

Well, thanks for having me. This has been a fun discussion.