Site icon Easy Prey Podcast

Website Attacks with John Graham-Cumming and Evan Johnson

Easy Prey

“People often get attacked for no good reason and that is pretty common.” - John Graham Click To Tweet

Website attacks are very common.  They are often not personal, but they can create a mess for website operators.  As website owners and operators, we need website privacy and security especially if we are collecting data and personal information.  

As I was making more money from my business in 2015,  the need for better performance became an issue.  I also needed to be able to keep my site up 24/7 and not have to deal with outages. My website started getting sustained 500Mbps denial service attacks.  I realized very quickly that this was not my expertise, but that is when I began my partnership with Cloudflare.  

Cloudflare provides services that increase the security performance of over 26 million internet properties around the world from individual blogs to governments to Fortune 500 companies.  Cloudflare offers services to accelerate internet applications and mobile experiences, mitigate DDoS attacks, prevent customer data breaches, stop malicious bot abuse, and more.  

Our guests on today’s show are John Graham and Evan Johnson. John Graham is a British software engineer and the current CTO at Cloudflare. Evan is a Product Security Manager at Cloudflare.  

We talk about attacks on websites, distributed denial of service attacks, and how to protect your own website.  If you want to keep your website up and running without skipping a beat, this is a must-listen episode.

“Attackers are scanning to see what is vulnerable, and if they find something vulnerable they will just hack it. It is not that they have particularly chosen you.” - John Graham Click To Tweet

Show Notes:

“Everyone should have a WAF, it is an extra layer that can really, really help. ” -John Graham Click To Tweet

Thanks for joining us on Easy Prey. Be sure to subscribe to our podcast on iTunes and leave a nice review. 

Links and Resources:

Thank you, guys. I appreciate you coming on the podcast today and being able to talk about what Cloudflare offers. I want to give you guys a little background, both the listeners and you guys, of how I ended up ultimately becoming a Cloudflare customer. I run whatismyipaddress.com. I've run that site, a few weeks ago, it'll be 20 years now. Back from the early days of the internet, pre-Google. Back in the AltaVista days.

Over the years, it became a side hustle, kind of a side gig. Ultimately, it became my full-time occupation. As it began earning more and more money, the need for better performance became an issue for me, the need to be able to keep the site up 24/7 and not have to deal with outages became bigger and bigger for me.

One of the things I originally started doing is, back when AWS launched CloudFront, I used them for CDN—which is Content Delivery Network—in order to get my static assets closer to my end users around the world so that it could have better performance on the site. One of the issues I did run across was when Lizard Squad, which was a hacking group, launched a Denial of Service tool that was open and available in the world. I started seeing consistent 50-megabit Denial of Service Attacks against my website. I don't think that it was necessarily intentional—“We don't like Chris. We don't like whatismyipaddress.com.” Just that kind of plain, “Well, I wonder if this service really works and I can take down this site.”

They did a good job. I was on a fairly low-end router, initially. Fifty megabits would saturate the router, the website would go down, my money would go down. I'm like, “Okay, this is not a good answer.”

I worked with a co-location facility. We swapped out the router. I’d see 100-megabit attacks, 200-megabit attacks. The router could absorb it. My servers would get a little bit sloshy and databases would get overwhelmed.

In 2015 or so, I started getting pretty much sustained 500-megabit Denial of Service Attacks. I have a record of 650-megabit Denial of Service Attacks. For anyone, the idea of how people work, most people probably have 100-megabit connection at home on the internet. That's basically, if you're watching a whole bunch of simultaneous 4K streams and it just overwhelms everything from the network hardware to the servers, and all that kind of stuff.

At that point, it was like, okay, I've got to do something different. This is not something I wanted. This is not my expertise—trying to protect against this stuff. I ultimately ended up looking at Cloudflare because you guys have a great offering of services that happen to work really well for me and are able to lower the threshold of what hits my servers. That's my history of how I've been working with Cloudflare.

I want to talk with you guys and find out what some of the risks are that Cloudflare helps website owners protect against. Obviously, I was talking about Denial of Services attacks. What are other kinds of things that website owners should be aware of that they need protection against?

John: I would say something about the Denial of Service thing, which is kind of interesting, which is, you were getting attacked for no good reason. We're just attacking you for the hell of it. That's really pretty common, actually, that people get online and someone just decides to try and knock your fliers—kind of online vandalism—which is part of the reason why we made DDoS mitigation free. We just saw many people get hit by it.

In the big scheme of things, these small attacks, they weren’t going to end up on the news, but they took websites offline. It hurt small businesses, hurt people with local blogs and services like yours. Actually, DDoS is surprisingly common no matter who you are. If you do have something online, get some servers like a Cloudflare type of thing to protect you because it is really important.

Yeah. One of the things that also happened was after I was working with you guys, I actually got several extortion emails basically saying, “Hey, if you don't pay up $50,000 in Bitcoin, we're going to launch a gigabit sustained DDoS Attack against you for the next 30 days.” How fun! “Pay up. By the way, you can't respond to this email.” I was like, “Okay, what do I do?” Called in and said, “Hey, should I worry about this?” The answer is, “No, we'll absorb it.”

John: No, you shouldn't worry about it. Yeah, that's right.

What are some of the other things that we should be worrying about as website operators?

John: Well, Evan, you run some websites and things. What do you worry about?

Evan: Yeah. There's lots of things to worry about. Everything from DDoS top-10 types of vulnerability, basic SQL injections, are still a big problem in the world. Process scripting issues. There's all sorts of basic DDoS top-10 type issues. Logic issues are something that people running complex services with complex backends and lots of business logic, they have to worry about logic flaws which are difficult to spot and you need to be able to find those. As an attacker, it's challenging but usually the impact is quite big.

The example that comes off the top of my head is there's been issues in the past of raised conditions and gift cards because this is the exact type of thing that is a logic issue where you can potentially use a gift card twice if you use it really fast. The impact of that is real dollars and tangible. You have to be careful about that type of thing.

One of the things I am starting to recommend to people—I’m putting my marketing hat on a little bit—is moving more and more of the application to the edge with workers with a product that we offer called the Cloudflare Workers, where you can actually run your application at the edge. It actually has real security benefits too because you have a distributed application. It's not much harder to DDoS. I've been recommending that to a lot of website operators as well.

I assume that also gives you the added benefit of anything that I'm moderating on my own servers is just an activity with you guys as opposed to, “Is this a legitimate user? Should I respond to this query?” I can really limit who I'm interacting with, so to speak.

Evan: The big benefit is not every application is a perfect use-case for Cloudflare Workers today. Things that are super high output, lots of requests, What Is My IP Address could be very good without too much knowledge of the backend going on. What Is My IP Address-type of website can be a good Workers candidate. The big benefit there is there's no backend server to overwhelm. It just moves the application to all of our servers.

That's a really neat solution to keep the hosting cost significantly down, the CPU cycles down, substantially. That's one of the things that can hit a lot of companies—just maintaining everything that has to be there to keep sites up and running.

John: Especially maintaining it for peak time. I thought one of the issues is how you scale. You suddenly get popular. Somebody mentions What Is My IP Address on Reddit, suddenly you've got an influx of traffic. We have a huge amount of capacity globally. It was quite easy for us to scale something like that and deal with it. It's great to have something scalable like that and not worry about your Elastic Load Balancer or how much you're going to pay Amazon to run the damn thing now that you’ve become successful. Our goal is not to extract the extra money from you.

Yeah. I've definitely had issues where I've been mentioned in the news in the early days when I was running—silly me, running it out of my house. “Oh gosh, the servers just can't keep up with the traffic.” It just picks up—totally feasible—but being able to hand that off to you guys would definitely relieve that stress.

I know you're talking about SQL injection. That's a pretty common issue that anyone who has a public-facing database application has to deal with. How much protection do you guys offer for SQL injection? I don't know how you necessarily quantify protection against SQL injection. Are you able to strip out the vast majority of stuff that might hit the servers?

John: Yeah, I think so. The WAF, this is what does that, which is the Cloudflare WAF. It has a whole bunch of rules that we use. The OWASP CRS, the standard rules for doing this stuff. The really big thing is, unless somebody has really got in it, decided they're going to attack you because they decided you're the enemy in some way, what they're going to be doing is using automated tools to scan. What you really want to do is to limit the ability of that scanner to even see something and then they won't bother; they'll just move on. A lot of what's happening with attacks on the internet is there's this background radiation of people just scanning to see what's vulnerable. If they find something vulnerable, they'll hack it. It's not that they particularly chose you, it's just that you happened to not patch something or have not done or prepared statements in SQL, all that kind of stuff.

I think you can protect a lot with a WAF because you just eliminate this huge wave that people just go around scanning for vulnerable applications. If something really comes after, then hopefully we can protect against that too. But then you're talking about a really persistent attacker who's going to try and find the soft underbelly of your application and find a way in.

Yeah. Anyone who has a database should always prepare for—I don't want to say eventually—but the contingency of if you do get your database breached, what are your legal obligations in dealing with that? For me, it's just my internal data, logs, and stuff like that. I'm not putting a heap of data. I'm not putting medical records at risk, I'm not putting bank account numbers at risk. If someone is storing more personally identifiable information, they really need to be able to have contingency of. If my database does get compromised, one, how can I recover it? Two, what are my legal obligations, notification, and things like that?

Evan: One of the big things that the WAF, to mention, is if you're a security team, I come from the mindset of I'm a part of the security team. A lot of times we'll discover security issues in our systems. The WAF is great if we discover, or somebody using our WAF discovers, the SQL injections in their website, it may go to one of their engineers. The engineer says it's going to take me three weeks to fix this. Instead of having to wait that three weeks, the security team can use the WAF and make a custom rule with our fire rolling product to block something really quickly. That's really one of the other big benefits of the WAF because sometimes you discover an issue and you want to mitigate it instantly. Three weeks doesn't cut it.

Yeah. That's definitely something I have seen on my site. There's something I can fairly easily identify, not necessarily as a database intrusion attempt or things like that. I don't initially want to publicly disclose what some of the attacks are that I've seen.

I've been really amazed that using your API to update a rule, or going to the website and creating the rule there, I can create a rule to protect myself against something that I'm seeing, and it's live within seconds. I don't have to be like, “Okay, let me put this in the support queue. Let me reach out to an engineering team.” Whether it's an internal team or an external team, that has to go through 60 levels of approval and it's really kind of neat that like, “Hey, I'm seeing this issue. I can identify it. I can do something and have it blocked almost immediately.”

John: Right, that's one of our key Intel technologies. I have this thing called Quicksilver, which is our distributed key value store globally. It's there to make those changes really fast for you. If you make a configuration change, a rule change out of DNS record, we should be able to push it out to the entire world. That usually ends about half-a-second globally. It's really, really fast. Yes, you can do that kind of stuff.

The other thing I would say about the WAF that's really important is that one of the things we observed is that when a new CVE comes out with some vulnerability, especially if it's in a web-based thing, attackers are going to start scanning for that very quickly. Very quickly is hours to single-digit days. It's incorporated into the latest scanning tool. It's actually crazy. It's really a big problem for people because everyone knows they should patch. Even the best organizations, which are like, “Yeah, we're going to patch the schedule. We're going to patch.” It's really hard to patch for us enough. That's why having a WAF can give you some breathing room to actually go and patch the backend systems—it’s really valuable. In general, WAF is something that everybody should have. One, it's an extra layer, which can really, really help.

For people who are not technical, WAF is a Web Application Firewall. It's a rule set that you can put in place to protect against known factors.

Evan: Yeah. Almost like antivirus for your website where known signatures are bad.

I assume that's really important with so many people. A third of the websites out there are running on WordPress. If there's a core function of WordPress that has a vulnerability, for me, for anyone who's running their little home website, they might be checking for plugin updates, and WordPress updates once a week. Maybe if they're lucky, every couple of days. Whereas I assumed that with you guys, it’s soon as you're aware of the vulnerability.

John: It applies to WordPress. It applies all over the place. Any piece of software you're using, it's likely going to be updated. Your iPhone gets updates. There's going to be updates all over the place. Yeah, WAF is really helpful for applications no matter what the technology.

Kind of an interesting question: What are some risks that websites face that you guys don't protect against? I'm not trying to put you on the spot and make you look bad. I have a couple of things that I'm thinking of. What are some of the things that Cloudflare or one of your competitors just can't protect against?

John: We don't protect against email. We don't look at stuff that comes in your email. If you click on a bad link in your personal email, I'm sorry, we can't protect you from that. That's a very different WAF. We're fundamentally protecting stuff as it comes over the web. In most instances, those going to websites or APIs and other big things.

We don't specifically protect against ransomware. You mentioned somebody saying, “Hey, you’ve got to pay us all this kinds of money.” Obviously, we can't protect against the consequences with someone's deeds or yours or something like that. But if you're thinking about malware in your computer in your office, that's slightly different from our business. If you connect to the public internet or you provide a service or a website, yeah, we can protect that.

Yeah. One of the things I was specifically thinking of was poor password management. If you've got a WordPress site and you've got half-a-dozen people who have access to it, you guys can't really protect against people who have poor password management where they've used Password123 halfway around the planet and then someone comes in and just logs into Bob's account who has administrator access.

John: Yeah. If your password is that bad in a password dump, then yes. If somebody comes in and just logs straight in, it's not something we're going to directly protect against. We do protect against the kind of credential-stuffing attacks we see where people are trying out a long list of usernames and passwords to break into stuff. We have a bot management, which helps a lot with that kind of thing—that sort of stuff.

Do use a good password and do have two-factor authentication everywhere because that stuff does help.

Yeah. What I really wanted to mention was there is no panacea, one product, one service, that's going to protect you across everything that you ever do in your life. You still need to practice common sense and do things on your own, not assume that I’ve got this service. They'll take care of everything. I don't have to worry about anything in my life.

John: Yeah. I suspect that if you have an alarm in your house, you do lock the front door. Yes, it's definitely something you should do.

Yeah, absolutely. What are some of the other benefits that Cloudflare can offer website owners in addition to the security features that we've already talked about?

John: Hopefully, we'll make it faster. We offer this thing called the CDN cache where we take the content that is static on our website and we move it to 200 cities all over the world. Wherever people are visiting a website, you should be able to get it from somewhere really close. The reason that matters is that the speed of light is fixed, unfortunately. If you go a long way, the speed of light really messes you up.

I was in Australia last week. Australia's really far away. Some websites felt really laggy. It was not because the internet is bad in Australia, it's just because it's a long way away. Moving things closer really matters. Caching CDN stuff is really, really, important.

Evan was just talking about Cloudflare Workers. Evan, tell us more about Cloudflare Workers so people understand what that's all about compared to caching.

Evan: Oh, yeah. I'm a huge fan of Workers. It's a way to execute code at the edge, at our edge, and our 200 points of presence around the globe. You can do caching with Cloudflare Workers. I believe there's an API that caches things at our edge. You can also store objects and store information at our edge as well. It's really great for heavy operations and moving an application out closest to where people are. I've built my personal website on it. I thought it was a great developer experience. We're actually using, pretty heavily, our security team right now to do some basic processing and some basic tooling to help all our developers at Cloudflare shift more secure code.

We actually use Workers to look at all the code that our developers are writing and leaving helpful comments on their PRs if we see it on their phone request telling them if there's anything we think can be improved about it. It's super flexible and very easy to write since you write it in JavaScript.

That's really cool. One of the things that I was noticing, I actually often caution people against, or tell them to look at extra carefully, is when companies are offering services for free. You have to say if you're not paying for this service, are you the service? I know that Cloudflare offers free DDoS protection service. How can you guys provide that service for free to users?

John: Your point is a good one, which is if a company's offering is free, then they're all making money in some other way. The important thing with Cloudflare is we actually offer paid service. We have a very, very large number of paying customers who are paying for this whole thing. If you're using our free service, then you're not the product there. You're being supported by the people who are paying for the service.

The reason you're interesting to us is many of those small customers paying us nothing end up being paying customers. They get used to the product. They love it. They say, “Right, there's extra services in Cloudflare that are not in the free tier and I'm going to start paying for them.” It might be Workers—$5 a month on Workers. It might be, “I really want the WAF, so I'm going to start paying for a pro plan.” We're a freemium model. We have a huge number of paying customers all the way from $5 a month up to very, very, very big household names. You don't need to be afraid of that.

The other side to think about is just following the money. Our customers are people like you. They have a website. You have a business on the web. The data that passes through that is your visitors’, is your data. We don't think of that as our data to exploit, turn into ads, or sell to some shady people. We think of that as your data in our care.

What we would do with it is we will analyze it so that we can provide you with analytics like you've seen in your dashboards—how many visits I have, that kind of stuff. We'll look for attacks. We use it for the cybersecurity stuff to say what new attack types are coming. Is the website under attack? Should we limit him? All that kind of stuff.

Fundamentally, we're just not in the business of, “We'll do something creepy with the data.” That would actually destroy the business we have, which is very successful. We went public last year. We'd actually shoot ourselves in the foot or maybe in the head if we started doing something creepy with people's data.

Yeah, I appreciate that. That's how it works. You're able to offer a service for free to people who may very well not be in the position where they'll be able to afford it. Probably very few people could afford the enterprise service. Corporations can afford it, but people who are just trying to figure out, “How do I have a web presence? How do I get myself out there? How do I start marketing my products and my services?” Them having a free solution is a great opportunity to get their foot in the door, have a little bit of peace of mind knowing that even if I'm not super fast at patching stuff, Cloudflare's got my back. Not that I don't have to patch.

John: Right, that's right. Doing it inexpensively—either free or one of our cheap plans—is a very good way to get a level of protection. Frankly, our DDoS protection was something that was only available to giant companies. We give it away for free because we built a network and a service that's capable of doing that. We have this mission, which is to help build a better internet, which sounds a bit hokey sometimes, but we really believe in it. We think, “Why was it okay that you were getting DDoS’d?” It wasn't okay. You shouldn't have to deal with that. You could not have gone to one of the big DDoS providers and paid them, it would've just bankrupted you. We really changed that by the way in which we built our service. No need to be paying fortune for this stuff. Come and try it out.

Yup. I can attest that their servers have been a total lifesaver for me that would've been a sleepless night here and there. It's definitely nice where I've got a good night’s sleep knowing that I don't have to worry about this stuff.

I guess there was one question of best practices. If someone has not been a Cloudflare customer, for whatever reasons, they're actually an intentional target of Denial of Service Attacks, or things like that, is there a best practice in transitioning to Cloudflare in that if I don't move my backend servers, they're still out there? They're still potentially vulnerable just because what's publicly facing is now behind Cloudflare? Is there a best practice for it? If there is a pattern of ongoing attacks, should you move your backend where you're hosting your website at the same time you move your Cloudflare?

John: Evan, you deal with all of this stuff.

Evan: Yeah. If they're that persistent, then totally that's a good idea. Also, best practices, I think you mentioned the first one earlier. I would make sure that you have a strong password in your account and two-factor authentication, because Cloudflare controls your DNS records and it controls so much about your website. It's very important—good hygiene with your Cloudflare account.

With your actual backend servers, you want to make sure that you orange-cloud things. By default, we refer to it as orange-clouding, internally, where by default, you don't have to worry about this as you onboard. You want to make sure that Cloudflare's actually proxying the request. We refer to orange-clouding as Cloudflare seeing the requests that are being proxied through our network. It's configurable on our dashboard and the DNS dashboard. Either orange-clouding, or we refer to it as gray-clouding, if the requests aren’t passing through a network.

We have features to make sure that the security between our edge and your servers is secure. Authenticated origin polls is a great way to make sure that the requests that are reaching your servers are coming from Cloudflare. We offer IPs on cloudflare.com/ips for you to put in your IP firewalls to make sure that the IP packets are only coming from Cloudflare. It really is a spectrum. We offer a bunch of different security services. Maybe they stopped on the onboard in the Cloudflare but you can take it as far as you want and be as maximal with your security as well. I believe now, with our newest product, Magic Transit, if you have IP space, we can take over that IP space and become your network.

Yeah. What's the Magic Transit product? I've not heard of this before.

John: Instead of you advertising your IP space, whatever IPs you have in your company, we will do it for you. The traffic will come to us. We can do the deed, the scrubbing, and all the other services that we provide. Then, we can pass the good traffic back home to you through some tunneling process, typically GRE. That's actually really good for larger companies. Just my own large block of IPs, they get a lot of different sorts of attacks against all sorts of infrastructure, and rather than just saying, “Hey, I just want to protect my website, my APIs,” or something, it's like, “Can you just protect everything?” Particularly for DDoS, for example, we can do that for them. That can be used as an always-on kind of thing. We’re always there. Or it can be, if I’m in trouble, they pull the big handle on the wall. Suddenly, we turn into them. That really allows you to use an existing IP infrastructure. Sometimes that's really important just from the policy perspective. They want to use their own IPs and only use Cloudflare IPs. Sometimes, it's important for legacy reasons where they might actually have this hardcoded for specific IPs. It's hard for them to change. Lots of reasons, and that’s been a very, very successful product.

I'll get really geeky because the network stuff really interests me. Does that mean you're basically acting as the ASN or the BGP and announcing the IP space for it?

John: That's right, that's right. We announce space. Traffic comes to us. We typically have GRE tunnels set back up to the real servers or real network to actually pass the good traffic through.

Cool; that sounds really neat. It's really neat that you offer everything from—I don't want to say vanilla services—very end user, to everyday manned services all the way to the stuff that is extremely technical, way above my pay grade. It's not just one product fits all. It's not like you have to like everything or nothing. You have this wide stream of services that makes sense for everybody.

John: Yeah. I would say we definitely do that. One of our key things is we try to take things that are complex and expensive, and make them easy-to-use and cheap, so that everybody gets access to all these really cool tools. That will provide things that the largest companies in the world want. You'll find many of those features in our pay-as-you-go product. No matter who you are, you should be getting access to this stuff. That's really what the whole democratizing thing is all about.

Yeah. I noticed, I think it was 2018, that you guys launched the 1.1 as a DNS and VPN service. Has that really taken off for you guys?

John: Yeah. It's absolutely huge, yes. The one that was resolved was a public DNS resolver, it’s the fastest there is, independently measured, not by us. Partly, because of the scope of our network, 200 cities worldwide. We can provide DNS wherever in the world.

Last year, we launched what we called WAF, which is our unVPN. It's a way of securing the connection from your mobile phone to the Cloudflare network. It doesn't matter where you're connecting to the internet, to some random coffee shop somewhere, guest WiFi at a company, at home, over 4G, whatever. We can secure that connection, make sure everything is being handled securely and make you safer. Those things are taken off, they're absolutely huge.

Definitely. Privacy and security have very much come into people's awareness over the last few years. That desire of, “I want to make sure no one's snooping on me, no one's in between me and the website I'm visiting doing weird things to my traffic.”

John: Yeah, absolutely. I think people really started to realize that using the internet for absolutely everything—from banking to dating—it really matters that they protect that and use things that are trustworthy. It's just become as much as we would lock our front door and close the windows when we go out when we're thinking, “Am I using secured websites? Is everything I'm doing secure and safe?”

Yeah. Do you have any other last bit of advice for website operators before we sign off today?

John: I always tell people, this is not necessarily just website owners but for everybody. If there's one password and two-factor you're going to use, it is on your personal email. If I can break into your personal email, I can probably reset the password on every other service you use under the sun. Please, if you're sitting, listening and thinking, “Yes, I know, I'm using my wife's maiden name as my password on my personal email.” Do not. Get up, go to your computer and fix that right now. This is such a huge risk. People don't think about it. Make sure it's unique, use two-factor auth, secure that first then you can work on it from there.

Yeah. I've got lots of people coming to me saying, “Hey, I've lost access to my email account. What do I do?” It gets very bad very quickly if that's the email address you use for your banking, your retirement accounts, and all that kind of stuff. It can go south very, very quickly, unfortunately. That is absolutely great advice.

John, Evan, thank you so much for coming on the podcast today. I really appreciate your time.

Evan: Thank you.

John: That was great. Thanks for having us. It's nice to chat about these things. Thanks for being a customer and that it works out so well for you. It's funny, I think I vaguely knew you're a customer, but I must've used your service hundreds of times. Thousands, maybe.

Yeah. That is one of the neat things about running a website. I'll be at an event somewhere, at some random thing. Someone will ask, “What do you do?” “I run a website.” “What website?” “It's really techy; you probably haven’t heard of it.” “No, no. What is it?” “Oh, it's whatismyipaddress.com.” “Oh! You're the whatismyipaddress.com guy? Wow! That's amazing.” I'm like, “This is kind of weird.”

Evan: I think everybody's been to that website in the world right now.

Exit mobile version