Effects, Indicators, and Mitigation of a DDoS Attack

Name: Marc Kranat – Title: Enterprise Firewall Support Supervisor

Marc Kranat:Hi, my name’s Marc Kranat. I’ve worked with the Sucuri Firewall for over five years now. I’m a security and privacy evangelist. I am a father of two, and I also am a part-time photographer’s assistant, as my wife is a photographer, and I’ll pass you over to Norton to introduce himself.

Northon Torga:Hello, everyone, my name is Northon. I’ve worked on Sucuri Firewall for about two years and a half now. I’m in Security Engineering Troubleshooting, and I work web reviews. I’m a Netflix and connoisseur coffee addict and I’m a passionate traveler. I love to travel to my country, which is Brazil.

Marc Kranat:Well first thing we need to know is how damaging DoS attacks are, and how cheaply they can be launched, the sources of where you can expect them to originate from, and who the targets are. For about 20 dollars, you can take down a dozen smaller unprotected sites on and off over a month period, paying for the service with PayPal or anonymously with Bitcoin. These attacks are mostly waged from insanely vulnerable IoT devices that make up the internet of things. These would include, security cameras, household appliances, such as smart TVs, remotes home lighting systems, refrigerators… Yup an army of fridges can be made to take down the website, and they are. Last month, over half a million in Bitcoin was recovered by law enforcement in the U.S. The operator with DoS for hire service pled guilty to launching over four million DoS campaigns, over a two year period.

Marc Kranat:The growth over time of the quantity and sparsity of DoS attacks is grown exponentially. Mostly due to the total lack of regulatory control of the internet of things. Their lack of security controls make them excellent recruits for the botnets. Controlled highjack group of devices with unique IP addresses, which can be redirected to make malicious requests against websites, and cause a DoS attack. According to the internet security company, Kaspersky, financial cost of DoS attacks to medium sized businesses were over 100 thousand U.S. dollars each. The cost, currently, growing at 20 percent a year. The cost represent, both, direct cost of the DoS, which lead to loss of business and opportunity but also loss of reputation among clients and partners. Victims can include anyone with a website or web application. Some of the more high profile cases have been DoS attacks for ransom or blackmail. These are straight up demands for payment to make some change in their business, or make a payment to stop an attack. Occasionally, a DoS attack is launched as cover, providing noise while a malicious hacker has some access to backend systems.

Marc Kranat:A couple years ago, I saw a combined DoS attack on a company’s website and office email server, which shared internet access with their voice over IP systems. The hacker also had remote to the Financial Director’s laptop, using some previously installed Trojan, which he was able to install using some fishing campaign. The bad guys were able to transfer huge sums to a foreign bank account, while the bank tried to contact them to question the transfers, but due to the DoS it was too late. They never got through in time. Attacks can be political, hacktivist, terrorist, and others of course, business competitors, or anyone with a financial or ideological motive to do damage to an organization for its own sake or the attackers benefit. Even video gaming leagues attack each others forums, taking the gaming outside the Xbox. And kids who have access to scripts, which can do damage, and often are just messing around rather than employing targeted or purposeful attack. These are often very short lasting, while a personal DoS attack from a state actor can continue for years against a specific target, even when fully mitigated by a firewall’s protection. A couple of years ago, I saw three, small, separately owned websites for a particular niche market, local businesses in a small town, all come under DoS attack at the same time. One of them was able to identify the attacker, while we’re on board a new entry to their market, who was also messing around with them on social media, being the likely culprit. This is very common. The purpose of a DoS or distributed denial of service attack can be to fully take down of a target or the slow draining of the resources of the target. Both in terms of resources, as well as financial. The target may feel rushed into a more expensive hosting plan.

Marc Kranat:Sometimes the attack can be disguised to look like a heavy load of real traffic to mount a malicious attack. The target will upgrade from say, a typical 30 dollar a month plan on a shared server to a dedicated server and all within the space of a year. The target assumed it was real traffic, but the cost of hosting grew to 800 dollars a month slowly over that year. That’s a cost increase of over 2,000 percent. In this particular case, the hosting company just did not understand how the content management system work, didn’t recognize the DoS, and assumed it was real traffic. Now the client has proper mitigation, downgrading to a sensible plan, and that host now brings us new clients for our WAF value regularly. Sucuri’s intrusion perpetuance in IDS, which is enabled by default, can detect and mitigate classic malicious application layer attacks without any interaction. The attacks being completely unnoticed by the intended victim. Slower attacks, the attacker in these cases were trying, and tie up as many networks sessions open currently. There by, not allowing new sessions to be open by genuine site visitors, leading to a complete denial of service. These attacks should be brought by default of the network layer. Again, a target will not see or notice nay ill effect if using Sucuri. If the site is unprotected, then the applications serve itself will not show any health issues, but net stat will show a huge number of open session in those cases. More common though, as we’re going to demonstrate, our application food attacks. There are a few variations of these, but generally it will mean that a malicious act will crawl the site, making repeated requests on any valid pages. This is how valid low tester works… would work. But the request come from a large number of bots and zombies. Good page caching helps deal with this sort of attack. Very well, but it will always affect server resources.

Marc Kranat:More problematic, or what we call 404 attacks. These are where the requests are made for pages do not actually exist in this site at all. Valid quest your site would only require a data based table to be partially served. In the case of a request for a nonexistent page, or none existent resource, some random string maybe, the server would have had to search the entire database table each time a request is made. Not only does that take more resources, but the deliver page, a new request resulting in 404 error. That could not have been served by cache, so the drain on the application means sources is that much higher than a typical mass request attack on valid pages. The result in the end is the site slowing down in response to valid requests and eventually responding with a 500 internal server. It is possible, at this point, that the application server, in this case, Apache will fail, lock up requiring the service to be restarted to bring it back online. We can see here that the DoS attack is taking place by using combination of the tail and H [inaudible 00:08:58] commands, and a couple of terminal windows. I’ll switch over to them later. Over to you Northon. Do your worst.

Northon Torga:Today we’re going to perform a DoS attack, which means eventually it’s just a single machine to attack another. But remember, we are not the bad guys. We are the good guys, so we have authorization for a bata center to perform such and attack. We’re going to be using [inaudible 00:09:25], so no website is going to be harmed. I’m going to show you a script that I wrote, actually not this script itself, but the window, so you can see the attack is starting. Just share my screen right now. I’m going to be launching the attack right now. Marc is going to be sharing his screen, so you can see the attacks happening. I’m going to divide the attack in three levels. The first level we should just slow down the website. With the second level, we’re going to slow down the website a bit more. And then we’re going to actually take the website down for a couple seconds, and you’ll see on my screen the actual resources page of the attack happening. Marc, if you could please share your screen.

Marc Kranat:I’ll switch back to my scene. You can see that the zero load on this launch, the tail of the access log. You can see there’s no traffic there currently. There’s currently half a gigabyte of two gigabyte being used. We can see there’s zero average load here.

Northon Torga:Remember this machine is proper VPS with dedicated resources, which is not actually the majority of the market, because most websites are hosting or sharing an environment, and does not have enough resource to spare. Here, we show you an unprotected machine, with no firewall or security software in place. The machine is going to receive an attack now directly on a level one, and I’m starting the attack right now. And look at the screen that Marc is sharing with you, you can see that CPU usage is going to increase and we should see some requests going to the access log. As you can see it’s a surge attack, we just searching for random streams on the watchpress and the CPU usage it’s already at 100% we see that the load average there was before zero, now it’s reaching five, which means that the key of focusing is already five times the amount of resources this server has to process. The website, still is available you can see that on the log, you see that 200 response code which means the website is available. But it’s going to take more time to open than usual. That kind of … that is usually used to deplete the resources used on the server and maybe affect the experience of the people visiting the websites. If you have, for example, an I-commerce site, people may give up on purchasing your products because the website is low not actually answering in expected time. The first level of attack should not stop the machine, should not cause any other issues, just slow the website a bit. You can see on the left side of the window that the MyScale which is the database, is pretty much using the entire CPU to perform the key numbers and to look for the search for the random strings that I’m currently sending.

Valentin Vesa:And this is still level one Northon?

Northon Torga:Yeah exactly. This is just level one, the unprotected machine is not actually been taken down, it’s just slowing a lot, and hurting the visitor experience.

Marc Kranat:Your host may not want you around very long if you have that sort of load on it.

Northon Torga:Exactly. Unless you have a proper VPS or dedicated machine you shouldn’t be able to handle that kind of traffic. We are able to do that right now because we are using a canon network data center. With a good machine will have two gigabytes of RAM, I’ll have a dedicated CPU there, but yeah, the machine has an incredible load, it’s 42 load average by now the CPUs age is just too much. Marc if you can tell us about the screen that you see on the left side, below average, before we were seeing zero right?

Marc Kranat:Yes.

Northon Torga:Now it’s 100.

Marc Kranat:Yes, now we’re at 100. I’ve been to the data center myself, I’ve installed a hardware firewall on a machine that was under a very similar attack a DL360 joules I think. It literally sounded like a helicopter was taking off, the fan starts spinning up, the hard drive is busy writing logs of the attack and the memory’s hitting that and the data center said.

Northon Torga:How are we going to … I’m just going to stop you Marc, just for a moment I’m going to show you the actual screen that I’m using here. You can see that the loading time of the website is reading 15 seconds. That’s the time stamp of the attack so you can see that we are reaching almost 10 – 15 requests per second, the website is still returning 200 so it is still available … taken a bit more than usual, I’m going to stop the attack now, I’m going to launch the second level. Of the attack as I said, I’m using that as more script so I just need to change to level two and in the level two it is not going to take down the website yet, but the load time is going to be much, much greater than it was before. Marc can you show your screen again please?

Marc Kranat:Yes, I can.

Northon Torga:The attack is starting so it should be starting to deplete the server resource more than you were before. We were seeing 46 on load of average, this should raise now with the attack which has more concurrency and more requests for second. Now it’s increasing as you can see we’re reaching 53, 57. The website is still available as you can see it’s still returning 200. Remember this is just a simple attack, you would just pretending to be an attack. We are not actually attacking, we are just simulating an attack on our simple website. The website is an unprotected machine running WordPress, with pretty much no helpful keys there, just some dumped content so we can actually have something on the database. And yet, the load average is already reaching 73. Imagine if this is a WordPress with pretty much nothing there on a dedicated environment, unlinked internal network, imagine if the attack was before on the sharing hosting with very limited resources and no defenses there and lots of content, for example we have a website with 50 plugins, lots of content, MyScale is not actually optimizing. On this machine that Marc is showing you, it has a special software to restart Apache when it crashes so we can see the 200 gap on the second level. But you can see that the machine is already reaching 84 off load average which is just too much. I’m going to change to my screen, so I can see the load time. Before you were seeing 14, 15 load time now it’s reaching 30, so the website is taking twice as much as it was before. The response though is still returning 200, so the website is still available but now taking around 30 second to load. According to research, usually the visitor won’t want to wait more than three or five seconds for the website to load. If the website is taking 30 seconds to load right now you should be not receiving that much valid visitors because we are give up on the website, it’s just taking too much to load. And remember that’s just the second level of attack on a single machine on internal network so it is very, very simple not sophisticated attack using just a single machine. Imagine, if like Marc said in the beginning, if you were using group net of internet of things, an army of smart TVs and grids attacking a website with multiples like fees, address, lots of networks involved. It would be much harder than we are seeing right now. The time is going to be staying on this 30 seconds of load time. That’s because the attack we are using now, the level attack we are using now, it’s just performing very limited process per second and I’m going to be stopping the attack now and I’m going to be launching the last level of attack against an unprotected machine. On the third level of attack it’s going to be different. Marc can you share your screen again please?

Marc Kranat:Yes.

Northon Torga:In about 30 seconds we should start to see 500 response code, which means the server is not being able to perform the attack and is just timing out, it’s giving up, it can’t process.

Marc Kranat:We’re into the 500s now.

Northon Torga:Screen of the second machine? Please?

Marc Kranat:This machine is half the memory resources. Just that we won’t need it, and … I’ll get the launch ready from the firewalls.

Northon Torga:Yep. Just so that we can show you that the machine has absolutely no load. Yep. That’s the second machine.

Valentin Vesa:For full disclosure, this is a Sucuri WAF protected machine.

Marc Kranat:We’ll set up about half the memory resources.

Northon Torga:Yep. You can see that there is no load there. The access logs shows no requests coming there. So now I’m going to be launching the attack against the machine. I’m going to change my green screen again. And I’m going launch now. We should see a load time. Read it fast because the firewall will actually handle this kind of requests, so the website’s holding requests. And with which the 100 requests there, where the firewall already detected the attack and stopped it right away. You can see I was returning 200 and the loading time was pretty good, but by now I shouldn’t be able to actually connect to the website anymore because the firewall may have already detected the attack and stopped it right away. Marc, can you show your screen so we can see the second machine, which is receiving the attack that’s supposed to be receiving the attack now? And … I see it already. 989 requests of attack and on Marc’s screen you cannot see any of these requests because the firewall already blocked me.

Marc Kranat:I can see them in the logs here.

Northon Torga:That’s the logs of the superior firewall.

Marc Kranat:This is the logs of the firewall. The DoS protection’s picked it up, and what it will have done now is it will have blocklisted that IP address … across the entire network. But then not even one request should have got through. It would have been recognized as malicious as we see here … it’s-

Northon Torga:So the machine has absolutely no load, so with the protection and half the resources, the machine is still running fine. The attack’s not going through.

Marc Kranat:The client, unless they check their firewall, will not … they would never have known of the attack.

Northon Torga:Exactly. The protection is automatic. The IDS detects the bad traffic, and the WAF stops it right there. The machine didn’t suffer anything. I can show you my screen now, so you can see-

Marc Kranat:Did you launch a level one or layer three? I doubt it makes to level…

Northon Torga:Because look at my screen now. We are receiving zero-zero response stop, which means we are not able to connect because the firewall blocking us.

Marc Kranat:You’ve been blocklisted from the entire network now.

Northon Torga:Exactly.

Marc Kranat:When we see one attack, we plan on them from everyone.

Northon Torga:Exactly. So … yeah, it wouldn’t matter because if I’m trying to do a second level right now, I’m already [inaudible 00:25:11] better because the firewall can block a small attack, a big attack, or whatever the kind of attack, or the size of the attack. The firewall would be able to handle most of the time … I mean, I work on the support for a couple of years, and I barely see one or two cases the firewall was not able to handle some kind of attack, but was really some kind of an application, like, the application attack [inaudible 00:25:38] or something like that. The time that I was in support, I can assure you the firewall can handle that kind of attack really easily.

Marc Kranat: We have solved pretty much sites that don’t use any other DoS mitigation and caching because our caching will catch, will serve the requests. It’s actually using the resources of the attacker. We would … none of that would have happened. Well, now they’re being blocked, they’re not seeing the network, but if we could serve at 200, I’d be a lot happier, then they can just waste their resources just attacking the systems.

Northon Torga:Exactly. So this is pretty much was a DsS attack looks like. Now I’m passing to Val.

Valentin Vesa:That was actually an extraordinary example, live example … live exercise, if I may say. I’m already seeing from Heidi Kopec, she had to leave the webinar, but she said, “Thank you so much for an excellent example of a DDoS attack.” So I’m assuming we’ve hit that proposed target of actually showing how it happens live. We have some questions. We don’t have much time, but I’ll still take at least two questions live. The first one that actually came on is from Ryan, and he’s asking, “Could you tell us what tools you are using to see these logs?” I’m assuming he means Marc’s machine.

Marc Kranat:Yes, I’m using a tail just to grab the end of the log and htop to see the request. I mean tail you’ll see the patterns. It just doesn’t look like real traffic. It’s obviously some random characters. It happens that WordPress actually resolves at a 200 rather than a 404 because when you put in an odd page, it’ll try and find the page for you, and it considers that 200.

Valentin Vesa:What we can also do, Ryan, is on the after-webinar page, the same page you where you registered to see the webinar, we will have most likely next week … we’ll have the recording and also we’ll have the resources side where you can see links to whatever tools we used. So I’m going to ask Marc and Northon to share those with us, and we’re going to add them to the actual webinar page just so you can have full disclosure on what tools were used, maybe even what the specs were on the machine and so on, so you can see how that could be or not. I’m really sorry we only have time for live … for another one question, but all of the other questions you guys send, we will be answering them, so I’m going to grab them all and send them to Marc and Northon. And we’ll have a Q&A sheet on the page as well, next to the webinar recording. I’m going to read this last question. Whoever wants to take it please, or maybe you want to both answer. “What’s the difference between …” I think that’s one of the main questions that came on because it was addressed by four different individuals, so “What is the difference between a DDoS and a DoS attack?” So double D, one D.

Northon Torga:I can answer that. A DoS attack usually comes from a single attacker, so in this example, we used just a single machine. So we were launching a DoS attack. When you have a DDoS attack, which is a distributed [inaudible 00:29:07] attacks, it usually means you would launch an attack from different attackers. It’s usually when you have, for example, botnet and you have lots of hired websites and maybe somebody with these devices, et cetera. And you use them to access a website, access an application just to take it down or slow it down. So DoS attack is the simplest kind, which is use just a single attacker. And a DDoS uses multiple machines to attack.

Valentin Vesa:Yeah, sorry. I was thinking, but there was no microphone on. Okay. Questions keep coming in, but we are really out of time, but, again, we will be answering all of the questions that came on. Also we’re going to link in the description of the webinar to our web page or web application description. If you guys want to reach out to us, please do this … you can continue to do this, and you can reach Marc or Northon directly. You can see on the screen usernames on Twitter. You can follow them, you can ask them. They love to answer questions. And if you guys like this, we’ll invite you to come back maybe to a next time when we’ll do something more interesting or something very different. But both of them love being in the webinars, so most likely we’re gonna see them again. Thank you, Marc. Thank you, Northon. By the way, we are joining you guys from three different countries, launching this would-be attack in a fourth country. So, just so you know, we are very distributed and we can be everywhere and all the places to protect you. I’m gonna let you guys say your goodbyes, and then we’re going to close.

Marc Kranat:Thank you for attending. And any questions, let me know, please.

Northon Torga:Thank you for attending as well. And please keep tuned on the blog as well. Our usual posts are good articles there, and especially talking about DDoS. I recently launched a video series instructing you how to improve your application to handle more traffic and especially that kind of videos.

Valentin Vesa:Thank you, everyone. And just the last comment, Northon, everybody mentioned your shades, so people like them. Have a great day everyone. Have a great afternoon, great night, whatever your local time is, and we’ll see you next time. Goodbye.