Enterprise Firewall Support Supervisor
Working in IT security and project management 20 years, 5 years with the Sucuri Firewall. Marc is a dedicated privacy and security advocate. Occasional Husky wrangler and part time photographers assistant. Personal website at 300m.com.
WAF/Monitoring/Backups – Support
Northon Torga is a *nix system engineer, member of the Sucuri Firewall team. You can find his personal website at ntorga.com.
Question #1: What’s the difference between a DDoS and DoS attack?
Answer: The extra D in DDoS stands for “distributed’ so a “Distributed Denial of Service Attack”. The effect of an attack is the same, but normally the host would identify such bad behavior generated from a single IP address far more easily. In the case of a DDoS attack, the multiple attackers are exploited machines/websites/IoT that are not even aware they are being leveraged for an attack.
Question #2: Could you tell us what tools you are using to see these logs?
Answer: To view the logs in real time, we used the native GNU command called “tail” with flag “-f” in a terminal window, as we are accessing remotely within an SSH session. This shows us in real time new entries on the Apache access log, which is where you would see evidence of a Layer 7 / Application (D)DoS attack.
Question #3: Thank you for all of that information, including links to resources you use! Would there be a difference if you forced all requests to be over http/2 protocol instead of http/1.1 or your curl requests? (I may not be understanding the use of these protocols)
Answer: HTTP/2 compared to HTTPS would have made no difference, as we were only requesting the default page. Why did I choose to use HTTPS? I like to remind people that the security in HTTPS is unrelated to protecting a site from this sort of abuse.
Question #4: Was that with the Emergency DDoS protection option turned on?
Answer: Yes. Emergency DDoS protection was enabled during the tests. Although the website could suffer on the first moments of the attack without Emergency DDoS enabled previously, the IDS would have definitely kicked in given a certain amount of requests. As “search” is a usually a well-known resource hungry feature, we recommend replacing it with a professional search system as described on our “How to Improve Your Website Resilience for DDoS Attacks” blog series (https://blog.sucuri.net/2018/06/how-to-improve-website-resilience-for-ddos-attacks-part-i.html) in order to decrease the unnecessary attack surface.
Question #5: Are there any tools that alarm the administrator (hopefully before the visitors) that a DDoS is running?
Answer: A site or a server is unlikely to know the difference between a Layer 7 / Application (D)DoS attack. All it would know is resources were being consumed. Valid visitors would notice a slowness, or unavailability, as soon as they access/refresh the website. There are server and application monitoring tools on the market that can alert you once a surge happens, however, the server may be completely unresponsive so you wouldn’t get any details except for connection timeout messages. The best course of action would be to prevent the DDoS from even reaching your website by using a Website Firewall such as the Sucuri Firewall. To further improve your website, we recommend reading “How to Improve Your Website Resilience for DDoS Attacks” blog series at https://blog.sucuri.net/2018/06/how-to-improve-website-resilience-for-ddos-attacks-part-i.html.
See all Questions & AnswersExpand
Name: Marc Kranat – Title: Enterprise Firewall Support Supervisor
Marc Kranat:Hi, my name’s Marc Kranat. I’ve worked with the Sucuri Firewall for over five years now. I’m a security and privacy evangelist. I am a father of two, and I also am a part-time photographer’s assistant, as my wife is a photographer, and I’ll pass you over to Norton to introduce himself.
Northon Torga:Hello, everyone, my name is Northon. I’ve worked on Sucuri Firewall for about two years and a half now. I’m in Security Engineering Troubleshooting, and I work web reviews. I’m a Netflix and connoisseur coffee addict and I’m a passionate traveler. I love to travel to my country, which is Brazil.
Marc Kranat:Well first thing we need to know is how damaging DoS attacks are, and how cheaply they can be launched, the sources of where you can expect them to originate from, and who the targets are. For about 20 dollars, you can take down a dozen smaller unprotected sites on and off over a month period, paying for the service with PayPal or anonymously with Bitcoin. These attacks are mostly waged from insanely vulnerable IoT devices that make up the internet of things. These would include, security cameras, household appliances, such as smart TVs, remotes home lighting systems, refrigerators… Yup an army of fridges can be made to take down the website, and they are. Last month, over half a million in Bitcoin was recovered by law enforcement in the U.S. The operator with DoS for hire service pled guilty to launching over four million DoS campaigns, over a two year period.
Marc Kranat:The growth over time of the quantity and sparsity of DoS attacks is grown exponentially. Mostly due to the total lack of regulatory control of the internet of things. Their lack of security controls make them excellent recruits for the botnets. Controlled highjack group of devices with unique IP addresses, which can be redirected to make malicious requests against websites, and cause a DoS attack. According to the internet security company, Kaspersky, financial cost of DoS attacks to medium sized businesses were over 100 thousand U.S. dollars each. The cost, currently, growing at 20 percent a year. The cost represent, both, direct cost of the DoS, which lead to loss of business and opportunity but also loss of reputation among clients and partners. Victims can include anyone with a website or web application. Some of the more high profile cases have been DoS attacks for ransom or blackmail. These are straight up demands for payment to make some change in their business, or make a payment to stop an attack. Occasionally, a DoS attack is launched as cover, providing noise while a malicious hacker has some access to backend systems.
Marc Kranat:A couple years ago, I saw a combined DoS attack on a company’s website and office email server, which shared internet access with their voice over IP systems. The hacker also had remote to the Financial Director’s laptop, using some previously installed Trojan, which he was able to install using some fishing campaign. The bad guys were able to transfer huge sums to a foreign bank account, while the bank tried to contact them to question the transfers, but due to the DoS it was too late. They never got through in time. Attacks can be political, hacktivist, terrorist, and others of course, business competitors, or anyone with a financial or ideological motive to do damage to an organization for its own sake or the attackers benefit. Even video gaming leagues attack each others forums, taking the gaming outside the Xbox. And kids who have access to scripts, which can do damage, and often are just messing around rather than employing targeted or purposeful attack. These are often very short lasting, while a personal DoS attack from a state actor can continue for years against a specific target, even when fully mitigated by a firewall’s protection. A couple of years ago, I saw three, small, separately owned websites for a particular niche market, local businesses in a small town, all come under DoS attack at the same time. One of them was able to identify the attacker, while we’re on board a new entry to their market, who was also messing around with them on social media, being the likely culprit. This is very common. The purpose of a DoS or distributed denial of service attack can be to fully take down of a target or the slow draining of the resources of the target. Both in terms of resources, as well as financial. The target may feel rushed into a more expensive hosting plan.
Marc Kranat:Sometimes the attack can be disguised to look like a heavy load of real traffic to mount a malicious attack. The target will upgrade from say, a typical 30 dollar a month plan on a shared server to a dedicated server and all within the space of a year. The target assumed it was real traffic, but the cost of hosting grew to 800 dollars a month slowly over that year. That’s a cost increase of over 2,000 percent. In this particular case, the hosting company just did not understand how the content management system work, didn’t recognize the DoS, and assumed it was real traffic. Now the client has proper mitigation, downgrading to a sensible plan, and that host now brings us new clients for our WAF value regularly. Sucuri’s intrusion perpetuance in IDS, which is enabled by default, can detect and mitigate classic malicious application layer attacks without any interaction. The attacks being completely unnoticed by the intended victim. Slower attacks, the attacker in these cases were trying, and tie up as many networks sessions open currently. There by, not allowing new sessions to be open by genuine site visitors, leading to a complete denial of service. These attacks should be brought by default of the network layer. Again, a target will not see or notice nay ill effect if using Sucuri. If the site is unprotected, then the applications serve itself will not show any health issues, but net stat will show a huge number of open session in those cases. More common though, as we’re going to demonstrate, our application food attacks. There are a few variations of these, but generally it will mean that a malicious act will crawl the site, making repeated requests on any valid pages. This is how valid low tester works… would work. But the request come from a large number of bots and zombies. Good page caching helps deal with this sort of attack. Very well, but it will always affect server resources.
Marc Kranat:More problematic, or what we call 404 attacks. These are where the requests are made for pages do not actually exist in this site at all. Valid quest your site would only require a data based table to be partially served. In the case of a request for a nonexistent page, or none existent resource, some random string maybe, the server would have had to search the entire database table each time a request is made. Not only does that take more resources, but the deliver page, a new request resulting in 404 error. That could not have been served by cache, so the drain on the application means sources is that much higher than a typical mass request attack on valid pages. The result in the end is the site slowing down in response to valid requests and eventually responding with a 500 internal server. It is possible, at this point, that the application server, in this case, Apache will fail, lock up requiring the service to be restarted to bring it back online. We can see here that the DoS attack is taking place by using combination of the tail and H [inaudible 00:08:58] commands, and a couple of terminal windows. I’ll switch over to them later. Over to you Northon. Do your worst.
Northon Torga:Today we’re going to perform a DoS attack, which means eventually it’s just a single machine to attack another. But remember, we are not the bad guys. We are the good guys, so we have authorization for a bata center to perform such and attack. We’re going to be using [inaudible 00:09:25], so no website is going to be harmed. I’m going to show you a script that I wrote, actually not this script itself, but the window, so you can see the attack is starting. Just share my screen right now. I’m going to be launching the attack right now. Marc is going to be sharing his screen, so you can see the attacks happening. I’m going to divide the attack in three levels. The first level we should just slow down the website. With the second level, we’re going to slow down the website a bit more. And then we’re going to actually take the website down for a couple seconds, and you’ll see on my screen the actual resources page of the attack happening. Marc, if you could please share your screen.
Marc Kranat:I’ll switch back to my scene. You can see that the zero load on this launch, the tail of the access log. You can see there’s no traffic there currently. There’s currently half a gigabyte of two gigabyte being used. We can see there’s zero average load here.
Northon Torga:Remember this machine is proper VPS with dedicated resources, which is not actually the majority of the market, because most websites are hosting or sharing an environment, and does not have enough resource to spare. Here, we show you an unprotected machine, with no firewall or security software in place. The machine is going to receive an attack now directly on a level one, and I’m starting the attack right now. And look at the screen that Marc is sharing with you, you can see that CPU usage is going to increase and we should see some requests going to the access log. As you can see it’s a surge attack, we just searching for random streams on the watchpress and the CPU usage it’s already at 100% we see that the load average there was before zero, now it’s reaching five, which means that the key of focusing is already five times the amount of resources this server has to process. The website, still is available you can see that on the log, you see that 200 response code which means the website is available. But it’s going to take more time to open than usual. That kind of … that is usually used to deplete the resources used on the server and maybe affect the experience of the people visiting the websites. If you have, for example, an I-commerce site, people may give up on purchasing your products because the website is low not actually answering in expected time. The first level of attack should not stop the machine, should not cause any other issues, just slow the website a bit. You can see on the left side of the window that the MyScale which is the database, is pretty much using the entire CPU to perform the key numbers and to look for the search for the random strings that I’m currently sending.
Valentin Vesa:And this is still level one Northon?
Northon Torga:Yeah exactly. This is just level one, the unprotected machine is not actually been taken down, it’s just slowing a lot, and hurting the visitor experience.
Marc Kranat:Your host may not want you around very long if you have that sort of load on it.
Northon Torga:Exactly. Unless you have a proper VPS or dedicated machine you shouldn’t be able to handle that kind of traffic. We are able to do that right now because we are using a canon network data center. With a good machine will have two gigabytes of RAM, I’ll have a dedicated CPU there, but yeah, the machine has an incredible load, it’s 42 load average by now the CPUs age is just too much. Marc if you can tell us about the screen that you see on the left side, below average, before we were seeing zero right?
Northon Torga:Now it’s 100.
Marc Kranat:Yes, now we’re at 100. I’ve been to the data center myself, I’ve installed a hardware firewall on a machine that was under a very similar attack a DL360 joules I think. It literally sounded like a helicopter was taking off, the fan starts spinning up, the hard drive is busy writing logs of the attack and the memory’s hitting that and the data center said.
Northon Torga:How are we going to … I’m just going to stop you Marc, just for a moment I’m going to show you the actual screen that I’m using here. You can see that the loading time of the website is reading 15 seconds. That’s the time stamp of the attack so you can see that we are reaching almost 10 – 15 requests per second, the website is still returning 200 so it is still available … taken a bit more than usual, I’m going to stop the attack now, I’m going to launch the second level. Of the attack as I said, I’m using that as more script so I just need to change to level two and in the level two it is not going to take down the website yet, but the load time is going to be much, much greater than it was before. Marc can you show your screen again please?
Marc Kranat:Yes, I can.
Northon Torga:The attack is starting so it should be starting to deplete the server resource more than you were before. We were seeing 46 on load of average, this should raise now with the attack which has more concurrency and more requests for second. Now it’s increasing as you can see we’re reaching 53, 57. The website is still available as you can see it’s still returning 200. Remember this is just a simple attack, you would just pretending to be an attack. We are not actually attacking, we are just simulating an attack on our simple website. The website is an unprotected machine running WordPress, with pretty much no helpful keys there, just some dumped content so we can actually have something on the database. And yet, the load average is already reaching 73. Imagine if this is a WordPress with pretty much nothing there on a dedicated environment, unlinked internal network, imagine if the attack was before on the sharing hosting with very limited resources and no defenses there and lots of content, for example we have a website with 50 plugins, lots of content, MyScale is not actually optimizing. On this machine that Marc is showing you, it has a special software to restart Apache when it crashes so we can see the 200 gap on the second level. But you can see that the machine is already reaching 84 off load average which is just too much. I’m going to change to my screen, so I can see the load time. Before you were seeing 14, 15 load time now it’s reaching 30, so the website is taking twice as much as it was before. The response though is still returning 200, so the website is still available but now taking around 30 second to load. According to research, usually the visitor won’t want to wait more than three or five seconds for the website to load. If the website is taking 30 seconds to load right now you should be not receiving that much valid visitors because we are give up on the website, it’s just taking too much to load. And remember that’s just the second level of attack on a single machine on internal network so it is very, very simple not sophisticated attack using just a single machine. Imagine, if like Marc said in the beginning, if you were using group net of internet of things, an army of smart TVs and grids attacking a website with multiples like fees, address, lots of networks involved. It would be much harder than we are seeing right now. The time is going to be staying on this 30 seconds of load time. That’s because the attack we are using now, the level attack we are using now, it’s just performing very limited process per second and I’m going to be stopping the attack now and I’m going to be launching the last level of attack against an unprotected machine. On the third level of attack it’s going to be different. Marc can you share your screen again please?
Northon Torga:In about 30 seconds we should start to see 500 response code, which means the server is not being able to perform the attack and is just timing out, it’s giving up, it can’t process.
Marc Kranat:We’re into the 500s now.
Northon Torga:Screen of the second machine? Please?
Marc Kranat:This machine is half the memory resources. Just that we won’t need it, and … I’ll get the launch ready from the firewalls.
Northon Torga:Yep. Just so that we can show you that the machine has absolutely no load. Yep. That’s the second machine.
Valentin Vesa:For full disclosure, this is a Sucuri WAF protected machine.
Marc Kranat:We’ll set up about half the memory resources.
Northon Torga:Yep. You can see that there is no load there. The access logs shows no requests coming there. So now I’m going to be launching the attack against the machine. I’m going to change my green screen again. And I’m going launch now. We should see a load time. Read it fast because the firewall will actually handle this kind of requests, so the website’s holding requests. And with which the 100 requests there, where the firewall already detected the attack and stopped it right away. You can see I was returning 200 and the loading time was pretty good, but by now I shouldn’t be able to actually connect to the website anymore because the firewall may have already detected the attack and stopped it right away. Marc, can you show your screen so we can see the second machine, which is receiving the attack that’s supposed to be receiving the attack now? And … I see it already. 989 requests of attack and on Marc’s screen you cannot see any of these requests because the firewall already blocked me.
Marc Kranat:I can see them in the logs here.
Northon Torga:That’s the logs of the superior firewall.
Marc Kranat:This is the logs of the firewall. The DoS protection’s picked it up, and what it will have done now is it will have blacklisted that IP address … across the entire network. But then not even one request should have got through. It would have been recognized as malicious as we see here … it’s-
Northon Torga:So the machine has absolutely no load, so with the protection and half the resources, the machine is still running fine. The attack’s not going through.
Marc Kranat:The client, unless they check their firewall, will not … they would never have known of the attack.
Northon Torga:Exactly. The protection is automatic. The IDS detects the bad traffic, and the WAF stops it right there. The machine didn’t suffer anything. I can show you my screen now, so you can see-
Marc Kranat:Did you launch a level one or layer three? I doubt it makes to level…
Northon Torga:Because look at my screen now. We are receiving zero-zero response stop, which means we are not able to connect because the firewall blocking us.
Marc Kranat:You’ve been blacklisted from the entire network now.
Marc Kranat:When we see one attack, we plan on them from everyone.
Northon Torga:Exactly. So … yeah, it wouldn’t matter because if I’m trying to do a second level right now, I’m already [inaudible 00:25:11] better because the firewall can block a small attack, a big attack, or whatever the kind of attack, or the size of the attack. The firewall would be able to handle most of the time … I mean, I work on the support for a couple of years, and I barely see one or two cases the firewall was not able to handle some kind of attack, but was really some kind of an application, like, the application attack [inaudible 00:25:38] or something like that. The time that I was in support, I can assure you the firewall can handle that kind of attack really easily.
Marc Kranat: We have solved pretty much sites that don’t use any other DoS mitigation and caching because our caching will catch, will serve the requests. It’s actually using the resources of the attacker. We would … none of that would have happened. Well, now they’re being blocked, they’re not seeing the network, but if we could serve at 200, I’d be a lot happier, then they can just waste their resources just attacking the systems.
Northon Torga:Exactly. So this is pretty much was a DsS attack looks like. Now I’m passing to Val.
Valentin Vesa:That was actually an extraordinary example, live example … live exercise, if I may say. I’m already seeing from Heidi Kopec, she had to leave the webinar, but she said, “Thank you so much for an excellent example of a DDoS attack.” So I’m assuming we’ve hit that proposed target of actually showing how it happens live. We have some questions. We don’t have much time, but I’ll still take at least two questions live. The first one that actually came on is from Ryan, and he’s asking, “Could you tell us what tools you are using to see these logs?” I’m assuming he means Marc’s machine.
Marc Kranat:Yes, I’m using a tail just to grab the end of the log and htop to see the request. I mean tail you’ll see the patterns. It just doesn’t look like real traffic. It’s obviously some random characters. It happens that WordPress actually resolves at a 200 rather than a 404 because when you put in an odd page, it’ll try and find the page for you, and it considers that 200.
Valentin Vesa:What we can also do, Ryan, is on the after-webinar page, the same page you where you registered to see the webinar, we will have most likely next week … we’ll have the recording and also we’ll have the resources side where you can see links to whatever tools we used. So I’m going to ask Marc and Northon to share those with us, and we’re going to add them to the actual webinar page just so you can have full disclosure on what tools were used, maybe even what the specs were on the machine and so on, so you can see how that could be or not. I’m really sorry we only have time for live … for another one question, but all of the other questions you guys send, we will be answering them, so I’m going to grab them all and send them to Marc and Northon. And we’ll have a Q&A sheet on the page as well, next to the webinar recording. I’m going to read this last question. Whoever wants to take it please, or maybe you want to both answer. “What’s the difference between …” I think that’s one of the main questions that came on because it was addressed by four different individuals, so “What is the difference between a DDoS and a DoS attack?” So double D, one D.
Northon Torga:I can answer that. A DoS attack usually comes from a single attacker, so in this example, we used just a single machine. So we were launching a DoS attack. When you have a DDoS attack, which is a distributed [inaudible 00:29:07] attacks, it usually means you would launch an attack from different attackers. It’s usually when you have, for example, botnet and you have lots of hired websites and maybe somebody with these devices, et cetera. And you use them to access a website, access an application just to take it down or slow it down. So DoS attack is the simplest kind, which is use just a single attacker. And a DDoS uses multiple machines to attack.
Valentin Vesa:Yeah, sorry. I was thinking, but there was no microphone on. Okay. Questions keep coming in, but we are really out of time, but, again, we will be answering all of the questions that came on. Also we’re going to link in the description of the webinar to our web page or web application description. If you guys want to reach out to us, please do this … you can continue to do this, and you can reach Marc or Northon directly. You can see on the screen usernames on Twitter. You can follow them, you can ask them. They love to answer questions. And if you guys like this, we’ll invite you to come back maybe to a next time when we’ll do something more interesting or something very different. But both of them love being in the webinars, so most likely we’re gonna see them again. Thank you, Marc. Thank you, Northon. By the way, we are joining you guys from three different countries, launching this would-be attack in a fourth country. So, just so you know, we are very distributed and we can be everywhere and all the places to protect you. I’m gonna let you guys say your goodbyes, and then we’re going to close.
Marc Kranat:Thank you for attending. And any questions, let me know, please.
Northon Torga:Thank you for attending as well. And please keep tuned on the blog as well. Our usual posts are good articles there, and especially talking about DDoS. I recently launched a video series instructing you how to improve your application to handle more traffic and especially that kind of videos.
Valentin Vesa:Thank you, everyone. And just the last comment, Northon, everybody mentioned your shades, so people like them. Have a great day everyone. Have a great afternoon, great night, whatever your local time is, and we’ll see you next time. Goodbye.
See Full TranscriptExpand
In the website security community, our name is known for fast site hack cleanup and responsible vulnerability disclosure. As thought leaders in website security, we are committed to sharing what we know. Follow our concise and helpful website security guides and tutorials so you can learn how to clean and secure your website.
In this webinar we will highlight the various activity, access, and error logs WordPress site administrators have at their fingertips. Plus, learn how logs can best be used to manage, troubleshoot, and most importantly, secure your sites.
In our latest webinar, we'll describe action items that can improve the security state of internet-connected devices we all use every day. These devices will include common household staples such as: WiFi Routers, iOS/Android devices, and personal computers.
Join us as we delve into the minds of hackers to explain targeted attacks, random attack, and SEO attacks. Find out why bad actors target websites.
A feature benefit guide for our agencies and end users. Why use our firewall? What kind of protection does it offer? How does it affect the efficiency and speed of my site? Will it affect my server's resources? Find out the answers to these questions and more in our webinar…..
Cross-site contamination happens when one hacked site infects other sites on a shared server. This webinar is for beginners and web professionals to understand cross-site contamination and how to prevent it…..
If you're considering security for your site or are new to our services, this webinar will guide you through Sucuri's simple setup processes. Potential notifications, support options for various scenarios, and ways that you can also work to keep your site malware-free will be discussed…..
Learn how you or your agency can account for security with your client projects. Presented by Sucuri Co-Founder, Dre Armeda, this webinar shows how you can get involved and help clients who are not aware of some of the security risks involved with managing a website…..
It's a move we've seen coming since early 2017. Chrome HTTP sites are now officially being marked as 'not secure'. With Chrome dominating 62.85% of the browser market space as of last month means that even small changes can have a big impact on website owners if ignored…..
In this fire chat, we're looking to find answers to some of the questions web agencies have been asking us for years, in hopes of shedding more light into how you, as an agency, need to respond to security threats your customers face…..
Website security is challenging, especially with a large network of sites. We want to help you understand how you can create a security plan and reduce the risk of a hack or security incident. In this session Dana covers the implications of a security breach and why security should be important to your agency. Dana shows you a tiered approach to we….