THE LAB #11: The Anti-Detect Anti-Bot matrix
Excerpt
An incomplete but still yes useful list of interesting resources on web scraping. Testing the most well-known web scraping tools in Python against Cloudflare, Kasada, PerimeterX, Datadome and Shape
This post is sponsored by Oxylabs, your premium proxy provider. Sponsorships help keep The Web Scraping Free and it’s a way to give back to the readers some value.
[
Oxylabs
In this case, for all The Web Scraping Club Readers, using the discount code WSC25 you can save 25% OFF for your residential proxy buying.
You can find all our partners and the discounts they apply to our readers on our Discord Server.
The web scraping landscape
[
Captcha
In these pages we said over and over that web scraping is getting more complex. Anti-bot softwares require more advanced solutions, leading to higher computing and memory costs. There’s also a less visible cost, which is the complexity of the web scraping infrastructure: since there’s no silver bullet or magic solution that fits every case, the modern web scraper needs a full array of tools in his belt to tackle different cases.
In this episode of The Lab we’ll see some of the tools I’m using daily to tackle the most common anti-bot solutions and how, through a quick test, they behave against them.
The chosen tools
[
Army of bots
As a python developer, basically my potential toolset of headful browsers is restricted to:
-
Selenium
-
Undetected Chromedriver
-
Playwright (in different sauces)
-
Pyppeteer + stealth
and that’s the reason why you won’t find puppeteer or cheerio in the following tests.
Selenium
Given that, you won’t find also Selenium: in my opinion, it has lost some appeal in the latest years, specifically from when Playwright has been released. It relies on standard webdrivers which are not meant to be used for web scraping and can be easily detected by anti-bot softwares.
Undetected Chromedriver
On the other hand, you can get a better result at a fraction of the complexity using the undetected_chromedriver python package. In this case, you’re still using a webdriver but that is modified and compiled with the final purpose to be used in web scraping projects.
Playwright
Playwright has been released in 2020 and at the moment it’s my favorite tool because of its flexibility and ease of usage. After the installation (via pip), you can start right with 3 different browsers, both in headful or headless mode. And if you need more, you can install other clients like Chrome (instead of the chromium bundled) or some compatible anti-detect browser like GoLogin, to get more options for your scrapers. Even Playwright is not meant for web scraping and there was a plugin for customizing the bundled browsers but it’s not been updated for a too long time and is no more effective, but playing around with the right combo of browser and settings, to me it’s my first choice.
Pyppeteer
It’s an unofficial porting in Python of Puppeteer, the original project of browser automation from where Playright took “inspiration”. I don’t find any reason actually to prefer it to Playwright, but it’s another option worth mentioning. It has a stealth module but, at least in these tests, it didn’t work as expected.
And what’s your favorite tool for web scraping in Python?
POLL CLOSED
The tested antibots
[
Not a robot
In this post, we’ll see how the tools mentioned before perform against the most well-known anti-bot solutions.
We will perform a generic page load test on 5 different websites, one per solution. It cannot be an exhaustive test, since every website can have a different setup and different rules to block or not suspicious traffic. On top of that, by loading only one page, we cannot test if the behavior of a spider written using one of the tools could be marked as a bot. And last but not least, there will be cases where some sections of websites (like the login pages) would be protected by stricter rules than the home page.
Given that, our test could be a good starting point to understand which tool is more convenient to start with.
Cloudflare
Cloudflare Bot Management solution utilizes a combination of signature-based detection and machine learning algorithms to accurately identify and classify bot traffic. The solution also offers rate-limiting, CAPTCHA challenges, and JavaScript challenges to mitigate the impact of bots on a website’s security, performance, and user experience.
It’s one of the most used and stronger solutions to by-pass if configured in a strict way.
PerimeterX
PerimeterX uses real-time behavior analysis and machine learning to detect and block bots in real-time while allowing legitimate traffic to pass through.
Compared to Cloudflare Bot Management, PerimeterX also focuses on real-time behavior analysis and machine learning, including at the same time the ability to detect advanced bots that use techniques like IP hopping, browser fingerprinting, and headless browsing.
In our tests, we’ll see it will be the easiest to bypass, at least for the website we considered.
Datadome
Datadome is another anti-bot solution with all the features mentioned before, for our tests it was the hardest to consistently bypass. Typically we could load the target page for the first time with every solution but if we try to test it for a second time from the same IP, the load will fail.
Kasada
I think it’s the youngest solution on the market between the ones tested here, and it’s the most recognizable. When loading inside your browser a website protected with Kasada for the first time, you should notice in the network tab of the developers’ tool window a 429 error. This is the “challenge” that Kasada sends to the browser and, if it is solved, then you get redirected to the target website. It is called a zero-trust security policy.
F5
In my filter bubble (fashion e-commerces) I don’t see many F5 protected websites, but when configured strictly it’s not that easy to bypass. It seems to rely heavily on AI to detect strange behavior in the users but even loading a single page of our testing website was not simple.
What’s the hardest anti-bot to bypass for you?
POLL CLOSED
The tests results
[
The Anti Detect Anti Bot Matrix
The last image says almost everything but I want to add my two cents before going into details.
I have the feeling that the anti-bot industry focused more on tackling the chromium webdriver, since probably is one of the most used and the one that leaks more information to the target server. Pyppetteer and Playwright with Chrome are the worst performers, while the undetected chromedriver gets much better results due to its customization.
Playwright with a headful execution of Firefox instead probably seems more genuine to the target server or, at least, less distinguishable, from genuine traffic.
Before having a look briefly at the code, just remember you can find the full code on our GitHub repository for paying users. If you are a paying subscriber and don’t have the access to the repository, feel free to write me at pier@thewebscraping.club for getting access (unluckily there’s no automation for it).
Undetected Chromedriver
[
Undetected Chromedriver tests
The script I’ve used it’s quite straightforward without any customization or particular options and got already these good results.
[
Undetected Chromedriver results
Playwright + Chrome
[
Playwright and Chrome tests
In this configuration, I’ve opted for one of the most successful setups I used in production until some months ago. We have a Chrome client (not a webdriver) with a persistent context, in order to store all the session data gathered from the navigation, like a real browser.
The awful results of the test confirmed what I’ve seen some months ago, the focus of the anti-bot industry seems to be on detecting real traffic from Chrome browsers.
[
Playwright and Chrome results
Playwright + Firefox
[
Playwright and Firefox tests
On the contrary, a plain script using Firefox can pass almost every anti-bot solution, with mixed results on Datadome because of the behavior of the test scraper.
[
Playwright and Firefox results
Playwright + GoLogin
[
Playwright and GoLogin tests
I used this setup starting the past month because of a website protected by Cloudflare in a particularly strict way. You need to buy a plan from GoLogin to use its service and to install their browser’s client on the machine but it worked like a charm and performed quite well also during these tests.
[
Playwright and Gologin results
Pyppetteer
[
Pyppetteer tests
This out-of-the-box solution performed badly on these tests but probably with some fine tuning things would improve.
[
Pyppetteer results
For today it’s all, remember you can find the full code on our GitHub repository. If you are a paying subscriber and can’t access the repository, feel free to write me at pier@thewebscraping.club.
The Lab - premium content with real-world cases
-
THE LAB #8: Using Bezier curves for human-like mouse movements
-
THE LAB #6: Changing Ciphers in Scrapy to avoid bans by TLS Fingerprinting
-
THE LAB #4: Scrapyd - how to manage and schedule a fleet of scrapers
-
THE LAB #2: scraping data from a website with Datadome and xsrf tokens
Our discord server is the place where we can share our experiences interactively or have a chat, find the bargain from our partners, and much more. I’d be glad to see you all there.