Aaaaaand more bots

Looking at my logs, to check on patterns, etc. *ahem* 500s.

And a unique IP starting doing one thing: crawling the mass repartition links at almost 100 requests per second. A quite aggressive filthy fucker. But that was fine because I rate limit per IP. Such a protection is primitive but at least, this is proof that it works since that bot ended up with a majority of 503s.

Digression: I still haven't found a way to counter those residential VPN addresses that crawl, although far far less aggressively. It is a constant stream of requests, though. Perhaps the same bot, and the people managing it got tired after realising that crawling the mass repartition links in one galaxy would require around 10 to the power of 14 HTTP requests. Good luck guys!

Then it started crawling for more suspicious, not content-related paths, such as:

• https://adlumens.org/.env.production.local
• https://adlumens.org/.env.old
• https://adlumens.org/.aws/credentials
• https://adlumens.org/app_dev.php/_profiler/phpinfo
• https://adlumens.org/swagger.json
• https://adlumens.org/server_info.php

And so on.

Now what happened is several things:

1. In the best of cases, it got 404s because of course none of those urls point to anything. If it crawled too quickly, 503s.
2. In all cases it provided me with a nice little list.
3. And I dearly hope that the person(s) behind this bot will see their cloud/infrastructure bill explode and that they enjoyed the gzip bombs that were returned on some of those urls. You're welcome, I hope it was worth it.

Oh, and fuck you. (the bots of course)

Yours truly.

Edit March 2nd

I have continued playing a little bit against bots, adding two itching powder versions:

• The tarpit: once an IP is identified as a bad bot, then it is sent into a universe where time flows… slowly :-P. The only cost for Adlumens is one opened socket for a little while, and that isn't much to pay! Hopefully the other side experiences problems.
• The endless stream: if that same IP insists and exceeds a certain rate, they are sent to another type of dimension, where a request is not only infinitely long-lived, but also returns nothing. Hopefully those connections stack up on their end for sweet cloud costs.