Bots and more bots

Published on May 22, 2025, 2:20 p.m.
Last changed on May 22, 2025, 2:21 p.m.

To whoever or whatever is crawling this site from various locations in the US (and elsewhere I'm sure), at least do it properly. Sheesh.


45.38.215.69 - - [22/May/2025:14:07:35 +0000] 
  "GET https://adlumens.org/%22/data-explorer/subsatellite/69a1019200491b050a9d0090cb9136a3bbb050aa0f100a3030b001e047a41397/%22   
  HTTP/2.0" 404 2015 0.006 "-" "..."
81.5.156.183 - - [22/May/2025:14:07:40 +0000] 
  "GET https://adlumens.org/%22/data-explorer/subsatellite/0901aaba1b0bb50198e0000f100b0991200012210005a1509c9d000a590a8200a20b9/%22
  HTTP/2.0" 404 2015 0.006 "-" "..."
104.233.38.73 - - [22/May/2025:14:07:46 +0000] 
  "GET https://adlumens.org/%22/data-explorer/subsatellite/000aaab02a09ba290001992b111b0a50303840e1003d210b9c20095b00a01f005490/%22
  HTTP/2.0" 404 2015 0.006 "-" "..."
159.148.130.90 - - [22/May/2025:14:07:51 +0000] 
  "GET https://adlumens.org/%22/data-explorer/subsatellite/0001aaba1600b5dc000f000011060b1b20001be10005a1500302000a990a9200a20b90/%22
  HTTP/2.0" 404 2015 0.006 "-" "..."
77.81.84.59 - - [22/May/2025:14:07:56 +0000] 
  "GET https://adlumens.org/%22/data-explorer/subsatellite/99a1017300301b050a9d0000cb0058a1bbb030aa0f120a6010b001e038a21608/%22
  HTTP/2.0" 404 2015 0.007 "-" "..."
45.41.135.142 - - [22/May/2025:14:08:01 +0000] 
  "GET https://adlumens.org/%22/data-explorer/subsatellite/000aaab02a00ba100001992b122b0a50304840e1003d210b0c20005b00a01f005590/%22
  HTTP/2.0" 404 2015 0.006 "-" "..."
etc, etc, etc.

I can't really complain as the crawling is very gentle with one request every 3-5 seconds. But still. Either properly parse the HTML (which I believe is ok?) or use a different regex to extract the urls. But remember:

From the fabulous Stack Overflow answer