(mid-)October status

Well, well, well, look at that. I haven't posted for what looks like an eternity but… I am currently knee-deep in the "Tech tree"… which will not be a tech tree, but more on that in a later post.

I just wanted to highlight a few things on user activity from the past few months.

Server log activity

nginx is not really busy filling up the file system at lightspeed (I wish), but a couple of clear trends have emerged:

1. Some users are around and actually stick around - about 10 per day and some do spend reasonable amounts of time on: the 3D explorer first, and second on the data explorer. Many of them do block analytics of any kind, so I have to (extremely roughly) triangulate my local logs and google analytics.
   This activity kind of makes sense: about 99.999999999999% of the contents of this site is procedurally generated. I can assure you the posts are not 8-}. Hopefully the upcoming science/tech pages will provide more content to grind to those users! Fingers crossed!
2. Bots are definitely around. During the month of July, the Amazon bot for Alexa (not sure but probably, both user agent and IP ranges checked out) started crawling the procedural pages for all four listed galaxies. And I do mean crawl deeply and in a comprehensive way; I guess I should be happy they (whoever they was) did not run hundreds or requests per second… So I did a quick back-of-the-envelope calculation and ended up with hilarious numbers: just for the Trass'ir galaxy, its 40-ish billion stars have around them on average 6 planets, with each about 4 satellites. So that's - give or take - 1,000,000,000,000 requests to run. Good luck to them, especially for data storage: assuming the html was parsed and turned into some json-ish blob, let's say… 1KB per page? So that's 10 terabytes? Contact me guys, this runs on less than 2MB of code!. Not to mention they miss many things which are not public data such as atmospheric scale heights and related compositions/states, tectonic plates, etc.
   Also, time: crawling at a very friendly rate of 10 requests per second would take 100,000,000,000 seconds: a few decades short of 3.2 millenia? Har har har.

Anyway… this bot gave up a while ago and has been replaced by another one(s?). All requests seem to come from EGIHosting and they have been going on for quite a while now. Again, at least those requests are not aggressive. However, the logs are sometimes funny:

• User agent: "Mozilla/5.0 (compatible; MSIE 10.0; Windows; U; Windows NT 6.2; Win64; x64 Trident/6.0)"
• User agent: "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 7_5_3) Gecko/20130401 Firefox/69.5"
• User agent: "Mozilla/5.0 (compatible; MSIE 8.0; Windows; U; Windows NT 6.2; x64; en-US Trident/4.0)"

They're not even hiding; they just have random UA selection… I guess if it has to come to it, I'll have to ban ip ranges or whole ASNs…

I wonder though… are these bots blind and stupid, i.e., they caught on a seemingly limitless supply of pages and keep on following links? Or are they more targeted and have a slightly more sinister purpose? Who knows…

Kind words received

I have to mention this because it was very nice and warmed by heart motivated me. Sorry, not sentimental like this. :-)

Here goes, and a very huge thanks + hope I will not dissapoint - to mister Rick from Italy:

<snip>i wanted to say that your project is amazing!
The only similar thing i can think of is Elite: Dangerous universe map. Keep workin on it, this is one of the most wonderful projects I've ever seen.<snip>

Rick, if you wish me to remove this quote, please do contact me like you did, I will happily oblige. And slash about 50% of the content of this already famelic post.