Procedural engine major updates

Phew! Well there it is at last, the first major upgrade to the procedural generation engine. This has been in the works for the past several months (no kidding) and, contrarily to what I wrote earlier, not only terrestrial planets were impacted, but all satellite types… and also stars!

There is quite a lot to say but I will try to be brief - if so inclined and/or curious about more technical details, you can head over to the Data explorer for actual data tables, clearer hierarchies, etc.

What hasn't changed

The very high-level structural cogs of the engine have not been changed (or in extremely minor ways): the galaxy, cube and system layers all remain identical compared to the previous version -- to this day.

I have some ideas around them involving clusters, stellar nurseries and other large structures but that will all come later - perhaps!

Global changes

TLDR: the data additions mentioned below and that drive the extremely vast majority of the data underlying the procedural engine can now be browsed in detail in the data explorer's following new sections:

• star classes and types
• satellite classes and types
• cores, mantles, crusts, atmospheres and nebulae
• mineral families and rock types
• elements, particles and compounds
• matter & reactions

Of course, the original Data explorer has enjoyed the upgrade too; and so has the 3D explorer.

Star classes and types

In Adlumens, stars are classified a little bit differently than in the real world. Some stars are still classified using the standard Harvard system, from O to M, with a mass range betweem 0.08 and 150 solar masses. These normal stars make up the vast majority of the stellar population, taking into account the usual probability of a star to appear via the engine.

Other stars are far more… alien. And require to be sorted in different categories altogether; ones submitted to slightly different rules and chances to appear. I have started documenting such exotic star types (from the almost normal Boson Star to the truly weird Kugelblitz star), but have not really started coding for them.

The result is that Adlumens's star classification shifts away from the real world a little bit; real-world stars become a subset of a wider family; In other words:

• The top-level star family is the star class
• One level below is the star type (for example Harvard classes)

In summary:

Important notes: due to several important stellar mass thresholds right in the middle of several classes, I have had to split them into several "subtypes":

• Ma-class and Mb class are a split made up to represent the threshold below which no giant phase occur for a star i.e., it goes straight into Helium white dwarf. I confess this is a not very welcome artificial split, but one I had to implement to be enable that limit. In any other aspect, Ma and Mb are mostly identical.
• O-class has been split into four mass ranges (it being so wide), which helps a lot into determining giant subtypes at runtime.
• B-class has similarly been split into two sub-ranges, for exactly the same reason as the O-class.

Last, stars have been normalised and now also have structured components: cores, mantles, crusts, atmospheres and nebulae. This brings them closer to satellites in terms of nature; before, they were very much black boxes and mathematical entities; they now are layered and data-driven. This will enable the engine to see them as any other type of resource as some later stage.

More randomisation control

Another aspect of procedural star generation is how the type is picked up at runtime.

Previously, a pert/scaled beta distribution was used, with a very heavy beta parameter, which tended to skew the distribution towards small masses, with a peak at 0.4 solar mass. While this worked perfectly, an undesirable side effect was that high (>6 solar) masses were extremely, extremely rare. Now even though this was realistic, I am a believer in metagame-driven artificial stellar variety so that model had to change. :-)

Since the engine is now far more "authoritative data"-driven that before, each star type now is assigned a score, which very simply represents its chances of being selected during generation. That score ranges from 9.9999999 to 0.0000001. The advantage that technique is that more control can be exerted over the type chances; not only via the number itself, but also via other future mechanisms (local space-time nature, etc).

The drawback? Well it is slightly slower than a simple beta distribution pick times a mass multiplier. But hey, one cannot make an omelette without breaking eggs!

Satellite classes and types

The big one, which has been the bulk of the upgrade and has stuck all its seven fat little fingers in all aspects of the code base. Ueeergh.

So:

• Satellites are organised as classes and types. If that sounds similar to the stellar hierarchy above, it is because it is. However, the satellite classification actually came first and contaminated stars. :-)
• Satellite classes are divided along the lines of the previous split: from telluric planets to gas giants and brown dwarves. Types are of course their children and try to reflect the sometimes extreme variety of planet and moons that can be found!
• Each star type is assigned three numbers: a minimum amount of satellites, a maximum number of satellites and a most likely number of satellites. A simple pert/beta random pick is performed to determine the final number.

As written in a previous post, all satellite types follow the "Top-down" approach:

1. for each type, come up with primordial compounds
2. in python, precompute how their components (core, mantle, etc.) interact with one another over time and store the chemical outcome. This tries to simulate erosion, deposition, chemical weathering, outgassing, photolysis, etc. However, only some of those interactions can be precalculated at python-time; others have to wait for runtime since flux, temperature and others are needed to establish the chemical outcome of interactions between a star (mostly) and compounds.

Satellite random pick is not 100% random

Of course some randomisation is applied when selecting the type assigned to a satellite. But before that pick can occur, a satellite must overcome several hurdles:

• Its type will come from its host star affinities with types: all stars cannot give their warmth to all satellites. Star<->Satellite affinites are expressed as ranges and the star's metallicity (Z) has a direct effect on the final affinity per type: some satellites have a greater chance of being generated with a high metallicity while others, well, others prefer a low score. This star/satellite affinity tool gives a rough overview of the base values used for type picking.
• Its mass (randomly generated) must match the type's range. For example, if the mass is 0.23 earths, the satellite cannot be a gas giant. Obvious.
• Its actual position in the list of satellites must match the type's range. Some satellites are far more comfortable at the beginning of such a list than others; while others prefer the far end…
• The incident stellar flux to which it is exposed must respect the type's accepted ranges.
• And last, its age must also be within the type's accepted values.

The use of "must" above is a little bit misleading; some criteria are actual enforced in a hard way, while others are more lenient. Why? To avoid the embarassing case where a type cannot be selected (which, during development, happened way too often! :-) )

The final result is a list of acceptable types, each assigned a weight depending on how they scored for each criteria; these weights are then used to generate a rust WeightedIndex, from which will finally be picked an actual type.

Satellite type randomisation

With the satellite type in hand, we can now start to generate its components. Depending on the class (note: the type's parent), the following rules apply:

Yes, I know this is not exactly a case of TMI at this stage -- but for the gruesome details (from satellite types to components), I want you to go to the Data explorer.

Subsatellites selection

Subsatellites (moons, rings) are of course also randomly picked, in a way very similar to satellites. In a similar fashion, not all satellites types can be hosts to all subsatellite types. Each Subsatellite is subjected to a partially random mechanism very similar to the parent satellite picking logic, where acceptable types are mixed with star-derived data, metallicity, etc.

Each satellite type is assigned three numbers: a minimum amount of subsatellites, a maximum number of subsatellites and a most likely number of subsatellites. A simple beta random pick is performed to pick the final number. Exactly like for stars.

Which satellite classes can host which subsatellites classes? Consult the table below:

Some rows feel obvious: rings, belts and accretion disk cannot host subsatellites. Others I wonder about: can brown dwarves really have gas dwarves satellites? Maybe, maybe not…

The actual data is more complex, as each satellite type actually has a mapping to the subsatellite types. For more info again ->Data explorer! :-)

A couple of notes as of the writing of this:

• I will also introduce extremely exotic satellite classes and types in the future. The current ones try to reflect Adlumens's dedication to pseudo-realism. Go check the data explorer and judge for yourself!
• There should definitely be sub-sub satellites for when, say, an gas dwarf or massive telluric planet orbits a brown dwarf. Mmmm… I still need to determine whether this would be worth running simple roche limits versus hill sphere calculations. Not too expensive because super simplistic, but still costing a little.

Elements, compounds and particles

This has been another major upgrade:

1. First because the number of compounds has massively increased. From 308, the list is now nearly 1500 compounds long.
2. Second, because a map of compounds reactions had to be created to keep track of which compound(s) could react with which compound(s) and under which conditions such as pressure, flux, etc.

Elements have not been touched that much; but I still need to come up with exotic (read: rare) isotopes that could prove valuable for factions. To be worked on.

And finally, particles have been introduced to be able to create exotic compounds in nice comfy places such as the cores of neutron stars or crusts of white dwarves. While this is fun in itself, it also paves the way for bridging natural-born compounds/alloys to artificial/manufactured ones via mid/end game sciences and techniques from the Tech tree.

Compounds

Increasing the number of compounds was motivated by two main ideas:

• Perhaps artificially increase the variety of stuff that can be found in planets, rings or belts. To satisfy pseudo-realism again.
• More importantly, support an interesting and more balanced spread over nearly 80 satellite types. What I did not want is to end up with richly detailed and varied types, while others lagged behind. I know the idea of a game is still far away, but the consequences of this would be an even stronger chance of near abandon of some types, at the benefit of others. To avoid at all costs.

Mineral families and groups and Rock types

For convenience, compounds that are solid in the default state have been assigned a mineral family, itself the child of a mineral "Group" While I confess this is an artificial way of splitting minerals, it has proved a useful tool to keep track of a (hopefully) balanced spread, again while trying to keep an eye on future game balance.
It's also good not only for immersion & fluff, but also for mapping with crew member skills/specialisations & research!

Rock types are still in and will be extensively used to determine resource/compound concentration at any given point on a planet/moon, whether client-side via wasm or server-side.

In the works

Some compounds do exist (i.e., do exist on our Good Old Earth) while others were invented, again striving for some realism (and some Easter Eggs, more private-jokey than anything else!). What's more, the addition of new stellar classes and types mentioned above warrants the following to happen - assume this is in the works:

• New materials beyond normal compounds should be created beyond the naturally occurring Carbon/oxygen Stellium or Neutronium. Even though the middle-late game will allow to manufacture them, they now also do occur naturally!
• Even more bizarre/alien/ultra high science materials can be born out of exotic stars and satellites. To be determined :-)

Components: cores, mantles, crusts, atmospheres and nebulae

In the previous version, these internal parts were limited to planets and moons. As described above, part of this upragde was to standardise all and apply them to all high-level entities such as stars and planets.
All entity types can have all components, as options -- typically:

struct Entity{
  pub core: Option<Core>,
  pub mantle: Option<Mantle>,
  pub crust: Option<Crust>,
  pub atmosphere: Option<Atmosphere>,
  pub nebula: Option<Nebula>,
}

Not references because these objects are owned by their parent entities.

Nebulae are new and represent not only actual planetary clouds formed by late-life stars, but also the overall stellar ejecta over time. They can also and will be applied to arbitrary exotic stars and planets… although that is still on the drawing board!

On the technical side

I will keep on ignoring requests for the source code - hell, a very rude - and utterly deluded - contact form was filled with a dummy email address demanding it lol. You can cram it buddy. However, I will try and dig in a little bit deeper than the functional, that you can browse in the data explorer anyway.

Level of detail

In some old posts, I explained the way the engine makes use of a dynamic LOD in order for the 3D explorer to work reasonably nicely.

Now while the concept is of course still used, level of detail has now become two-dimensional. Here's the old list:

Compare to the new matrix:

It is a tiny bit more complex but allows querying the engine in a far more flexible way; it also saves some computation times because the depth branching is easier to control. All in all a wonderful improvement. IMHO.

Workflow

This one is a little bit convoluted and clumsy. :-) But it works well. I choose to share it here because it reflects how even the code has become far more data-driven than in the previous version, under which whatever happened inside rust's binary was kind of a black box. Now almost everything is driven by what lives in the database. In a nutshell:

1. Leverage libreoffice calc to play with large spreadsheets. By large is meant 1300 rows by 30 columns, for example. Now even though this may seem like a paleolithic way of working, I did it consciously, not wanting to waste considerable time on meta tools to manage data. I needed something ready and fast.
2. Next comes the CSV export; I currenly juggle with 50 of them - some as small as 4KB, others larger at 2MB.
3. Next custom made scripts import the CSVs in one go; it is also possible to use individual django management commands. In fact, each CSV file is associated with its dedicated command. Then, launching them all in the right order is as easy as chaining several call_command('constant_load_star_classes'). I decided to build custom bulk up/insertion tools because Django's native data import tools had become way too slow given the number of rows involved. In some instances, I went from Django's initial 30 minutes upsert to custom CSVs loading that took around 1 minute. A useful improvement :-)
4. With the data now in the database, it's tweaking time: as I mentioned before, heavily annotated querysets to find out holes, illogic values and so on -- in a word, leveraging a database's ability to look sideways. When such errors are found, back to the spreadsheet and GOTO 1
5. Data good? then time for BSON export. We are nearing the binary world now. Those BSON files are lazily loaded by the rust engine. Some of you will remember that I used to print huge rust files and I thought it was nice. It was, but when the data volume started to grow, those rs files started to be even less readable and maintainable…
6. Parallel to the BSON dumps is made a 100% JSON dump. Why? For readability: when serde panics it is easier to check a human-readable file than a binary one. Obvious.
7. Then compile the binary. Pray.

Django models

The number of models has slightly grown from:

• Atmosphere template
• Compound
• Core template
• Crust template
• Element
• Mantle template

… to (roughly from low-level to high level):

The above table omits several many-to-many tables, enums and the like. This increase is I believe significant, and is reflected the following numbers:

• Around 500,000 database rows in total
• About 80MB of BSON and JSON files.

A 2-minute test script

I put this incredibly quick and dirty script here because I have used it extensively during the procedural engine upgrade. Main goal: to spot the low hanging fruit caused by Panics and so easy to spot 500s.

HOST = 'https://localhost:2600'

cube_error_urls = []
satellite_error_urls = []

async def build_urls():
    test_range = 10
    xs = ys = zs = range(-test_range, test_range+1)
    urls = [
        f'{HOST}/data-explorer/24a6770e-88b4-4add-ae63-2571ba5e6f19/{x}/{y}/{z}'
        for x in xs
        for y in ys
        for z in zs
    ]
    return urls


async def crawl_satellite(client, text):
    soup = BeautifulSoup(text, "html.parser")
    urls = [a["href"] for a in soup.select(".data-explorer-satellites a")]
    print(len(urls))

    for url in urls:
        s = time.monotonic()
        r = await client.get(f'{HOST}{url}')
        e = time.monotonic()
        print(f'    {r.status_code} - {(e - s):3f} - {url}')

        if r.status_code != 200:
            satellite_error_urls.append(url)
            continue


async def go(client):
    urls = await build_urls()
    for url in urls:
        s = time.monotonic()
        r = await client.get(url)
        e = time.monotonic()
        print(f'\n{r.status_code} - {(e - s):3f} - {url}')

        if r.status_code != 200:
            cube_error_urls.append(url)
            continue

        await crawl_satellite(client, r.text)

    print('\nCUBE ERRORS:')
    print(cube_error_urls)
    print('\nDETAILS ERRORS:')
    print(satellite_error_urls)


async def init():
    async with httpx.AsyncClient(verify=False) as client:
        await go(client)

The result is 1000 cubes crawled:

• For overall status: even though when the level of detail is still shallow at cube page level, errors can still occur, resulting in a 500.
• For individual satellite status when errors happen upon components building (cores, atmospheres, etc.)
• Same goes for subsatellites.

This is very violent when run on local and would not work on production -- for obvious reasons linked to bots and the awful way in which they abuse public resources on the Internet.

Other updates

These are perhaps less important, some of them from yours truly… others kindly suggested by you:

UI and the meta around it

• The ability to toggle fullscreen mode in the 3D explorer (from luna). Switching to fullscreen of course worked before (there is afik no way to intercept it), but would keep the top header, footer, etc. visible. These now disappear when in fullscreen mode. Limitation: one cannot trigger F11 properly while F12 is active and docked. I guess that will do for now…
• In the works: the ability, from the data explorer, to click and go to the corresponding 3D explorer location (from deziheyward).
• In the works: when logged in, the ability to bookmark cubes, systems, stars or satellites from the data explorer (from uhhhdie).
• In the works: new dialog mechanics. HTML dialogs to be now far more dynamic than before (from adlumens).
• Rework of the SASS fragmentation/spread and creation of a quick and dirty "SJS" framework for gulp aggregation/piping (from adlumens). The result is a much easier management of scattered .sjs files via simple @include "..." directives. Not production-grade or anything but works well enough for me and is fast :-).

Procedural engine

Mass repartition of elements and compounds per component (from adlumens)

Component is either core, mantle, crust, atmosphere or nebula; this is familiar by now. A layer is a new concept inherited from the atmospheric scale height idea; it still applies to atmospheres of course, but is now applied to all other components. In summary: this splits cores, mantles, etc. into depth or altitude layers. This is useful for several things:

• For all components, to know in which state and quantities elements and compounds are found, and at which layer. Although this is for the future at this stage, this is incredibly useful for resource discovery and extraction.
• For atmospheres, we have 6 altitude layers to play with during rendering. I hope I can leverage that for interesting visual aspects (volumetric or not!)
• Knowing that all components have 6 layers (0-indexed), 0 being core-ward and 5 being outside-ward, a crust's outer layer (5) is interesting because it now has the ability to keep track of which elements/compounds it has in terms of quantities and states. The goal is of course to make use of that data for rendering planetary surfaces and also for resource management/detection/etc.

Tools & experiments

The planetary surface tools has temporarily been turned off as it is undergoing what I hope is going to be a nice upgrade! I had to take it out so the lengthy process of upgrading it would not delay (even more) the availability of the new procedural engine.

I am working on the following aspects:

• Full random generation: it sounds easy at first, but this cannot be true anymore, as the surface will now be linked to the material/compounds composition. So not as random as it looks, and internally needs to loop back to pseudorandom/manual satellite generation.
• From a satellite ID: here it becomes easier as the engine is able to rebuild a satellite's characteristics from this ID.
• …and of course the integration in the 3D explorer. Which will probably require more workers :-)

In conclusion

I sincerely hope you will find the upgrade enjoyable; I know I do but then again I am slightly biased. :-)

To the South American bots now getting 404s: time to start over again, after what… 200k requests?