A game with thousands of rules has a particular fear. Imagine you change one number for the Lantern Line, and three versions later someone notices that Iron Mandate fleets have quietly stopped firing in the siege missions. The change was small, the damage was far away, and nothing in between complained. This note is about the net we put under every release so that this cannot happen without a loud sound.
Fingerprints of a battle
The idea is old and plain. A battle in Cosmic Starship is a program with one input, a random seed, plus whatever the pilot does. If the pilot is a script that does the same thing every time, then the same code must produce exactly the same battle. So we play a fixed set of battles with a scripted pilot and, at a series of moments during the fight, write down a short fingerprint of the whole field: every ship, where it is, how much hull and shield it has, every shot in the air. We call the list of fingerprints a battle’s hash. Two versions of the game that play the same battle the same way have the same hash. One differing digit anywhere tells us that something changed.
The set has 93 battles. It covers every kind of mission, one ship from every civilization with two different seeds each, the presets of the Who Would Win? simulator, a sample of campaign missions, and a veteran’s real save, with all its mods and upgrades, because the strange interactions live in a rich save and not in a fresh one. It takes about a minute to run on an ordinary machine, which matters: a test that takes an hour is a test that gets skipped.
What it is for, and what it is not for
The net is not a way of finding bugs. It cannot tell whether a battle is good or whether a ship is balanced. It answers one question only: did this change do what I meant it to do, and nothing else? That is the right question when we are not trying to change the game. When we split the code into an engine and a game, over three releases, the answer had to be “nothing changed”, and the net could say so with confidence nobody could have got by reading the code. All 93 hashes stayed identical across the whole split.
When we do mean to change the game, the net behaves differently, and this is just as useful. Release 3.37 added a new civilization, the Saltglass Guild, with new ships, a new trait and two new steps in the damage calculation. Every existing battle had to stay identical, because no existing ship uses any of it. They did: 93 of 93. If one had differed, we would have known the new damage steps were leaking into old fights.
The order of the damage steps
A hit in the game is not one multiplication. The damage passes through a chain of named steps in a fixed order: crew bonuses, formation bonuses, armour, vulnerability marks, cloaks and so on. The order matters, because percentages applied in a different order give a different number once there are caps and floors. A tidy-minded cleanup that swaps two harmless-looking steps would change every battle that uses them.
So the order is under its own test. A small program lists the steps and compares them with the expected list, name by name. When we added the two Saltglass steps the list went from 34 to 36 entries, and we edited the expected list in the same change. The edit is deliberate and visible, and that is the point: nobody can reorder the steps by accident, because the test shouts.
Where the net has holes
The hash covers the rules. It does not cover what the player sees or hears. A ship could be drawn as a grey box, a sound could play twice, or the game could fail to start without 3D graphics, and every hash would stay the same. For those we built separate nets, which the engine note describes. And the net fails honestly in one more way: it can only compare against a version we trust. Matching an old version proves that we did not break things, not that things were right in the first place.
Our habit now is to run it before every release and put the result in the validation file that ships with the plugin, next to what was not tested. That second list is shorter than we would like, and it is the more interesting one.