Testing a Godot game without opening it

Games get tested by playing them. That works for feel, and feel is most of what matters in a game, so it is a reasonable default. It stops working when the game is a simulation, because the failures that matter stop announcing themselves.

I build a space colony sim in Godot. It has crops, livestock, power and water networks, illness, hauling, mining, a work rota and a fleet, and every one of those touches at least two others. The test suite runs headless, without a renderer, and it now stands at 1,740 checks across 61 suites. This is how it is put together and, more usefully, what it found.

The failures do not look like failures

Here are four real bugs from this project. I have put what a person playing the game would have seen next to what was actually happening.

Four real bugs. The left column is what playing the game would have shown you.

Take the first one. Every crop grower cached its configuration in _ready(), but on some build paths the node enters the tree before initialize() has assigned its item type. So growers cached the default item's data, which is a bed, and every data-driven behaviour quietly defaulted off. Soil beds stopped watering themselves. Hydroponic racks lost their grow-speed bonus. Tree plots stopped enforcing tree-only planting.

You would never have noticed, because colonists water crops by hand as a fallback. The fallback did its job perfectly and hid the fault underneath it. The crops grew. The colony was fine. A feature I had written and believed was working had never once run.

The second is worse in a way I find funnier. The animal-harvest logic lived in process_work(), and nothing calls process_work(). It is an unused stub on the base class. A farmer would walk to the pen, enter the working state, play the working animation, and stand there for the rest of the game doing nothing at all. It looks exactly like a colonist working.

None of these four produced an error, a crash, a warning, or a visibly wrong frame. Three of them made the game slightly easier or slightly duller in ways no player would attribute to a bug. You cannot find these by playing, however long you play, because there is nothing to see.

One Godot process per suite

The mechanics are simple. Each suite is a scene with a script that runs checks, prints one summary line in a fixed format, and quits. A bash script runs all 61 and adds up the counts.

The part worth copying is that each suite gets its own Godot process.

Sixty-one process launches is the cost. A failure that means what it says is what you get for it.

Godot autoloads are global singletons, and a colony sim leaves a great deal behind: rooms in the game manager, entries in the resource networks, spawned colonists still walking about. Run two suites in one process and the second one inherits all of it. Then a test passes because an earlier suite happened to leave a colonist in a convenient place, and you have built the thing you were trying to avoid: a check that cannot fail, telling you nothing.

Sixty-one process launches is genuinely slower. A full run takes fourteen minutes and fifty-two seconds on my machine, almost all of it engine startup. That is a fine trade. The suite runs while I do something else, and when it comes back the result means what it says.

One detail that cost me an afternoon: the exit code is not trustworthy here. This project links a Steam GDExtension whose native library is absent under --headless, so Godot returns non-zero even on a clean quit(0). The runner therefore judges each suite by parsing its printed summary line, and treats a missing summary as a failure. If your test runner reads only the exit code, check that a crash and a clean pass actually produce different ones.

Where the checks are

One bar per suite, from a real run. The chart is linear, so the outlier stays an outlier.

The distribution is lopsided and that is not an accident. The incidents suite holds 266 checks on its own because reactive triggers have the largest surface in the game: every condition that can fire an event, and every gate that stops it firing at the wrong moment. Life support is 71, colony events 63, save and load 56 across two suites.

The long tail matters just as much. A suite with four checks in it is almost always a bug that came back once. That is the rule I hold to and the reason the number climbed from six suites to sixty-one: a bug found is a regression test written, before the fix, every time. Writing the test first is not discipline for its own sake. It is the only way to know the test would have caught it, and a regression test that passes before you fix anything is not a regression test.

What it actually saves

I want to be careful here, because "hours saved" is the kind of claim that gets made without a stopwatch.

The honest version is not that headless tests are faster than playing. It is that for this class of bug, playing does not work at any speed. Three of the four bugs above have no observable symptom. The fourth, the vanishing produce, would show up eventually as a fridge that fills more slowly than it should, at which point you are debugging a statistical impression rather than a fault.

What the suite does buy me, measurably, is the freedom to change things. This is a solo project with 641 commits and a lot of interlocking systems. Refactoring hauling when hauling feeds the feeders, which keep the livestock alive, which the kitchen depends on, is not a change I would make on a Sunday afternoon on the strength of a ten-minute play session. With the suite it is a change I make, run, and either keep or throw away fifteen minutes later.

This is not really about games

The pattern generalises, which is why it is on this site rather than in a devlog.

A fallback that hides the failure it is covering. A method that is never called and looks identical to one that is. A loader that reports success and produces nothing. A resource that is drawn from a global total instead of a live connection, so the check passes whether or not the thing is connected.

Those are all production bugs. I have met every one of them outside a game, wearing different clothes. The reason they survive is the same in both places: the system carried on, nothing went red, and there was no test asking the specific question that would have failed.