NEON CITYmaking of
05 / 07

Quality

Green tests are only the start

Thousands of unit tests, mutation tests that check whether the tests catch bugs at all, a visual smoke run in a clean browser and a pixel comparison with the baseline. And the owner still plays it before anything merges.

5.1 Gates before a merge

Quality gates diagram, left to right: unit tests, mutation tests, clean export with build and distribution test, visual smoke in headless Chrome, pixel comparison with the baseline, performance gates, Pages preview, staging with playtest, and merge

Each gate answers a different question. Unit tests: does the code do what it claims. Mutations: would the tests even notice if it didn’t. Clean export: does what works in the working directory also work from the repository alone. Smoke: does the game start and look right. Performance: does it fit the budget. Playtest: is it any good.

5.2 Mutation tests

The reviewer breaks the code on purpose – flips a condition, changes a constant, brings back an old bug – and checks whether any test notices. A mutant that survives means a hole in the tests, even when every test is green.

After TRAM-1 the old, wrong winding of the D5 corridor joined the mutation set. The ordinary-match test catches it immediately – the same bug won’t quietly come back.

Versioning is checked separately: a change to a generator rule must change the version, and a change to an input that doesn’t affect output must leave it alone – 15 of 15 trials behaved as expected.

Mutations after the second TRAM-1 review: 17 of 17 caught (selection)
MutationResult
Old D5 winding restoredcaught
Left/right door sides swappedcaught
Damage from a standing tramcaught
No “once per pass” limitcaught
Shortest routes instead of longestcaught

5.3 Visual smoke and the pixel criterion

The smoke run starts the production build from a clean export of the repository in headless Chrome, with a local Worker and no network access beyond the machine. Every console error and every 404 response is a failure. Known environment issues are listed in the report, not masked.

Screenshots are taken across a matrix of windows – from a 667×375 phone to a 2560×1080 monitor, by day and at night, at High and Low quality. When a change must not touch the existing mode, a hard criterion applies: 0.000 % of pixels differing by more than 8/255 from the baseline.

Smoke screenshot on a 667×375 phone at night: weapon, HUD and touch buttons
667×375 – phone, touch controls. Map: © OpenStreetMap contributors (ODbL).
Smoke screenshot on a 2560×1080 monitor at night: a wide view of the square with weapon and HUD
2560×1080 – ultrawide monitor. Map: © OpenStreetMap contributors (ODbL).
A horde of 120 zombies at night seen from a distance on a city square
Horde of 120 – the GPU measurement scene. Map: © OpenStreetMap contributors (ODbL).

5.4 Performance gates

Measurements from the TRAM-1 sign-off (threshold alongside)
MeasurementResultThreshold
Simulation tick, p95 (PLAY-1 scenario)1.344 ms3 ms
Enemy field, p950.744 ms1.5 ms
Łódź r600, 3 moving trams, 60+ bots, p951.047 ms3 ms
LocalServer: 169.0 → 40.9 ms4.13×≥ 3×

Measurements have conditions: mains power, a quiet window and CPU load below a threshold. When a condition isn’t met, the measurement is skipped and documented in the report – failed measurements are never replaced with zeros or older results.

5.5 The work in numbers

  • 869commits in 15 iterations (client + backend)
  • 179task reports in the repositories
  • 1634 + 937backend and client tests, 0 failures
  • 4.13×LocalServer speed-up (threshold: 3×)
Commits per iteration, iterations 1–15Stacked bars: client and backend repositories. Peak on iteration 9: 136 commits. 050100150 clientbackend iteration 1: client 15, backend 12, total 27 1 iteration 2: client 21, backend 14, total 35 iteration 3: client 23, backend 25, total 48 3 iteration 4: client 13, backend 10, total 23 iteration 5: client 12, backend 10, total 22 5 iteration 6: client 12, backend 4, total 16 iteration 7: client 47, backend 6, total 53 7 iteration 8: client 40, backend 7, total 47 iteration 9: client 107, backend 29, total 136 1369 iteration 10: client 48, backend 43, total 91 iteration 11: client 45, backend 41, total 86 11 iteration 12: client 18, backend 23, total 41 iteration 13: client 39, backend 30, total 69 13 iteration 14: client 57, backend 9, total 66 iteration 15: client 86, backend 23, total 109 15
Commits per iteration, iterations 1–15. Stacked bars: client and backend repositories. Peak on iteration 9: 136 commits.
Show the data as a table
Iterationclientbackendtotal
iteration 1151227
iteration 2211435
iteration 3232548
iteration 4131023
iteration 5121022
iteration 612416
iteration 747653
iteration 840747
iteration 910729136
iteration 10484391
iteration 11454186
iteration 12182341
iteration 13393069
iteration 1457966
iteration 158623109
Σ583286869

Numbers aggregated from git history (all branches) and task reports; no branch names or commit contents.