8. Streaming and persistence

Phase 2 in detail. Eleven questions the roadmap entry named but did not answer — chunk lifetime, GPU resource lifetime, ownership, write-back, residency, the world format, and how a player reaches a world at all — plus the parameter values they leave open.

Why these are here rather than in the roadmap

Phase 2 is the phase that turns a fixed test world into a real one, and most of its cost is in decisions rather than code. Two of them — Q1 and Q2 — change interfaces rather than adding code behind them, so they are the ones worth getting right before the phase starts. The rest are ordinary but easy to discover late.

The design context for all of this is section 5; this page is the engineering that follows from it.

Decided Q1 — Chunk lifetime under concurrent access

A mesh job reads its chunk plus a two-cell skin into the surrounding neighbourhood (risk 4). The main thread meanwhile edits and evicts chunks continuously. The Chunk::version counter from risk 2, gap 3 solves staleness — a job returns a mesh built from pre-edit data, versions differ, discard and requeue. It does nothing about the chunk's memory disappearing mid-read, which is a use-after-free rather than a stale mesh.

(a) Stop-the-world — edits and evictions only when no job is in flight. Provably correct, but draining the queue before every eviction reintroduces the stalls this phase exists to remove.
(b) Reference-counted pins — a job pins the 27 chunks it reads, eviction defers while pinned. Stall-free, but one job blocks eviction of 27 chunks.
(c) Bordered snapshot — copy the region the job needs into per-worker scratch before it starts; the job then touches nothing shared.
(d) Per-chunk reader-writer locks — lock ordering across 27 chunks is a deadlock hazard.

Decided: (c), the bordered snapshot. The snapshot is not 27 whole chunks — it is one chunk plus a two-cell skin, so (32 + 4)³ = 46,656 cells, roughly 93 KB at density plus material. That is a fifth of the 430 KB weld scratch already budgeted per worker (risk 4), and it is reused per worker the same way.

Why it wins: it collapses three problems into one. No data race, no use-after-free, and the version check becomes trivially correct because the snapshot is taken atomically with the version. Eviction then needs no coordination with the job system at all — which is what makes Q6 simple.

Decided Q2 — GPU resource lifetime without a stall

Today waitIdle is the only mechanism, and IRenderer.h says so outright: "Call before destroying or recreating resources an in-flight frame may still reference." Buffers are unique_ptr<IBuffer> and free on reset. Once evictions are continuous, every one of them would need a full GPU stall — precisely the stutter this phase exists to remove.

(a) Keep waitIdle — status quo, and catastrophic under streaming.
(b) Deferred deletion queue — the renderer tracks a frame counter and the frames-in-flight count N; retiring a buffer tags it with the current frame, and anything tagged ≤ (current − N) is freed.
(c) Per-buffer fences — more precise, more bookkeeping, unnecessary when N is 2 or 3.
(d) Buffer pooling — recycle rather than destroy, killing allocation churn too.

Decided: (b). One interface addition, and the implementation is contained entirely in the Vulkan backend. waitIdle survives for shutdown and hot-reload, which are genuinely rare.

(d) is a deferred improvement, to revisit if allocation churn measures badly. Note it still needs (b) underneath — a recycled buffer must not be rewritten while in flight either.

Pair this with the construction damage buffer

The other addition IRenderer needs is a frequently-updated per-draw buffer, for construction crack damage (section 10, question 4). Two additions to the same interface, and both now land in this phase — section 10's question 10 moved the construction damage model here precisely so the interface is opened once rather than twice.

The damage buffer is allocated lazily per damaged chunk and freed through this same deletion queue, so the two features share their resource lifetime as well as their interface pass.

Decided Q3 — Who owns the world lifecycle

Today Engine::run() builds a VoxelVolume inline, meshes it once, uploads and loops. Phase 2 needs generate-or-load → stream → save → unload as a real state machine — a restructuring of Engine's ownership rather than an addition to it.

(a) Inline in run() — fastest, but run() is already the frame loop plus input plus HUD plus hot-reload.
(b) A World class owning storage, residency, jobs and persistence; Engine holds one and calls update(playerPos, dt).
(c) Split three ways — store, streamer, live chunk map.

Decided: (b), with (c)'s first seam pre-drawn. One World type owns the lifecycle, but the format sits behind a narrow interface from the start — load chunk, store chunk, read header — because that is the piece that must be versioned and is the most likely to be swapped. The same instinct that produced IRenderer: one boundary that matters, not five speculative ones.

Decided Q4 — Eviction and write-back

(a) On eviction only — minimal I/O, but players stay near what they are building, so a crash loses most of a session.
(b) Write-through per edit — loses nothing, unacceptable for a per-swing density change.
(c) Periodic flush, plus on eviction, plus on quit — bounded loss window, bounded I/O.
(d) Edit journal compacted periodically — strongest crash safety, most machinery.

Decided: (c). Loss is bounded to seconds, I/O is bounded, and it composes with eviction naturally — eviction just forces that chunk's flush. (d) is kept as a potential future improvement if crash-loss ever proves painful.

"Dirty" and "modified" are different flags

They have different lifetimes and conflating them is a classic bug. Dirty means needs-remesh. Modified means differs-from-generated and needs-saving. A chunk remeshed because its neighbour changed is dirty but not modified — so keying the save on the dirty flag writes chunks that never changed, and keying the remesh on modified drops meshes that needed rebuilding.

Construction damage counts as modified. A chunk whose only change is a half-hammered wall still differs from generated and still needs writing back.

Two representations of construction damage, one of which is state

Damage has to survive a save/load cycle — a wall left half-hammered must still be half-hammered next session. But only one of its two forms is state:

The sparse DamageMap is the source of truth and serialises — a handful of entries per chunk, so almost nothing in the format.
The dense 32 KB per-chunk GPU buffer is a derived cache. Never written to disk; rebuilt from the sparse map when the chunk becomes resident (section 10, question 5).

Built: damage is one section, not part of each chunk

The map serialises as a single whole-world section (damage.vcs) rather than travelling inside each chunk's payload. Damage is sparse by construction: under D5 terrain damage is density and is already in the cells, so this map only ever holds construction cells that have been struck and not yet broken — a small and highly transient set. One section costs less than a single chunk's payload, needs no format version bump, and leaves the chunk encoding alone.

The half that is easy to miss: a strike that does not break the voxel changes no cell, so nothing routes through VoxelWorld::set and nothing marks the chunk. It has to be marked by hand, or every swing short of the last one is lost on the next save. The modified flag therefore means this chunk's stored state is stale, which is a slightly wider claim than "its cells changed".

Decided Q5 — The residency margin

The mesher reads two cells into its neighbours, so a chunk cannot be meshed unless its neighbours are resident.

(a) One radius, missing neighbours read as air — every chunk remeshes when a neighbour arrives, giving a remesh wave and visible popping at the frontier. Also wrong under D9, where non-resident does not mean non-existent — the data is on disk.
(b) Two radii — load within R+1, mesh within R. One ring of loaded-but-unmeshed chunks.
(c) Skin-only frontier — store just the two-cell shell of the outer ring.

Decided: (b). For R = 12 the extra ring is roughly 17% more columns, and it removes an entire class of pop and remesh-wave.

(c) is kept in reserve as a memory optimisation. Worth measuring before dismissing: world height multiplies the ring's cost, and the height is 384.

Decided Q6 — Hysteresis, and what actually bounds memory

(a) Nothing — load at R, evict at R; a player pacing across the boundary thrashes.
(b) Two thresholds — evict at R plus a margin.
(c) Time-based — evict after T seconds out of range.
(d) LRU under a memory budget — evict least-recently-needed when over cap, regardless of distance.

Decided: (b) plus (d). Distance hysteresis handles thrashing cheaply, but the LRU cap is what actually protects memory — and it is needed because both view distance and world height feed the resident set, and a user can raise both (risk 5, sub-decision 3). (c) is subsumed by (d) in practice.

Decided Q7 — LOD artifacts in the world format

D11's terminal step produces the coarse terrain mesh and the simplified construction LOD models. Both have to live somewhere the loader can find them.

(a) Sidecar files — rebuild independently, but the two can drift out of sync silently.
(b) One container, one version — consistency guaranteed, but regenerating LOD rewrites everything.
(c) One container, independent LOD version stamp recording which terrain version it derived from.

Decided: (c). LOD is derived data, and derived data goes stale — the LOD step will improve, and regenerating it should not mean regenerating the world. Stamping the terrain version it came from makes a mismatch detectable and loud, which is the same principle already applied to the format version itself.

Decided Q8 — Reaching a world

(a) Path or command-line argument — zero UI, roughly what exists today.
(b) Minimal in-engine menu — list worlds, create-new with seed and size, progress bar.
(c) Full front-end — management, deletion, thumbnails.

Decided: open a default world, plus a debug-only path override. The executable opens the default world with no arguments. A command-line argument supplies a different world path, and that argument is available only in debug builds — matching the existing convention, where shader/texture hot-reload and the per-frame log levels both compile out under NDEBUG. See Building and running.

(b) arrives when generation stops being instant, and is out of scope here — the HUD is a hand-rolled stroke font, so any menu is real work. But the directory layout and world naming are decided in this phase, alongside the format: retrofitting a layout means migrating worlds.

First run

When the default world does not exist yet, it is generated and then serialised under the default name, so subsequent runs load it rather than regenerating. Generation shows the progress bar D9 already requires.

Which makes first-run the first real exercise of the whole format path — generate, write, and read back — and is why Q10's round-trip test is worth having before it rather than after.

Decided Q9 — Spawn point

(a) World centre, dropped from the sky — may land inside a mountain or over water.
(b) Centre, raised to the surface by sampling the column's surface height.
(c) A generator-chosen spawn — a stage picks somewhere flat, dry and cave-free and records it in the world header.

Decided: (b) now, (c) once the infrastructure step exists. (b) is a few lines and correct for a terrain-only world. Once cities and roads exist, spawn placement becomes a content decision that belongs in the pipeline as an output — and writing it to the header means it costs nothing at load time.

Decided Q10 — Proving a world survives a round trip

(a) Nothing — rely on it visibly working.
(b) Generate → save → load → compare chunk hashes, reusing the FNV-1a hash risk 2 already specifies for the mesh verifier.
(c) (b) plus a golden-world fixture committed with an expected hash.

Decided: (b). Cheap, and it catches silent corruption — which is the failure that matters most, since the stored world is the durable artifact and nothing else can reconstruct it. Fits voxel_tests if the format layer stays free of renderer dependencies, which Q3's narrow format interface already encourages.

Q11 makes this test more valuable than it first looks: once chunks are compressed, the round trip also proves the codec round-trips. That is exactly the kind of bug that would otherwise surface as terrain quietly wrong in one region of one world.

(c) is deferred deliberately. A golden fixture makes every intentional generator improvement a test failure needing a regenerated fixture. That is technically correct under D9 — a distributed world must not change when the generator does — but it is friction during phase 1's tuning. Revisit once the generator stabilises.

Decided Q11 — How the world artifact is compressed

A 2 km world at 384 height is 64 × 64 × 12 = 49,152 chunks at 128 KB each — ~6.4 GB raw. A 4 km world is ~25 GB; even 1 km is 1.6 GB. Unreasonable on disk regardless of whether worlds are ever shared, so this is a requirement rather than an optimisation.

It is also an unusually favourable problem. Most of the world is solid rock at density 255 or air at 0, materials change in broad strata, voxelClass is Terrain nearly everywhere, and the genuinely varying region is a few voxels thick around the surface and around caves.

Two requirements that come before the codec

Both apply whichever compressor is chosen, and both are nearly free.

Split the fields before compressing — SoA, not AoS. Voxel is four interleaved bytes, so a run in that stream only continues while all four fields stay equal. Near the surface density changes every voxel while material stays constant, so runs break constantly on the one field that is varying. Transposing to planes — all densities, then all materials, then all class and type — lets each compress on its own terms: long 255 and 0 runs in density, long within-stratum runs in material, near-constant class. This multiplies the ratio several times over and costs only a transpose.
Elide uniform chunks entirely. A chunk that is one voxel value throughout — all air, or one material at full density — stores as a small header record rather than 128 KB. In a 384-tall world most vertical chunks in a column are exactly that, with only the one or two straddling the surface plus cave-bearing ones carrying real content. A uniformity check at write time, and nothing else.

How large a fraction is uniform should be measured, not assumed — it depends entirely on how thick D10's strata end up and how pervasive the caves are. The estimate worth planning against is 40–60%, and worth checking early because it dominates everything else here.

Then the codec

(a) None — 6.4 GB. Not viable.
(b) Per-chunk RLE on the SoA planes — no dependency, near memory speed in both directions, and run-length is precisely what this data's structure rewards.
(c) Per-chunk general-purpose compression (zstd or lz4) on the SoA planes — better ratios, especially across the varying surface band where RLE degrades. Costs a dependency, though vcpkg already carries the pattern.
(d) Per-region blocks — best ratios, by compressing many chunks together for cross-chunk redundancy.

Decided: (b), measured, escalating to (c) only if the ratio disappoints.

Per-chunk rather than per-region, because that is what the streamer reads. Residency loads chunks, and Q5's R + 1 border ring loads single chunks at the frontier — where a whole-region decompress would be pure waste.

And RLE rather than zstd first, because ratio is not the only axis. Compression sits directly in generation's critical path: D9 already makes world creation a visible up-front wait, and pushing 6.4 GB through a compressor at ~200 MB/s adds around 30 seconds to it. RLE runs near memory speed; zstd at level 1 does roughly 500 MB/s. With both pre-requirements above, (b) should take 6.4 GB to a few hundred megabytes — enough that the problem stops existing.

Record the codec per chunk, so this choice stays reversible

A codec byte in each chunk's index entry means escalating from RLE to zstd later needs no format version bump and does not invalidate existing worlds — the two coexist, old chunks read as RLE, new ones as zstd.

That serves D9's versioning requirement directly: version bumps should be reserved for changes that genuinely cannot be read, not for improvements. And it is the main reason the simpler codec is safe to pick first.

It generalises to per-section too, which matters because the LOD artifacts from Q7 are different data — the coarse mesh is float-heavy vertex and index data that RLE handles poorly. Give that section its own codec, or leave it uncompressed until it is measured; the format does not need to care yet.

Parameters

Values that the decisions above leave open. Most are empirical — they need real terrain on real hardware to settle — so these are starting points to measure from, not conclusions. World height (384) and chunk size (32³) are fixed elsewhere and repeated here for arithmetic.

#	Parameter	Value	What it decides
1	Voxel scale	1.0 m	Underpins every distance and memory figure. Was implicit; now stated. Makes `kEditReach = 8` read as 8 m
2	Chunk size	32³	128 KB per chunk. Rebuild cost scales with the cube of the side, and D5 makes remeshes frequent, so growing it is unattractive
3	World size ladder	1 / 2 / 4 km, default 2 km	1 km fits densely in RAM so it never exercises streaming honestly; 4 km is reference scale; 2 km forces real streaming at ~1.6 B voxels
4	Residency radius `R`	12 chunks (384 m), 8–20 configurable	How far chunks are loaded and meshed. ~800 MB resident at height 384 including the load ring
5	Load radius	`R` + 1 — derived	One ring loaded but unmeshed, so every meshed chunk has its full 2-cell border (Q5)
6	Evict radius	`R` + 2 — derived	A 32 m dead band. A player would have to pace 32 m to thrash, which is not a pacing motion (Q6)
7	Voxel memory budget	1.5 GB	The LRU cap. `R` clamps down to fit it. 1.5 GB permits about `R` = 17
8	Frames in flight `N`	1 (current)	Depth of Q2's deletion queue. Keyed by frame index regardless, so raising `N` later needs no rework
9	Upload budget	4 meshes/frame	~26 MB/s at 60 fps and ~110 KB per mesh. Deliberately conservative — raise until frame time suffers. Counts meshes, not chunks
10	Worker threads	`hardware_concurrency` − 2, min 1	Same pool serves bulk up-front generation and runtime meshing
11	Flush interval	30 s	Q4's periodic write-back. Bounds crash loss to half a minute of mining
12	`kBand` / max Δdensity	40 / 64	The band-limited contract (D11). 40 spans the ramp over ~3.2 voxels; the 64 bound leaves headroom for caves and ore whose gradients differ from the surface

#7 must be revisited whenever the voxel changes size

The budget is meaningful only relative to sizeof(Voxel), which is 4 bytes today — type, colorIndex, density, voxelClass. Any change to that layout changes what 1.5 GB buys, and therefore the R it permits.

There is a known lever here: VoxelClass is one bit and VoxelType is a small enum, so they pack into a single byte for 3 bytes total — a straight 25% cut across the whole working set. Worth knowing it exists before raising the budget rather than after.

Invariants

Enforced in code so a bad combination is unrepresentable rather than merely discouraged:

load = R + 1 and evict = R + 2 — derived, never set independently.
R × chunkSize < farPlane. At 32 m per chunk against the current 1000 m far plane, that caps R at 31.
R clamps down to fit the memory budget, and says so when it does. Without this the LRU evicts chunks the residency rule immediately reloads — permanent thrash, worst exactly when the machine is already short of memory.
R × chunkSize > kEditReach — automatic at any sane R, but assert it.

One setting fewer than the plan used to have

Risk 5's sub-decision 3 made the full-resolution density radius a separate user setting, floored above reach distance and clamped to the streaming radius. That was written when a runtime LOD hierarchy existed and there was a coarse-but-resident band between the two.

The static distant mesh removed that band. There are now only two tiers — resident full-resolution voxels, and the pre-generated distant mesh — so the full-resolution radius and the residency radius are the same knob. The "floor above reach distance" constraint becomes automatic, since even R = 8 is 256 m against an 8 m reach.

What this adds to phase 2

Two interface changes, one restructuring, and seven pieces of ordinary work:

	Kind	Touches
Q1 bordered snapshot	New per-worker scratch	Meshing job setup
Q2 deferred deletion	Interface change	`IRenderer` + Vulkan backend
Q3 `World` ownership	Restructuring	`Engine`, new `World`, format interface
Q4 flush policy	Ordinary	`World`, chunk flags
Q5 two radii	Ordinary	Residency
Q6 hysteresis + LRU cap	Ordinary	Residency
Q7 LOD version stamp	Ordinary	Format
Q8 default world + debug override	Ordinary	`main.cpp`, format
Q9 surface spawn	Ordinary	`World`
Q10 round-trip test	Ordinary	`voxel_tests`
Q11 SoA + uniform elision + per-chunk RLE	Ordinary	Format layer

And one decision arriving from phase 3

From phase 3 a chunk carries two meshes — MC terrain and cube construction — on two pipelines, with independent dirty state (section 9, question 1). Three things above are shaped by it:

Q1's job result type — a mesh job produces two meshes, or is parameterised by which one it is building. The bordered snapshot serves both, since they read the same field.
The upload budget counts meshes, not chunks — otherwise a chunk carrying both blows the per-frame cap by 2×. Built: and it counts uploads rather than results — an empty mesh creates no buffers, so it stalls nothing and rides free, which matters because around 43% of a generated world's chunks mesh to nothing.
Q2's deletion queue granularity — retirement is per mesh, since one can be rebuilt while the other is untouched.

All three are cheap here and awkward to retrofit, which is why the decision was taken before this phase starts rather than when phase 3 reaches it.