Release 0.9 records fireEngine’s first frame with more than one CPU participant. The normal application still loads, animates, and presents AnimatedCube, but the path beneath it now separates mutation, immutable frame input, command recording, submission, and presentation by ownership and lifetime.

This document describes the complete system at tag 0.9. It stands on its own: understanding it does not require the 0.7 or 0.8 architecture pages. Stable foundations are repeated where they remain important, while experiments that did not survive into the release are intentionally absent.

The pinned 0.9 source tree is the authority on types and functions, and the 0.9 release notes summarise the shipped result. The generated public documentation and internal documentation track the current tutorial revision rather than this tag, so both will drift from this page.

The organising decision

Every concurrency boundary in 0.9 follows from one rule: mutable ownership stays serial; parallel recording receives only a frozen view and the authority needed to encode its own commands.

The application finishes animation and transform mutation before it freezes a SceneDrawList. The renderer resolves that external view into a RecordingInput containing immutable packets and plain Vulkan handles. Only then may the coordinator and helper read disjoint spans concurrently. Neither recording job receives authority to mutate the scene, compile resources, submit work, present, or destroy the RAII owners behind those handles; the coordinator resumes those serial responsibilities after recording joins.

serial owners                       read-only recording transaction

SceneContent
    |
    +-- mutate animation and transforms
    |
    v
SceneDrawList --------------------> RecordingInput
immutable application view          resolved packets + plain handles
                                             |
                         +-------------------+-------------------+
                         v                                       v
                    coordinator                              helper
                    own command pool                         own command pool
                         +-------------------+-------------------+
                                             |
                                             v
                                      ordered execution

============================ Vulkan ownership boundary ===================
Device + compiled graph + presentation + frame slots stay on the coordinator

This is a capability boundary, not merely a promise to use const correctly. Vulkan-Hpp RAII wrappers can expose operations that affect GPU state through apparently read-only C++ objects. Worker-facing packets therefore copy raw, non-owning handles by value instead of borrowing resource owners.

The second participant is a consequence of that structure rather than its foundation. The same immutable path records correctly with one participant, and the release keeps that path as the normal choice for smaller workloads.

System map

The format-neutral content and Vulkan façade from 0.8 remain. Version 0.9 adds an application layer for scenarios and measurement, strengthens the scene and preparation boundaries, and divides the renderer’s internal work by lifetime.

src/app/
├── load ──────> gltf/ ──────> content/
├── mutate ────> animation/ ─> scene/
├── freeze ────> scene/ ─────> SceneDrawList
└── render ────> render/ ─────> render/detail/

graphics/ supplies assets, preparation, and pipeline requirements
platform/ supplies the GLFW window, surface, and FramebufferExtent
math/ supplies transforms and application-owned camera calculations
core/ supplies logging and non-Vulkan internal helpers

Area	Responsibility	Knows about
math/	Vectors, quaternions, transforms, matrices, camera mathematics	no engine layer
graphics/	Format-neutral assets, typed IDs, pipeline requirements, preparation	maths; scene draw views in preparation implementation
scene/	Forest ownership, insertion-time identity, transforms, immutable draw views	maths, graphics IDs, animator binding
animation/	Rotation samples, validation, and serial playback into local transforms	maths and scene
content/	One composition of assets, hierarchy, and animations	graphics, scene, animation
gltf/	Translation of the supported glTF slice into SceneContent	content, plus graphics and scene types internally; fastgltf and stb
render/	Vulkan-free camera and renderer façade, configuration, outcomes, timings	graphics, scene, maths, platform values
render/detail/	Resource compilation, immutable packet resolution, Vulkan ownership, recording, submission, and presentation	Vulkan, VMA, Slang output
platform/	GLFW lifetime, native window, events, framebuffer dimensions, surface creation	GLFW; Vulkan only at the surface seam
core/	Logging and internal hashing support	standard library

core/ is cross-cutting and omitted from the other rows’ dependency lists. The existing source-level relationship between graphics/ preparation and scene/ draw views remains; the directory graph is not presented as a strict acyclic stack.

The mutation, freeze, record contract

CPU pipeline depth remains one. One application thread drives each frame through a fixed sequence:

elapsed time
    |
    v
advance animations             mutate local transforms
    |
    v
update world transforms        resolve the hierarchy
    |
    v
build draw list in arena       freeze ordered IDs + world matrices
    |
    v
Renderer::drawFrame()
    |
    +-- compile RecordingInput before acquisition
    +-- wait for the selected frame slot
    +-- acquire one swapchain image
    +-- record one or two secondary command buffers
    +-- execute them from one primary command buffer
    +-- submit, then present

No application mutation overlaps recording. No second CPU frame is being prepared while the first records. Two Vulkan frame slots instead allow the CPU to prepare and submit a later frame while the GPU may still be executing the previous slot.

This distinction creates three independent depths:

Depth	0.9 value	Controlled by
CPU frames being prepared concurrently	1	application protocol
submitted Vulkan frames that may remain outstanding	2	frame-slot cycle
presentation queue depth	driver-selected	swapchain image count

A swapchain with three images does not imply three frames in flight. Frame-slot selection advances after successful graphics submission; swapchain-image selection comes independently from acquireNextImage().

Scene identity and immutable draw views

Scene still owns a forest of immovable SceneNode values through unique pointers and keeps a dense, non-owning SceneNode* registry. Version 0.9 makes registration an insertion invariant:

• addRoot() validates and registers the complete incoming subtree;
• addChild() accepts a scene-local ID or a verified scene-owned node;
• every node receives its dense SceneNodeId when it enters the scene; and
• parent nodes enter the registry before their descendants.

updateWorldTransforms() now only resolves transforms. It performs no hidden structural registration pass, which makes the mutation phase more precise and keeps stable identity independent of frame traversal.

Draw-list storage also moves out of the returned value:

SceneDrawListArena                         SceneDrawList
owns reusable vector<DrawItem>             small non-owning value
high-water allocation survives frames ---> span<const DrawItem>
                                           dependency hash

Scene::buildDrawItems() rewrites the arena in stable depth-first order and returns an immutable view. The span remains valid until that arena builds another list or is destroyed. Its elements are read during preparation or one synchronous draw call; they are not retained until GPU completion because the command buffers contain the encoded values by then.

The dependency hash covers the ordered RenderObjectId sequence, including duplicates, but excludes transforms. Animation can therefore change every world matrix without presenting a new resource dependency.

Preparation and strongly committed resources

Preparation remains explicit and device-free, with one additional input: the pipeline’s format-neutral vertex-layout requirement.

RenderAssets + SceneDrawList + PipelineDescription
                         |
                         v
                 RenderPreparation
          validate + select + cache plan
                         |
                         v
              RenderPreparationPlan
 meshes, images, textures, materials, prepared render objects
                         |
                         v
                  ResourceCompiler
          allocate + stage + upload candidate
                         |
                         v
              CompiledResourceGraph
                         |
                  successful only
                         v
                CompiledResources

The plan key contains the asset collection’s identity and revision, the exact ordered render-object dependencies, and PipelineDescription. The current pipeline description names one interleaved vertex layout. Preparation proves that every selected mesh matches it before Vulkan allocation begins.

ResourceCompiler owns a dedicated transient command pool, upload command buffer, and fence. Setup uploads no longer borrow a frame slot, so compilation and per-frame submission have separate synchronization owners. Compilation produces a complete CompiledResourceGraph containing buffers, images, samplers, the persistent white fallback, and resolved draw packets.

CompiledResources does not partially mutate that graph while work can fail. It accepts the complete candidate only after compilation and upload succeed. The old graph therefore remains coherent if candidate construction throws, and replacement occurs only after submitted work has been retired. Internal declaration order ensures texture borrowers die before their image owners. The draw-packet table contains plain handles, so it introduces no RAII ownership relationship of its own.

An unchanged preparation generation returns before retirement, allocation, or upload. Transform-only frames do not call prepare() at all in the normal loop; even if they did, their unchanged ordered dependencies would reuse the same plan and compiled graph.

Recording input is a restricted capability

SceneDrawList is immutable transport, but any caller can construct a span and IDs. It is not proof that those IDs were prepared. RecordingInputCompiler provides the stronger renderer-internal boundary:

external, descriptive                internal, recording-ready

SceneDrawList                  --->  RecordingInput
  RenderObjectId                        RecordingState
  world transform                      pipeline + uniform + frame values
                                       viewport + scissor + formats

CompiledResourcesView         --->    span<const RecordingDraw>
  packet lookup only                   buffers + image handles
                                       index count + draw constants

Compilation validates that every draw belongs to the prepared generation and matches the selected vertex layout, resolves it through a packet-only CompiledResourcesView, copies current transforms and material values, and returns a RecordingInput backed by a reusable packet arena.

The result is deliberately non-copyable and non-movable. Only its compiler can construct it, the compiler must not build another input while consumers remain, and every synchronous recording participant must finish before the call returns. Those constraints make the transaction visible in the types, while compiler reuse retains an explicit lifetime precondition.

RecordingState and RecordingDraw contain plain Vulkan handles, small values, and no RAII owners. A recording participant can bind and encode the supplied resources, but it cannot reset another context, replace the compiled graph, allocate memory, submit to a queue, or present an image.

Frame slots, recording contexts, and presentation images

The renderer separates ownership according to what chooses and retires it:

Renderer::Impl
├── Device
├── MemoryAllocator
├── ResourceCompiler
├── PresentationState
│   ├── Swapchain
│   │   └── render-finished semaphore per image
│   ├── DepthBuffer[2]                 indexed by frame slot
│   ├── Pipeline
│   └── present fence per image
├── FrameResources[2]
│   ├── FrameSlot
│   │   ├── frame uniform buffer
│   │   ├── image-available semaphore
│   │   └── submission fence + pending bit
│   ├── coordinator primary context
│   └── secondary context[2]
├── RenderPreparation + CompiledResources
├── RecordingInputCompiler
└── SecondaryRecordingWorker

FrameSlot owns submission synchronization and frame-uniform storage. RecordingContext separately owns one externally synchronized transient command pool and either a primary, secondary, or no command buffer. Separating them lets each recording participant reset only its own pool without receiving submission authority.

Depth attachments are selected by frame-slot index because two outstanding frames must not write the same depth image. They nevertheless live inside PresentationState because their extent and format must remain compatible with the swapchain and pipeline. Presentation replacement recreates both depth attachments with the swapchain-compatible group while leaving frame-slot synchronization and recording contexts alive.

Swapchain images use a different index. Each owns a render-finished semaphore, and each presentation request is paired with a fence through swapchain maintenance. The slot fence retires graphics submission; the per-image present fence proves that presentation has released the image’s semaphore and other presentation resources.

One or two secondary recording participants

The production path records geometry into secondary command buffers inherited into one dynamic-rendering instance. Participant selection is workload-based:

resolved draw count
       |
       +-- fewer than 10,000 draws --> coordinator only
       |
       +-- 10,000 draws or more ----> coordinator + one helper
                                      at least 5,000 each

The threshold is an internal empirical policy recorded in RendererInfo, not a claim that every workload crosses over at exactly that point. A diagnostic configuration can force one or two participants so both paths remain reproducible and testable below the automatic threshold.

For a split, the coordinator receives the first contiguous half, rounded up, and the helper receives the remainder. Each resets its own pool, begins its own secondary with complete dynamic-rendering inheritance, establishes all fixed state, and records its draws. The primary begins rendering once both are complete and executes their command buffers in original draw order.

immutable RecordingDraw[]

coordinator: first range  ---> secondary 0
helper:      second range ---> secondary 1
primary:     execute secondary 0, then secondary 1

The helper is one persistent, single-purpose thread rather than a task system. The coordinator publishes one copied job through a semaphore. Completion is published atomically; the coordinator polls for at most 50 microseconds and then uses an atomic wait. Destruction requests one final wake and joins the thread, and the renderer declares the helper after the recording contexts so reverse destruction stops it before those pools disappear.

Failure follows the same join rule as success. The helper captures its exception and publishes completion; the coordinator observes completion before rethrowing it. A scope guard also joins outstanding helper work if the coordinator’s own recording throws. No job arguments or arena storage can expire while the other participant may still read them.

Command-buffer-local state

Every command buffer begins from unknown binding state and owns an independent DrawBindingState cache. The fixed preamble binds the pipeline, viewport, scissor, and frame uniform. Per draw, the cache emits only changes:

Input change	Command emitted
vertex or index buffer differs	bind vertex and index buffers
sampler or image view differs	push the sampled-image descriptor
every draw	push model/material constants and call drawIndexed

The cache never crosses a command-buffer boundary. Both participants therefore pay the first-bind cost and can independently skip redundant state without depending on which range another participant recorded.

CommandRecordingMode::eDirectPrimary remains an attribution control. It records the same draw packets directly into the primary command buffer while preserving an empty worker-pool reset for comparison. It is a diagnostic structure, not the normal rendering path.

The frame protocol

Renderer::drawFrame(drawList, camera) has one ordered transaction:

1. Resolve the immutable RecordingInput, including the application-owned camera and extent-dependent view projection.
2. Wait for the next cycled frame slot’s submission fence.
3. Write that slot’s frame uniform.
4. Acquire a swapchain image with the slot’s image-available semaphore.
5. Wait and reset the acquired image’s presentation fence when necessary.
6. Reset the coordinator pool and record the selected command structure.
7. Reset the slot fence, submit the primary, mark the slot pending, and advance the slot cycle.
8. Present with the acquired image’s render-finished semaphore and present fence, then return a RenderResult.

Recording input is compiled before image acquisition. Invalid IDs, incompatible layouts, or a bad camera therefore fail before a successful acquisition can leave a signalled semaphore with no submission to consume it. An out-of-date acquisition submits nothing, leaves the frame fence signalled, and does not advance the frame-slot cycle.

The camera is a Vulkan-free value owned by the application and passed with each frame. The renderer derives the projection from the current extent without retaining mutable camera policy. Framebuffer Y inversion is expressed by a checked negative-height Vulkan viewport; projection mathematics remains right-handed with zero-to-one depth.

Presentation and the platform boundary

Presentation replacement retains the 0.8 whole-group protocol. Before the old group is destroyed, the renderer retires submitted work from both frame slots and waits for every presentation fence. A complete replacement PresentationState is then built using the old swapchain as a retirement hint. Compiled scene resources, upload state, recording contexts, and the helper survive the replacement.

Version 0.9 narrows the public resize path. Window::framebufferExtent() returns an engine-owned FramebufferExtent { width, height }, and the application passes that value to recreatePresentation(). Conversion to vk::Extent2D occurs only inside the Vulkan implementation. A zero extent is a transient outcome rather than an error; the event loop waits until the window becomes drawable.

Window::createVulkanSurface() remains the deliberate GLFW-Vulkan integration seam. Platform code is therefore not entirely Vulkan-free, but ordinary window sizing and the public renderer surface no longer exchange Vulkan types. Instance-extension discovery and validation-messenger construction now live in render/detail/instance_support; core/ has no Vulkan dependency in 0.9.

Failure and outcome model

Failures are classified by what the caller can do with them:

Mechanism	Meaning
std::invalid_argument	malformed content, camera, or requested configuration
std::logic_error	valid API used in an invalid state, such as drawing before preparation or referencing an unprepared object
std::runtime_error or internal vk::SystemError	file, allocation, upload, device, or presentation capability failed
RenderResult	normal presentation outcome: presented, presented but suboptimal, or not presented because out of date
bool recreatePresentation(...)	replacement completed, or the framebuffer is temporarily zero-sized

Worker failure does not introduce a second error channel. It is captured on the helper, synchronized with the coordinator, and rethrown on the calling thread. This preserves the serial public contract while ensuring the worker has finished before stack objects or recording arenas unwind.

Strong replacement boundaries apply to both stable resource compilation and presentation construction. A failed candidate does not become the current owner graph. Destructors make defensive retirement attempts and log cleanup failures rather than throwing.

Measurement is part of the architecture

Concurrency choices are exposed through reproducible controls rather than a single headline frame time. Release builds can generate fixed 1,000- and 10,000-instance workloads, force participant counts, or select the direct primary control. The application separates transform update and draw-list construction from renderer-owned phases.

RendererCpuTimings records frame-slot waits, acquisition and presentation waits, recording-input compilation, uniform updates, pool resets, primary and secondary recording, secondary execution, submission, and presentation. Participant blocks record their own reset and recording intervals. The coordinator-observed secondary region includes dispatch and join overhead; participant CPU durations remain diagnostics and are not added as though their overlapping time were sequential.

The shipped policy uses two participants only at the workload where both measured implementations benefited: 10,000 synthetic draws, or 5,000 per participant. The diagnostic override remains available because the one-thread path is both the production fallback and the control needed to revisit that policy on another driver.

Public façade and internal implementation

renderer.hpp contains configuration, camera input, Vulkan-free renderer information, timing values, and presentation outcomes, but no Vulkan type. Renderer remains a non-copyable façade over one implementation pointer, and the application owns no device resource directly.

The ownership and capability types that make recording safe live under render/detail/: ResourceCompiler, CompiledResourceGraph, CompiledResourcesView, RecordingInputCompiler, RecordingInput, RecordingContext, FrameSlot, DrawBindingState, and SecondaryRecordingWorker. Public Doxygen excludes detail headers and source files; internal Doxygen includes them, and CI checks that boundary.

The public API is explicitly serialized: callers must not invoke member functions concurrently on one Renderer. Parallelism is an internal implementation detail confined to the recording transaction.

Build and test architecture

The build produces the same three principal targets with a larger verification surface:

fireEngineTutorialEngine                 static library
       |
       +--> fireEngineTutorialTests      61 device-free Catch2 cases
       |
       +--> fireEngineTutorial           application, smoke, and benchmark paths
               |
               +--> 11 standard Vulkan registrations
               +--> 5 Debug synchronization-validation registrations

61 + 11 + 5 = 77 CTest registrations in Debug

Device-free tests cover the existing content path plus insertion-time scene identity, camera validation, pipeline-aware preparation, binding-cache decisions, recording-input provenance and packet resolution, and helper dispatch, completion, stale-failure clearing, and exception propagation.

The application registrations retain the four integration scenarios and add direct-primary, forced split, forced fallback, mixed-resource, and automatic policy coverage. All Vulkan scenarios share one CTest resource lock, fail on a validation error, and have a 30-second timeout. Five Debug variants enable synchronization validation for replacement and command-structure paths. The basic smoke scenario still runs from an isolated working directory.

Linux CI builds and runs all 77 Debug registrations through Xvfb and Lavapipe, then builds a separate Release executable and records the fixed benchmark matrix as evidence rather than a pass threshold. Hosted macOS and Windows jobs verify the AppleClang and MSVC builds but do not claim device execution on those target drivers. Formatting, clang-tidy, terminology, and documentation, including the public/internal boundary check, remain separate jobs.

The release verification also exercised the split and mixed-resource paths under ThreadSanitizer. That is recorded evidence for those executions, not a proof that all possible races are absent.

Seams the design has not closed

CPU preparation is still serial. Animation, transform resolution, draw-list construction, recording-input compilation, queue submission, and presentation do not overlap across CPU frames. Two frame slots add CPU/GPU overlap, not pipeline-parallel simulation.

The scene remains pointer-scattered and append-only. IDs are assigned at insertion, but nodes and transforms are individually allocated. There is no removal, reparenting, compaction, or ID reuse, and the snapshot phase remains a material serial cost for large synthetic scenes.

A draw-list view carries a lifetime convention. SceneDrawList is a public span that can be fabricated and expires on the arena’s next build. RecordingInput supplies the stronger renderer-internal transaction, but the application must still keep each draw-list view within its arena lifetime.

Preparation still has coarse invalidation. Its collection identity is an address plus revision, and any asset revision rebuilds the selected compiled graph. The release does not incrementally preserve unaffected buffers or images after arbitrary asset mutation.

The component model still permits one role per node. Imported mesh primitives use synthetic children so an animator and several render objects can share one source transform. There is no multi-component storage or removal model.

The helper is intentionally narrow. It accepts one second contiguous draw range, supports at most two recording participants, and cannot schedule other work. The 5,000-draw-per-participant rule comes from homogeneous synthetic forward draws rather than a general cost model.

Presentation and forward-pass ownership remain concentrated. The file-local PresentationState and much of the forward recording sequence live inside renderer.cpp. That is workable for one pass, but adding a genuinely different pass will put pressure on those names and responsibilities.

The platform surface seam still knows Vulkan. FramebufferExtent removes Vulkan from ordinary resize data and core/ is clean, but GLFW surface creation still requires Vulkan-Hpp in the public window boundary.

Deliberate omissions at 0.9

Version 0.9 deliberately does not add:

• CPU pipeline depth greater than one;
• concurrent queue submission or presentation;
• more than two command-recording participants;
• a general scheduler, work stealing, or parallel physics;
• scene-node removal, reparenting, compaction, or ID reuse;
• contiguous transform storage;
• additional graphics or compute passes;
• shadow mapping, lighting, or a render graph;
• broader glTF content, material, or animation support; or
• automatic incremental reuse within a changed compiled resource graph.

These are boundaries of the released design rather than missing pieces of the parallel-recording path.

Where 0.10 takes this

The current 0.10 plan adds one directional shadow map before the existing forward pass. That is a direct test of 0.9’s architectural claim: immutable frame input should be able to produce two restricted pass views without giving either recorder access to scene mutation or resource ownership.

The planned release keeps two frame slots, serial queue submission, and serial CPU recording between passes. Each slot gains its own fixed-resolution shadow map; a depth-only shadow primary writes it, and the forward pass samples it. Pass-specific pipelines, packets, timings, and resources are intended to remain explicit rather than introducing a render graph for two known passes.

That pressure also gives the remaining forward-only seams a concrete reason to change. Presentation ownership can become a named internal type, recording input can separate shadow and forward capabilities, and the existing pipeline and timing names can say which pass they describe. Whether pass-level recording concurrency or a render graph is worthwhile remains a later question to be answered from the two-pass result, not assumed by 0.9.