Renderer modernization

This plan upgrades Opal from its current Canvas2D renderer to production-scale GPU backends without coupling game data to one graphics API or forcing a risky big-bang rewrite.

Invariants

• Scene, prefab, component, Flow, and asset IDs remain backend-independent.
• Canvas2D remains a working reference backend until GPU output has parity.
• Editor-only drawing never enters the shipped player import closure.
• Every migration slice keeps the full test suite green and adds render-output or performance coverage for the code it moves.
• Backend selection is runtime capability/configuration, not persisted content.

Phase 0 — measurable baseline

Implemented:

• A backend-neutral per-frame metric collector in src/game/rendering/render-metrics.js.
• Canvas2D publishes logical workload metrics for submitted/rendered/culled objects, sprites, component render hooks, procedural visuals, skeletons, visible tiles, and particles.
• The editor Stats overlay displays those metrics during play.

These are logical workload counters, not fake GPU statistics. A future backend can extend the same snapshot with values such as gpu.drawCalls, triangles, texture bytes, pipeline switches, upload bytes, and GPU frame time.

Phase 1 — draw-command seam

In progress:

• Versioned backend-neutral ordinary-sprite commands now carry transform, pivot, flip, opacity, tint, highlight, and shadow intent.
• Texture-free primitive commands describe centered rectangles, circles, and stars with opacity and normal/additive blending. VFX emits these generic commands instead of bypassing Render2D with particle-specific Canvas calls.
• Indexed textured-mesh commands carry compact positions, region-relative UVs, indices, transforms, filtering, opacity, tint, and blend intent. The contract is animation-system agnostic, so skeletons, terrain strips, warped UI, and custom components share it.
• Backend-neutral rectangular clip scopes are strict ordering boundaries and nest by intersection. Queue-owned balancing restores leaked scopes at frame completion or abort, so a faulty component cannot clip later objects/frames.
• Stable render-target handles describe backend-owned offscreen color storage. Target passes and composites are strict ordering boundaries, nest safely, and recover leaked target or child-clip scopes at frame completion or abort.
• Target composites carry a normalized, backend-neutral effect block for exposure, contrast, saturation, hue rotation, blur, vignette, bright extraction, and tone mapping. Identity composites preserve the original fast path, while effect-bearing composites are counted as explicit post-process passes.
• Target composites also accept immutable material intent. The first sampled input is any second render target used as an alpha or inverted-alpha mask; both handles remain backend-neutral and feedback-looping self references are rejected while commands are normalized.
• A backend-independent effect-chain planner expands bloom into ordinary target commands: downsampled bright extraction, blur, tone-mapped base composition, and additive glow. Its transient-target pool retains stable handles at the maximum concurrent demand and reuses them across chains and frames.
• Component definitions can declare renderMode: "render2d" and receive a resource-aware command writer. Its bound-texture API uploads once and submits many source regions without leaking decoded browser objects into commands.
• TileMap 2D uses that writer for camera-visible atlas cells. Cells sharing a texture and sampler instance together on WebGL2; direct component/tool calls retain the Canvas reference path.
• The Canvas2D backend executes those commands with output parity.
• createRenderer accepts an injected Render2D backend and exposes its identity and capabilities; Canvas2D remains the default.
• Backends are validated against the current command version and required capabilities at renderer creation, so incompatible implementations fail fast.
• Backend identity is visible in the live Stats panel and render snapshot.
• Skeleton region slots submit ordinary sprite commands and CPU-skinned mesh slots submit textured-mesh commands in authored draw order. Loading placeholders and older backends retain an explicit ordered Canvas fallback.
• An ordered command queue now groups uninterrupted sprite, primitive, or mesh runs, caps queue growth, and inserts explicit barriers around remaining Canvas2D component, procedural, skeleton, speech, and editor drawing.
• Backends that declare batching receive each compatible ordered sprite run via drawSpriteBatch; Canvas2D remains the immediate parity executor.

Next:

1. Extend the small Render2D command vocabulary with richer blend state and stencil-backed nested masks; sampled target masks are now implemented.
2. Migrate UI and finally editor overlays.
3. Keep component onRender Canvas compatibility while migrating built-ins to the backend-neutral writer. Deprecate direct canvas access only after the built-in and plugin ecosystem has migrated.

Gate: recording-canvas golden tests produce the same transforms, order, clips, and blend modes before and after each primitive migrates.

Phase 2 — textures and materials

In progress:

• Stable, immutable runtime texture handles are separate from authored asset records and decoded browser resources.
• Texture uploads, revision deduplication, retry, release, and clear lifecycle operations now cross the backend contract explicitly.
• Backend residency can be invalidated and rebuilt from the same handles after device/context loss without touching authored asset identity.
• Sprite commands support pixel source regions, so an atlas can render frames without creating a cropped image asset for every frame.
• Stable texture-region handles separate sprite commands from atlas coordinates; named regions can be repacked while retaining identity through revisioned backend updates.
• Deterministic multi-page atlas planning and Canvas page construction now support padding, power-of-two sizing, edge extrusion, and stable registration back into the texture/region registries.
• Immutable sampler descriptions cover min/mag/mipmap filtering, address modes, and anisotropy. Canvas2D honors nearest/linear filtering; future GPU backends receive the complete description.
• The Canvas2D reference backend owns its texture residency map, and the live Stats panel reports resident/ready textures, estimated decoded bytes, and uploads for the current frame.
• Project switches clear texture residency, preventing decoded art from one project leaking into another renderer session.
• URL-backed decoded sources also share residency without an authored asset key. Fresh play worlds can construct fresh Image objects for animation frames without allocating another GPU texture for the same URL. Procedural Light 2D textures use an appearance signature, so identical color/falloff variants share one generated source, texture handle, and additive mesh batch.
• Backend texture residency now has a runtime-only LRU budget (256 MiB by default). Stable handles and decoded registry sources survive eviction, while only backend storage and its uploaded region state are released. A later draw re-uploads through the same handles, current-frame textures are protected, and a short grace window avoids double-buffer and adjacent-frame churn. Resident bytes, evictions, reloads, budget, and temporary over-budget working sets are visible through renderer metrics.
• Sprite Material 2D and Light 2D are authored, serialized engine components rather than demo-only target commands. Materials reference an optional image asset for tangent-space normals plus a named lighting layer, ambient/normal strength, a four-light budget, filtering, and color grading. Lights author color, intensity, world-space range/height, falloff, and layer. Runtime-only image URLs, decoded textures, target handles, and backend resources never enter project data.
• A pure lighting-plan adapter filters by named layer and range, ranks the most relevant contributors deterministically, and maps world lights into a sprite's target-local space. The directional path translates that plan into Render2D passes with HDR light targets. Flat surfaces automatically use one stage-sized HDR light field per receiver-layer mask instead: every matching light is accumulated once and each rotated/scaled sprite samples its world region through a portable affine UV transform. Lighting Scope can force the scalable layer field or the precise per-sprite path. Sprites without the material—or with no matching active light—stay on the ordinary zero-target fast path, so old projects benefit without a migration or added cost.

Next:

1. Integrate opt-in atlas construction with asset preload/build workflows and establish measured page-size and rebuild budgets.
2. Add worker-driven decode state and platform-aware residency budget presets.
3. Add distortion inputs and typed custom scalar/vector parameters to the immutable target-material description. Normal/light authored adapters are now implemented; graphics-API objects remain private to backends.
4. Wire browser device/context-loss events to the implemented residency invalidation/rebuild path.

Gate: scene reload, context loss, missing assets, and budget eviction preserve stable asset identity and never mutate authored project data.

Phase 3 — GPU backends

In progress:

• An automatically preferred WebGL2 backend uploads stable texture handles, consumes texture regions and sampler descriptions, and draws compatible adjacent sprites with hardware instancing.
• Solid rectangles, circles, and stars use a separate texture-free instanced pipeline. Analytic fragment coverage keeps circles/stars smooth, and normal versus additive groups composite correctly against existing scene content.
• Sprite shadows and rounded selection highlights now render inside the same expanded instanced quad as the sprite. A weighted silhouette kernel supplies soft arbitrary-shape shadows, while an analytic rounded-box distance field supplies highlight strokes and glow without adding a draw call or changing authored ordering.
• Dynamic indexed textured meshes use a dedicated WebGL2 pipeline. Adjacent meshes sharing a texture, sampler, and blend mode expand into one dynamic triangle buffer and draw call; Canvas2D retains a triangle-affine reference executor for parity and fallback.
• Rectangular scopes execute as nested Canvas paths and transformed WebGL2 scissors. The hybrid compositor remains inside the Canvas clip as well, which preserves parity while avoiding fragment work outside the visible rectangle.
• Offscreen passes render in target-local coordinates and composite through one transformable command. Canvas owns the reference storage; WebGL2 allocates framebuffer-backed color textures, renders compatible sprite, primitive, and mesh batches directly into them, and samples target composites on the GPU. If an older Canvas-only command appears mid-pass, the backend materializes the framebuffer once and completes the pass through ordered Canvas fallback.
• Effect-bearing target composites use a dedicated WebGL2 post-process shader; the Canvas2D backend supplies the same command contract through its filtered and pixel-processed reference surface. The shared chain API now supplies ACES and Reinhard tone mapping plus downsampled bloom without exposing a WebGL framebuffer, Canvas surface, or game-specific component.
• Pooled temporary targets avoid per-frame framebuffer/canvas allocation. Live counters distinguish creates, resizes, reuse, retained bytes, logical chain passes, GPU shader passes, and Canvas pixel-reference passes.
• Render-target descriptors now carry portable standard or hdr precision intent. WebGL2 negotiates RGBA16F storage only when floating-point color attachments and blending are available, and retries RGBA8 on the GPU if the capability or allocation is unavailable. Canvas2D accepts the same request through a measured standard-color reference fallback. Bloom automatically requests HDR transient targets; games can opt scene or lighting targets into HDR without persisting a WebGL enum or backend resource.
• Target-material masks sample two framebuffer textures in one WebGL2 composite shader. Canvas2D executes the same alpha and inverted-alpha modes with pooled reference surfaces. If either sampled target has already fallen back to Canvas, the hybrid backend materializes the other once and preserves authored ordering rather than silently ignoring the mask.
• Target materials also accept a backend-neutral normal target and up to four independent light layers. Each layer carries its own HDR light target, normalized direction, and diffuse strength; ambient and normal strength stay shared by the material. The fixed portable ceiling fits WebGL2's guaranteed texture-unit budget alongside scene and mask inputs. WebGL2 accumulates the layers in one composite shader, while Canvas2D runs the equivalent bounded pixel formula on pooled reference surfaces. The original single lightTargetHandle shape remains supported as normalization shorthand. Invalid handles, incomplete input pairs, and source feedback loops are rejected while the command is normalized, before either backend sees them.
• Flat authored materials now batch matching lights into shared HDR layer fields. These remove the per-sprite four-light selection ceiling and replace repeated per-light receiver targets with one additive field per unique layer mask. Normal-mapped materials stay on directional per-sprite lighting by default, while an explicit shared scope offers a scalable quality tradeoff. Canvas2D understands transformed light-field sampling in the portable target material contract and remains on its bounded reference path; WebGL2 enables shared fields automatically.
• WebGL2 flat receivers shade the original sprite directly from the shared field. The portable command carries only a field handle, affine UV transform, ambient value, and color grading. This removes each flat receiver's source pass, flat-normal pass, and target composite; normal-mapped receivers and Canvas2D retain the precise portable target path.
• Shared layer fields are camera-local and chunk-aligned rather than permanently stage-sized. Their world bounds move in stable 128-unit increments, their resolution follows viewport density, and each dimension stays within a backend-neutral 2048-texel budget. Large and zoomed-out worlds therefore pay for the streamed view instead of allocating an HDR texture for the whole map; affine receiver sampling preserves rotation, pivot, and scale across the shifted field origin.
• Floating-point intermediate passes no longer clamp untone-mapped shader output to 0–1, allowing emissive and additive energy to survive through HDR extraction, blur, masks, and later lighting before final tone mapping.
• Live metrics distinguish requested target memory, actual HDR GPU memory, active HDR targets, and precision fallbacks. Existing projects require no migration: targets without a precision request remain standard, and backend selection and negotiated formats remain runtime-only.
• The backend renders to a transparent GPU surface and composites at existing command barriers. Shadows, selection highlights, procedural rendering, and other commands that have not migrated yet execute through an ordered Canvas2D fallback without changing scene or component data.
• The editor and standalone player now prefer WebGL2 automatically. WebGL2 initialization failure returns to Canvas2D, while ?renderer=canvas2d remains an explicit reference/debug override; backend selection is never serialized into a project. The Canvas reference path enforces a per-frame software-lighting pixel-work budget, so a stage-sized receiver degrades to its ordinary sprite instead of stalling the main thread with multi-megapixel getImageData passes.
• GPU-specific counters report draw calls, instances, triangles, composites, texture upload bytes, Canvas fallback draws, direct field-lit sprites, and non-blocking GPU frame time alongside the shared logical metrics. WebGL2 uses disjoint timer queries when available and never stalls the CPU for a result.
• Canvas backing-store resolution is independent from logical layout. Editor play and the standalone player start at native Retina quality, lower render scale only after sustained pressure, and restore detail slowly when headroom returns. Device-specific scale remains runtime state, never project data.
• renderer-tech-demo.html is the first live stress/parity scene. Its monitor builds animated normal, cool/warm HDR light, and mask targets entirely through public Render2D APIs, and can switch to renderer=canvas2d as a visual reference.

Next:

• Evolve high-volume VFX from CPU command generation to optional GPU simulation without changing its public API.
• Implement a WebGPU executor behind the same command and material contracts.
• Add GPU particle buffers and optional shader-side bone deformation incrementally. HDR targets already preserve additive and emissive energy above 1.0 through lighting and bloom before the final tone-map pass.

Gate: backend parity scenes plus automated frame-time, draw-call, upload, and texture-memory budgets. Canvas2D stays available as a reference/fallback until the supported browser matrix says otherwise.

Phase 4 — production scale

• Worker-based asset decode and optional render submission where supported.
• Chunk-aware texture residency tied to world streaming.
• Expand the implemented dynamic-resolution scale into authored quality tiers for shadows, post effects, particles, and streamed fields.
• Add GPU frame captures alongside the implemented timestamp queries.
• Headless content validation and repeatable benchmark runs in CI.

First benchmark scene

Before choosing budgets, build one representative scene containing:

• several parallax layers and large translucent backgrounds;
• a camera-culled tilemap;
• many ordinary sprites with mixed sorting groups;
• multiple deformed skeleton characters;
• normal and additive particles;
• gameplay UI, speech, masks, and representative post effects.

Record target hardware, resolution, frame-time percentile, memory peak, scene load time, and each render metric. Budgets should come from that target rather than arbitrary engine-wide constants.