Skip to content

Opal Art Pipeline ​

This document defines the technical direction for Opal's artist-facing source pipeline. The goal is dependable iteration for production art, not guided content creation.

Product principles ​

  • Source files remain editable in artists' existing tools.
  • Asset IDs are stable references, independent from paths and display names.
  • Reimport is inspectable: Opal can explain what changed before it mutates the project.
  • Missing or broken sources degrade visibly, never destructively.
  • Derived outputs are reproducible and retain the source and import settings that created them.
  • Batch operations and automation use the same pipeline as the editor UI.

Automatic project organization ​

Accepted for editor creation and imports: the asset library owns the default destination policy (projectFolderNameForAsset in assets/library.js). Creation tools pass an omitted or empty folder to use it. New assets go to the standard folder for their type; a stable-ID overwrite retains its current folder. Explicit folder IDs, including literal root, remain authoritative for restoration and intentional placement. Shared assets remain shared.

The browser's project-root import means automatic organization; a drop into a concrete folder means that folder. Current-project synthetic root IDs never become persisted folder IDs. Custom folders, source-folder paths, and asset-pack grouping are preserved. No opening-time rearrangement or schema migration is required.

Each write captures its project before asynchronous preparation. Folder-backed batch imports create missing standard folders in the same transaction as the complete asset dependency set. Gameplay packages resolve destinations in the new project rather than reusing the source project's folder IDs.

Complete project packages contain the canonical authoring files (opal.project, source graphs/scripts/settings, scenes, folders, and binary assets). Import validates the archive and reference closure, prepares independent IDs once, then commits the whole project: cloud uses existing staged object uploads and activation, IndexedDB uses one transaction across all owning stores, and Desktop creates a new authoring folder. Retry retains the prepared identity and resumes missing cloud objects. Other asset batches still use multipart metadata plus raw binary parts, retaining one atomic asset-closure commit. Project import no longer serializes numeric byte arrays or performs one folder lookup per imported asset.

Portability decision (accepted 2026-09-25 for the user-requested backup/import work): reuse the Desktop/cloud authoring schema instead of adding another project schema. The ZIP codec validates paths, sizes and checksums before exposing files; limits are 10,000 entries and 1 GiB of expanded content, with the existing cloud 150 MiB per-file limit enforced before upload. Selective DEFLATE saves space on authored text and raw media; already compressed media stays stored. Archive bytes remain memory-resident, but STORE reads borrow views and immutable captures avoid repeated whole-project copies. Retry identity is retained for the current editor session. Revisit streaming and persistent resume if measured project sizes outgrow these bounds.

See Assets and importing for the folder standard.

Milestone 1: source snapshot and diff ​

Implemented for local project asset folders:

  1. Scan supported source files and record SHA-256 content hashes, size, modification time, and MIME type.
  2. Compare the snapshot with imported asset records.
  3. Classify each source as added, modified, moved, missing, restored, or unchanged.
  4. Infer moves only for a unique source/record content-hash match.
  5. Apply the diff while preserving asset IDs across modifications and moves.
  6. Retain the last imported payload for missing sources and surface a non-blocking Project Health warning.

GET /api/projects/:id/assets/diff previews the next sync without changing records. POST /api/projects/:id/assets/sync applies that same model and returns applied counts plus the scanned summary.

Asset state model ​

Each folder-sourced asset stores:

  • sourceFolderFile: project-relative source path
  • sourceHash and sourceHashAlgorithm: identity of the imported bytes
  • sourceSize and sourceMtimeMs: scan metadata
  • sourceMissing and sourceMissingSince: retained-import state
  • stable asset id: the reference used by scenes, prefabs, and graphs

Paths organize sources; they are not identity. Content hashes support conservative move detection; they are not a long-term substitute for explicit source identity.

Next milestones ​

2. Asset Changes workspace ​

  • Review grouped additions, reimports, moves, ambiguous matches, and missing sources.
  • Apply or ignore individual changes, or sync a folder as a batch.
  • Show thumbnail, dimensions, alpha bounds, color-space changes, and dependent scenes/prefabs.
  • Preserve a short local import history and allow rollback to the previous payload.

3. Import recipes and derived data ​

  • Store per-asset import settings separately from source metadata.
  • Add deterministic processors for trimming, padding, slicing, pivots, atlases, compression, and platform variants.
  • Separate source assets from generated artifacts and rebuild only invalidated outputs.
  • Run processors through a versioned job graph with cancellation and atomic commits.

4. Explicit source identity ​

  • Introduce sidecar metadata with a generated source UUID for moves that cannot be inferred from content.
  • Detect copies, source replacement, and conflicts without path heuristics.
  • Support relinking a missing asset to a new source while retaining its asset ID and import recipe.

5. Toolchain integration ​

  • Watch folders with debounced, race-safe scans instead of relying only on polling.
  • Expose headless scan, diff, validate, and reimport commands for CI and build farms.
  • Add processor/plugin APIs and structured diagnostics.
  • Track dependencies so a source change invalidates only affected atlases, animations, prefabs, and builds.

Safety requirements before broad format support ​

  • Stage new payloads before swapping records so failed imports leave the previous version intact.
  • Validate content independently from filename extensions.
  • Bound concurrency, file sizes, decode work, and generated output sizes.
  • Reject path escapes and ignore symlinks in project source folders.
  • Make sync serialization explicit so overlapping scans cannot race record writes.
  • Include processor version and settings in derived-artifact cache keys.