SCUA

How-to

Ship a program's data with it (res: and pack files)

A tool ships templates and schemas, a game ships sprites and levels, a site ships pages. SCUA keeps that data apart from the user's files. It has its own path prefix, res:, and the ordinary fs verbs read it:

import fs
import json

let level = json.decode(fs.read_text("res:maps/keep.json")?)?   -- the program's own data
fs.write("slot1.json", json.encode({ level = "keep" }))?           -- the user's file, under --allow-fs

A path that starts with res: is a resource. Every other path is a host file, the same as it always was.

  • The two are never mixed. A user's templates/ folder can never replace the templates your tool ships, and fs.grep("TODO", ".") over the user's project never finds your tool's own files.
  • Resources are read-only. fs.write, fs.mkdir, fs.rename, fs.remove and fs.open for writing all return an Error that tells you to write to a host path instead.

#During development: a res/ folder

Put the data in a folder named res at the project root. That is the folder holding scua.toml, or, with no scua.toml, the folder holding your script:

mygame/
  game.scua
  res/maps/keep.json        read as res:maps/keep.json
  res/sprites/hero.png      read as res:sprites/hero.png

That is all. scua game.scua reads res: from that folder with no flag, and edits show up on the next read.

Declare that your project needs a scua that understands res:. An older scua would read res:x as an ordinary file name and report it as missing; with a floor, it says to upgrade instead:

# scua.toml
[package]
scua-version = "0.36"

res: names match case exactly, even on macOS and Windows, whose own file systems would quietly accept Sprites/Hero.png for sprites/hero.png. That way, a path that works while you develop also works once the data is packed. When the case is wrong, the error says so:

Error(not found: res:Sprites/hero.png (res/ has a name that differs from it only in case — res: names match case exactly, as a pack does))

Read the data as res:maps/keep.json, not as the host path res/maps/keep.json. The host path works while you develop, because the folder is there, but a shipped program has no res/ folder beside it. scua points this out three ways:

  • A literal res/ path in a read is a compile warning while the project has a res/ folder:

    scua: game.scua:4: warning: "res/maps/keep.json" is the project's resource folder read as a host file; it is not there once shipped — write "res:maps/keep.json"

    --deny-warnings makes it an error. Writes never warn, so a level editor can save into res/.

  • A computed host path that lands in res/ gets the same warning under scua debug, once a run, at the line that read it.

  • A host read that misses names the res: path when your resources have the file:

    Error(maps/keep.json: no such file or directory (a host path; the program's resources have res:maps/keep.json))

A path that no run would ever accept is a compile error, exit 3: res://x, res:/x, RES:x, user:x, and res: joined onto a host path such as out/res:x.

Some files are hidden in every res: layer, because operating systems and tools create them, not you:

  • .DS_Store, Thumbs.db, desktop.ini and Icon\r;
  • anything inside .git/;
  • inside a pack, anything inside __MACOSX/.

So fs.list("res:levels") never puts .DS_Store on your level-select screen.

res/ must be a real folder, not a symlink. --res=none turns it off, and --res=PATH uses another folder or a pack in its place.

#Packs

A pack is a zip archive. Build one from your res/ folder:

$ scua res build res/ -o base.pak
wrote base.pak: 4 entries (0 deflated), 504 bytes; skipped 1
  content digest d3839beeabbfefeebf5395450af159ccaebe96f126fdb04252bac02a1b8e9e0d
  sha256         429e58041abae0a2aae8954763841fce181cd8cc3147717ed4eaf8b1b71e0cd6

scua res build writes the same bytes for the same input every time. Set SOURCE_DATE_EPOCH to choose the timestamp it records. It also:

  • skips the hidden names above;
  • refuses symlinks, and names that would not unpack on every OS (:, \, con.txt, a trailing dot, two names that differ only in case, Straße beside STRASSE included);
  • warns about a name that is not in NFC, such as café.png written as e and a separate accent, which macOS has made in the past. A program asking for res:café.png would not find it. --nfc writes every name composed;
  • stores every file uncompressed unless you ask, with for example --deflate="*.json,*.txt".

Uncompressed is the fastest to read. A pack made with zip works too, if you make it from inside the folder (cd res && zip -0 -r -X ../base.pak .).

Mount a pack with --mount. A later --mount goes above an earlier one, and the higher layer wins. This is how patches work:

$ scua --mount=base.pak --mount=patch1.pak --allow-fs=save game.scua
$ scua --mount="patches/*.pak" --allow-fs=save game.scua     # every .pak there, patch2 before patch10
  • A relative --mount path resolves against the program file's folder, not against wherever the command was typed. A launcher works from any working directory.
  • Quote a glob. * and ? are allowed in the last part of the path only.
  • A glob directory with no matching files mounts nothing. A directory that does not exist stops the run.
  • --mount=DIR mounts a folder as a read-only layer.
  • --mount=base.pak#sha256=HEX refuses to run unless the file's SHA-256 matches.
  • A policy file can list the mounts in place of the flags: mount = ["base.pak", "patches/*.pak"] in the --allow-file file. A --mount on the command line replaces that whole list.

The project's res/ folder, or the pack given with --res, is the bottom layer, and every --mount sits above it.

Every pack is checked before anything is read from it. A pack that fails a check stops the run before a line of your program runs, with exit status 2 and a code that names the check:

  • E-PACK-FORMAT, E-PACK-CORRUPT and E-PACK-PATH for the archive itself;
  • E-PACK-DUP for the same pack mounted twice;
  • E-PACK-DIGEST for a pin that does not match;
  • E-PACK-WRITABLE and E-PACK-GLOB for the mount itself.

Pack format rules:

  • Refused: names with .., absolute names, duplicate names, entries that disagree with the archive's directory, encrypted entries, and compression other than deflate.
  • Also refused, as duplicates: two names that differ only in case, and two names that are the same once written in NFC, such as café.png with é and with e and a separate accent. Only one of them could be unpacked on macOS or Windows. Case is compared with Unicode's full case folding, so Straße and STRASSE are the same name.
  • Damaged data is caught on read, by the file's CRC: the read returns Error("damaged: …").
  • A file changed while mounted fails every later read with changed while mounted. Restart to pick up a new pack.

Keep shipped packs outside the folder you grant with --allow-fs. A --mount inside it is refused (E-PACK-WRITABLE), because the program could rewrite the pack for its next run.

#Mods: mounting from the program

A program can mount packs itself, up until it starts its first actor:

import fs

match fs.find("mods", { filter = ["*.pak", "*.zip"], kind = "file" })
  Ok(entries) -> for e in entries do
    match fs.mount(e.path, { root = "*" })
      Ok(layer) -> print(`mod {e.name}: overrides {layer.overrides} files`)
      Error(why) -> print(`skipped {e.name}: {why}`)
    end
  end
  Error(_) -> nil
end

How fs.mount behaves:

  • What it mounts. A pack the program can already read. It needs --allow-fs for a host path, or a res: path to a pack stored inside another pack.
  • Where the new layer goes. It goes above every other layer.
  • A broken mod returns an Error, which you can skip.
  • { root = "*" } serves a zip whose entries are all inside one folder from inside that folder. This is what Finder's "Compress" and zip -r MyMod produce. { root = "DIR" } names the folder.
  • After the first actor(…) it is refused. Every actor then reads the same files for the same path. To turn a mod on or off, restart.

#When the program has no --allow-fs

A program with resources but no --allow-fs can import fs and read res:. Reading a host file then stops the run with exit status 4, the status for "a grant is missing":

scua: tool.scua:3: host path config.json: this run was not granted --allow-fs, so only res: paths are readable — re-run with --allow-fs=DIR

#Finding out where a file came from

fs.where names the layer that serves a path, and the layers it hides:

import fs

match fs.where("res:maps/keep.json")
  Ok(w) -> print(`served by {w.layer.name} ({w.layer.origin}), hiding {len(w.shadows)} other copies`)
  Error(why) -> print(why)
end

The other inspection tools:

  • fs.mounts() returns the whole view, top layer first. Every layer record has the same fields:
    • name, kind, origin, path;
    • digest, a content digest that changes only when a file's name, size or contents change;
    • sha256, bytes, entries, skipped, damaged, overrides, wrapper, stale.
  • scua --show-res … prints the same view to stderr before the program runs.
  • scua res list FILE lists a pack.
  • scua res check FILE runs every check on every entry.

A read that misses says where it looked, and names a near miss when there is one:

not found: res:maps/e1m1.json (looked in patch1.pak, base.pak)
not found: res:maps/keep.json (mod.zip has MyMod/maps/keep.json — every entry is inside MyMod/; rebuild with `scua res build MyMod/ -o mod.zip`, or mount with { root = "*" })

#Reading a large file a piece at a time

fs.open works on every res: file, packed or not, stored or deflated. Read it with fs.read_line or fs.read_chunk, the same as a host file (see Read and write files). Only one buffer's worth of the file is in memory at a time.

The read that reaches the end checks the file's CRC. When the file is damaged, that read returns Error("damaged: …") where it would have returned the end, so the program has already seen the bytes before the check fails. A program that must not act on damaged data reads the file whole instead.

#Reading part of a file

fs.read(path, { offset = o, len = n }) reads up to n bytes starting at o. It works on host files, and on res: files stored uncompressed. Use it for audio loop points or large assets.

It checks the pack's structure, but it does not check the CRC: a CRC covers the whole file, not a part of it.

#Data inside a .scuapp

An application bundle can carry its res/ folder inside it. Then there is one file to ship:

$ scua game.scuapp                        # reads res: from inside the bundle; no --allow-fs needed
$ scua --allow-fs=save game.scuapp        # and writes saves to save/
$ scua res check game.scuapp              # checks every file the bundle carries

How it behaves:

  • The bundle's files are the bottom layer. --mount packs go above them, so patches and mods work as they do with a res/ folder.
  • --res=PATH or --res=none replaces them.
  • Every file is checked against the SHA-256 the bundle records for it, on top of the pack checks above. A file that does not match reads as Error("damaged: …"). A bundle whose archive does not match its records stops the run before it starts.
  • A bundle's data is not loaded at launch. Only its code is, so large assets cost nothing until they are read.

scua-pkg's bundle --app writes these bundles from a later release. Until then, ship base.pak beside the bundle and run it with --res=base.pak.

#Saves and the packs they ran with

A save of a partition (actors.snapshot) records the res: layers the run had: each layer's name, where it came from, and a digest of its contents. Reviving the save compares them with the layers the reviving run has:

  • A layer the save had is missing, such as a mod the player has since removed. The revive is refused with Error({ kind = "res_missing", missing = ["mods/castle.pak"] }), so a game can tell the player which mods the save used. actors.revive(blob, { res_missing = "allow" }), or the command-line flag --res-missing=allow, revives anyway, and fs.mounts().missing lists what was left out.
  • A layer changed, or is new. The revive goes ahead, and the load report in the log says res_changed. { refuse_res_changed = true }, or --strict-res, refuses it instead with Error({ kind = "res_changed", changed = [...] }).
  • A save from before 0.36 recorded nothing, so it revives as it always did.

The digest covers each file's name, size and CRC. It tells you a pack changed, but it is not a security check: pin a pack (--mount=base.pak#sha256=HEX) when you need to know it is the one you shipped.

#Serving a site from a pack

scua serve site.pak serves a pack the way scua serve DIR serves a folder.

#What res: does not do

  • import never reads res:. Code ships in a one-file bundle.
  • Packs are never written to.
  • Native libraries that open files by path cannot see res:. Hand them bytes from fs.read, or use their memory-loading functions.

#Not built yet

This is an evaluation build of the feature. These are not available yet:

  • Writing assets into a .scuapp bundle. scua reads them, and scua-pkg writes them from a later release. For now, ship base.pak beside the bundle and run it with --mount=base.pak, or --res=base.pak.