Persistence, reconstruction, and crash recovery

T3 Code does not have one persistence boundary. SQLite owns the orchestration event log, command receipts, projection tables, authentication, and several runtime records. JSON and secret files own configuration. Image files own attachments. Terminal and provider logs have independent buffering and retention. Hidden Git refs own checkpoint trees. A provider process owns work that may already be running outside all of them.

Recovery quality follows the coordination protocol between those stores—not the strongest store named in the operation.

SQLite is the domain core

The server chooses a Bun or Node SQLite implementation at runtime, creates the database directory, opens state.sqlite, and configures the connection before application traffic begins.

const makeRuntimeSqliteLayer = Effect.fn("makeRuntimeSqliteLayer")(function* (
  config: RuntimeSqliteLayerConfig,
) {
  const runtime = process.versions.bun !== undefined ? "bun" : "node";
  const loader = defaultSqliteClientLoaders[runtime];
  const clientModule = yield* Effect.promise<Loader>(loader);
  return clientModule.layer(config);
}, Layer.unwrap);
 
const setup = Layer.effectDiscard(
  Effect.gen(function* () {
    const sql = yield* SqlClient.SqlClient;
    // CLI and server write from separate processes; wait rather than fail with SQLITE_BUSY.
    yield* sql`PRAGMA busy_timeout = 5000;`;
    yield* sql`PRAGMA foreign_keys = ON;`;
    yield* sql`PRAGMA journal_mode = WAL;`;
    yield* runMigrations();
  }),
);
 
export const makeSqlitePersistenceLive = Effect.fn("makeSqlitePersistenceLive")(function* (
  dbPath: string,
) {
  const fs = yield* FileSystem.FileSystem;
  const path = yield* Path.Path;
  yield* fs.makeDirectory(path.dirname(dbPath), { recursive: true });
 
  return Layer.provideMerge(
    setup,
    makeRuntimeSqliteLayer({
      filename: dbPath,
      spanAttributes: {
        "db.name": path.basename(dbPath),
        "service.name": "t3-server",
      },
    }),
  );
}, Layer.unwrap);
 
export const SqlitePersistenceMemory = Layer.provideMerge(
  setup,
  makeRuntimeSqliteLayer({ filename: ":memory:" }),
);

The Node implementation wraps one DatabaseSync connection with a semaphore, so operations through that service are serialized. WAL permits a separate CLI and server process to coordinate with less reader/writer blocking, while the explicit busy timeout waits for locks instead of immediately returning SQLITE_BUSY.

Migrations are a numbered program

Migrations.ts imports every migration and constructs a statically ordered map. At this pinned revision the manifest contains 41 migrations. Startup invokes the migration runner after the SQLite pragmas; it does not discover filenames and sort them at runtime.

The first migrations establish the architectural core:

Selected migration tests exercise upgrades from older shapes. There is no injected process-crash test halfway through the full 41-step startup runner.

Projectors reconstruct from cursors—with a 1,000-event ceiling

The projection pipeline registers nine projectors. For each committed event it runs every projector sequentially; each projector’s SQL changes and cursor update share a transaction. Startup reads every projector’s stored cursor and asks the event store for the tail beginning at lastSequence + 1.

The surprising part is the read default. readFromSequence defaults limit to 1,000 and treats that number as the total remaining result count, even though it pages internally. Bootstrap calls it without supplying a larger or unlimited value. A projector that is 1,001 events behind applies only the first 1,000 during that startup and the server proceeds with the projector still behind.

flowchart TB
  U{restore marker?} -->|yes| R[restore state.sqlite + WAL + SHM]
  U -->|no| O[open SQLite]
  R --> O
  O --> P[busy timeout + FK + WAL]
  P --> M[run 41 migrations]
  M --> C[read each projector cursor]
  C -->|default total limit 1000| F[fold persisted events]
  F --> Q[load optimized command model]
  Q --> K[park reactor roots]
  K --> S[reconcile projected active sessions]
  S --> A[prepare listeners + activate]
  A -. lazy .-> L[provider adoption · terminal history open]
  A -. "source audit: no general repair" .-> N[attachments · hidden refs · secret pairs · missed reactor work]

Existing bootstrap tests fold three events and verify that rerunning at the stored cursor is idempotent. No test creates 1,001 events behind one projector. A valuable regression test is mechanical: append 1,001 events, reset one projector cursor to zero, bootstrap once, and assert whether its cursor reaches the global head.

After projection bootstrap, the engine loads an optimized command read model from SQL: projects, threads, plans, sessions, latest turns, and projector states. It does not reload every message, activity, or checkpoint. Snapshot safety uses the minimum required projector cursor as its sequence watermark, so clients are not told that a newer partially projected head is complete. That safe watermark does not itself finish a projector left behind by the cap.

The service updater has an explicit recovery journal

Managed server updates are the clearest crash-aware persistence protocol in the repository. The active child first persists a pending update record. After the handoff delay, the launcher stops that child, stages and syncs copies of state.sqlite, state.sqlite-wal, and state.sqlite-shm, and then starts the trial. It does not write the restore marker before the trial.

The marker belongs to the rollback path: immediately before copying backup files over the live triplet, the launcher creates and syncs a marker inside the backup. On a later boot, pending service state without that marker resumes the trial; a marker makes startup retry the interrupted rollback restoration first.

Commit writes terminal service state before discarding the backup. Rollback restores the triplet, records the terminal state, discards the backup, and then restarts the old child. Tests verify commit only after prepared, reject a rollback with the wrong update id, and restore a trial-mutated main/WAL/SHM set.

No test kills the launcher at every instruction between pending-state persistence, backup staging, marker creation, partial triplet replacement, directory sync, terminal-state write, and backup deletion. The protocol is designed for retry, but those injected-crash cases remain valuable coverage.

Cross-store operations are small sagas

The matrix makes an important distinction: “atomic write” usually protects one file replacement. It does not make a sequence spanning several files and SQLite atomic. Among the inspected cross-store paths, only the service-update flow has an explicit durable phase marker and a startup reconciler across its named resources.

Settings JSON and secret files

Settings load into a cache guarded by a semaphore and publish a hot change stream. A missing file becomes defaults. Malformed JSON is logged and also falls back to defaults; the file is not quarantined or repaired automatically. Redacted settings resolve their referenced secret files, and a missing secret materializes as an empty value.

An update mutates secret files first, normalizes the redacted public shape, atomically renames the settings JSON, then replaces the cache and publishes. This ordering prevents plaintext secrets from entering settings JSON, but it creates cross-file failure windows:

• adding a secret and then failing the JSON write leaves an orphan secret;
• removing a secret and then failing the JSON write leaves the old redacted pointer with no value;
• the pinned startup-path audit found no reconciler that proves every redacted pointer and secret file form one committed version.

The JSON helper writes a same-directory temporary file and renames it, without an explicit file or directory sync. Secret set similarly writes a temporary file, applies 0600, renames, and reapplies permissions; exclusive create syncs its new file, but the multi-file settings update still has no shared journal.

Tests verify redaction, roundtrip materialization, race handling, and permissions. They do not inject a crash between secret mutation and settings JSON replacement.

Attachments cross the transaction in both directions

Attachment normalization validates an image, creates a random attachment id, and writes its file before the HTTP or WebSocket path dispatches the command. A rejected command therefore leaves a possible orphan. Retrying an already accepted command id repeats normalization first, creates another random file, and then learns from the receipt that the older command already succeeded.

Projection cleanup has the inverse ordering problem. A projector records reference changes and advances its SQL cursor transaction, then performs scheduled filesystem deletions with errors logged and swallowed. During engine dispatch, that cleanup can occur while the engine’s larger transaction still has later events to process; a later projection failure can roll SQL back after the file has disappeared. During startup bootstrap, a crash after cursor commit but before cleanup means the cursor prevents that cleanup from being selected again.

The real revert-cleanup test creates files and verifies removal. A differently named rollback test never creates its supposed source file; it proves projection-row rollback and nonexistence, not rollback of a filesystem deletion. No general orphan garbage collector was found at this revision.

Terminal history restores text, not a process

Terminal history is capped at 5,000 lines by default and persisted through a keyed coalescing worker after a 40 ms debounce. Output updates memory, schedules the history write, and then reaches live subscribers. The worker directly overwrites the history file; it does not use the settings atomic-rename helper.

Explicit terminal close persists immediately and drains the keyed worker. General manager scope finalization kills active sessions but does not explicitly flush and drain every history key. A hard crash can therefore lose the debounce window. Because the worker directly overwrites the target instead of using an atomic rename, an abruptly interrupted write also carries a partial-file risk; the exact filesystem outcome is an inference, not a behavior asserted by the normal write path.

On first open after restart, the manager reads and sanitizes saved text and then starts a new PTY. Saved text does not encode the old shell process, working job, input buffer, or terminal emulator state. Tests cover caps, sanitization, deletion, inactive-history eviction, and legacy filename migration—not crash injection or scope-finalizer flush.

Provider logs are diagnostic, not recovery truth

Provider runtime NDJSON logging declares itself best-effort. It rotates at 10 MiB with ten files by default, applies age and total-size retention, buffers writes, and deliberately excludes high-rate canonical deltas and progress events. Drain clears the pending batch even when writes fail, while setup failure degrades to a no-op logger. A hard crash can lose the buffered window.

The ordinary server logger emits pretty stdout plus the observability trace logger. The pinned logging-path audit found no separate authoritative server.log event ledger. Local trace rotation and optional OTLP export support diagnosis. Durable “work log” activities are a different system: runtime ingestion dispatches activity commands into SQLite projection rows, and revert can prune those derived rows.

Checkpoints live in hidden Git refs

Checkpoint refs use refs/t3/checkpoints/<base64url-thread>/turn/<count>. Capture creates an isolated temporary Git index, seeds it from HEAD, runs git add -A, writes a tree and parentless commit, then updates the hidden ref. SQL checkpoint metadata is dispatched afterward, so a crash can leave an orphan ref.

Restore reads the hidden tree into worktree and index, cleans untracked files, then resets the index toward HEAD. Consequently the implementation preserves file content on a successful restore but does not preserve the original staged-versus-unstaged boundary. The CheckpointStore comment promising “workspace and staging state” is too strong.

Revert is an unjournaled saga: restore workspace, refresh, roll the provider back, delete later refs, then dispatch durable completion. A crash can leave any prefix of those mutations. The inspected thread-deletion handler does not delete checkpoint refs, and the pinned startup-path audit found no ref/SQL reconciliation pass.

Tombstones are not a retention policy

Several mechanisms remove different kinds of data and must not be merged under the word “cleanup”:

The event-store API exposes append and reads, not event-history deletion. The receipt repository exposes upsert/get, not a retention pass. Provider-binding deletion exists in the repository interface, but a pinned-source search found no production caller; that is an absence-of-code inference, not a permanent product promise.

What startup repairs

Evidence and missing crash tests