Part IV · Transactional domain core and post-commit deliveryProjections
Chapter 26source checked

Projection tables and read models

New events fold through ordered SQLite projectors before the enclosing command commits; bootstrap replays committed events from per-projector cursors, and queries expose deliberately different read-model shapes.

What this chapter resolves
  • Trace one durable event through every projection cursor and its SQL boundary.
  • Distinguish the safe snapshot watermark from the durable event-store head.
  • Separate full snapshots, command state, shell summaries, detail pages, and live replay.

An orchestration event is not directly a UI row. The durable event store records a global sequence; a projection pipeline folds that event into several SQLite read tables; query services compose those tables into views for a particular reader. This is a synchronous part of normal command acceptance, not a background eventually-consistent worker that happens after a command has returned.

Nine folds share one event order

Each named projector has its own projection_state cursor. For every event, the pipeline considers every projector in this exact sequence:

  1. Projects
  2. Messages
  3. Proposed plans
  4. Activities
  5. Sessions
  6. Turns
  7. Checkpoints
  8. Pending approvals
  9. Threads

The final placement is meaningful. The thread projection refreshes summary fields after the message, plan, activity, session, turn, and approval tables have been updated. It can therefore compute navigation-oriented facts such as pending approval/input counts and whether a plan is actionable from current dependent rows.

Figure 26.1 · A command folds one ordered event through SQLite read sidesdashed detours are filesystem work outside projector SQL rollback
Ordered projection foldDiagram loading

Zoom with the controls, +/−, or Ctrl/⌘ + trackpad scroll. Enable Pan to drag, use two-finger scrolling, or use the arrow keys. 0 fits the diagram; Esc leaves Pan or expanded view.

Ordered projection fold
Text equivalent

The engine's outer SQL transaction appends an event, then runs projects, messages, plans, activities, sessions, turns, checkpoints, pending approvals, and threads in order. Each projector first completes its projection row and cursor SQL. After the messages projector, an optional attachment-pruning filesystem detour can occur before plans. After the final threads projector, an optional attachment-deletion detour can occur before the accepted receipt. The outer SQL transaction commits only after those steps, making the event, projection rows, cursors, and receipt durable. Filesystem mutations cannot be rolled back if later SQL fails. Committed events are then published to live consumers.

Figure 26.1. A newly appended global event sequence enters nine sequential projector applications while the outer transaction is still open. Each projector completes its row and cursor SQL before any attachment effect it scheduled. A message fold can prune files before later projectors run; the final thread fold can delete files before the accepted receipt and outer commit. The sequence becomes durable only at that outer commit, while the earlier filesystem mutations are not enlisted in rollback.
apps/server/src/orchestration/Layers/ProjectionPipeline.ts:1609–1678 ↗verbatim · typescript · b4586260
          return;
        }
 
        case "thread.turn-interrupt-requested": {
          if (event.payload.turnId === undefined) {
            return;
          }
          const existingTurn = yield* projectionTurnRepository.getByTurnId({
            threadId: event.payload.threadId,
            turnId: event.payload.turnId,
          });
          if (Option.isSome(existingTurn)) {
            yield* projectionTurnRepository.upsertByTurnId({
              ...existingTurn.value,
              state: "interrupted",
              completedAt: existingTurn.value.completedAt ?? event.payload.createdAt,
              startedAt: existingTurn.value.startedAt ?? event.payload.createdAt,
              requestedAt: existingTurn.value.requestedAt ?? event.payload.createdAt,
            });
            return;
          }
          yield* projectionTurnRepository.upsertByTurnId({
            turnId: event.payload.turnId,
            threadId: event.payload.threadId,
            pendingMessageId: null,
            sourceProposedPlanThreadId: null,
            sourceProposedPlanId: null,
            assistantMessageId: null,
            state: "interrupted",
            requestedAt: event.payload.createdAt,
            startedAt: event.payload.createdAt,
            completedAt: event.payload.createdAt,
            checkpointTurnCount: null,
            checkpointRef: null,
            checkpointStatus: null,
            checkpointFiles: [],
          });
          return;
        }
 
        case "thread.turn-diff-completed": {
          // Mid-turn diff updates produce placeholder checkpoints; record the
          // checkpoint, but don't settle a turn its session is still running.
          const session = yield* projectionThreadSessionRepository.getByThreadId({
            threadId: event.payload.threadId,
          });
          const turnStillRunning =
            Option.isSome(session) &&
            session.value.status === "running" &&
            session.value.activeTurnId === event.payload.turnId;
          const existingTurn = yield* projectionTurnRepository.getByTurnId({
            threadId: event.payload.threadId,
            turnId: event.payload.turnId,
          });
          const nextState = event.payload.status === "error" ? "error" : "completed";
          yield* projectionTurnRepository.clearCheckpointTurnConflict({
            threadId: event.payload.threadId,
            turnId: event.payload.turnId,
            checkpointTurnCount: event.payload.checkpointTurnCount,
          });
 
          if (Option.isSome(existingTurn)) {
            yield* projectionTurnRepository.upsertByTurnId({
              ...existingTurn.value,
              assistantMessageId: event.payload.assistantMessageId,
              state:
                turnStillRunning || existingTurn.value.state === "interrupted"
                  ? existingTurn.value.state
                  : nextState,
              checkpointTurnCount: event.payload.checkpointTurnCount,
Read this as: The projector work and its per-projector state cursor are in SQL work; filesystem attachment cleanup is deliberately invoked afterward and logs failures instead of undoing the projection.

Cursor means “safe through here,” not “the newest event”

The event store has a monotonically increasing, global sequence; each aggregate also has its own stream version. Projection restart uses the former. On bootstrap, each projector reads only events after its own last_applied_sequence and advances that cursor after its SQL fold. A projector that has nothing to do for an event still records that it safely considered the event.

The query layer does not advertise the highest event in the log as a snapshot sequence. It checks the required projector state and returns their minimum. If a required state row is missing, the safe sequence is zero. That conservative watermark prevents a snapshot composed from several tables from claiming it includes an event that one required read side has not reached.

apps/server/src/orchestration/Layers/ProjectionSnapshotQuery.ts:195–258 ↗verbatim · typescript · 12c8e828
  threadId: ThreadId,
  runtimePayload: Schema.Unknown,
});
const ThreadIdLookupInput = Schema.Struct({
  threadId: ThreadId,
});
const TurnStartMessageLookupInput = Schema.Struct({
  threadId: ThreadId,
  messageId: MessageId,
});
const ThreadActivityKindsLookupInput = Schema.Struct({
  threadId: ThreadId,
  activityKinds: Schema.Array(Schema.String),
});
const ThreadActivityIdsLookupInput = Schema.Struct({
  activityIds: Schema.Array(ProjectionThreadActivity.fields.activityId),
});
// Windowed reads order turns by the stable keyset (anchor, turn key), where
// anchor is requested_at and turn key is
// COALESCE(turn_id, ''). Both are event-derived, so cursors survive the
// revert projector's row-id rewrite and full projection rebuilds.
const ThreadTurnWindowLookupInput = Schema.Struct({
  threadId: ThreadId,
  // Exclusive keyset upper bound. Sentinels "~"/"" mean unbounded ("~" sorts
  // after every ISO timestamp).
  beforeAnchorAt: Schema.String,
  beforeTurnKey: Schema.String,
  userTurnLimit: Schema.Number,
  maxRawTurns: Schema.Number,
});
const ProjectionTurnWindowRowSchema = Schema.Struct({
  // The turn's timeline anchor, used to bound rows that have no turn linkage
  // (user messages and turnless activities) to the same page window.
  anchorAt: Schema.String,
  turnKey: Schema.String,
});
const ThreadTurnRangeLookupInput = Schema.Struct({
  threadId: ThreadId,
  // Turn-linked rows are bounded by the keyset range [min, before) over
  // (anchor, turn key); turnless rows by the matching [minAnchorAt,
  // beforeAnchorAt) time range. Unbounded ends use sentinels: "" for the
  // lower bound, "~" (sorts after ISO dates) for the upper bound.
  minAnchorAt: Schema.String,
  minTurnKey: Schema.String,
  beforeAnchorAt: Schema.String,
  beforeTurnKey: Schema.String,
});
const ProjectionProjectLookupRowSchema = ProjectionProjectDbRowSchema;
const ProjectionThreadIdLookupRowSchema = Schema.Struct({
  threadId: ThreadId,
});
const ProjectionThreadCheckpointContextThreadRowSchema = Schema.Struct({
  threadId: ThreadId,
  projectId: ProjectId,
  workspaceRoot: Schema.String,
  worktreePath: Schema.NullOr(Schema.String),
});
const FullThreadDiffContextLookupInput = Schema.Struct({
  threadId: ThreadId,
  checkpointTurnCount: NonNegativeInt,
});
const ProjectionFullThreadDiffContextRowSchema = Schema.Struct({
  threadId: ThreadId,
  projectId: ProjectId,
Read this as: The snapshot helper requires its required projectors and returns the minimum sequence among them. It intentionally does not read the event-store head.
Interactive fold lab

One event stream, nine projector cursors

Step through three durable events. Every registered projector is considered in order, including no-ops; the safe snapshot watermark is the slowest required cursor, not the event-store head.

No sample event has been folded. All projector cursors are at 0.

Event-store head0
Safe snapshot sequence0
Current eventnone
Normal command control flowouter SQL begins → append → projector SQL + cursor → optional filesystem cleanup → accepted receipt → outer commitCleanup runs before commit but is not enlisted in SQL rollback.
  1. Projectsprojection_projects · project fields · snapshot-required
    0
  2. Messagesprojection_thread_messages · message bodies · snapshot-required
    0
  3. Plansprojection_thread_proposed_plans · proposed plans · snapshot-required
    0
  4. Activitiesprojection_thread_activities · activity log · snapshot-required
    0
  5. Sessionsprojection_thread_sessions · runtime session · snapshot-required
    0
  6. Turnsprojection_turns · turns + checkpoint metadata · auxiliary cursor
    0
  7. Checkpointsno table · registered no-op · snapshot-required
    0
  8. Approvalsprojection_pending_approvals · approval state · auxiliary cursor
    0
  9. Threadsprojection_threads · shell summary (last) · snapshot-required
    0
Start with the cursor, not a promise of eventual consistency.

On a normal command, all nine folds run sequentially before the command transaction commits.

Static reference: the normal third event leaves every required cursor at 3, so the safe snapshot sequence is 3.

1,001-event bootstrap case: the default total read limit stops every lane at 1,000 while the event-store head is 1,001.

OrderProjectorRead sideWatermark roleAt sequence 3
1Projectsprojection_projectsrequired3
2Messagesprojection_thread_messagesrequired3
3Plansprojection_thread_proposed_plansrequired3
4Activitiesprojection_thread_activitiesrequired3
5Sessionsprojection_thread_sessionsrequired3
6Turnsprojection_turnsauxiliary3
7Checkpointsno tablerequired3
8Approvalsprojection_pending_approvalsauxiliary3
9Threadsprojection_threadsrequired3

Checkpoint is a naming trap

Checkpoint data is currently carried by turn projection rows, rather than an active independent checkpoint table projector. The projector named projection.checkpoints returns no work. The detail snapshot query obtains checkpoint-oriented rows from the turn read side. Treat the name as a tracked pipeline slot, not evidence of a separate materialized checkpoint store.

One database, several deliberately incomplete views

“The snapshot” is not one universal object. The full hydrated snapshot composes projects, threads, messages, plans, activities, sessions, checkpoints, turns, and projector state in a single query transaction. The engine instead starts from a lighter command model that intentionally omits message bodies, activities, and checkpoints. The navigation shell is leaner still: project and active-thread summary rows. Thread detail is a dedicated, possibly paged read.

Comparison of projection query shapes
ReaderPurposeWhat it intentionally leaves out
Full hydrated snapshotauthoritative composed read for complete statenothing from its selected projection families; it is the expensive shape
Command read modelengine bootstrap and command decisionsmessages, activities, and checkpoints
Shell snapshotproject/thread navigation and live shell refetchesthread bodies, activity history, plans, checkpoint/turn detail
Thread detail snapshotone selected conversation, optionally windowedolder turn windows beyond its cursor until requested

The HTTP /orchestration/snapshot endpoint returns the command-model shape, not the full hydrated snapshot. That is an API-level reminder not to infer a UI payload just from a type named “read model.” Shell and thread readers have their own endpoints and WebSocket stream behavior.

Live repair uses global cursors; page merging has another watermark

WebSocket resume is bounded. A client’s afterSequence is a global event sequence. The handler attaches a live buffer before replay or snapshot work, then either replays a non-negative global gap through a captured head or falls back to a fresh snapshot when the gap is absent, impossible, or exceeds 1,000 events. Buffered live input can overlap the replay; client reducers deduplicate by global sequence.

Thread detail adds threadSequence to a windowed page. It is the greatest deliverable detail-event sequence for that thread at or below the snapshot sequence. Its purpose is to delay merging a historical page until the live reducer has caught up. It is not the afterSequence accepted by the subscription API, and it cannot replace the global cursor.

apps/server/src/ws.ts:1394–1521 ↗verbatim · typescript · 9423f5c9
                      message: "Failed to dispatch orchestration command",
                      cause,
                    }),
              ),
            ),
            { "rpc.aggregate": "orchestration" },
          ),
        [ORCHESTRATION_WS_METHODS.getWorkflowScript]: (input) =>
          observeRpcEffect(
            ORCHESTRATION_WS_METHODS.getWorkflowScript,
            readWorkflowScript({ scriptPath: input.scriptPath }),
            { "rpc.aggregate": "orchestration" },
          ),
        [ORCHESTRATION_WS_METHODS.getTurnDiff]: (input) =>
          observeRpcEffect(
            ORCHESTRATION_WS_METHODS.getTurnDiff,
            checkpointDiffQuery.getTurnDiff(input).pipe(
              Effect.mapError(
                (cause) =>
                  new OrchestrationGetTurnDiffError({
                    message: "Failed to load turn diff",
                    cause,
                  }),
              ),
            ),
            { "rpc.aggregate": "orchestration" },
          ),
        [ORCHESTRATION_WS_METHODS.getFullThreadDiff]: (input) =>
          observeRpcEffect(
            ORCHESTRATION_WS_METHODS.getFullThreadDiff,
            checkpointDiffQuery.getFullThreadDiff(input).pipe(
              Effect.mapError(
                (cause) =>
                  new OrchestrationGetFullThreadDiffError({
                    message: "Failed to load full thread diff",
                    cause,
                  }),
              ),
            ),
            { "rpc.aggregate": "orchestration" },
          ),
        [ORCHESTRATION_WS_METHODS.searchThreads]: (input) =>
          observeRpcEffect(
            ORCHESTRATION_WS_METHODS.searchThreads,
            projectionSnapshotQuery.searchThreads(input).pipe(
              Effect.mapError(
                (cause) =>
                  new OrchestrationSearchThreadsError({
                    message: "Failed to search threads",
                    cause,
                  }),
              ),
            ),
            { "rpc.aggregate": "orchestration" },
          ),
        [ORCHESTRATION_WS_METHODS.subscribeShell]: (input) =>
          observeRpcStreamEffect(
            ORCHESTRATION_WS_METHODS.subscribeShell,
            Effect.gen(function* () {
              // Coalesce the live shell stream per aggregate over a small window
              // so bursts of high-frequency events (streaming message deltas,
              // activity appends) collapse into a single shell refetch and never
              // serialize a brand-new thread's `thread.created` behind hundreds
              // of per-event DB reads. See coalesceShellStream.
              // Attach live delivery into a scope-bound buffer BEFORE loading any
              // snapshot or draining catch-up, otherwise an event published while
              // the snapshot query is in flight is lost (it is past the snapshot's
              // sequence but the live subscription is not attached yet). Every
              // path below emits from this same buffered live tail. Overlapping
              // events are deduped by sequence on the client.
              const liveBudget = yield* makeLiveStreamBudget();
              const liveBuffer = yield* Queue.unbounded<
                RetainedLiveItem<ShellLiveInput>,
                OrchestrationGetSnapshotError
              >();
              let liveBufferClosed = false;
              const closeLiveBuffer = (error?: OrchestrationGetSnapshotError) =>
                Effect.gen(function* () {
                  if (liveBufferClosed) {
                    return;
                  }
                  liveBufferClosed = true;
                  liveBudget.release(yield* Queue.clear(liveBuffer).pipe(Effect.orDie));
                  if (error) {
                    yield* Queue.fail(liveBuffer, error);
                  }
                  yield* Queue.shutdown(liveBuffer);
                });
              yield* Effect.addFinalizer(() => closeLiveBuffer());
              yield* liveBudget.failed.pipe(
                Effect.catchTags({ OrchestrationGetSnapshotError: closeLiveBuffer }),
                Effect.forkScoped,
              );
              yield* Effect.forkScoped(
                orchestrationEngine.streamDomainEvents.pipe(
                  Stream.map(toShellEvent),
                  Stream.runForEach((event) =>
                    liveBudget.retain({ kind: "event" as const, event }, event).pipe(
                      Effect.flatMap((item) => Queue.offer(liveBuffer, item)),
                      Effect.uninterruptible,
                    ),
                  ),
                  // Stop the PubSub consumer even if RPC delivery is waiting
                  // for an ACK and never pulls the failed buffer again.
                  Effect.raceFirst(liveBudget.failed),
                  Effect.catchTags({ OrchestrationGetSnapshotError: () => Effect.void }),
                ),
                { startImmediately: true },
              );
              const coalesceRetainedInputs = (
                items: ReadonlyArray<RetainedLiveItem<ShellLiveInput>>,
              ) =>
                coalesceShellLiveInputs(items.map((item) => item.value)).pipe(
                  Effect.flatMap((output) => liveBudget.replace(items, output)),
                );
              const bufferedLiveStream = Stream.fromQueue(liveBuffer).pipe(
                Stream.groupedWithin(SHELL_COALESCE_MAX_CHUNK, SHELL_COALESCE_WINDOW),
                Stream.mapEffect(coalesceRetainedInputs),
                Stream.flatMap((items) => Stream.fromIterable(items)),
              );
 
              const loadSnapshot = projectionSnapshotQuery.getShellSnapshot().pipe(
                Effect.tapError((cause) =>
                  Effect.logError("orchestration shell snapshot load failed", { cause }),
                ),
                Effect.mapError(
                  (cause) =>
                    new OrchestrationGetSnapshotError({
Read this as: Thread subscription attaches its scoped live input before choosing replay or snapshot. Its replay bound uses the global gap, then filters events for the selected thread.

Pull requests, ordering, and project presentation

The projection schema keeps several user-facing facts independently queryable: manual active-thread order, unsettled timestamps, project auto-pull policy, project icons, branch-associated reviews, and the many-to-many link between threads and pull requests. This is why a review can remain attached to a thread even when it is not the branch’s primary review, and why a stack can be rendered as related layers rather than flattened into one URL.

The migration sequence makes the ownership clear: migrations 42 and 48 introduce singular and branch-aware review state; migration 50 replaces that shape with plural thread pull requests. The projector, snapshot query, and clients then agree on the same graph. Browse the current migration directory.

Source trail

T3
Source-locked editionRead against 859304b78 · 10 Sept 2026
Book search

Find a concept, module, or source path

Type two or more characters.