Projection tables and read models
New events fold through ordered SQLite projectors before the enclosing command commits; bootstrap replays committed events from per-projector cursors, and queries expose deliberately different read-model shapes.
What this chapter resolves
- Trace one durable event through every projection cursor and its SQL boundary.
- Distinguish the safe snapshot watermark from the durable event-store head.
- Separate full snapshots, command state, shell summaries, detail pages, and live replay.
An orchestration event is not directly a UI row. The durable event store records a global sequence; a projection pipeline folds that event into several SQLite read tables; query services compose those tables into views for a particular reader. This is a synchronous part of normal command acceptance, not a background eventually-consistent worker that happens after a command has returned.
Nine folds share one event order
Each named projector has its own projection_state cursor. For every event, the
pipeline considers every projector in this exact sequence:
- Projects
- Messages
- Proposed plans
- Activities
- Sessions
- Turns
- Checkpoints
- Pending approvals
- Threads
The final placement is meaningful. The thread projection refreshes summary fields after the message, plan, activity, session, turn, and approval tables have been updated. It can therefore compute navigation-oriented facts such as pending approval/input counts and whether a plan is actionable from current dependent rows.
Zoom with the controls, +/−, or Ctrl/⌘ + trackpad scroll. Enable Pan to drag, use two-finger scrolling, or use the arrow keys. 0 fits the diagram; Esc leaves Pan or expanded view.
Text equivalent
The engine's outer SQL transaction appends an event, then runs projects, messages, plans, activities, sessions, turns, checkpoints, pending approvals, and threads in order. Each projector first completes its projection row and cursor SQL. After the messages projector, an optional attachment-pruning filesystem detour can occur before plans. After the final threads projector, an optional attachment-deletion detour can occur before the accepted receipt. The outer SQL transaction commits only after those steps, making the event, projection rows, cursors, and receipt durable. Filesystem mutations cannot be rolled back if later SQL fails. Committed events are then published to live consumers.
return;
}
case "thread.turn-interrupt-requested": {
if (event.payload.turnId === undefined) {
return;
}
const existingTurn = yield* projectionTurnRepository.getByTurnId({
threadId: event.payload.threadId,
turnId: event.payload.turnId,
});
if (Option.isSome(existingTurn)) {
yield* projectionTurnRepository.upsertByTurnId({
...existingTurn.value,
state: "interrupted",
completedAt: existingTurn.value.completedAt ?? event.payload.createdAt,
startedAt: existingTurn.value.startedAt ?? event.payload.createdAt,
requestedAt: existingTurn.value.requestedAt ?? event.payload.createdAt,
});
return;
}
yield* projectionTurnRepository.upsertByTurnId({
turnId: event.payload.turnId,
threadId: event.payload.threadId,
pendingMessageId: null,
sourceProposedPlanThreadId: null,
sourceProposedPlanId: null,
assistantMessageId: null,
state: "interrupted",
requestedAt: event.payload.createdAt,
startedAt: event.payload.createdAt,
completedAt: event.payload.createdAt,
checkpointTurnCount: null,
checkpointRef: null,
checkpointStatus: null,
checkpointFiles: [],
});
return;
}
case "thread.turn-diff-completed": {
// Mid-turn diff updates produce placeholder checkpoints; record the
// checkpoint, but don't settle a turn its session is still running.
const session = yield* projectionThreadSessionRepository.getByThreadId({
threadId: event.payload.threadId,
});
const turnStillRunning =
Option.isSome(session) &&
session.value.status === "running" &&
session.value.activeTurnId === event.payload.turnId;
const existingTurn = yield* projectionTurnRepository.getByTurnId({
threadId: event.payload.threadId,
turnId: event.payload.turnId,
});
const nextState = event.payload.status === "error" ? "error" : "completed";
yield* projectionTurnRepository.clearCheckpointTurnConflict({
threadId: event.payload.threadId,
turnId: event.payload.turnId,
checkpointTurnCount: event.payload.checkpointTurnCount,
});
if (Option.isSome(existingTurn)) {
yield* projectionTurnRepository.upsertByTurnId({
...existingTurn.value,
assistantMessageId: event.payload.assistantMessageId,
state:
turnStillRunning || existingTurn.value.state === "interrupted"
? existingTurn.value.state
: nextState,
checkpointTurnCount: event.payload.checkpointTurnCount,Cursor means “safe through here,” not “the newest event”
The event store has a monotonically increasing, global sequence; each aggregate
also has its own stream version. Projection restart uses the former. On bootstrap,
each projector reads only events after its own last_applied_sequence and advances
that cursor after its SQL fold. A projector that has nothing to do for an event still
records that it safely considered the event.
The query layer does not advertise the highest event in the log as a snapshot sequence. It checks the required projector state and returns their minimum. If a required state row is missing, the safe sequence is zero. That conservative watermark prevents a snapshot composed from several tables from claiming it includes an event that one required read side has not reached.
threadId: ThreadId,
runtimePayload: Schema.Unknown,
});
const ThreadIdLookupInput = Schema.Struct({
threadId: ThreadId,
});
const TurnStartMessageLookupInput = Schema.Struct({
threadId: ThreadId,
messageId: MessageId,
});
const ThreadActivityKindsLookupInput = Schema.Struct({
threadId: ThreadId,
activityKinds: Schema.Array(Schema.String),
});
const ThreadActivityIdsLookupInput = Schema.Struct({
activityIds: Schema.Array(ProjectionThreadActivity.fields.activityId),
});
// Windowed reads order turns by the stable keyset (anchor, turn key), where
// anchor is requested_at and turn key is
// COALESCE(turn_id, ''). Both are event-derived, so cursors survive the
// revert projector's row-id rewrite and full projection rebuilds.
const ThreadTurnWindowLookupInput = Schema.Struct({
threadId: ThreadId,
// Exclusive keyset upper bound. Sentinels "~"/"" mean unbounded ("~" sorts
// after every ISO timestamp).
beforeAnchorAt: Schema.String,
beforeTurnKey: Schema.String,
userTurnLimit: Schema.Number,
maxRawTurns: Schema.Number,
});
const ProjectionTurnWindowRowSchema = Schema.Struct({
// The turn's timeline anchor, used to bound rows that have no turn linkage
// (user messages and turnless activities) to the same page window.
anchorAt: Schema.String,
turnKey: Schema.String,
});
const ThreadTurnRangeLookupInput = Schema.Struct({
threadId: ThreadId,
// Turn-linked rows are bounded by the keyset range [min, before) over
// (anchor, turn key); turnless rows by the matching [minAnchorAt,
// beforeAnchorAt) time range. Unbounded ends use sentinels: "" for the
// lower bound, "~" (sorts after ISO dates) for the upper bound.
minAnchorAt: Schema.String,
minTurnKey: Schema.String,
beforeAnchorAt: Schema.String,
beforeTurnKey: Schema.String,
});
const ProjectionProjectLookupRowSchema = ProjectionProjectDbRowSchema;
const ProjectionThreadIdLookupRowSchema = Schema.Struct({
threadId: ThreadId,
});
const ProjectionThreadCheckpointContextThreadRowSchema = Schema.Struct({
threadId: ThreadId,
projectId: ProjectId,
workspaceRoot: Schema.String,
worktreePath: Schema.NullOr(Schema.String),
});
const FullThreadDiffContextLookupInput = Schema.Struct({
threadId: ThreadId,
checkpointTurnCount: NonNegativeInt,
});
const ProjectionFullThreadDiffContextRowSchema = Schema.Struct({
threadId: ThreadId,
projectId: ProjectId,One event stream, nine projector cursors
Step through three durable events. Every registered projector is considered in order, including no-ops; the safe snapshot watermark is the slowest required cursor, not the event-store head.
No sample event has been folded. All projector cursors are at 0.
- Projects
projection_projects· project fields · snapshot-required - Messages
projection_thread_messages· message bodies · snapshot-required - Plans
projection_thread_proposed_plans· proposed plans · snapshot-required - Activities
projection_thread_activities· activity log · snapshot-required - Sessions
projection_thread_sessions· runtime session · snapshot-required - Turns
projection_turns· turns + checkpoint metadata · auxiliary cursor - Checkpoints
no table· registered no-op · snapshot-required - Approvals
projection_pending_approvals· approval state · auxiliary cursor - Threads
projection_threads· shell summary (last) · snapshot-required
On a normal command, all nine folds run sequentially before the command transaction commits.
The event store is at 1,001, but bootstrap calls its reader with the default total limit of 1,000. The stream completes with every projector cursor one event behind; this path has no second catch-up call in the same bootstrap.
Static reference: the normal third event leaves every required cursor at 3, so the safe snapshot sequence is 3.
1,001-event bootstrap case: the default total read limit stops every lane at 1,000 while the event-store head is 1,001.
| Order | Projector | Read side | Watermark role | At sequence 3 |
|---|---|---|---|---|
| 1 | Projects | projection_projects | required | 3 |
| 2 | Messages | projection_thread_messages | required | 3 |
| 3 | Plans | projection_thread_proposed_plans | required | 3 |
| 4 | Activities | projection_thread_activities | required | 3 |
| 5 | Sessions | projection_thread_sessions | required | 3 |
| 6 | Turns | projection_turns | auxiliary | 3 |
| 7 | Checkpoints | no table | required | 3 |
| 8 | Approvals | projection_pending_approvals | auxiliary | 3 |
| 9 | Threads | projection_threads | required | 3 |
Checkpoint is a naming trap
Checkpoint data is currently carried by turn projection rows, rather than an active
independent checkpoint table projector. The projector named projection.checkpoints
returns no work. The detail snapshot query obtains checkpoint-oriented rows from the
turn read side. Treat the name as a tracked pipeline slot, not evidence of a separate
materialized checkpoint store.
One database, several deliberately incomplete views
“The snapshot” is not one universal object. The full hydrated snapshot composes projects, threads, messages, plans, activities, sessions, checkpoints, turns, and projector state in a single query transaction. The engine instead starts from a lighter command model that intentionally omits message bodies, activities, and checkpoints. The navigation shell is leaner still: project and active-thread summary rows. Thread detail is a dedicated, possibly paged read.
| Reader | Purpose | What it intentionally leaves out |
|---|---|---|
| Full hydrated snapshot | authoritative composed read for complete state | nothing from its selected projection families; it is the expensive shape |
| Command read model | engine bootstrap and command decisions | messages, activities, and checkpoints |
| Shell snapshot | project/thread navigation and live shell refetches | thread bodies, activity history, plans, checkpoint/turn detail |
| Thread detail snapshot | one selected conversation, optionally windowed | older turn windows beyond its cursor until requested |
The HTTP /orchestration/snapshot endpoint returns the command-model shape, not the
full hydrated snapshot. That is an API-level reminder not to infer a UI payload just
from a type named “read model.” Shell and thread readers have their own endpoints and
WebSocket stream behavior.
Live repair uses global cursors; page merging has another watermark
WebSocket resume is bounded. A client’s afterSequence is a global event sequence.
The handler attaches a live buffer before replay or snapshot work, then either
replays a non-negative global gap through a captured head or falls back to a fresh
snapshot when the gap is absent, impossible, or exceeds 1,000 events. Buffered live
input can overlap the replay; client reducers deduplicate by global sequence.
Thread detail adds threadSequence to a windowed page. It is the greatest
deliverable detail-event sequence for that thread at or below the snapshot sequence.
Its purpose is to delay merging a historical page until the live reducer has caught
up. It is not the afterSequence accepted by the subscription API, and it cannot
replace the global cursor.
message: "Failed to dispatch orchestration command",
cause,
}),
),
),
{ "rpc.aggregate": "orchestration" },
),
[ORCHESTRATION_WS_METHODS.getWorkflowScript]: (input) =>
observeRpcEffect(
ORCHESTRATION_WS_METHODS.getWorkflowScript,
readWorkflowScript({ scriptPath: input.scriptPath }),
{ "rpc.aggregate": "orchestration" },
),
[ORCHESTRATION_WS_METHODS.getTurnDiff]: (input) =>
observeRpcEffect(
ORCHESTRATION_WS_METHODS.getTurnDiff,
checkpointDiffQuery.getTurnDiff(input).pipe(
Effect.mapError(
(cause) =>
new OrchestrationGetTurnDiffError({
message: "Failed to load turn diff",
cause,
}),
),
),
{ "rpc.aggregate": "orchestration" },
),
[ORCHESTRATION_WS_METHODS.getFullThreadDiff]: (input) =>
observeRpcEffect(
ORCHESTRATION_WS_METHODS.getFullThreadDiff,
checkpointDiffQuery.getFullThreadDiff(input).pipe(
Effect.mapError(
(cause) =>
new OrchestrationGetFullThreadDiffError({
message: "Failed to load full thread diff",
cause,
}),
),
),
{ "rpc.aggregate": "orchestration" },
),
[ORCHESTRATION_WS_METHODS.searchThreads]: (input) =>
observeRpcEffect(
ORCHESTRATION_WS_METHODS.searchThreads,
projectionSnapshotQuery.searchThreads(input).pipe(
Effect.mapError(
(cause) =>
new OrchestrationSearchThreadsError({
message: "Failed to search threads",
cause,
}),
),
),
{ "rpc.aggregate": "orchestration" },
),
[ORCHESTRATION_WS_METHODS.subscribeShell]: (input) =>
observeRpcStreamEffect(
ORCHESTRATION_WS_METHODS.subscribeShell,
Effect.gen(function* () {
// Coalesce the live shell stream per aggregate over a small window
// so bursts of high-frequency events (streaming message deltas,
// activity appends) collapse into a single shell refetch and never
// serialize a brand-new thread's `thread.created` behind hundreds
// of per-event DB reads. See coalesceShellStream.
// Attach live delivery into a scope-bound buffer BEFORE loading any
// snapshot or draining catch-up, otherwise an event published while
// the snapshot query is in flight is lost (it is past the snapshot's
// sequence but the live subscription is not attached yet). Every
// path below emits from this same buffered live tail. Overlapping
// events are deduped by sequence on the client.
const liveBudget = yield* makeLiveStreamBudget();
const liveBuffer = yield* Queue.unbounded<
RetainedLiveItem<ShellLiveInput>,
OrchestrationGetSnapshotError
>();
let liveBufferClosed = false;
const closeLiveBuffer = (error?: OrchestrationGetSnapshotError) =>
Effect.gen(function* () {
if (liveBufferClosed) {
return;
}
liveBufferClosed = true;
liveBudget.release(yield* Queue.clear(liveBuffer).pipe(Effect.orDie));
if (error) {
yield* Queue.fail(liveBuffer, error);
}
yield* Queue.shutdown(liveBuffer);
});
yield* Effect.addFinalizer(() => closeLiveBuffer());
yield* liveBudget.failed.pipe(
Effect.catchTags({ OrchestrationGetSnapshotError: closeLiveBuffer }),
Effect.forkScoped,
);
yield* Effect.forkScoped(
orchestrationEngine.streamDomainEvents.pipe(
Stream.map(toShellEvent),
Stream.runForEach((event) =>
liveBudget.retain({ kind: "event" as const, event }, event).pipe(
Effect.flatMap((item) => Queue.offer(liveBuffer, item)),
Effect.uninterruptible,
),
),
// Stop the PubSub consumer even if RPC delivery is waiting
// for an ACK and never pulls the failed buffer again.
Effect.raceFirst(liveBudget.failed),
Effect.catchTags({ OrchestrationGetSnapshotError: () => Effect.void }),
),
{ startImmediately: true },
);
const coalesceRetainedInputs = (
items: ReadonlyArray<RetainedLiveItem<ShellLiveInput>>,
) =>
coalesceShellLiveInputs(items.map((item) => item.value)).pipe(
Effect.flatMap((output) => liveBudget.replace(items, output)),
);
const bufferedLiveStream = Stream.fromQueue(liveBuffer).pipe(
Stream.groupedWithin(SHELL_COALESCE_MAX_CHUNK, SHELL_COALESCE_WINDOW),
Stream.mapEffect(coalesceRetainedInputs),
Stream.flatMap((items) => Stream.fromIterable(items)),
);
const loadSnapshot = projectionSnapshotQuery.getShellSnapshot().pipe(
Effect.tapError((cause) =>
Effect.logError("orchestration shell snapshot load failed", { cause }),
),
Effect.mapError(
(cause) =>
new OrchestrationGetSnapshotError({Pull requests, ordering, and project presentation
The projection schema keeps several user-facing facts independently queryable: manual active-thread order, unsettled timestamps, project auto-pull policy, project icons, branch-associated reviews, and the many-to-many link between threads and pull requests. This is why a review can remain attached to a thread even when it is not the branch’s primary review, and why a stack can be rendered as related layers rather than flattened into one URL.
The migration sequence makes the ownership clear: migrations 42 and 48 introduce singular and branch-aware review state; migration 50 replaces that shape with plural thread pull requests. The projector, snapshot query, and clients then agree on the same graph. Browse the current migration directory.