Events, receipts, idempotency, and the post-commit gap

T3 Code persists two related ledgers for every accepted command:

• the event ledger says what happened, in global order;
• the receipt ledger says what result belongs to this command id.

They commit together, but they answer different questions. An event supports replay and projection. A receipt prevents one command id from creating a second domain transaction for the same aggregate. Neither ledger records whether a provider harness ultimately completed the work triggered by that intent.

The public event envelope

Every event variant combines the same base envelope with a type-specific payload.

The event store therefore has two orders with different visibility:

• a global sequence, exposed on every event and used for resumable reads;
• a per-stream stream_version, computed and stored internally.

The per-stream version is not an expected-version token in the command protocol. As Chapter 9 showed, one serialized command worker prevents concurrent decisions inside a server process.

A receipt is smaller than the command it deduplicates

export const OrchestrationCommandReceipt = Schema.Struct({
  commandId: CommandId,
  aggregateKind: OrchestrationAggregateKind,
  aggregateId: Schema.Union([ProjectId, ThreadId]),
  acceptedAt: IsoDateTime,
  resultSequence: NonNegativeInt,
  status: OrchestrationCommandReceiptStatus,
  error: Schema.NullOr(Schema.String),
});

For an accepted command, resultSequence is the sequence of the last event in its batch. For an invariant rejection, the same schema is used differently: no event is appended, status is rejected, error carries the rejection, and resultSequence is the command model’s current snapshot sequence. The field named acceptedAt is populated for both statuses.

This is deduplication by caller discipline: the caller must treat a command id as the permanent name of one intent. The server catches reuse across aggregates, but it cannot catch changed intent within the same aggregate.

Accepted events, projections, and receipt share one transaction

        const committedCommand = yield* sql
          .withTransaction(
            Effect.gen(function* () {
              const committedEvents: OrchestrationEvent[] = [];
              let nextCommandReadModel = commandReadModel;
 
              for (const nextEvent of eventBases) {
                const savedEvent = yield* eventStore.append(nextEvent);
                nextCommandReadModel = yield* projectEvent(nextCommandReadModel, savedEvent);
                yield* projectionPipeline.projectEvent(savedEvent);
                committedEvents.push(savedEvent);
              }
 
              const lastSavedEvent = committedEvents.at(-1) ?? null;
              if (lastSavedEvent === null) {
                return yield* new OrchestrationCommandInvariantError({
                  commandType: envelope.command.type,
                  detail: "Command produced no events.",
                });
              }
 
              yield* commandReceiptRepository.upsert({
                commandId: envelope.command.commandId,
                aggregateKind: lastSavedEvent.aggregateKind,
                aggregateId: lastSavedEvent.aggregateId,
                acceptedAt: lastSavedEvent.occurredAt,
                resultSequence: lastSavedEvent.sequence,
                status: "accepted",
                error: null,
              });
 
              return {
                committedEvents,
                lastSequence: lastSavedEvent.sequence,
                nextCommandReadModel,
              } as const;
            }),
          )
          .pipe(
            Effect.catchTag("SqlError", (sqlError) =>
              Effect.fail(
                toPersistenceSqlError("OrchestrationEngine.processEnvelope:transaction")(sqlError),
              ),
            ),
          );
 
        commandReadModel = committedCommand.nextCommandReadModel;
        for (const [index, event] of committedCommand.committedEvents.entries()) {
          yield* PubSub.publish(eventPubSub, event);
          if (index === 0) {
            yield* Metric.update(
              Metric.withAttributes(
                orchestrationCommandAckDuration,
                metricAttributes({
                  ...baseMetricAttributes,
                  ackEventType: event.type,
                }),
              ),
              Duration.millis(Math.max(0, (yield* Clock.currentTimeMillis) - envelope.startedAtMs)),
            );
          }
        }
        return { sequence: committedCommand.lastSequence };

flowchart LR
  D[decide event batch] --> T
  subgraph T[one SQL transaction]
    E[append all events] --> P[apply SQLite projections]
    P --> R[upsert accepted receipt]
  end
  T -->|commit| M[replace in-memory model]
  M --> H[publish event 1..N]
  H --> A[return last sequence]
  H --> X[hot reactor streams]
  X --> V[provider + side effects]
  T -.process dies after commit.-> G[durable intent; hot event missing]

Three observations follow directly from this ordering:

1. A successful commit never leaves an accepted receipt without its event batch or transactional projections.
2. Hot publication is not part of that atomic unit.
3. The RPC result waits for the publication loop, but publication only offers events to in-memory subscribers; it does not wait for provider work to finish.

Duplicate behavior is a five-row matrix

Receipt lookup happens before the decider. Aggregate comparison happens before receipt-status replay.

An accepted same-aggregate retry does not replay the original events into the hot stream. It returns the stored sequence immediately. This is desirable after a lost RPC response when the original publication completed. It cannot repair a commit-to-publication crash gap.

Persistence failures inside the accepted transaction are different. The transaction rolls back, no accepted receipt remains, and the failure is not recorded as a domain invariant rejection. A retry can run the decision and transaction again.

The hard-crash gap is real

After SQL commit, the engine first replaces its in-memory model, then publishes each committed event sequentially. The pinned production-code audit found no outbox row whose delivery acknowledgement commits with the events.

The failure handler does provide a narrower repair path while the process is still alive. For most dispatch failures, it reads the durable tail after the command’s starting sequence, folds that tail back into the command model, and republishes it. That can reconcile a recoverable post-commit failure in the same process. It cannot run after a hard process crash, and it is not a durable outbox protocol. If a prefix was already published before a recoverable failure, replaying the durable tail can offer that prefix again; reconciliation is not exactly-once delivery.

Explore receipts and crash windows

The lab starts with an accepted receipt. Change its status, target aggregate, body, and original delivery window. In particular, compare “response lost” with “hard crash after commit”: both can look like silence to the first caller, but only one preserves the original hot publication.

What a sequence acknowledgement proves

The right mental model is deliberately narrow: receipt idempotency protects one serialized domain commit. It is not payload equality, not normalization idempotency, not saga idempotency, not durable reactor delivery, and not provider exactly-once execution.