ARC 3Understand Replication and RepairChapters 10–14

Chapter 11 — The Remote Path of One Write

Chapter objective: After this chapter, a developer can observe local commit, delta encoding, transport, remote validation, and application separately, and explain why online propagation does not replace repair.

Learning objectives

  1. Trace a Register update through five stages from local mutation to remote application.
  2. Explain collection locator, schema, and delta-envelope responsibilities.
  3. Separate control-plane messages from data-plane transfer.
  4. Define a repairable boundary for missed propagation.

Prerequisites

  • Register operations from Chapter 5 are complete.
  • Member views and data content are distinct.
  • A successful local put includes no remote acknowledgement.

Case progress

Route Publisher updates the healthy payment route to 10.0.0.8 on node-a. The local API sees it immediately while node-b briefly retains the old address. The team inspects each propagation stage instead of calling the whole condition "slow synchronization."

Five stages and five observation points

A delta from local commit to remote application

  1. Local commit: the collection processes resolver/metadata rules and becomes readable locally.
  2. Collection location: the change carries tenant, application, and collection identity.
  3. Envelope encoding: delta and schema/capability information enter the wire envelope.
  4. Data channel: sending, framing, retries, or loss occur at the transport boundary.
  5. Remote application: the peer validates target and capability, then applies REMOTE_DELTA to the collection.

Stage 1 can return local success while Stage 5 happens later or never happens on the online path.

Identity precedes payload correctness

Two collections that both store RouteHint cannot automatically accept each other's deltas. RuntimePlatformSyncService addresses a collection route/locator and routes Register, Lease, or CRDT messages after capability negotiation.

A propagation contract answers:

  • Which logical service sent the message?
  • What is the target collection locator?
  • Does the payload belong to Register, Lease, or CRDT?
  • Are schema and codec compatible?
  • Is this an online delta, replay item, or snapshot chunk?

Ambiguity in any answer can turn "message delivered" into an incorrect claim that state was applied safely.

The boundary of online delta

Online delta optimizes low-latency propagation under normal conditions. It is not a durable business-event log.

Condition Possible online outcome Follow-up responsibility
receiver temporarily offline no real-time application compare digest and repair after recovery
control message lost difference remains undiscovered briefly anti-entropy sweep detects it later
incompatible envelope application rejected record the reason and correct version/capability
duplicate arrival idempotence or metadata arbitration needed collection merge/resolver handles it
out-of-order arrival old value cannot replace new lineage, HLC, or collection semantics handle it

A ChangeStream consumer therefore cannot infer complete history. The stream observes collection changes and does not provide message-queue audit semantics.

Counterexample and fault injection

Drop the data path from node-a to node-b and perform a local put. A reads the new route, B retains the old route, and local success remains valid. After network recovery, repair closes the gap. An assertion against A proves only local semantics. An assertion against B without event origin cannot distinguish online delta from later repair.

Experiment

cd submodule/dsm
mvn -q -pl dsm-integration-test -am \
  -Dtest=TwoNodeIntegrationTest,ChaosIntegrationTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

Inspect delta-trace.json and confirm that every stage records input, output, an observable signal, and a continuation after failure.

Experiment acceptance card

Field Content
Command The focused Maven tests above; node --test tests/chapter-assets.test.mjs
Input or fault local update, remote node, missed online message, and recovery
Observable result local success occurs first; the normal path reaches the peer; repair catches a missed update
Evidence level E3: in-process multi-node and chaos integration tests
This experiment does not prove Cross-host throughput, real-network retry limits, or exactly-once business events

Review

  • Local commit and remote visibility are separate completion points.
  • Locator, collection type, and schema constrain the remote target together.
  • Online delta favors low latency and does not guarantee complete history.
  • Missed messages are an expected input; repair detects and closes gaps continuously.

Next

Chapter 12 lets a late node-b compare digests and select replay or snapshot with an explainable decision.