Chapter 11 — The Remote Path of One Write
Chapter objective: After this chapter, a developer can observe local commit, delta encoding, transport, remote validation, and application separately, and explain why online propagation does not replace repair.
Learning objectives
- Trace a Register update through five stages from local mutation to remote application.
- Explain collection locator, schema, and delta-envelope responsibilities.
- Separate control-plane messages from data-plane transfer.
- Define a repairable boundary for missed propagation.
Prerequisites
- Register operations from Chapter 5 are complete.
- Member views and data content are distinct.
- A successful local
putincludes no remote acknowledgement.
Case progress
Route Publisher updates the healthy payment route to
10.0.0.8 on node-a. The local API sees it
immediately while node-b briefly retains the old address.
The team inspects each propagation stage instead of calling the whole
condition "slow synchronization."
Five stages and five observation points
- Local commit: the collection processes resolver/metadata rules and becomes readable locally.
- Collection location: the change carries tenant, application, and collection identity.
- Envelope encoding: delta and schema/capability information enter the wire envelope.
- Data channel: sending, framing, retries, or loss occur at the transport boundary.
- Remote application: the peer validates target and
capability, then applies
REMOTE_DELTAto the collection.
Stage 1 can return local success while Stage 5 happens later or never happens on the online path.
Identity precedes payload correctness
Two collections that both store RouteHint cannot
automatically accept each other's deltas.
RuntimePlatformSyncService addresses a collection
route/locator and routes Register, Lease, or CRDT messages after
capability negotiation.
A propagation contract answers:
- Which logical service sent the message?
- What is the target collection locator?
- Does the payload belong to Register, Lease, or CRDT?
- Are schema and codec compatible?
- Is this an online delta, replay item, or snapshot chunk?
Ambiguity in any answer can turn "message delivered" into an incorrect claim that state was applied safely.
The boundary of online delta
Online delta optimizes low-latency propagation under normal conditions. It is not a durable business-event log.
| Condition | Possible online outcome | Follow-up responsibility |
|---|---|---|
| receiver temporarily offline | no real-time application | compare digest and repair after recovery |
| control message lost | difference remains undiscovered briefly | anti-entropy sweep detects it later |
| incompatible envelope | application rejected | record the reason and correct version/capability |
| duplicate arrival | idempotence or metadata arbitration needed | collection merge/resolver handles it |
| out-of-order arrival | old value cannot replace new | lineage, HLC, or collection semantics handle it |
A ChangeStream consumer therefore cannot infer complete history. The stream observes collection changes and does not provide message-queue audit semantics.
Counterexample and fault injection
Drop the data path from node-a to node-b
and perform a local put. A reads the new route, B retains
the old route, and local success remains valid. After network recovery,
repair closes the gap. An assertion against A proves only local
semantics. An assertion against B without event origin cannot
distinguish online delta from later repair.
Experiment
cd submodule/dsm
mvn -q -pl dsm-integration-test -am \
-Dtest=TwoNodeIntegrationTest,ChaosIntegrationTest \
-Dsurefire.failIfNoSpecifiedTests=false testInspect delta-trace.json and confirm that every stage
records input, output, an observable signal, and a continuation after
failure.
Experiment acceptance card
| Field | Content |
|---|---|
| Command | The focused Maven tests above;
node --test tests/chapter-assets.test.mjs |
| Input or fault | local update, remote node, missed online message, and recovery |
| Observable result | local success occurs first; the normal path reaches the peer; repair catches a missed update |
| Evidence level | E3: in-process multi-node and chaos integration tests |
| This experiment does not prove | Cross-host throughput, real-network retry limits, or exactly-once business events |
Review
- Local commit and remote visibility are separate completion points.
- Locator, collection type, and schema constrain the remote target together.
- Online delta favors low latency and does not guarantee complete history.
- Missed messages are an expected input; repair detects and closes gaps continuously.
Next
Chapter 12 lets a late node-b compare digests and select
replay or snapshot with an explainable decision.