Chapter 4 — From Local Commit to Remote Visibility
Chapter objective: After completing this chapter, a developer can run a two-node replication experiment and use event origin to prove that remote visibility follows local commit.
Learning objectives
- Connect two memberships and register the same spec on both nodes.
- Track local put, delta, and remote apply.
- Verify eventual visibility with a condition instead of fixed
sleep. - State the evidence boundary of an in-process two-node test.
Prerequisites
- Completion of Chapters 2–3.
- Ability to distinguish locator, entry key, and schema.
Case progress
Gateway node A publishes fulfillment-primary. Node B
eventually reads the same address from its Register. Order facts remain
outside DSM.
From local commit to remote apply
Both sides need the same cluster/service communication domain,
collection locator, compatible schema, and sync routing. Ordering
matters: local put can return before the peer processes
REMOTE_DELTA.
registerA.put(new RouteHint("fulfillment-primary", metadata, "10.0.0.8:8080"));
await(() -> registerB.get("fulfillment-primary").isPresent(), 2_000);
assertEquals("10.0.0.8:8080",
registerB.get("fulfillment-primary").orElseThrow().address());The condition expresses the acceptance target. A fixed
Thread.sleep(500) can be too short and flaky or long enough
to hide abnormal replication latency.
Let event origin carry evidence
ChangeOrigin.REMOTE_DELTA distinguishes remote apply
from a local operation. A test that asserts both value and origin is
stronger than polling get alone. ChangeStream
is still not a durable event log and cannot preserve complete history
while a subscriber is offline.
Counterexample and fault injection
Register the collection only on node A, or give node B a different
schemaId. Connected membership alone does not route and
apply collection data. A missing route or incompatible schema should
surface explicitly; a timeout is not evidence that eventual consistency
is merely slow.
Experiment
Run the focused two-node test:
cd submodule/dsm
mvn -q -pl dsm-integration-test -am \
-Dtest=TwoNodeIntegrationTest#runtimeFirstRegisterDeltasReplicateAcrossTwoNodes \
-Dsurefire.failIfNoSpecifiedTests=false testThe test covers A→B put, A→B remove, B→A put, and remote event origin.
Experiment acceptance card
| Field | Content |
|---|---|
| Command | Focused TwoNodeIntegrationTest command above |
| Input or fault | Two fake-membership nodes, bidirectional writes, and removal |
| Observable result | The peer observes state within a 2-second condition window and
origin is REMOTE_DELTA |
| Evidence level | E3: controlled two-node protocol integration |
| Not proven | Cross-process networking, loss repair, production discovery, or linearizability |
Review
- Local commit and remote visibility occur at different times.
- Membership alone is insufficient; collection routing and schema also match before application.
- Eventual conditions should poll the target state instead of sleeping for a guessed duration.
REMOTE_DELTAproves online propagation origin, not complete event history.
Next
Chapter 5 develops the full Register contract for put, get, all, remove, and local visibility.