Chapter 19 — Locate Replica Differences with Observable Signals
Chapter objective: After this chapter, a developer can combine diagnostics, low-cardinality metrics, and traces into falsifiable troubleshooting hypotheses instead of watching one "sync failed" counter.
Learning objectives
- Read identity, state, member view, and collection summaries from
RuntimeDiagnostics. - Use outcome/reason metrics to separate membership, propagation, repair, Lease, and consumer faults.
- Correlate one cross-node propagation with trace context.
- Avoid high-cardinality labels and unbounded diagnostic snapshots.
Prerequisites
- Repair and backpressure from Chapters 12–13 are complete.
- The evidence ladder from Chapter 18 is understood.
- Diagnostic data is an observation, not a new source of business truth.
Case progress
node-b has an older route hint than node-a.
Before restarting anything, the team checks Runtime state, member
visibility, locator registration, message-drop/repair outcomes, and a
trace for that propagation.
Three observation surfaces form one diagnostic loop
Diagnostics: current state
runtime.diagnostics() returns an immutable snapshot with
RuntimeInfo, lifecycle state, cluster view, total entries,
per-collection diagnostics, Lease-specific diagnostics, and last error.
It describes what this node sees now, not a time-series history.
Metrics: accumulating outcomes
DsmMetrics records sync latency, cluster size,
authentication failure, message drop, repair outcome, Lease operations,
fencing rejection, and ChangeStream overflow. Stable low-cardinality
reason/outcome labels support aggregation and alerts.
Trace: path of one action
TraceContextHolder puts current trace context into
outbound platform envelopes. Two-node integration verifies propagation
on a Register delta. A trace correlates one path; it does not replace
metric trends or a diagnostic snapshot.
An executable troubleshooting path
| Step | Question | Primary signal |
|---|---|---|
| 1 | Is Runtime ready? | state, isReady,
lastErrorMessage |
| 2 | Is the node in the right communication domain? | cluster view, service/cluster mismatch metric |
| 3 | Is the locator registered with a compatible schema? | collection diagnostics, message-drop reason |
| 4 | Did online delta arrive? | sync latency, REMOTE_DELTA, trace |
| 5 | Did repair detect and handle the gap? | digest/repair outcome, replay/snapshot reason |
| 6 | Is only the observer behind? | ChangeStream overflow/callback timeout |
Each step produces a falsifiable next hypothesis. Normal membership, rising delta drops, and later repair success support online loss rather than a bad locator.
High cardinality is another failure mode
Arbitrary entryKey, full exception text, or trace ID in
a metric tag creates time series proportional to business data.
DsmMetrics requires stable reason/outcome labels. Specific
keys and exception context belong in structured logs, audit, or
traces.
Counterexample and fault injection
- Overflow a ChangeStream and confirm current entries remain visible in diagnostics while the metric rises.
- Inject 50% control-plane loss and observe a later repair outcome.
- Put a Register under trace context and confirm the outbound envelope carries it.
- Add a random entry key as a tag in the exercise asset and observe cardinality review rejection.
Experiment
cd submodule/dsm
mvn -q -pl dsm-runtime,dsm-sync,dsm-metrics-micrometer,dsm-integration-test -am \
-Dtest=DefaultDsmRuntimeTest,MicrometerDsmMetricsTest,TraceContextHolderTest,RuntimePlatformSyncServiceTest,ChaosIntegrationTest \
-Dsurefire.failIfNoSpecifiedTests=false testUse diagnostic-runbook.json to record a snapshot,
metric, trace, and next hypothesis for four symptoms.
Experiment acceptance card
| Field | Content |
|---|---|
| Command | The focused Runtime/metrics/trace/chaos tests above plus book assets |
| Input or fault | stale replica, message loss, repair, overflow, and trace context |
| Observable result | current state, outcome trend, and single path corroborate without replacing one another |
| Evidence level | E2 observation implementations plus E3 controlled trace/chaos |
| This experiment does not prove | Metrics-backend capacity, alert thresholds, production sampling, or on-call effectiveness |
Review
- Diagnostics answers current state, metrics shows trends, and trace follows one path.
- Troubleshooting begins with identity, locator, and reason instead of restart.
- Metrics use low-cardinality outcomes/reasons; keys stay in logs and traces.
- Agreement among observation surfaces does not make a business fact correct.
Next
Chapter 20 routes every remote message through admission, authentication, decryption, replay prevention, and abuse control before sync/collection routing.