ARC 5Own Runtime OperationsChapters 19–22

Chapter 19 — Locate Replica Differences with Observable Signals

Chapter objective: After this chapter, a developer can combine diagnostics, low-cardinality metrics, and traces into falsifiable troubleshooting hypotheses instead of watching one "sync failed" counter.

Learning objectives

  1. Read identity, state, member view, and collection summaries from RuntimeDiagnostics.
  2. Use outcome/reason metrics to separate membership, propagation, repair, Lease, and consumer faults.
  3. Correlate one cross-node propagation with trace context.
  4. Avoid high-cardinality labels and unbounded diagnostic snapshots.

Prerequisites

  • Repair and backpressure from Chapters 12–13 are complete.
  • The evidence ladder from Chapter 18 is understood.
  • Diagnostic data is an observation, not a new source of business truth.

Case progress

node-b has an older route hint than node-a. Before restarting anything, the team checks Runtime state, member visibility, locator registration, message-drop/repair outcomes, and a trace for that propagation.

Three observation surfaces form one diagnostic loop

Diagnostic loop from symptom to falsifiable hypothesis

Diagnostics: current state

runtime.diagnostics() returns an immutable snapshot with RuntimeInfo, lifecycle state, cluster view, total entries, per-collection diagnostics, Lease-specific diagnostics, and last error. It describes what this node sees now, not a time-series history.

Metrics: accumulating outcomes

DsmMetrics records sync latency, cluster size, authentication failure, message drop, repair outcome, Lease operations, fencing rejection, and ChangeStream overflow. Stable low-cardinality reason/outcome labels support aggregation and alerts.

Trace: path of one action

TraceContextHolder puts current trace context into outbound platform envelopes. Two-node integration verifies propagation on a Register delta. A trace correlates one path; it does not replace metric trends or a diagnostic snapshot.

An executable troubleshooting path

Step Question Primary signal
1 Is Runtime ready? state, isReady, lastErrorMessage
2 Is the node in the right communication domain? cluster view, service/cluster mismatch metric
3 Is the locator registered with a compatible schema? collection diagnostics, message-drop reason
4 Did online delta arrive? sync latency, REMOTE_DELTA, trace
5 Did repair detect and handle the gap? digest/repair outcome, replay/snapshot reason
6 Is only the observer behind? ChangeStream overflow/callback timeout

Each step produces a falsifiable next hypothesis. Normal membership, rising delta drops, and later repair success support online loss rather than a bad locator.

High cardinality is another failure mode

Arbitrary entryKey, full exception text, or trace ID in a metric tag creates time series proportional to business data. DsmMetrics requires stable reason/outcome labels. Specific keys and exception context belong in structured logs, audit, or traces.

Counterexample and fault injection

  • Overflow a ChangeStream and confirm current entries remain visible in diagnostics while the metric rises.
  • Inject 50% control-plane loss and observe a later repair outcome.
  • Put a Register under trace context and confirm the outbound envelope carries it.
  • Add a random entry key as a tag in the exercise asset and observe cardinality review rejection.

Experiment

cd submodule/dsm
mvn -q -pl dsm-runtime,dsm-sync,dsm-metrics-micrometer,dsm-integration-test -am \
  -Dtest=DefaultDsmRuntimeTest,MicrometerDsmMetricsTest,TraceContextHolderTest,RuntimePlatformSyncServiceTest,ChaosIntegrationTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

Use diagnostic-runbook.json to record a snapshot, metric, trace, and next hypothesis for four symptoms.

Experiment acceptance card

Field Content
Command The focused Runtime/metrics/trace/chaos tests above plus book assets
Input or fault stale replica, message loss, repair, overflow, and trace context
Observable result current state, outcome trend, and single path corroborate without replacing one another
Evidence level E2 observation implementations plus E3 controlled trace/chaos
This experiment does not prove Metrics-backend capacity, alert thresholds, production sampling, or on-call effectiveness

Review

  • Diagnostics answers current state, metrics shows trends, and trace follows one path.
  • Troubleshooting begins with identity, locator, and reason instead of restart.
  • Metrics use low-cardinality outcomes/reasons; keys stay in logs and traces.
  • Agreement among observation surfaces does not make a business fact correct.

Next

Chapter 20 routes every remote message through admission, authentication, decryption, replay prevention, and abuse control before sync/collection routing.