Appendix C — Troubleshooting Decision Tree

Preserve evidence first, then narrow the fault in the order identity → lifecycle → membership → route → delta → repair → observer. Restart changes evidence and is not the first diagnostic action.

Entry decisions

Symptom Inspect first Next hypothesis
process alive, requests fail state, isReady, lastErrorMessage Runtime unready or startup configuration failed
peer completely invisible cluster/service, member view, port communication domain, seed, or network mismatch
one collection differs locator, schema, collection diagnostics locator/fingerprint mismatch
occasional online loss message-drop reason, sync trace delta path loss; repair pending
difference remains after repair digest, replay window, snapshot result source choice, regression guard, or handler failure
Lease rejects mode, stability, quorum, token unstable view, missing quorum, or old token
subscriber misses events overflow/callback timeout observer lag rather than collection-state loss
remote message rejected admission/auth/decrypt/replay/rate limit gate configuration or hostile behavior
federation unhealthy target identity, relay/binding status missing remote locator or unavailable relay

Fast inspection order

  1. Record revision, node identity, time window, and exact locator.
  2. Capture diagnostics before clearing errors or restarting.
  3. Check low-cardinality operation/outcome/reason metrics.
  4. Trace one operation; keep trace ID in log/trace rather than metric tags.
  5. Review security audit without exposing token, key, or sensitive payload.
  6. Compare peer capability/schema before choosing replay, snapshot, or configuration repair.
  7. Record which business successes remain irreversible and which effects need compensation.

Common false greens

Minimal evidence record

revision:
environment/topology:
claim:
command/probe:
observed result:
does not prove:
next action / owner: