Chapter 12 — Digest-Based Difference Detection and Replica Repair

Chapter objective: After this chapter, a developer can detect differences with a digest, explain replay/snapshot selection, and reject a repair that would move the requester backward.

Learning objectives

1. Connect digest, anti-entropy, and repair.
2. Select replay or snapshot from retention and estimated cost.
3. Detect regression when the requester is newer or more complete at the same sequence.
4. Separate current-state repair from business-event replay.

Prerequisites

• The delta path from Chapter 11 is understood.
• Online messages can be lost while collections still need eventual convergence.
• Repair is a background process outside the local write transaction.

Case progress

node-b misses ten route-hint updates while offline. On return it reports its collection digest. The source decides whether retained deltas can close the gap or a current-state snapshot is more appropriate; B does not request every business event.

From difference to repair plan

A digest summarizes state so peers can compare observed sequence, entry count, or bucket hash. It carries no complete values and is not automatically authoritative; it supplies input to negotiation.

AdaptiveRepairPlanner follows four boundaries:

1. No difference produces NONE.
2. A requester that is ahead, or more complete at the same sequence, does not receive an older snapshot.
3. When the requester remains inside the source's replay window, estimated replay and snapshot costs are compared.
4. When replay is unavailable, snapshot restores current state.

Replay and snapshot differ in more than size

Adaptive choice uses latency, throughput, and history in a peer profile. Without a profile, the planner falls back to a fixed strategy, which is more predictable than reacting to one slow request.

Repair cannot move state backward

If requester sequence is 101 while the source has 100, a hash difference does not authorize the source to overwrite the requester. At an equal sequence, a requester with more entries also rejects a smaller source snapshot. AdaptiveRepairPlannerTest fixes both cases at RepairMode.NONE.

Anti-entropy is a continuing control loop

exchange digest → detect difference → plan repair → transfer → apply → compare again

Control messages can also be lost. ChaosIntegrationTest injects 50% control-plane loss and shows that later sweeps can still detect and repair the difference. Completion of one request only proves one operation; equal final digests establish the repair outcome.

Counterexample and fault injection

• Put requester sequence ahead of the source and observe regression rejection.
• Move the requester outside replay retention and observe snapshot selection.
• Supply a profile that estimates snapshot as faster and observe snapshot even while replay is possible.
• Drop the first control round and observe compensation by a later sweep.

Experiment

cd submodule/dsm
mvn -q -pl dsm-sync,dsm-integration-test -am \
  -Dtest=AdaptiveRepairPlannerTest,ChaosIntegrationTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

For each requester/source pair in repair-scenarios.json, select NONE, REPLAY, or SNAPSHOT and state the reason for avoiding regression.

Experiment acceptance card

Review

• Digest detects a difference; a repair plan decides how to close it.
• Available replay is not always cheaper, and a snapshot is not automatically authoritative.
• An ahead or more complete requester is protected from regression.
• Anti-entropy repeats; current-state convergence does not replay business history.

Next

Chapter 13 turns to observers: a slow subscriber can overflow the local change buffer even when collection replicas converge.

Evidence links

• AdaptiveRepairPlanner
• AdaptiveRepairPlannerTest
• CollectionDigest
• ChaosIntegrationTest