Chapter 18 — Scope of Layered Test Evidence

Chapter objective: After this chapter, a developer can select evidence for a domain port, real Runtime, multi-node protocol, or network black box and refuse to substitute a lower-layer green result for higher-layer acceptance.

Learning objectives

1. Build an E1–E4 evidence ladder.
2. Record environment, pinned revision, proven claims, and open claims at each layer.
3. Distinguish real collections, controlled fake membership, and real network ports.
4. Define a minimal faithful validation set for all three collection types.

Prerequisites

• The Chapter 15 example and Chapters 16–17 assembly/isolation are complete.
• The Chapter 14 chaos test has run.
• Test count is not treated as a production-readiness score.

Case progress

The fulfillment team has domain, single-Runtime, and chaos tests. Release review asks whether these results prove real ports, client protocols, and process isolation. The answer is stated one evidence layer at a time.

Evidence strength describes scope, not points

E4 is closer to deployment reality than E3 and still leaves topology, capacity, access, alerts, and operations to production acceptance.

Minimal evidence by collection

Register

• E1: the route hint is rebuildable guidance, not an order fact.
• E2: put/get/remove, resolver, and ChangeStream overflow.
• E3: two-node delta, tombstone, schema mismatch, and repair.
• E4: Chapter 23's Redis-shaped lab observes replication through a real TCP port.

Lease

• E1: the domain result retains uncertain, reason, and fencing token.
• E2: a guard rejects the old token after acquire/transfer.
• E3: autonomous dual holder and quorum fail-closed behavior.
• E4: the actual protected resource validates the token; this book defines the boundary without claiming a ready-made business system.

CRDT

• E1: counts support operations, not billing authority.
• E2: single-Runtime merge and algebra cases.
• E3: multi-node local updates, duplicate/reordered delivery, and repair convergence.
• E4: exposure through a client protocol depends on the application and is not proven by a CRDT unit test.

Every evidence card fixes four facts

1. Environment and revision: source, JDK, and Maven context.
2. Input/fault: the actual injected condition.
3. Observation: test counts, failures, errors, skips, and key behavior.
4. Open claim: the question left to the next layer.

Skipped and Tests run: 0 are not passes; compilation alone is not behavior evidence.

Counterexample and fault injection

• Select a nonexistent Maven test while disabling failIfNoSpecifiedTests and observe a false green with zero tests.
• Use an E1 fake result to claim network repair and name the missing execution surface.
• Use controlled E3 multi-node results to claim real TCP compatibility and identify the process/port gap.
• Record "all passed" without a revision and observe the loss of reproducibility.

Experiment

Run book-owned E1/E2:

cd examples/order-fulfillment-control-plane
mvn clean test

Run pinned-source E2/E3:

cd submodule/dsm
mvn -q -pl dsm-integration-test -am \
  -Dtest=RuntimeIntegrationTest,TwoNodeIntegrationTest,ChaosIntegrationTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

Use evidence-ladder.json to confirm that every level contains nonempty proves and doesNotProve fields.

Experiment acceptance card

Review

• Test level describes execution scope rather than a quality score.
• Every layer states both proven and open claims.
• A fake, one Runtime, multi-node chaos, and a real port do not replace one another.
• Revision and fault input precede any green conclusion.

Next

Arc 5 turns to runtime responsibility. Chapter 19 adds diagnostics, metrics, and trace surfaces for locating distributed differences.

Evidence links

• DSM examples: Spring Boot application test
• RuntimeIntegrationTest
• TwoNodeIntegrationTest
• ChaosIntegrationTest