Chapter 10 — Member Views and Stability
Chapter objective: After this chapter, a developer can distinguish membership events, currently visible members, and a stable member view, and explain why a high-risk Lease cannot act on one observation.
Learning objectives
- Explain discovery, suspicion, departure, and recovery signals.
- Distinguish
activeMembers(),clusterSize(), and stable cluster size. - Explain how
serviceIdprevents unrelated logical clusters from discovering each other. - Protect QUORUM Lease decisions with a membership-stability window.
Prerequisites
- The two-node replication experiment in Chapter 4 is complete.
- Lease fencing and QUORUM mode from Chapter 7 are understood.
- A temporary lack of response is not treated as permanent departure.
Case progress
After node-b joins the fulfillment cluster,
node-a first sees a membership event and only later
observes the same member count over consecutive rounds. During this
unstable window, a worker tries to claim orders-17. The
team needs to choose the membership view used as the quorum
denominator.
One observation is not a global fact
ClusterMembership exposes both membership observations
and DSM control/data channels: self(),
activeMembers(), clusterSize(),
membershipRoundInterval(), and event/data listeners. These
calls report one node's current observation, not a linearizable cluster
roster.
| Signal | What it shows | What it does not show |
|---|---|---|
NodeJoined |
this node received join information | every node now has the same view |
NodeSuspected |
failure detection suspects a node | the node has permanently failed |
NodeLeft |
this node confirmed or received a departure | the other side of a partition agrees |
activeMembers() |
currently reachable members | the original cluster size has changed |
clusterSize() |
the implementation's cluster-size view | that size has stayed stable for several rounds |
The test double deliberately separates clusterSize()
from activeMembers(). During a partition, the original size
remains while visible membership shrinks. A QUORUM check therefore
cannot redefine one isolated node as the complete cluster and allow a
write.
The purpose of a stability window
MembershipStabilityTracker observes member count across
rounds. stableClusterSize() remains -1 until
the count reaches quorumStabilityRounds; any size change
resets the counter.
With three required stable rounds:
| Round | Observed size | Stable count | stableClusterSize() |
|---|---|---|---|
| 1 | 2 | 1 | -1 |
| 2 | 2 | 2 | -1 |
| 3 | 2 | 3 | 2 |
| 4 | 3 | 0 | -1 |
| 5–7 | 3 | 1–3 | eventually 3 |
A QUORUM Lease returns membership-unstable while stable
size is unknown and quorum-unavailable when a stable view
lacks a majority. This fail-closed behavior pauses new leases instead of
calculating a majority from a drifting denominator.
serviceId is
the first isolation boundary
Development, test, and production clusters may share one network.
Equal transport addresses do not make them one logical service.
Membership implementations isolate discovery and messages with
serviceId; nodes with different values stay out of each
other's membership lists.
serviceId is also unsuitable as a business tenant
identifier. It defines a cluster broadcast domain. Chapter 17 adds
tenant/application isolation at the collection-locator layer.
Counterexample and fault injection
Partition one node from the other two in a three-node cluster. Its
activeMembers() can contain only itself. If visible count
1 becomes total size, the majority threshold also becomes
1 and the isolated node incorrectly considers itself
writable. Retaining the cluster size or waiting for a stable view causes
the QUORUM Lease to reject the claim.
Experiment
Verify the membership failure model and the Lease gate:
cd submodule/dsm
mvn -q -pl dsm-test-support,dsm-runtime -am \
-Dtest=FakeClusterMembershipChaosTest,MembershipStabilityTrackerTest,InMemoryDsmLeaseRegisterTest \
-Dsurefire.failIfNoSpecifiedTests=false testClassify every item in membership-observations.json as
an event, visible membership, total size, or stable view. No single
event supports a claim that the cluster is globally consistent.
Experiment acceptance card
| Field | Content |
|---|---|
| Command | The focused Maven tests above;
node --test tests/chapter-assets.test.mjs |
| Input or fault | loss, delay, partition, suspicion, crash/recovery, and member-count change |
| Observable result | visible membership can shrink; stable size requires consecutive rounds; unstable QUORUM Lease rejects |
| Evidence level | E2: single-process membership and Lease behavior tests |
| This experiment does not prove | Cross-host network quality, multicast reachability, or suitable production failure thresholds |
Review
- A member view is a local observation, not an instantaneous global fact.
- Visible membership and original cluster size serve different purposes.
- High-risk quorum decisions wait for consecutive stable rounds.
serviceIdisolates logical clusters; it does not replace tenant isolation.
Next
With a stable membership view, Chapter 11 follows one
route-hints.put from local commit to remote visibility.