Stateful clusters

one datastore plan prints the steps and one datastore apply carries out the ones belonging to the machine it runs on. A three-member SoliDB cluster was brought up from nothing on one workstation with it: the seed reporting one, then two, then three healthy nodes, each confirmed before the next started, with the keyfile arriving on tmpfs at mode 0400 and the memory ceiling reaching every cgroup.

SoliKV shards now too. Slot ownership propagates, and the plan assigns an even split as its last step — verified against two real nodes: a half each, both agreeing on the split, and a redis-cluster client writing eight keys through either node and reading them all back through the other, with the data genuinely split four and four.

SoliES clusters now too, through the same two commands. Its log replication was always there — this page claimed otherwise and was wrong; see the correction below. What was missing was the configuration surface to reach it, and a start shape for a service that votes.

Nothing on this page is refused any more. The two refusals that remain are about the shape of a request — fewer machines than members, and no keyfile — not about a service being unable to do what it claims.

apply is per-machine, and deliberately so for a database. A SoliDB member is pinned: it owns a local data directory, and a scheduler that moved it would move a name and leave the bytes. So each machine carries out its own steps and the command for the others is printed rather than guessed at. The scheduler places mobile workloads across the cluster; it is not the right tool for this one.

SoliES replicates its log. This page said it did not, and that was wrong. The claim — that AppendEntries carried an always-empty entries list and that try_propose had no caller — came from a doc comment at the top of messages.rs that still described an earlier step. The code had moved on: try_propose is called from the node loop, and build_append_for slices real entries out of the log.

Measured against two real broker processes, because reading more code was not going to settle a claim that came from reading code: the follower refuses a write and names the leader, twenty-one records written through the leader arrive on the follower in the same order, and with the follower killed the leader stops committing — a majority of two cannot be one. That last check is the one that separates consensus from two logs that happen to agree.

What genuinely did not exist was any way to tell a broker it was a cluster member. Three config fields for the node id, bind address and peers were declared and never read; resolve_raft handed back an empty address map; and connect_raft_transports had no caller. Its port scheme was raft_bind + partition_index, which works for exactly one topic and silently collides for the second — and a port per Raft group cannot be configured in advance, because topics are created at runtime. So the groups are multiplexed now: one listener, one connection per peer, and a group name in every frame.

es-broker --data-dir /var/lib/solies \
  --raft-node-id 1 --raft-bind 10.0.0.1:9300 \
  --raft-peer 2=10.0.0.2:9300 --raft-peer 3=10.0.0.3:9300 \
  --raft-shared-secret "$SECRET"

The membership is this node plus exactly the peers that have an address. There is deliberately no way to name a member without giving its address: a member that counts toward a quorum but cannot be reached is a vote the node will never collect, and the cluster would stall with every process reporting healthy.

A voting service gets its own start shape. Every other plan here starts a seed alone and points each later member at what is already running. That is right for a service which replicates without voting, and backwards for Raft: a node started alone is a cluster of one that elects itself, and a peer named in a later process does not join it. So the plan starts all members with the whole membership, and confirms each of them only after the last one is up — a quorum cannot form before then, so confirming between starts would wait for something that cannot happen.

start     es-1 on rk-01 as raft id 1, with all 3 members
start     es-2 on rk-02 as raft id 2, with all 3 members
start     es-3 on rk-03 as raft id 3, with all 3 members
confirm   es-1 is replicating, not merely running
confirm   es-2 is replicating, not merely running
confirm   es-3 is replicating, not merely running
publish   es-prod at 10.0.0.1:9200

The raft_id is written in the spec, not derived from a member's position in the file. A Raft node id is durable identity: taken from the ordering, sorting the members alphabetically would renumber a live cluster, and two nodes swapping identities is the worst thing that can happen to a replicated log. The planner refuses a missing id, a duplicate one, id 0 — the “no leader known” sentinel — and two members sharing a consensus address.

Verified against three real brokers started from the plan's own declared command lines: a record written through the leader arrives on a follower, killing one member of three does not stop writes, and killing two does.

confirm now confirms, for SoliES. It could not before, because nothing outside a broker could tell a replicating member from one that merely runs — /healthz answers a static “healthy”, /readyz a static “ready”, and /metrics says nothing about consensus. So SoliES grew a GET /raft, and two things about it were wrong until three brokers were run rather than the code read again:

• Every follower reported replicating: false. A follower keeps no match_index — that is leader state — so demanding a quorum from every role called a healthy cluster broken on two nodes out of three. A leader is replicating when a majority of live members have caught up; a follower, when it knows who leads, because that is exactly what it can attest to.
• A leader that had lost two of three members still reported a quorum of three. match_index records what a peer once acknowledged and goes on saying so after the peer dies. A quorum is a statement about liveness, not history.

When the spec names no admin token the step is reported as not performed, with the curl that performs it. Reporting “confirmed” for a check that was skipped is the one outcome worse than not checking. And the token is per member, because SoliES keeps its key store in each broker's own data directory and generates key secrets server-side — one token cannot be valid on three brokers, which was found by watching one confirm the local member and collect two 401s.

A SoliES topic is local to the broker that created it. Until the same topic exists on every member, that topic's partitions cannot reach a majority. It fails closed — the write is refused and says why — so the plan says it as a note rather than refusing the cluster. An operator should be told, not left to find out from a timeout.