Consensus

One Raft group per region, holding only the facts whose divergence would break something.

What is allowed in

Every fact kept here costs a quorum round trip to change, so the bar is specific rather than general: would two divergent copies cause a double-run, a lost write, or a port conflict? If yes, it goes in Raft. If a stale answer is merely unhelpful — load average, free memory, which node holds which artifact — it goes in the observed-state store, which is eventually consistent and enormously cheaper.

The one rule that makes replication mean anything

Applying a log entry must be a pure function of (state, entry). Not "mostly pure" — a single SystemTime::now() inside the state machine and two replicas drift apart, silently, for as long as nobody compares them.

Which is awkward, because TTLs and lease expiry are the whole point of the store and both are about time. The resolution: the proposer stamps the clock once, into the entry, and every replica applies that stamp. A replica replaying the log at 3am reaches the state the leader reached at noon.

pub struct Command {
    pub at_ms: u64,   // stamped by the proposer, once
    pub op: Op,
}

With one clamp on top: the clock only ever moves forward. A leader whose clock jumped backwards — an NTP step, a VM restored from a snapshot, a handover between machines with skew — would otherwise un-expire a lease a previous leader had already let go, and the cluster would run two schedulers. That case is a test, not a comment.

And because a rule like this decays the moment nobody is watching, the test suite reads kv.rs as text and fails if it so much as mentions SystemTime, Instant::now or a random number outside its own tests. There is no runtime assertion that can catch this class of bug; the compiler will not either.

Layering, and why the seams are where they are

openraft has no 1.0 — 0.9 is the stable line and 0.10 is still in alpha. So the parts that encode our correctness are kept out of reach of it: Kv is testable with no Raft and no network, Journal with no Raft, and nothing outside this crate ever names an openraft type. A migration is then one crate's work rather than a sweep through the scheduler, the agent and the CLI.

The journal

An append-only file of length-prefixed records — appends, votes, truncations, purges — replayed into memory at open.

• Append, never rewrite. Raft's guarantee is that an acknowledged entry is on stable storage. A design that rewrites the whole file has a window where the old copy is gone and the new one is not yet fsynced; a crash there loses decisions the cluster promised to keep. Appending has no such window.
• The vote lives in the same file. openraft requires all storage writes to be serialised — saving a vote must not overtake an append. Two files would need an ordering discipline nothing enforces. One file gets it for free: the order records appear in is the order they happened.
• A torn tail costs exactly the unacknowledged record. Replay stops at the first record it cannot read whole and truncates there. Safe by definition: a partial record is one whose append never returned, so no leader ever counted it. Tested by writing a length prefix with no body behind it and reopening.
• Compaction is the one rewrite, and it writes to a sibling and renames — the original survives a crash mid-compaction. Its test reopens from scratch rather than trusting the in-memory copy, because that is the failure mode it exists to catch.

Reads: two of them, named differently on purpose

A single get that silently degraded to the local view would be the convenient choice and a very expensive one: the stale answer is right almost always, and the time it is wrong is the time two nodes start the same singleton.

The same split applies to leadership. is_leader() confirms with a quorum; believes_self_leader() is this node's opinion. A deposed leader still believes it leads until it hears otherwise — fine for a status line, wrong before doing something only one node may do.

Leases

A named lease is how the cluster picks one of anything — one scheduler, one ACME orderer, one VIP holder — without a second election protocol. Taking it is an ordinary log entry, so the single-winner property comes from Raft rather than from a lock.

rk-01 → acquire("scheduler", ttl 60s) → granted
rk-02 → acquire("scheduler", ttl 60s) → refused, held by rk-01 until …
rk-03 → acquire("scheduler", ttl 60s) → refused, held by rk-01 until …

A refusal names the holder. Blind backoff would make a wedged holder indistinguishable from a busy cluster, and that is an hour of debugging every time.

Only the holder may release. A node returning from a partition still believing it is the scheduler would otherwise drop its successor's lease — and then two schedulers run, which is the exact failure the lease exists to prevent.

Renew well inside the TTL. A holder that renews at the deadline has already lost it once the write's quorum round trip is counted.

Guards, and why they are on versions

Repointing an alias is the rollback primitive, so it must not half-apply and must not apply at all if the alias moved underneath. Both come from putting the precondition in the entry:

txn(
  guards:     [VersionIs("alias/x.soli.app", observed)],
  on_success: [Put("alias/x.soli.app", "dpl_B"),
               Put("alias/x.soli.app/previous", "dpl_A")],
  on_failure: [],
)

On a value guard this is subtly broken. Between the read and the write the key can go A → B → A; a value guard sees nothing happened and applies over two intervening writes. Versions are globally monotonic and never reused, so a version guard sees them. The pair of tests that pins this asserts the value guard succeeds — because that is the bug, and a test that only checked the fix would not say why the fix exists.

Watching

Every member emits a stream of the changes it applies — followers included, and that is the point. Hundreds of workers subscribing to the leader would make it the bottleneck; they watch a nearby control node instead.

Expiry is an event, not an absence. Expired and Deleted are distinct: "the scheduler released this" and "the scheduler stopped renewing it" call for different reactions, and collapsing them costs an incident. It is also why a leader ticks on a timer — on a quiet cluster, nothing else would advance the clock, and lapsed state would stay visible until the next unrelated write.

Joining

A new member arrives as a learner: it receives the log but does not vote. Promoting straight to voter grows the quorum before the newcomer has caught up — a cluster of three adding a fourth would need 3 of 4 while the fourth is still replaying.

The transport

Raft RPC goes through the same HMAC envelope as gossip and artifact transfer, under a key of its own: the control key, which only nodes admitted as control hold. It has to: an unauthenticated AppendEntries rewrites the cluster's history, and before the keys were split every worker could produce one this listener accepted.

• A request is addressed to the target's Raft id, so one captured on its way to a member does not verify at another.
• A reply is addressed to the nonce of the request it answers. It used to travel bare, so anything on the path could answer a vote request with “granted”; now a reply verifies for one request only, which also makes a replayed reply useless.
• Both bodies are encrypted, AES-256-GCM under a key derived from the control key. Secrets are sealed at rest regardless; the workload specs, allocations and alias table were readable by anyone on the path between two control nodes.
• A stranger gets bounded work. The port faces the same network as everything else, so connections are capped, the first frame must authenticate within a deadline that starts at accept, and a declared length is checked before it becomes an allocation.

Who is added to the group at all is the control roster, roster/control/<id>, written from an attestation made with the control key — never from what a node gossips about its own role. See Adding and removing nodes.

A connection is dropped on any error rather than reused. A stream that failed mid-frame is out of sync with its peer, and continuing on it pairs a response with the wrong request — which surfaces as a leader accepting a vote reply it never asked for.

Everything a peer can do wrong maps to unreachable, not retry now. A peer that answers garbage or refuses our HMAC will not start behaving on an immediate retry, and treating it as retryable turns one misconfigured node into a hot loop against it.

All peer I/O goes through a Dial trait rather than TcpStream::connect. That costs four lines today and is what makes a deterministic simulation of a partitioned, clock-skewed cluster possible later. Retrofitting it after the fact is not a refactor, it is a rewrite.

Tuning

An election timeout under roughly five heartbeats makes a node call an election every time a GC pause lands on a heartbeat, and the cluster spends its life electing instead of working.

Restoring a group from a backup