Architecture

Gossip for liveness, Raft for decisions, and a hard rule about which one answers what.

Regions, not one global cluster

At hundreds of nodes across datacenters, a single Raft group is the wrong tool: WAN latency lands in the write path, and one partitioned datacenter freezes everything. So each region gets its own Raft group and its own scheduler.

Workers are never in the Raft membership. That is what keeps hundreds of nodes viable: consensus cost stays fixed at three to five members no matter how large the fleet gets.

A partitioned region keeps serving; it just cannot schedule anything new. Its peers report it as unreachable rather than pretending it is healthy.

The invariant worth memorising

Three components could plausibly answer "is rk-07 up?". If all three are allowed to, they will disagree, and every incident review will open by arguing about which one was right. So, by fiat:

• SWIM is the only liveness authority. Nothing else decides up or down.
• The Raft roster is the only membership authority. Admission is a join token minted for a role; who votes is roster/control/<id>, written from an attestation made with the control key. A role or Raft address a node gossips about itself decides nothing.
• In the roster but not in SWIM means down. In SWIM but not in the control roster means a worker, whatever it claims to be.

Liveness must never be pushed through Raft. Routing heartbeats through consensus is the most reliable way to melt a Raft cluster.

What goes in Raft, what does not

The rule: consensus for decisions the cluster makes, eventual consistency for facts a node observes about itself. Anything where two divergent copies would cause a double-run, a lost write, or a port conflict belongs in Raft.

It is called the observed-state store, never the "config store". The word config invites people to put decisions in it, and every decision that lands there is a future outage.

The Raft side is built: leader election, replication, restart from disk, linearizable reads, transactions and leases. Consensus covers what it guarantees and where the sharp edges are.

The cluster bus

Every datagram is wrapped in an HMAC-SHA256 envelope carrying a version, a timestamp and a nonce, under the key for its traffic — gossip, fetch or control, which are held by different nodes (see Security). Five things make it hold up:

• Length-prefixed signing. Fixed-width fields plus a length-prefixed payload, so no choice of payload can be made to reproduce a different message's signed bytes.
• A nonce cache that is actually checked. A freshness window alone does not stop a replay inside the window — it only bounds how long the attack stays possible.
• A channel tag is signed. Gossip, Raft and artifact fetch are different conversations; with the tag inside the MAC, an envelope captured on one does not verify on another, whether or not they share a key today.
• The recipient is signed. An envelope addressed to node A and replayed to node B inside the window used to verify at B, whose nonce cache had never seen it. Now B refuses anything addressed to somebody else, and a reply is addressed to the nonce of the request it answers.
• Failure is per-datagram. A forged packet is counted and discarded; the receive loop keeps running. A loop that dies on one bad packet is a one-packet denial of service.

One byte in front of each payload demultiplexes SWIM traffic from application traffic. It sits inside the signed envelope, so it cannot be tampered with.

Crates