soli-one

The trust model is split, and the change is not compatible with clusters bootstrapped before it. The single shared secret, which let every worker sign Raft traffic and open every operator secret, is now a keyring with separate gossip, fetch and control keys, plus a per-worker node key; federation has a key of its own, outside the keyring. Join tokens (soli2) carry the role they admit — one token --role control|worker — and a worker is given nothing that votes or unseals. The agent's role is read from its keyring, control nodes are enrolled into a roster in Raft rather than believed from gossip, and proxy API keys moved to a proxy-keys file. Security has the key table and the reasons.

An existing cluster cannot acquire the new keys in place: the old secret and old tokens are refused with a message saying so. It has to be re-bootstrapped — one bootstrap on a fresh founder, one token --role … and one join for every other node, workloads and secrets redeclared. Take a one backup first.

Artifact replication is driven now. This note used to name it as a decision nobody invoked, so a rack going down cost copies in silence. Every twenty seconds the node holding the scheduler lease asks each peer which artifacts it holds, places copies across racks, and writes the answer; every node then fetches what it should hold and releases what it should not.

Verified with three agents in three racks: an artifact ingested on one node was on all three shortly after, with nothing telling the other two to copy it, and killing a rack produced “wanted 3 copies, placed 2” rather than silence.

Idle workloads go to sleep, and a request wakes them. This is what makes unlimited preview environments affordable: a Soli deployment is a long-lived process that holds its memory around the clock, and previews are mostly not being looked at.

Suspension is desired state — the reconcile loop is told the workload should be asleep, rather than being fought by a second thing that stops it — and the mode is stop, not freeze, because freezing holds the cgroup and every byte in it. It is off unless a node is given --idle-suspend-after-min: taking running workloads down is not a behaviour to acquire by being upgraded.

The signal is a per-app last-request time the proxy now records. Three things it deliberately does not do: it never sleeps a workload the proxy said nothing about (an unreachable proxy would otherwise put a whole fleet to bed), never one that started within the grace period (redeploying something idle would otherwise send it straight back to sleep under the person who deployed it), and never one whose timestamp is in the future (clock skew keeps it awake).

Verified against a real agent and a proxy under test control: the workload came up, went to sleep with its unit stopped, stayed asleep across passes without flapping, and came back when the timestamp went fresh.

The WAN gossip pool carries region summaries, so cross-region routing has data to decide with. It is a second socket and a much slower timer than LAN gossip — per-node facts between regions is a transit bill for information nobody reads at that resolution — and what may cross is an allowlist, because the failure mode of a denylist is a new message type crossing by default until the invoice arrives.

Only control nodes join, and only those given the federation's WAN key (--wan-key-file or the wan-key credential), which no worker holds. Frames are believed only from configured --wan-peer region=addr peers, and each peer speaks for one region. The role in the frame is checked too, and is worth least — it is the sender's own claim, so it catches a misconfigured node rather than an attacker.

A summary carries a degraded reason, and it is set from whether the region has a Raft leader rather than from whether it answers. “Healthy” and “serving but cannot schedule” are identical in capacity numbers, and the difference is the whole story. one regions shows it, and one regions --route <name> asks where a request would go.

Verified with two regions on one workstation: each learned about the other, one reported itself degraded because it had no leader, and routing answered local, no-leader and unknown as three distinct things rather than one error. A worker was refused the pool outright.