Security

There is no private network. Everything below follows from that.

built, nothing drives it yet

The peer-allowlist policy and its audit are implemented and tested in soli-one-bootstrap. Nothing installs the generated rules yet, so a deployment today still has whatever firewall was configured by hand.

The shipped firewalld service cannot be followed

packaging/firewalld/soli-one.xml opens 7300–7304 and says, correctly:

Open this to the cluster's own network, never to the internet.

On OVH VPS there is no cluster network. vRack is offered for dedicated servers, Hosted Private Cloud and Public Cloud instances; the VPS range is not on that list. Every VPS has a public address and nothing else, so gossip, Raft and artifact transfer all cross the public internet.

The advice is therefore unfollowable, and the obvious next move — firewall-cmd --add-service=soli-one — opens a consensus implementation to everyone. The HMAC envelope means an outsider cannot inject, but an exposed gossip port hands out the cluster's membership from its traffic pattern alone.

A source allowlist, derived from the roster

Rules are generated from the membership, not maintained beside it. A firewall maintained separately is a firewall that disagrees with the roster, and the direction it disagrees in is usually “a node that was removed can still connect”.

# Generated by soli-one from the cluster roster. Do not edit.
add rule inet filter input tcp dport 80 accept comment "http — ACME HTTP-01"
add rule inet filter input tcp dport 443 accept comment "https"
add rule inet filter input ip saddr 203.0.113.8 tcp dport 7302 accept comment "raft"

A cluster port with an Any source is a bug, not a configuration. The audit reports it as one.
The public ports stay public. 443 is the product, and 80 must answer Let's Encrypt from addresses it does not publish — so it cannot be allowlisted either.
A joiner is allowed on the join path only. Gossip and the cluster API, not Raft and not artifact transfer. The alternative — opening the seed port to the world for the duration — is the easy answer and the wrong one.

SoliDB used to fail open

works today

Fixed, with tests. Described here because the shape of the bug is worth remembering.

db/src/cluster/transport.rs signed cluster messages when a secret was configured. With no secret, both halves fell through:

_ => Ok(payload.into_bytes()),          // sent unsigned
_ => Ok(serde_json::from_slice(data)?), // accepted unverified

So a cluster started without a keyfile had no authentication on its replication bus, and looked identical to one that did. The asymmetry was the dangerous part: a node with a secret rejects unsigned messages, while a node without one accepts both — one misconfigured member is an open door into the replicated state, invisible from every other node. A forged Leave evicts a node; a forged JoinRequest adds one.

Both arms now refuse, and the error names the setting to fix. The rule is the same one the workload credential probe follows: less isolation still works, no secret does not.

Which node holds which key

breaking change — clusters from before it are re-bootstrapped

A cluster used to share one secret. It authenticated gossip, Raft, artifact fetch and the WAN pool, and the seal on operator secrets was derived from it — so every worker could sign an AppendEntries, open every one secret, and speak for its region to other regions. Joining as a worker was joining as everything.

/etc/soli-one/secret is now a keyring: a header line, then one key per line. one bootstrap writes the founder's, which holds all of them; one join writes the one the joiner's token was minted for.

soli-one keyring v1
name rk-02
gossip 3f…
fetch 91…
node 5c…

Key	Held by	Authenticates
`gossip`	every member	SWIM and the facts broadcast, 7300
`fetch`	every member	artifact transfer, 7303 — workers pull replicas too
`control`	control nodes	Raft, 7302; the seal on operator secrets; the attestation that puts a node on the control roster
`node`	one worker, with the `name` it was admitted under	the secrets delivered to that worker, and nothing else

A worker's node key is derived from the control key and its admitted name. Any control node can compute it for any worker, so nothing per-node is replicated; the worker receives its own at the join and cannot compute another's. The name is bound too: an agent started with a --name other than the one in its keyring refuses to start, because it would be sent secrets it could not open.

The federation key is not in the keyring. Regions are separate clusters with separate keyrings, so a key derived from one cluster's control key would have to be copied into the others'. It is configured per federation, on the control nodes that take part — see below.

There is no upgrade in place. A file holding the old single secret is refused at start, by name, with a message that says to re-bootstrap. Splitting it after the fact would leave every worker holding what is now the control key under another label, which is the finding this exists to fix.

Roles are decided by the token, and kept in Raft

one token --role control|worker, default worker — the role that hands over least. The seed admits from its own record of the token, never from what the joiner says. Tokens are soli2_…; a soli1 token, which carried no role, fails as old rather than as corrupt.
Only a control node admits. Both answers need the control key — a control joiner is handed it, a worker a node key derived from it — so a node without it serves no joins. Only a control joiner is given a TLS identity of its own.
The agent's role follows its keyring. --role is optional. --role control without the control key is refused; --role worker on a node that holds it runs, with a warning, since the key is on its disk whatever the role.
The control roster lives in Raft, roster/control/<id>. Every member holds the gossip key, so a gossiped role or Raft address is the sender's claim; a worker that wrote Control into it used to be added as a learner and promoted to voter. Now a node reaches the roster only by gossiping an attestation made with the control key over its id, name and Raft address, and the leader adds only rostered nodes. On a control node, a node not on the roster is shown as a worker with no Raft address, whatever it claims.
one node remove leaves a tombstone, roster/removed/<id>, so the attestation the removed node can keep broadcasting does not enrol it again. A removed machine rejoins under a fresh node id.

Operator secrets on a worker

A worker cannot unseal anything from the Raft log, so its secrets are pushed: the node holding the scheduler lease finds the allocation placed on that worker, re-seals the value to the worker's node key, and sends one gossip frame addressed to it. The node, tenant, job, secret name and issue time are bound in as associated data, and a delivery is good for 60 seconds. A worker claiming another's name in gossip is sent that node's deliveries and cannot open them; a replayed one fails the issue-time check. Pushed rather than requested, so no endpoint answers “give me secret X” to whoever asks.

wired, not yet exercised

Workers do not yet run cluster-placed jobs, so this path carries nothing until they do. A file-declared job on a worker that names a secret fails to start, with an error naming the secret — on purpose: a workload that comes up healthy and secretless is the failure nobody notices.

Raft, sealed both ways

Raft on 7302 is keyed by the control key, so a worker cannot even follow the log. A request is addressed to its target; a reply is addressed to the nonce of the request it answers — it used to travel bare, and anything on the path could answer a vote request with “granted”. Both bodies are encrypted with AES-256-GCM under a key derived from the control key: the log carries workload specs, allocations and the alias table, which were readable by anyone on the path between two control nodes. The listener bounds its connections, checks a declared frame length before allocating it, and gives the first frame a deadline that starts at accept.

Underneath all of it, the envelope (v2) signs a channel tag and a recipient. The tag stops an envelope captured on one conversation — gossip, Raft, fetch — from verifying on another; the recipient stops one addressed to node A from verifying at node B inside the replay window, where B's nonce cache has never seen it.

Federation

The WAN pool on 7301 has its own key: 32 random bytes, hex, shared by the control nodes of every region in the federation and by nothing else. It comes from --wan-key-file or the systemd credential wan-key ($CREDENTIALS_DIRECTORY/wan-key); without it, a control node takes no part in federation. Peers are --wan-peer region=addr, frames are believed only from configured peers, each peer speaks for one region and cannot overwrite another, and the directory holds at most 256 regions.

Where secrets live

Secret	Where	Why not elsewhere
Cluster CA private key	Local disk, `0400`, root	Raft is replicated to every control node and copied into every snapshot. A key placed there is in every backup forever
Certificate private keys	Raft, sealed with AES-256-GCM	They have to reach every node to terminate TLS. The key name is bound in as associated data, so a bundle moved onto another domain's key fails to open
Workload secrets	Raft, sealed with the control key; on a worker, pushed sealed to its node key; then a systemd credential, tmpfs, `0400`	Never the environment: `/proc/<pid>/environ` and `podman inspect` both read it. Never the artifact: every backup of it would contain them. The one plaintext hop, the file `LoadCredential=` reads, is under `/run` — a tmpfs, so it never reaches a disk or a filesystem backup
The backup bundle	Sealed to an operator key, off the cluster	It contains the keyring, so sealing it with a key from that keyring would be a lock with its key inside the box — and the one situation a bundle exists for is the one where the cluster it came from is gone
The package signing key	CI secret; never on a node	Nodes verify with the public half (`gpgcheck=1`) and metadata signatures (`repo_gpgcheck=1`). A node that could sign could serve itself a package
Join tokens	Cluster stores a hash	A leaked cluster state must not yield a usable token

Sealing protects material at rest — a leaked snapshot, a copied data directory, an old backup. Inside the cluster the boundary is the key split, not the seal: every control node can open every operator secret, because the scheduler may place a workload on any of them, and a worker can open exactly the secrets delivered to it. Anyone with the agent socket on a control node can still reach the plaintext through the workloads it declares. Saying the seal did more would be the more comfortable claim and the false one.

One construction, in soli-one-seal, for all of it: AES-256-GCM with a key derived by HKDF-SHA256. Three implementations of one AEAD would be three chances to get a nonce wrong. Each purpose — certificates, operator secrets, the backup bundle — binds its own HKDF domain, and every domain in the system is listed in one file, so two features sharing a separator by accident is visible there rather than discovered when a secret turns out to open as a certificate key.

The agent socket is the authorisation boundary

an audit finding, and three that came with it

Anything that can connect to /run/soli-one/agent.sock can declare a unit, read the sealed cluster state, and run a command with one exec. So the socket's mode is the whole boundary — and it was whatever the umask happened to leave. Under systemd that is 0755, which keeps group and world out; one UMask= line in the unit, or an agent started from a shell with umask 002, and every tenant workload would have had it. The directory above it granted traversal to the soli-one group, which is the account workloads run as. A safe state held by a default is not a decision.

Now: the socket is chmod 0600 explicitly after bind, and /run/soli-one is 0700 root with no group — nothing in that group has business there. one exec carries the job's User= and Group=, so an operator debugging tenant A runs their command as A, not as root inside A's cgroup; there is a system-bus test for it. And a node's self-declared name is checked at both ends of the wire before it can become an ssh destination — see below.

A node name is an ssh destination

demonstrated, then closed on both ends

one logs --all and one node upgrade reach nodes by name over SSH, from the operator's own machine. ssh parses its arguments with getopt, so a destination that begins with - is an option — and -oProxyCommand=… runs a command on the machine invoking it. A member gossips its own name, and nothing validated it on receipt. That was a path from a compromised node to code execution on an operator's laptop: the one direction of trust the rest of the design keeps shut. Demonstrated on a workstation before it was fixed.

Closed twice, independently. The CLI refuses any destination that is not a plain [user@]host before ssh sees it, and passes -- so even a name that slipped through is a hostname to ssh and not an option. And the registry drops a peer whose name is not a hostname at the first node that hears it, so no node stores or forwards it. Either alone is enough; both exist because the failure is on the far side of a trust boundary.

SELinux

a bug this caught, late

The agent itself needs no policy: a binary in /usr/bin is bin_t, and the targeted policy transitions init_t + bin_t → unconfined_service_t, so StartTransientUnit, D-Bus and FUSE all work under enforcing with nothing installed.

What breaks is containers. /var/lib/soli-one is var_lib_t and container_t may only read container_ro_file_t, so a podman workload handed an artifact bind mount gets EACCES with no useful message. packaging/selinux/soli-one.fc reasoned that out carefully — and the RPM never installed it. Nothing noticed, because the acceptance run supervised a plain shell command: a run with zero AVC denials said nothing at all about the case the labels exist for.

The package installs and registers them now, and acceptance.sh reads the expected type out of the .fc itself rather than repeating it, so editing one without the other fails the run. That check is only meaningful where SELinux is real, which is vm.sh test under KVM — and it says out loud which of the two it is doing.