Operations

Bootstrap, upgrade, back up — and how this gets tested without a datacenter.

Running a local cluster today

Three agents on one machine, which is enough to exercise gossip, failure detection and the aggregate views.

one bootstrap --cluster-id lab --config-dir /tmp/lab/etc

for i in 1 2 3; do
  soli-oned --bind 127.0.0.1:7${i}00 \
    --raft-bind 127.0.0.1:7${i}02 --artifact-bind 127.0.0.1:7${i}03 \
    --secret-file /tmp/lab/etc/secret \
    --data-dir /tmp/lab/n$i --socket /tmp/lab/n$i.sock \
    --name rk-0$i --dc par1 --rack r$i --cluster-size 3 \
    $( [ $i -eq 1 ] && echo "--config-dir /tmp/lab/etc" ) \
    $( [ $i -ne 1 ] && echo "--seed 127.0.0.1:7100" ) &
done

one --socket /tmp/lab/n1.sock top

All three run from the founder's keyring, so all three are control nodes: rk-01, the one given --config-dir, forms the Raft group, and the other two are enrolled from the attestation they gossip. Sharing a founder's keyring is admitting a machine as control — fine on one workstation, and never how a real node joins; that is one token --role … and one join, on Adding and removing nodes. The explicit Raft and artifact ports are what let three agents share a host.

gotcha

Unix socket paths are capped at 108 bytes by the kernel. Keep --socket short — the agent checks this and says so, because the raw error names neither the limit nor the path.

Node identity

Each node generates a stable id on first start and persists it to <data-dir>/node-id. If that file is present but unreadable as an id, the agent refuses to start rather than minting a replacement: a node that silently changes identity looks like a brand new machine to the whole cluster and orphans everything assigned to the old one.

The incarnation number is the wall clock at start, so a restarted node always supersedes its own stale entry instead of fighting it for an address.

Rolling upgrades

works today, and the test asked for exists

Because workloads are transient units owned by PID 1, replacing the agent binary and restarting it does not signal them. The new process enumerates units by prefix and re-adopts what it finds.

This note used to ask for an explicit test, on the grounds that the whole upgrade story rests on the property and a refactor could quietly break it. That test is a_replacement_engine_adopts_instead_of_restarting: one engine starts a workload and is dropped, releasing the state store's lock exactly as a stopped agent would; a second engine over the same data directory must report the running unit as adopted and must start nothing. It needs a real service manager, so it skips loudly where there is no bus, and ONE_ROCKY_BENCH=1 turns that skip into a failure so the bench cannot pass by not running.

The enumeration it depends on is now a parse rather than a pattern, in both directions. The scan asks systemd for one-* and the parser builds the same prefix from the same constant instead of a literal — two spellings would mean the scan finding units the parser cannot read, and the reconcile loop warning about each of them on every pass forever.

Backup

works today

Three things must survive, and none of them can be reconstructed from anything else: the replicated state, the cluster CA private key, and the keyring. Lose the first and every alias, workload spec and port lease goes with it. Lose the second and the cluster still serves and can never admit another node — the repair is a write, and a cluster that cannot write cannot be repaired.

umask 077; head -c 32 /dev/urandom > /root/soli-one-backup.key

one backup --out /var/backups/soli-one --recipient /root/soli-one-backup.key
one restore --from <bundle> --inspect            # no key needed
one restore --from <bundle> --recipient /root/soli-one-backup.key

The bundle is sealed to an operator key, not to anything in the keyring. It contains the keyring, so sealing it with a key from there would be a lock with its key inside the box — and the one situation a bundle exists for is the one where the cluster it came from is gone. Keep the recipient file somewhere the cluster is not.

The manifest — when, from which node, at which revision — stays in the clear and is authenticated as the AEAD's associated data. So --inspect tells you whether the only bundle you have is the right one without the key, and editing a byte of it makes the payload refuse to open rather than open under a lie.

soli-one-backup.timer ships with the package and is not enabled: the destination of a file holding the CA key is a decision, not a default. See /etc/soli-one/backup.env.

An untested restore is not a backup. The test is the deliverable here, not this page: crates/soli-one-consensus/tests/restore.rs populates a cluster, backs it up, restores onto a different node id, and asserts the aliases, specs and leases come back and that the node reaches leadership alone and accepts a new write. That last part is what fails when the restored membership is not rewritten to the surviving node — and the symptom is a node that starts, looks healthy, and cannot commit.

Metrics and logs

works today

one metrics                      # Prometheus text, over the agent socket
one logs <job> --all             # every node holding an allocation

No listening port. The agent's only interface is a unix socket whose permissions are its authentication, and an agent that binds a second port to publish its own liveness has made itself more likely to need publishing. soli-one-metrics.timer writes the text into node_exporter's textfile collector, which is the mechanism a Prometheus deployment already has for exactly this.

Nothing per-workload carries a label. A label whose values are workload ids turns thirty preview environments into thirty time series per metric, and the cardinality grows with the product's success. Per-workload questions are one ps and one logs.

Upgrades

works today

one node upgrade                 # every alive node, one at a time
one node upgrade rk-02 --timeout 180

Drain, dnf upgrade over SSH, wait for the agent to come back and account for its workloads, undrain, then the next node. Nothing restarts: the workloads are PID 1's children, and the package restarts only the agent.

It stops at the first node that does not come back, and leaves that node drained. A rolling upgrade that continues through a failure is a way to take a fleet down one node at a time, on purpose, at speed.

Secrets

works today

printf %s "$PASSWORD" | one secret put db-password
one secret ls                    # names, never values
one secret rm db-password

Sealed before it enters the replicated log, and delivered to a workload that names it (secrets = ["db-password"]) as a systemd credential: a tmpfs file, mode 0400, owned by the unit's user, unmounted when the unit stops. Never an environment variable — that leaks through podman inspect and /proc/<pid>/environ.

There is no get. A secret's destination is a workload, not a terminal, and an operator who can read every secret back out of the cluster is a laptop that is now as sensitive as the cluster. Rotating is put again.

The seal is keyed by the control key, which only control nodes hold, so every control node can open every secret — the scheduler may place the workload on any of them. A worker holds no control key: the node holding the scheduler lease re-seals each secret to that worker's own node key and pushes it, addressed, bound to the job, and valid for 60 seconds, so a worker can open the secrets of what is placed on it and nothing else. Workers do not yet run cluster-placed jobs, so that path carries nothing until they do; a file-declared job on a worker that names a secret fails to start with an error naming it. Beyond that, what the seal buys is that the Raft log, its snapshots, and every backup of them hold no plaintext. Security has the details.

The acceptance run

works today

The scenario the architecture is sold on, scripted end to end against the installed RPM on Rocky Linux 9.3 — not against anything the test harness constructs.

packaging/rpm/build.sh --verify
packaging/rocky/acceptance.sh

==> starting the agent through systemd, from the RPM's unit
   ok  soli-oned.service is active (Type=notify, so this means it signalled ready)
==> the workload is running
   ok  workload pid 23539, parent PID 1, uid 998
   ok  MemoryMax reached the cgroup: 67108864
   ok  the agent (23521) is not the workload's parent
==> kill -9 the agent
   ok  the workload is STILL RUNNING with the same pid 23539
==> systemd restarts the agent, which re-adopts
   ok  adopted, not restarted: still pid 23539
==> removing the LAST one is refused by the mass-deletion guard
   ok  refused, and said so
==> empty slices are reaped, not left to accumulate
   ok  no empty tenant or job slices

The uid is 998, not 0: the workload runs as the soli-one account the package created, even though its supervisor is root. And the pid is unchanged across the agent's death — adopted, not restarted, which is the difference between a supervisor you can upgrade and one you cannot.

Three things this run found

The unit said Type=notify and nothing ever sent READY=1. systemd waited 90 seconds, killed the agent, and restarted it — forever. Every unit test passed throughout; only a real systemd could show it.
The journal capability probe only knew about persistent storage. Stock Rocky 9 has no /var/log/journal, so journald writes to /run/log/journal — and one logs would have refused to run on a machine where journalctl works perfectly.
systemd never reaps an empty slice. A one-<tenant>-<job>.slice stays active with zero tasks until reboot. On a node running ephemeral preview environments that is one leaked cgroup directory per deployment, forever. The agent now collects them, after two consecutive empty passes so it cannot race a start.

The cluster's address

built, not wired in yet

One VIP, held by one node, moved when that node stops holding its lease. Tested against consensus, and the ip layer tested against a real interface on the bench — but soli-one-vip is a library that no binary links yet. Neither soli-oned nor one runs it, so a cluster today has no VIP; put the addresses in front of it by other means until this is wired.

The whole component is one rule: the address is bound if and only if the lease is held. Four states, four answers, and no fifth case for "we cannot tell" — an address kept on the strength of an unanswered question is how a partitioned node keeps serving traffic it should have given up.

Release is eager, claim is patient. A handover therefore has a gap where nobody answers — a second of 503s — and that is strictly better than a window where two machines answer and nobody can tell which.

Two nodes answering for one IP is worse than none answering: clients see whichever ARP reply arrived last, connections land on alternating machines, and every symptom points somewhere else. So release is eager and claim is patient — a handover has a gap where nobody answers, which is a second of 503s, and that is strictly better than a period where two machines answer and nobody can tell.

measured on Rocky 9

Moving an IP is useless if the switches still have the old MAC cached. Rocky 9 ships arp_notify=0 and does not ship arping, so the sysctl is the whole mechanism — it is set before every bind.

And ip's errors are judged by the postcondition, never by the message. Matching "already assigned" would break on the next iproute2 version — Rocky 9 says Error: ipv4: Address already assigned. where older ones said RTNETLINK answers: File exists — and again under a non-C locale.

Regions

works today

Region registry, routing, and the policy for what may cross a WAN link. The gossip pool that carries it is next.

Federation is deliberately thin: regions learn that each other exist and who leads them, and nothing else. Two rules give it its shape, and both are code rather than convention — a convention about what crosses a WAN link is one that will be broken by whoever adds the next field.

Only control nodes gossip across the WAN, and only those given the federation key. Several hundred workers in a second pool turns a link with real money attached into a mesh whose traffic grows with the square of the fleet. The key is its own — --wan-key-file, or the systemd credential wan-key — shared by the control nodes of every region and by nothing else, so a worker in any region holds none of it; without it a control node takes no part. Frames are believed only from configured --wan-peer region=addr peers, each of which speaks for one region, and the directory stops at 256 regions.
Facts do not cross; a summary does. One RegionSummary per region — under 128 bytes — instead of N NodeFacts for information the far region cannot act on. It is an allowlist, because a denylist lets a new message type cross by default and nobody notices until the bill arrives.
Nothing is executed remotely. A request naming another region is forwarded to that region's leader. Carrying it out here would mean writing to a Raft group this node is not a member of, and any design that made it possible would be one where a WAN partition produces two answers.

"Mid-election" and "not a region" are different answers, and a stale region stops being routable rather than being counted as capacity — a total that includes a datacenter this node cannot reach is a scheduler promising what it cannot deliver.

Testing at three levels

Level	Tool	What it proves	Runs
Unit and integration	`cargo test`	Everything provable without privileges	every push — 900 tests
The real path	Rocky 10 and Rocky 9 containers, systemd as PID 1	systemd, D-Bus, journald, cgroups, podman, FUSE, `ip`, the RPM	every push — 900 tests + acceptance
Confinement	Rocky VM under KVM	SELinux `enforcing`, firewalld	by hand, per release

The first two are CI jobs, and the bench sets ONE_ROCKY_BENCH=1 so a test that would quietly self-skip fails the job instead — without that, cargo test on a runner with no system bus passes while proving much less than it appears to.

The third cannot be a CI job: SELinux is a kernel feature, the runner has none, and --privileged disables confinement anyway. It is a person, per release, and the honest reading of the run below is narrower than it looks — the workload it supervised was a shell command, so it says nothing about the file contexts a container needs to read an artifact. Those went uninstalled by the package for months behind exactly this reassuring zero. acceptance.sh checks them explicitly now.

packaging/rocky/vm.sh up      # boot a real Rocky; ROCKY_RELEASE=10 for the other one
packaging/rocky/vm.sh test    # install the RPM, run the acceptance

==> and under what confinement
   SELinux: Enforcing
   AVC denials since boot: 0
   ok  enforcing, and nothing was denied

Acceptance passed on Rocky Linux release 9.8 (Blue Onyx)

The container bench cannot do this, and not because of --privileged: SELinux is a kernel feature and a container shares the host kernel. A workstation whose kernel has no SELinux — selinuxfs absent from /proc/filesystems, LSM stack lockdown,capability,landlock,yama,apparmor — can never run enforcing in a container, however privileged.

worth checking before concluding

The VM needs /dev/kvm. Group membership is the usual route and not the only one — an ACL granting the user rw works identically. getfacl /dev/kvm is one command, and assuming the group is the only path is how a gap that could have been closed stays open.

What it confirmed: the agent runs under enforcing with no policy module at all — /usr/bin/soli-oned is bin_t, and the targeted policy transitions init_t + bin_t → unconfined_service_t, which may call StartTransientUnit. Zero AVC denials across the whole acceptance. And the firewalld service ships correctly: firewall-cmd --add-service=soli-one opens all seven ports.

The second level is not optional and cannot be faked on Ubuntu: StartTransientUnit, privilege drop and the io controller all behave differently, so validating them elsewhere proves nothing.

The third is still untestable in a hosted CI runner — no nested virtualisation, no SELinux — so without a self-hosted Rocky runner or a nightly lab job it will regress. That is a scheduling problem now rather than an unknown.

Everything is authenticated

A node presenting the wrong secret is not merely ignored — its datagrams never reach the membership state machine at all. There is a test for exactly that, and another asserting a forged packet does not stop the receive loop from delivering the legitimate one behind it.