1,100+ tests · rocky 10 · selinux enforcing gossip :7300 raft :7302 api :7304 rocky 10 / systemd 257

Installation

Rocky Linux 10. Two binaries: an agent on every node, a CLI wherever you work.

works today

The RPM builds, installs and runs on Rocky Linux 10, and still on 9. The whole scenario below — install, configure, start, run a workload, kill the agent with -9, watch the workload survive — is a script, packaging/rocky/acceptance.sh, and CI runs it on every push alongside the full test suite against a real systemd as PID 1 on both releases.

Rocky 10 is the target

works today

Rocky Linux 10 is what soli-one is set up and documented on: the tutorial runs on it, and the Soli runtimes the cluster installs — soli, soli-proxy, solidb, solikv — are published for Rocky 10 only. They are those projects' release binaries, built against glibc 2.39, which is Rocky 10's and not Rocky 9's.

soli-one itself still runs on Rocky 9. Its package is built on Rocky 9 and runs on 9 and 10. The binaries need at most GLIBC_2.34 — measured, not assumed — and Rocky 10 ships 2.39, so forward compatibility is free. The reverse is not: build on 10 and the result links against 2.39 and does not start on 9. That is why packaging/rpm/build.sh refuses to build anywhere but inside a Rocky container, and why the container is 9 even though both releases are supported.

Rocky 10 needs no code changes: nothing in the tree reads /etc/os-release or branches on a release number, and systemd capabilities are probed from the bus rather than derived from a version, so every gate is a >= that systemd 257 satisfies.

One thing to check at your provider rather than here: RHEL 10 raised the CPU baseline to x86-64-v3, so the oldest and cheapest instance generations may not run Rocky 10 at all.

The package channel

works today
rpm --import <base>/el10/RPM-GPG-KEY-soli-one
cat > /etc/yum.repos.d/soli-one.repo   # packaging/repo/soli-one.repo
dnf -y install soli-one

Signed packages and signed repository metadata: gpgcheck=1 refuses a package this key did not sign, and repo_gpgcheck=1 refuses metadata it did not sign. Without the second, someone who can write to the bucket can serve yesterday's metadata and have every node reinstall a version you have already fixed — no package is tampered with in that attack, so gpgcheck alone does not see it.

one cluster create puts a new machine on the channel while it is already logged in, when ONE_REPO_BASE is set. Installing from a file and never configuring a repository is how a fleet ends up with five versions and no way to tell which.

Or the tarball, for a machine with no channel

works today
curl -LO <release>/soli-one-linux-amd64.tar.gz
curl -LO <release>/soli-one-linux-amd64.tar.gz.sha256
sha256sum -c <<<"$(cat soli-one-linux-amd64.tar.gz.sha256)  soli-one-linux-amd64.tar.gz"
tar -xzf soli-one-linux-amd64.tar.gz -C /usr/local/bin soli-oned one

Two binaries and nothing else: no unit, no tmpfiles, no SELinux contexts, no firewalld service. Use it for a workstation that only needs one --host, or for a node you are configuring some other way — not as the normal way onto a cluster node, where everything the RPM installs besides the binaries is the part that makes confinement and the directory modes correct.

Unpacked out of the signed RPM by CI rather than built beside it, so the tarball and the package cannot contain different bytes. It carries a .sha256 because, unlike the RPM, it has no signature of its own — verify it before you run it.

Build the package

packaging/rpm/build.sh --verify

dist/soli-one-0.1.0-2.el9.x86_64.rpm
dist/soli-one-cli-0.1.0-2.el9.x86_64.rpm

--verify installs it into the Rocky bench and checks what actually landed: the binaries start, systemd-analyze verify is clean on the unit, the soli-one account exists, and /run/soli-one is 0700 while /var/lib/soli-one is 0711 — traversable so a workload's dynamic user can reach artifacts/, not listable, and with every sensitive directory inside it at 0700.

dnf install ./dist/soli-one-*.rpm

# Deliberately not started by the install. A package that starts a cluster
# agent would have it join whatever seeds a leftover config names, on a host
# nobody has finished configuring — and its keyring does not exist yet.

Build from source

cd one
cargo build --release

target/release/soli-oned --help
target/release/one --help
build on rocky, not on ubuntu

A binary linked against Ubuntu 24.04's glibc 2.39 does not start on Rocky 9, which ships glibc 2.34. Measured on this codebase, not assumed:

$ objdump -T target/debug/soli-oned | grep -o 'GLIBC_[0-9.]*' | sort -Vu | tail -1
GLIBC_2.39
$ docker run --rm rockylinux:9 ldd --version | head -1
ldd (GNU libc) 2.34

The stack's CI convention builds x86_64-unknown-linux-gnu on ubuntu-latest, which would ship a broken RPM on day one — one that installs cleanly and then fails at systemctl start. So the toolchain lives in the same image as the bench, and building it wrong by accident is not possible.

The Rocky bench

Rocky with systemd as PID 1, a real system bus, and the pinned toolchain. It is where the parts a developer laptop cannot reach get tested, and CI runs it on both 9 and 10 on every push. It also makes the container's root mount shared, because a private one — Docker's default — leaves every systemd credential directory silently empty; without that, nothing on this bench had ever delivered a secret. And it runs packaging/rocky/distbuild.sh: two workloads declared through the installed agent with their configs as secrets, and a cold cargo build whose every cacheable compile must run on the server the agent started.

packaging/rocky/bench.sh all      # build, start, run the suite inside

900 tests, on Rocky 9 (systemd 252) and Rocky 10 (systemd 257)
user bus (laptop)system bus (bench)
User= — privilege dropnoyes
ProtectSystem= — mount sandboxusually noyes
the io controllernot delegatedyes

Those three matter more than the count: their absence is invisible. A unit that silently ignores User= still starts, still serves, still passes its health check — and runs every tenant as root.

works today

SELinux enforcing and firewalld need a second bench, a real Rocky VM under KVM: packaging/rocky/vm.sh test. The full acceptance has passed there with zero AVC denials — and the note at the end of this page says what that run covered, which is less than “everything”.

It is the one gate CI cannot hold, so it is a per-release step a person takes rather than something a push proves.

A container could never do it. SELinux is a kernel feature and a container shares the host kernel — a workstation whose kernel has no SELinux cannot run enforcing in one, however privileged.

The keyring

/etc/soli-one/secret is the node's keyring, and it is written by one bootstrap on the first node and by one join on every other — never by hand. It holds the keys the node's role needs: gossip and artifact fetch on every member, the control key on control nodes only, and on a worker a node key bound to its name. The table, and why, is on Security.

The agent refuses a keyring readable by group or others, and refuses a file that is not a keyring at all. That includes the single shared secret earlier versions told you to generate with openssl rand: it is refused by name, and the answer is to re-bootstrap, because there is no way to split it into the new keys in place.

Starting the first node

one bootstrap --cluster-id soli-prod
soli-oned \
  --bind 0.0.0.0:7300 \
  --config-dir /etc/soli-one \
  --secret-file /etc/soli-one/secret \
  --data-dir /var/lib/soli-one \
  --socket /run/soli-one/agent.sock \
  --name rk-01 --dc par1 --rack r1

No --role: a keyring from one bootstrap holds the control key, so the node runs as control. --config-dir is what makes it serve joins and, because it can see ca-key.pem, form the Raft group.

Joining the rest

# on rk-01, or any node admitted as control
one token --node rk-02 --seed 10.0.0.11:7300 --role control   # or --role worker, the default

# on rk-02
one join --token soli2_…
soli-oned ... --name rk-02 --seed 10.0.0.11:7300

The token decides the role, and the keyring one join writes carries it. One reachable seed is enough. Everything else is learned through gossip — a node told only about rk-01 discovers rk-03 without being told it exists. Getting started has the whole sequence with its outputs.

Ports

None of these collide with the rest of the Soli stack, or with the proxy's 20000-30000 allocation range.

PortProtoUse
7300UDP + TCPSWIM gossip, LAN pool
7301UDP + TCPSWIM gossip, WAN pool (control nodes)
7302TCPRaft (control nodes)
7303TCPArtifact transfer
7304TCPCluster API (mTLS)
works today

A firewalld service definition ships in the RPM at /usr/lib/firewalld/services/soli-one.xml: firewall-cmd --add-service=soli-one. Never open these to the world — bind them to a zone restricted to peer CIDRs. The HMAC envelope is the real boundary and the firewall is defence in depth, not the other way round; an exposed gossip port still leaks the cluster's membership from the traffic pattern alone.

SELinux

In enforcing mode the agent itself works with no policy at all: /usr/bin/soli-oned is labelled bin_t, and the targeted policy transitions it to unconfined_service_t, which may call StartTransientUnit and mount FUSE.

What breaks is the container interaction — artifacts under /var/lib/soli-one are var_lib_t, and container_t can only read container_ro_file_t. That is a three-line file-context module, not a full policy — it ships as packaging/selinux/soli-one.fc. setenforce 0 is not an acceptable workaround: it removes exactly the defence in depth that running untrusted tenant code requires.

Both the chunk store and the artifact tree carry the same label, and that is deliberate. An SELinux label is a property of the inode: with hardlinks between the two trees, relabelling one relabels the other and the order of a restorecon walk starts to matter. That is the second reason materialisation uses reflink rather than hardlink.

Verified on a real Rocky 9.8 VM: the agent supervises workloads under enforcing with no policy module at all, and the whole acceptance run produced zero AVC denials. /usr/bin/soli-oned is bin_t, and the targeted policy transitions init_t + bin_t → unconfined_service_t — which is exactly the prediction the design rested on, now measured rather than reasoned.

what that run did not cover

The workload that VM supervised was a shell command. The file contexts above exist for a different case — a container handed an artifact bind mount — and the RPM was not installing them at all. So “zero AVC denials” was a true statement about a run that never exercised the thing the labels are for, and the reassuring number sat beside the untested case for months.

acceptance.sh now checks the contexts directly, reading the expected type out of soli-one.fc rather than repeating it — so editing one without the other fails the run — and it says out loud when it is running somewhere they cannot be checked at all, which is every container.