Installation
Rocky Linux 10. Two binaries: an agent on every node, a CLI wherever you work.
The RPM builds, installs and runs on Rocky Linux 10, and still on 9. The whole
scenario below — install, configure, start, run a workload, kill the agent
with -9, watch the workload survive — is a script,
packaging/rocky/acceptance.sh, and CI runs it on every push
alongside the full test suite against a real systemd as PID 1 on both
releases.
Rocky 10 is the target
Rocky Linux 10 is what soli-one is set up and documented on: the
tutorial runs on it, and the Soli runtimes the cluster
installs — soli, soli-proxy,
solidb, solikv — are published for Rocky 10
only. They are those projects' release binaries, built against
glibc 2.39, which is Rocky 10's and not Rocky 9's.
soli-one itself still runs on Rocky 9. Its package is built
on Rocky 9 and runs on 9 and 10. The
binaries need at most GLIBC_2.34 — measured, not assumed — and
Rocky 10 ships 2.39, so forward compatibility is free. The reverse is not:
build on 10 and the result links against 2.39 and does not start on 9. That
is why packaging/rpm/build.sh refuses to build anywhere but
inside a Rocky container, and why the container is 9 even though both
releases are supported.
Rocky 10 needs no code changes: nothing in the tree reads
/etc/os-release or branches on a release number, and systemd
capabilities are probed from the bus rather than derived from a
version, so every gate is a >= that systemd 257 satisfies.
One thing to check at your provider rather than here: RHEL 10 raised the CPU
baseline to x86-64-v3, so the oldest and cheapest instance
generations may not run Rocky 10 at all.
The package channel
rpm --import <base>/el10/RPM-GPG-KEY-soli-one
cat > /etc/yum.repos.d/soli-one.repo # packaging/repo/soli-one.repo
dnf -y install soli-one
Signed packages and signed repository metadata:
gpgcheck=1 refuses a package this key did not sign, and
repo_gpgcheck=1 refuses metadata it did not sign. Without the
second, someone who can write to the bucket can serve yesterday's metadata
and have every node reinstall a version you have already fixed — no package
is tampered with in that attack, so gpgcheck alone does not see
it.
one cluster create puts a new machine on the channel while it is
already logged in, when ONE_REPO_BASE is set. Installing from a
file and never configuring a repository is how a fleet ends up with five
versions and no way to tell which.
Or the tarball, for a machine with no channel
curl -LO <release>/soli-one-linux-amd64.tar.gz
curl -LO <release>/soli-one-linux-amd64.tar.gz.sha256
sha256sum -c <<<"$(cat soli-one-linux-amd64.tar.gz.sha256) soli-one-linux-amd64.tar.gz"
tar -xzf soli-one-linux-amd64.tar.gz -C /usr/local/bin soli-oned one
Two binaries and nothing else: no unit, no tmpfiles, no SELinux contexts, no
firewalld service. Use it for a workstation that only needs
one --host, or for a node you are configuring some other way —
not as the normal way onto a cluster node, where everything the RPM installs
besides the binaries is the part that makes confinement and the directory
modes correct.
Unpacked out of the signed RPM by CI rather than built beside it, so the
tarball and the package cannot contain different bytes. It carries a
.sha256 because, unlike the RPM, it has no signature of its
own — verify it before you run it.
Build the package
packaging/rpm/build.sh --verify
dist/soli-one-0.1.0-2.el9.x86_64.rpm
dist/soli-one-cli-0.1.0-2.el9.x86_64.rpm
--verify installs it into the Rocky bench and checks what
actually landed: the binaries start, systemd-analyze verify is
clean on the unit, the soli-one account exists, and
/run/soli-one is 0700 while
/var/lib/soli-one is 0711 — traversable so a
workload's dynamic user can reach artifacts/, not listable, and
with every sensitive directory inside it at 0700.
dnf install ./dist/soli-one-*.rpm
# Deliberately not started by the install. A package that starts a cluster
# agent would have it join whatever seeds a leftover config names, on a host
# nobody has finished configuring — and its keyring does not exist yet.
Build from source
cd one
cargo build --release
target/release/soli-oned --help
target/release/one --help
A binary linked against Ubuntu 24.04's glibc 2.39 does not start on Rocky 9, which ships glibc 2.34. Measured on this codebase, not assumed:
$ objdump -T target/debug/soli-oned | grep -o 'GLIBC_[0-9.]*' | sort -Vu | tail -1
GLIBC_2.39
$ docker run --rm rockylinux:9 ldd --version | head -1
ldd (GNU libc) 2.34
The stack's CI convention builds x86_64-unknown-linux-gnu on
ubuntu-latest, which would ship a broken RPM on day one — one
that installs cleanly and then fails at systemctl start. So
the toolchain lives in the same image as the bench, and building it wrong
by accident is not possible.
The Rocky bench
Rocky with systemd as PID 1, a real system bus, and the pinned toolchain. It is
where the parts a developer laptop cannot reach get tested, and CI runs it on
both 9 and 10 on every push. It also makes the container's
root mount shared, because a private one — Docker's default —
leaves every systemd credential directory silently empty; without that,
nothing on this bench had ever delivered a secret. And it runs
packaging/rocky/distbuild.sh: two workloads declared through the
installed agent with their configs as secrets, and a cold cargo build whose
every cacheable compile must run on the server the agent started.
packaging/rocky/bench.sh all # build, start, run the suite inside
900 tests, on Rocky 9 (systemd 252) and Rocky 10 (systemd 257)
| user bus (laptop) | system bus (bench) | |
|---|---|---|
User= — privilege drop | no | yes |
ProtectSystem= — mount sandbox | usually no | yes |
the io controller | not delegated | yes |
Those three matter more than the count: their absence is invisible. A unit
that silently ignores User= still starts, still serves, still
passes its health check — and runs every tenant as root.
SELinux enforcing and firewalld need a second bench, a real
Rocky VM under KVM: packaging/rocky/vm.sh test. The full
acceptance has passed there with zero AVC denials —
and the note at the end of this page says what that run covered, which is
less than “everything”.
It is the one gate CI cannot hold, so it is a per-release step a person takes rather than something a push proves.
A container could never do it. SELinux is a kernel feature and a
container shares the host kernel — a workstation whose kernel has
no SELinux cannot run enforcing in one, however privileged.
The keyring
/etc/soli-one/secret is the node's keyring, and it is written by
one bootstrap on the first node and by one join on
every other — never by hand. It holds the keys the node's role needs:
gossip and artifact fetch on every member, the control key on control nodes
only, and on a worker a node key bound to its name. The table, and why, is
on Security.
The agent refuses a keyring readable by group or others, and refuses a file
that is not a keyring at all. That includes the single shared secret earlier
versions told you to generate with openssl rand: it is refused by
name, and the answer is to re-bootstrap, because there is no way to split it
into the new keys in place.
Starting the first node
one bootstrap --cluster-id soli-prod
soli-oned \
--bind 0.0.0.0:7300 \
--config-dir /etc/soli-one \
--secret-file /etc/soli-one/secret \
--data-dir /var/lib/soli-one \
--socket /run/soli-one/agent.sock \
--name rk-01 --dc par1 --rack r1
No --role: a keyring from one bootstrap holds the
control key, so the node runs as control. --config-dir is what
makes it serve joins and, because it can see ca-key.pem, form
the Raft group.
Joining the rest
# on rk-01, or any node admitted as control
one token --node rk-02 --seed 10.0.0.11:7300 --role control # or --role worker, the default
# on rk-02
one join --token soli2_…
soli-oned ... --name rk-02 --seed 10.0.0.11:7300
The token decides the role, and the keyring one join writes
carries it. One reachable seed is enough. Everything else is learned through
gossip — a node told only about rk-01 discovers
rk-03 without being told it exists.
Getting started has the whole sequence with
its outputs.
Ports
None of these collide with the rest of the Soli stack, or with the proxy's 20000-30000 allocation range.
| Port | Proto | Use |
|---|---|---|
| 7300 | UDP + TCP | SWIM gossip, LAN pool |
| 7301 | UDP + TCP | SWIM gossip, WAN pool (control nodes) |
| 7302 | TCP | Raft (control nodes) |
| 7303 | TCP | Artifact transfer |
| 7304 | TCP | Cluster API (mTLS) |
A firewalld service definition ships in the RPM at
/usr/lib/firewalld/services/soli-one.xml:
firewall-cmd --add-service=soli-one. Never open these to the
world — bind them to a zone restricted to peer CIDRs. The HMAC envelope is
the real boundary and the firewall is defence in depth, not the other way
round; an exposed gossip port still leaks the cluster's membership from the
traffic pattern alone.
SELinux
In enforcing mode the agent itself works with no policy at all:
/usr/bin/soli-oned is labelled bin_t, and the targeted
policy transitions it to unconfined_service_t, which may call
StartTransientUnit and mount FUSE.
What breaks is the container interaction — artifacts under
/var/lib/soli-one are var_lib_t, and
container_t can only read container_ro_file_t. That is a
three-line file-context module, not a full policy — it ships as
packaging/selinux/soli-one.fc. setenforce 0 is not an
acceptable workaround: it removes exactly the defence in depth that running
untrusted tenant code requires.
Both the chunk store and the artifact tree carry the same label, and that is
deliberate. An SELinux label is a property of the inode: with
hardlinks between the two trees, relabelling one relabels the other and the
order of a restorecon walk starts to matter. That is the second
reason materialisation uses reflink rather than hardlink.
Verified on a real Rocky 9.8 VM: the agent supervises workloads under
enforcing with no policy module at all, and the
whole acceptance run produced zero AVC denials.
/usr/bin/soli-oned is bin_t, and the targeted
policy transitions init_t + bin_t →
unconfined_service_t — which is exactly the prediction the
design rested on, now measured rather than reasoned.
The workload that VM supervised was a shell command. The file contexts above exist for a different case — a container handed an artifact bind mount — and the RPM was not installing them at all. So “zero AVC denials” was a true statement about a run that never exercised the thing the labels are for, and the reassuring number sat beside the untested case for months.
acceptance.sh now checks the contexts directly, reading the
expected type out of soli-one.fc rather than repeating it —
so editing one without the other fails the run — and it says out loud
when it is running somewhere they cannot be checked at all, which is every
container.