Filesystem
Immutable artifacts addressed by content — not a distributed POSIX filesystem, and not Ceph.
The artifact store is built and tested: content addressing, chunking,
manifests, ingest, materialize, verification, and two-phase garbage
collection with leases. Drive it with one artifact.
Artifacts move between nodes over an authenticated socket, transferring only the chunks the receiver lacks.
Secrets, delivered as systemd credentials — a tmpfs at
$CREDENTIALS_DIRECTORY, mode 0400, owned by the workload's
user.
Artifact replication. Both of the pieces this note used to call missing exist. The placement rule — given an artifact, its holders and the node roster, decide where the copies belong, spreading across failure domains and never moving a copy already in the right place — now runs on a timer in the agent, and it has real holders to read.
The possession index is not what was planned, and the reason is worth keeping. The design here was a per-node announcement: every node writes what it holds into the replicated store, and the timer reads the union. That version was written, compiled, passed its tests, and did nothing at all — because only the leader can write. A follower's write comes back as “forward this to the leader” and nothing forwards it, so each follower's announcement failed quietly and the leader saw an index containing only itself.
So the direction is inverted: the node holding the scheduler lease is the one that already writes, and it asks each peer what it holds instead of waiting to be told. Same information, one writer. Any design where “each node reports X” has this defect on a Raft store, and it is invisible from the tests — a single-node test has a leader for every write.
Datacenter gateways. Cross-region artifact transfer through
a small set of forwarding nodes, so a region fetching an artifact pulls it
once over the WAN rather than once per node. Nothing is built; the WAN link
policy in soli-one-federation decides what may cross a region
boundary, but no artifact traffic uses it.
What is deliberately out of scope
The deployment model is "immutable artifact, mutable alias": a build produces an artifact that never changes, and domains are aliases repointed at artifacts. Rollback is repointing an alias.
What that needs is efficient distribution of read-only file trees — not a
coherent multi-writer write(). Building a distributed POSIX filesystem
to get it would be a project in itself, and operating CephFS would end the claim
that the whole thing self-hosts from a handful of binaries.
The artifact store
- BLAKE3 content addressing, FastCDC chunking. Files up to 1 MiB are stored whole; larger ones get content-defined boundaries, so an insertion near the top of a file does not shift every boundary after it.
- Verify then rename. Bytes are hashed as they are staged and only enter the store if the hash matches what was asked for. A peer is never trusted, including one holding a valid certificate — content addressing is only a security property if the address is actually checked.
- Never repair silently. A blob whose content stops matching its name is quarantined and reported, not quietly refetched over. Silent repair hides a failing disk until several replicas have gone the same way.
- Replication N=3, placement-aware across racks and datacenters. A copy already in the right place is never moved — the artifact is immutable, so an existing replica is exactly as good as a new one, and re-placing each pass would make the store rewrite itself forever. Surplus copies are dropped only once the policy is met: deleting first would take the count below N for the width of a transfer, which is the window a second failure needs.
- A shortfall is always named. Three copies across two racks is reported even when the fleet has only two racks — it is a true statement about that artifact's availability, and the fix (add a rack, or lower the copy count) is exactly the decision an operator should be making. Silence would mean the only signal arrives on the day the rack goes down.
Measured on this codebase
Ingesting one/crates, then re-ingesting it with one file edited:
$ one artifact ingest crates
b3:8c7f00f9694838c4902b1e85216711bc9c0db7dca0de55f05c7be21a888666f3
30 files, 13 dirs, 0 symlinks, 191K logical
30 chunks stored, 0 already present (0% deduplicated), 191K written
$ one artifact ingest v2 # one file changed
b3:cb68b61cfa427a5e5e008f9c5bcb9fdccae8a3dfbd78ece5cd95e6dcc18aebcb
30 files, 13 dirs, 0 symlinks, 191K logical
1 chunks stored, 29 already present (96% deduplicated), 7.9K written
A redeploy that touched one file stored 7.9 K instead of 191 K. That
ratio is what the whole layer exists for, and one artifact ingest
prints it every time so it stays visible rather than assumed.
The round-trip property
Materializing an artifact and re-ingesting the result must produce the same id. If it ever does not, the manifest is losing something the filesystem carries — a permission bit, a symlink, an empty directory. It is asserted as an equation in the test suite, and it is why modes are recorded and directories are stored even when empty.
Where the copies go
Every node holding every artifact is simple, correct, and quietly unaffordable: three hundred nodes and a few hundred deployments is tens of terabytes of duplicate chunks, most never read on most nodes. N copies placed across failure domains is the same availability for a hundredth of the disk.
A replica is only a replica if it can fail independently, so placement spreads by the widest domain it can — datacenter, then rack — narrows only when it must, and says when it had to. It is deterministic, so two control nodes compute the same answer without talking to each other; a tie broken by hash order would make them disagree and the store would oscillate between two placements forever.
A node is skipped if storing there would leave it without headroom. Trading a replica for a full disk costs more than the replica was worth — the workloads are the point, and they need the space.
Moving an artifact between nodes
A small request/response protocol on port 7303. Every request is wrapped in the same HMAC envelope the gossip bus uses, under the fetch key every member holds and nothing else does, so only cluster members can pull artifacts off a node.
That is authorization, not integrity. Content is verified by hashing it on arrival, before it is allowed into the store — a peer holding a valid credential is still never trusted about what bytes it sends. Content addressing is only a security property if the address is checked, and checking it is what makes the transport uninteresting.
Measured between two stores, on a redeploy that changed one file of twenty:
first fetch: 20 chunks transferred
second fetch: 1 chunk transferred, 19 already localThe order is load-bearing: a lease is taken before the first byte arrives. Between "the collector decided this chunk is unreachable" and "the fetch completes", nothing references it, and the lease is the only thing covering that window. Taking it afterwards would leave the race open — and a test asserts no leases are left behind once a fetch finishes, because the opposite failure makes a store slowly uncollectable.
The server is read-only by construction: there is no request that writes. A compromised peer cannot use this to plant content; it would have to find a hash collision.
Garbage collection
Mark and sweep, not refcounts. A refcount has to be written before its referrer exists and cleared after it stops existing, and any crash in between leaves it wrong forever. Recomputing reachability each pass means a crash costs one delayed cycle instead of permanent drift.
Two races, one mechanism each:
- A fetch starts for a blob the sweep is about to delete. The fetcher takes a lease before the first byte arrives, and a leased digest is a root. No coordination with the collector needed.
- Something becomes referenced between marking and deleting. Deletion is two-phase: an unreachable object is condemned first and only removed on a later pass if it is still unreachable. One that came back into use is reprieved. A single-phase sweep would need a lock across the entire walk to be correct.
$ one artifact gc --keep b3:cb68b6… --dry-run
reachable 31 condemned 2 deleted 0 (0B) reprieved 0 too young 0
An empty root set is indistinguishable from "nothing is in use", so running
without --keep warns loudly rather than collecting the store.
Materialize with reflink, not hardlink
This looks like a micro-optimisation and is actually a security decision. An SELinux label is a property of the inode. With hardlinks, the artifact tree and the chunk store are the same inodes, so relabelling one relabels the other and the order starts to matter. Worse, a container that can write into its own artifact tree corrupts every other deployment sharing that chunk.
Reflink (FICLONE, on by default on Rocky 9's XFS) gives distinct
inodes, distinct labels and distinct permissions at zero copy cost.
The /soli view
/soli/
artifacts/<digest>/ read-only, materialized on demand
config/ observed-state store
secrets/<workload>/ tmpfs, visible only to that workload
data/<workload>/ bind to the pinned local directory
proc/ the aggregate cluster view
nodes/<node>/ per-node drill-down
soli-one-fuse is a library: it mounts and is read back
through real syscalls on the bench, but neither soli-oned
nor one links it, so no node has a /soli today.
Everything below describes the crate, not a running cluster.
Mounted read-only, nosuid, nodev, and without
allow_other — a view any user could read would expose the
cluster's shape to every process on the box. There is no write
implementation at all, and not "not yet": the only thing a writable
/soli could offer is a second way to change cluster state, and
a second way to change cluster state is a second thing that can disagree
with Raft.
proc/ is plain text, one record per line — the whole reason to
have a filesystem view rather than a subcommand is that
grep and awk work. one … -o json
covers structure.
FUSE is never on the critical path. Workloads get their artifacts
by direct bind mount from /var/lib/soli-one/artifacts/;
/soli is a view for humans and scripts. If the mount dies, nothing
stops serving.
The inverse would turn every mount bug into a production outage: a wedged FUSE
mount leaves processes in uninterruptible D state, which no signal
clears and only a reboot resolves. It ships disabled by default until it has
soaked.
Secrets
Encrypted at rest in the replicated state, sealed under a cluster key. Delivered
to a workload through a tmpfs mount written at start,
0400, owned by the workload's uid.
Never in the environment — that leaks through podman inspect and
/proc/<pid>/environ. Never baked into the artifact — every backup
of that artifact would then contain them.
The guarantee is exactly this and no more: root on a node can read every secret that node is entitled to, and nothing else. It is achieved by scoping delivery, not by cleaning up afterwards.
Stateful data
solidb is pinned to its node with its local data directory. Its
availability comes from its own replication, already wired in db/,
not from this cluster manager. one orchestrates the instances and
restarts them; it does not move RocksDB.
solikv used as a cache is relocatable — losing a cache is acceptable.
Configured to persist, it is pinned.
A pleasant consequence of content addressing: RocksDB SSTs are immutable, so consecutive database checkpoints share most of their chunks. Ingesting a checkpoint into the store gives incremental, deduplicated, verified, replicated backups with no backup-specific code.