Getting started

Three machines, one cluster, one database cluster on top of it — from nothing.

every command below was run

This page is a transcript, not a design. Each command was executed on a real machine and the outputs are what it printed, abbreviated only where a temporary path would fill the line. Where something does not work, it says so instead of leaving the step out — a page that skips a service reads as an omission, and a page that says why is usable.

Steps 1 to 4 were edited after the trust model was split: the token now carries a role and the keyring replaced the single bus secret, and the outputs below show the new fields. The sequence is the one that was run.

What you need first

Three machines that can reach each other on TCP and UDP 7300, and the soli-one package installed on each — see Installation. Nothing else: no shared filesystem, no pre-shared secret, no configuration management.

The three machines below are called rk-01, rk-02 and rk-03. The first one is where the cluster is created, and that matters more than it looks — see the note about which machine mints tokens.

1. Create the cluster — on rk-01 only

one bootstrap --cluster-id soli-prod

cluster soli-prod created in /etc/soli-one

  ca.pem          the certificate joining nodes pin
  ca-key.pem      0400 — the trust root. Everything else follows from it
  secret          0400 — the keyring: gossip, fetch and control keys
  seed-cert.pem   what this node presents on the join port
  seed-key.pem    0400 — without it, a join could be impersonated

  fingerprint     cdf35e6c563f1ffedc241200d74f9ec34be69121d5f418aa0d75ac25b1031052

Nothing is running yet: this wrote files and started no process.

It started nothing on purpose. Generating a certificate authority is the expensive, irreversible step; starting a process is the routine one. Keeping them apart means you can look at what landed on disk before anything listens on a port.

Run it twice and it refuses. Regenerating a certificate authority invalidates every node certificate and every outstanding join token at once, and running a command again is the most natural thing to do when the first attempt seemed not to work.

2. Start the agent on rk-01

systemctl start soli-oned

The unit passes --config-dir /etc/soli-one, which is what makes this node serve joins. It logs which of the two things it is doing:

INFO soli_oned: soli-oned starting node=217d0c2b… name=rk-01 role=control
INFO soli_oned: serving joins: this node holds the cluster CA and can delegate identities addr=0.0.0.0:7300 founder=true
INFO soli_one_bootstrap::exchange: join listener ready addr=0.0.0.0:7300
INFO soli_oned: agent socket ready socket=/run/soli-one/agent.sock

Gossip is UDP on 7300 and the join exchange is TCP on the same port. Two sockets, two protocols, one firewall rule.

No --role anywhere: rk-01's keyring holds the control key, so it runs as a control node and forms the Raft group. The role a node runs as is read from its keyring, never from a flag it could be started with by mistake.

3. Mint a token for rk-02 — on rk-01

one token --node rk-02 --seed 10.0.0.1:7300 --role control

token for rk-02, valid until unix 1785697503:

  soli2_soli-prod_10.0.0.1:7300_cdf35e6c…1052_rk-02_control_1785697503_01e1…d65a1_3bfd05b9

Run this on rk-02:

  one join --token <the line above>

It admits that one node, from that one address, once. Shown here and
nowhere else — the cluster keeps only a hash of it.

The long field in the middle is the certificate authority's fingerprint. It is the reason this is a token and not a password: it lets the joining machine tell your cluster from anything else answering at that address. It is not secret and is meant to be compared out loud.

--role control, because three machines is a three-voter Raft group: each of them holds the control key and votes. Without the flag the token admits a worker — a node that runs workloads and holds the gossip and fetch keys plus a node key of its own, but nothing that votes or opens an operator secret. Every node after the third would normally be one. Security has the table of which role holds which key.

One hour maximum, one machine, one use. A join is a person at a terminal; if a token has to outlive the session it was minted in, the thing to fix is the automation.

4. Join, on rk-02

one join --token soli2_soli-prod_10.0.0.1:7300_…

joined soli-prod as rk-02; material written to /etc/soli-one

  ca.pem       the certificate this node pinned, and now trusts
  secret       0400 — the keyring for a control node
  cluster-id   soli-prod
  node-id      rk-02
  seed-cert.pem   what this node presents on its own join port
  seed-key.pem    0400 — this node can now admit the next member

Nothing is running yet: this wrote files and started no process.
Start the agent, pointing it at the peers the seed named:

  soli-oned --config-dir /etc/soli-one --secret-file /etc/soli-one/secret --seed 10.0.0.1:7300

Then start it:

systemctl start soli-oned

Repeat 3 and 4 for rk-03. Then, from any node:

one nodes

NAME       DC        RACK    STATE     CPU   LOAD              MEM
rk-01      default   default alive      16   1.13         6.4G/27G
rk-02      default   default alive      16   1.05         6.4G/27G
rk-03      default   default alive      16   1.05         6.4G/27G

3 nodes  3 alive  0 not alive

What happened during that join, and why the order matters

Nothing secret left rk-02 until it had decided rk-01 was its cluster. In order:

The token is parsed locally. Expired, or for another machine, and it stops here — no packet is sent, so anything listening at the seed's address learns nothing from a join that never happened.
A TLS handshake, which rk-01 can only complete because it holds a key the pinned certificate authority vouches for.
Only then is the token's secret sent.

Reverse steps two and three and a joining node hands a working credential to whatever answered the port. That is not a subtle weakness; it is the attack, and it is the order a person writes naturally, because sending the token is what joining feels like.

The handshake is what makes the check real. A certificate authority is public, so comparing a fingerprint over a plain connection proves nothing — anything in the path can relay the genuine one, watch the comparison pass, and collect the secret. See Security.

works today

Any control node can admit the next one. This page used to say every token had to come from rk-01, because minting and admitting both needed ca-key.pem. That was a cliff rather than a limitation: lose the founding machine and no node could ever join again.

A node joining as control is now handed its own TLS identity — seed-cert.pem and seed-key.pem, signed by the CA — as part of its admission. It arrives over a connection the joiner has already pinned to the CA and only after the joiner has proved it holds the token secret, so the key goes to a node the operator minted for and to nothing else. With it, rk-02 serves its own join port, mints tokens for its own address, and admits rk-03.

A worker is given no identity and no control key, and admits nobody: admitting means handing over either the control key or a node key derived from it, and a worker holds neither.

What still only rk-01 can do is delegate. ca-key.pem is deliberately not copied: a node admitted by rk-02 can admit others in turn only if it too was given an identity, and only the CA holder can sign one. So admitting a node hands over the ability to admit, never the ability to mint identities. Replicating the CA key instead would have been three lines and would have put the key that signs the whole cluster's trust on every machine in it.

Verified with three agents on one workstation, with the founder killed before the third join: rk-03 joined through rk-02, ca-key.pem never left rk-01, and rk-03 was admitted without an identity of its own — which one join says out loud rather than leaving to be discovered.

5. A SoliDB cluster on top

Write a spec once, and use the same file on every machine:

# /etc/soli-one/db-prod.toml
service    = "solidb"
cluster_id = "db-prod"
data_dir   = "/var/lib/solidb"
memory_mb  = 4096

[[member]]
node_id = "db-1"
machine = "rk-01"
address = "10.0.0.1:6746"

[[member]]
node_id = "db-2"
machine = "rk-02"
address = "10.0.0.2:6746"

[[member]]
node_id = "db-3"
machine = "rk-03"
address = "10.0.0.3:6746"

No replication_port: SoliDB multiplexes replication on the member's own port, which is one fewer port to open on a public interface. Every field is checked — a misspelling is an error, not a silently ignored default, because memory_gb instead of memory_mb would start a database with no memory ceiling at all and report it healthy.

Look at the plan first:

one datastore plan --spec /etc/soli-one/db-prod.toml

solidb cluster db-prod — 3 member(s)

  keyfile   generate for db-prod (local, never in Raft)
  keyfile   deliver to rk-01 as a systemd credential
  data      rk-01:/var/lib/solidb/db-1 (pinned, never relocated)
  keyfile   deliver to rk-02 as a systemd credential
  data      rk-02:/var/lib/solidb/db-2 (pinned, never relocated)
  keyfile   deliver to rk-03 as a systemd credential
  data      rk-03:/var/lib/solidb/db-3 (pinned, never relocated)
  start     db-1 on rk-01 — seed, alone
  confirm   db-1 is replicating, not merely running
  start     db-2 on rk-02 -> 1 peer(s)
  confirm   db-2 is replicating, not merely running
  start     db-3 on rk-03 -> 2 peer(s)
  confirm   db-3 is replicating, not merely running
  publish   db-prod at 10.0.0.1:6746

Nothing was done: `plan` reads and prints.

The seed starts alone. Starting all three at once with a full peer list lets each one decide it is the founder before it reaches the others: three single-node clusters, each convinced it is healthy, and no error anywhere. Split brain at birth is the hardest kind to notice, because there was never a moment when it was right.

Then, on rk-01:

one datastore apply --spec /etc/soli-one/db-prod.toml

  keyfile   created /etc/soli-one/db-prod.key (0400)
            copy this exact file to every other machine before applying there
  keyfile   delivered as a systemd credential
  data      /var/lib/solidb/db-1 ready, pinned to rk-01
  start     db-1 declared — seed, alone
  confirm   the seed reports 1 healthy node(s) after db-1 (needed 1)

Not done by this build:
  publish   db-prod
      no cluster-wide state to publish to yet: soli-oned does not run the
      consensus group, so apps must be told the address directly

On the other machines, in this order:
  one datastore apply --spec /etc/soli-one/db-prod.toml --machine rk-02
  one datastore apply --spec /etc/soli-one/db-prod.toml --machine rk-03

Copy the keyfile there first: the members cannot exchange a single
cluster message without the same secret on both ends.

Copy db-prod.key to rk-02, run the command it printed, then do the same for rk-03. Each one confirms before the next starts, and both views have to agree:

  confirm   the seed reports 2 healthy node(s) after db-2 (needed 2)
  confirm   db-2 agrees: 2 healthy node(s) from its own view
  confirm   the seed reports 3 healthy node(s) after db-3 (needed 3)
  confirm   db-3 agrees: 3 healthy node(s) from its own view

Tell your apps where db-prod is — nothing else will:
  http://10.0.0.1:6746
  http://10.0.0.2:6746
  http://10.0.0.3:6746

Any of them: every member accepts reads and writes. The seed is only
special while the cluster is being built.

That confirmation is a real check, not a wait. A sleep satisfies “running”, and so does a set of databases that each hold their own copy of everything — which is the failure the step exists to catch. It reads solidb_cluster_healthy_nodes from the seed's public /metrics, and refuses to continue if the count has not risen.

It asks both the seed and the member, and requires them to agree. That pairing is not belt-and-braces: the two used to disagree silently. A joined member reported one healthy node while the seed reported two, because the seed's answer naming the peers was sent to a doubled address, failed to resolve, and the failure was thrown away. Checking only the seed would still pass against that; checking only the member would miss one that never reached the seed at all.

What apply does not do

It is per-machine, and for a database that is the right shape rather than a limitation. A member is pinned — it owns a local data directory, and moving the workload would move a name and leave the bytes behind. So each machine carries out its own steps and the command for the others is printed rather than guessed at.

Workloads that can move are a different story: one run --cluster declares one for the whole cluster, the scheduler picks where each instance goes, and every chosen node starts it. Nothing writes a file.

The keyfile is delivered as a systemd credential: a tmpfs path, mode 0400, owned by the database's user. Never the environment, which leaks through /proc/<pid>/environ; never the artifact, which would put the secret in every backup of it.

SoliKV and SoliES clusters

all three plan and apply now

SoliKV plans and applies like SoliDB now. Slot ownership propagates, and the plan assigns an even split as its last step — verified against two real nodes, with a redis-cluster client writing eight keys through either node and reading them all back through the other. This page said sharding did not work; that was true when it was written and is not any more.

SoliES replicates its log — also contrary to what this page used to say. The old text claimed AppendEntries carried an always-empty entries list and that nothing appended to the log; both came from a stale doc comment rather than the code. Two real brokers replicate, and the check that settles it is the third one: kill the follower and the leader stops committing, because a majority of two cannot be one.

one datastore plans and applies solies too, with a start shape of its own: every member starts with the whole membership, and each is confirmed only after the last one is up, because a quorum cannot form before then. Its raft_id is written in the spec rather than taken from a member's position in the file — a Raft node id is durable identity, and sorting the members would otherwise renumber a live cluster.

[[member]]
node_id = "es-1"
machine = "rk-01"
address = "10.0.0.1:9200"
raft_id = 1                                   # never derived, never changed
admin_token_file = "/etc/soli-one/es-1.token" # only to confirm; per member

The token is per member because SoliES keeps its key store in each broker's own data directory and generates key secrets server-side, so one token cannot be valid on three brokers. Name none and the confirmation is reported as not performed, with the curl that performs it — claiming “confirmed” for a check that was skipped is the one outcome worse than not checking.

And create every topic on every member: topic metadata is local to the broker that created it, so until it exists everywhere, that topic's partitions cannot reach a majority. The write is refused with the reason rather than silently accepted, and the plan says so as a note.

Two bugs this page found

fixed, and worth knowing about

Writing this page meant running every command, and running them turned up two defects that had the same shape: the seed's view was correct and the member's was not, so nothing looked wrong from where an operator looks.

Full sync never worked. A batch of documents was encoded with bincode, which cannot decode a self-describing value: encoding succeeded, decoding failed every single time, and a joining node received nothing while the seed reported it healthy. The payload is JSON now.

The join answer went to a doubled address. --advertise 10.0.0.1:6746 had the port appended a second time, giving 10.0.0.1:6746:6746. That does not resolve, and the send was on a discarded result — so the joiner was a member from the seed's point of view and knew nobody from its own. --advertise now accepts a host with or without a port, and the failure is logged.

Both are why the confirmation step above asks the seed and the member. A check that had only asked one of them would have passed through both bugs.

Before this cluster holds anything you would miss

two minutes, and the only step that cannot be done later

Three things in the cluster cannot be reconstructed from anything else: the CA private key you generated in step 1, the keyring beside it, and the replicated state. Lose the first and the cluster keeps serving and can never admit another node.

umask 077; head -c 32 /dev/urandom > /root/soli-one-backup.key
# and a copy of that file somewhere the cluster is not

one backup --out /var/backups/soli-one --recipient /root/soli-one-backup.key
systemctl enable --now soli-one-backup.timer   # daily, after you have set backup.env

The bundle is sealed to that key rather than to anything in the keyring, because it contains the keyring — sealing it with that would be a lock with its key inside the box, and the situation a bundle exists for is the one where the cluster is gone. Check what you have with one restore --from <bundle> --inspect, which needs no key at all.

Where to go next

Adding & removing nodes — a third machine, and taking one out again without breaking the quorum.
Security — why the join is ordered the way it is, and the peer allowlist an OVH VPS forces.
Stateful clusters — the two refusals, and why a database is not an app with three replicas.
Workloads — running something other than a database.
CLI reference — every command and flag.