Creating a cluster

Two machines that have never met, over a network neither controls.

works today

one bootstrap, one token and one join exist and were run against three real agents: a cluster is created, two nodes join it, and one nodes shows three alive. The exchange happens over TLS the pinned certificate authority has to authenticate — without that the fingerprint check below would be decoration, because a certificate authority is public and anything in the path can relay the genuine one.

A node admitted as control is handed its own CA-signed TLS identity as part of the admission, so it can serve its own join port and admit the next member. The cluster no longer depends on the machine that ran one bootstrap still being alive. What that machine keeps is ca-key.pem, and therefore the sole ability to issue an identity: admitting a node passes on the ability to admit, never the ability to mint. A worker is given neither the identity nor the control key, and admits nobody. See Getting started for the whole sequence, and Security for which keys each role receives.

The problem, stated once

Everything else in this page is downstream of one question: how does a new machine come to trust the cluster, and the cluster come to trust it?

It is tempting to treat the join token as a password — the joiner proves it belongs. That is half the problem, and the half that is usually skipped is the dangerous one.

Check that is missing	What an attacker does
The cluster does not authenticate the joiner	Anyone who reaches the seed port joins, and a member reads every secret, every artifact and every workload spec
The joiner does not authenticate the cluster	An attacker on the path answers the join, becomes the cluster from that node's point of view, and receives its workloads and their secrets

The second is worse, and it is the one that gets skipped, because skipping it works: the node joins, everything comes up, and nothing looks wrong.

The token carries the fingerprint

So a join token is not a password. It carries the cluster CA fingerprint, and the Token type cannot be constructed without one. Trust-on-first-use is not discouraged here — it is unrepresentable. A design whose unsafe mode requires deleting a field is a design whose unsafe mode does not happen by accident at three in the morning.

soli2_soli-prod_203.0.113.7:7300_<ca fingerprint>_rk-02_worker_<expiry>_<secret>_<crc>

What each field is for:

The fingerprint, untruncated. It is not shortened to keep the token tidy: a pinned fingerprint's whole job is second-preimage resistance, and trading that for 48 characters on a string nobody types is a bad trade.
The secret, stored cluster-side as a hash. A leaked cluster state must not yield a usable token.
An expiry, capped at one hour. A join is an operator at a terminal; if a token has to outlive that session, the thing to fix is the automation. A long-lived control token is a long-lived root credential for the whole cluster.
One node id, so a token read over a shoulder is useless from any other machine.
The role it admits, control or worker — one token --role, default worker, the one that hands over least. A worker receives the keys every member needs and nothing that votes in Raft or opens a secret. The seed admits from its own record of the token, never from what the joiner says; the role is in the text so the operator can see which kind of credential they are pasting. The version prefix is soli2 because of this field, and a soli1 token is refused as old.
A checksum, so a truncated paste reports “truncated” rather than “invalid” — which reads as “expired” and gets answered by minting two more cluster-root credentials before anyone looks at the clipboard.

The ordering rule

Nothing irreversible happens until everything reversible has succeeded.

On this infrastructure “irreversible” is not a figure of speech. Ordering a VPS creates a billed subscription; a DNS record with a long TTL is cached by resolvers nobody controls; and a CA private key, once used to sign, cannot be replaced without re-issuing every certificate in the cluster. So the plan front-loads the checks, and a run that dies half way costs nothing.

verify    OVH credentials and clock, region eu
verify    DNS zone solisoft.net is manageable
provision rk-01 as vps-4  ** BILLED **
harden    rk-01: key-only ssh, no root login
install   soli-one on rk-01
ca        generate for soli-prod (private key stays local)
firewall  rk-01: peer allowlist from the roster
bootstrap raft on rk-01 — the cluster exists after this
firewall  rk-02: peer allowlist from the roster
token     mint for rk-02, bound to 203.0.113.8:7300
join      rk-02 -> 203.0.113.7:7300 (pins the CA fingerprint)
confirm   rk-02 is replicating, not merely configured
dns       *.soli.app in solisoft.net

Four orderings in there are load-bearing:

Harden before install. A fresh VPS answers password SSH on a public address. Installing first leaves a window where the machine holds cluster credentials and still accepts a password.
Firewall before token. A token lives one hour; minting it for a node that cannot yet reach the seed spends that hour on a connection that was never going to succeed.
Confirm after every join. add_learner returns Ok for a node it never reached — measured on openraft 0.9.25. Reporting that as success gives a cluster that believes it has three members and replicates to one.
DNS last. A record published before the cluster answers is cached as a failure by resolvers nobody controls, for as long as the negative TTL says.

Re-running is safe, and that matters most for the billed step

A half-finished bootstrap gets resumed, and resuming is the ordinary case. With the machines already present the plan contains no billed step at all — ordering three more VPS because someone pressed up-enter is the expensive version of an idempotency bug.