Adding and removing nodes

Changing a running cluster's membership, and the arithmetic that decides whether it survives.

every command below was run

Getting started creates a cluster and joins a second machine to it. This page is what comes after: a third node, and taking one out again. It does not repeat the first join — two pages describing the same command drift, and the one you are reading would be the stale one.

Three things called membership, and only one of them is a vote

Most confusion here comes from treating one word as one fact. A node can be in any combination of these three, and each is changed by something different:

Fact	Kept by	What changes it	What it decides
Liveness	SWIM gossip	The node answering probes, or stopping	Whether `one nodes` calls it alive
Voting membership	The control roster and the Raft configuration, both in Raft	`one join` with a `--role control` token (indirectly), `one node remove`	Quorum — whether the cluster can write at all
Schedulability	A cordon key in replicated state	`one node drain` / `undrain`	Whether new work is placed there

A drained node still votes. A dead node still votes. That second one is the important one, and it is why removal exists: a machine you deleted from your provider's console is still a member of the Raft configuration, and its absence counts against every quorum from then on.

Adding a node

The same three commands as the second node, with two things to know. The token decides what the node is: --role control for a node that votes in Raft and holds the control key, --role worker — the default — for one that runs workloads and holds neither. And tokens are minted on a control node that serves joins, the one named in --seed: the founder, or a node admitted as control, which is handed a TLS identity for exactly this. The token's hash is recorded on that node, so a token minted anywhere else names a seed that cannot redeem it. A worker holds no control key and admits nobody.

1. Mint the token — on a control node

one token --node rk-03 --seed 10.0.0.1:7300 --role control --config-dir /etc/soli-one

token for rk-03, valid until unix 1785858081:

  soli2_demo_10.0.0.1:7300_49dae656…808eba_rk-03_control_1785858081_c1e5655c…e429310_1be86e3b

Run this on rk-03:

  one join --token <the line above>

It admits that one node, from that one address, once. Shown here and
nowhere else — the cluster keeps only a hash of it.

The seed records a hash of it, and the role, under <config-dir>/pending/rk-03.json in mode 0600, and deletes that file when the node is admitted — before the admission is sent, so a crash between the two refuses a replay rather than allowing one.

Three control nodes is the usual shape: enough for a quorum that survives one failure, few enough that a commit is cheap. Every node after that is a worker.

2. Join — on the new machine

one join --token soli2_demo_10.0.0.1:7300_…

Nothing secret leaves the joining machine until the seed has proved, through a TLS handshake, that it holds a key the pinned certificate authority vouches for. join writes the keyring for the node's role to /etc/soli-one/secret, the pinned CA and, for a control node, its TLS identity, and prints the soli-oned command to run. It starts nothing, for the same reason bootstrap does not: the result can be read before anything listens.

3. Start the agent, and let the cluster notice

soli-oned --config-dir /etc/soli-one --secret-file /etc/soli-one/secret

No --role: the agent runs as whatever its keyring allows. Asking for --role control on a worker's keyring is refused.

Promotion to voter is not a command you run. A control node gossips an attestation made with the control key over its id, name and Raft address; the leader checks it, writes the node to the control roster (roster/control/<id>), adds it as a learner, waits until its log has caught up, and only then promotes it. What a node says in gossip about its role decides nothing — every member holds the gossip key, so a claim there is only a claim. A worker never reaches the roster. That order is the whole point: a node admitted straight to voter grows the quorum before it holds any of the data the quorum is counted over, so a cluster of two healthy nodes plus one empty newcomer needs the newcomer to agree about entries it does not have.

measured, not assumed

add_learner answers Ok for a node the leader cannot reach at all. So the promotion waits on a real replication check rather than on that answer, and a node that never catches up stays a learner — visible, not voting, and not counted against a quorum.

one nodes

Removing a node

Two steps, and the first is optional only when you have already given up on what the node is running.

one node drain rk-03        # stop placing new work there; it keeps what it runs
one node remove rk-03      # take it out of the Raft configuration, for good

remove also takes the node off the control roster and leaves a tombstone, roster/removed/<id>. A removed node still holds the control key and can keep broadcasting the attestation that enrolled it; the tombstone is what stops the leader enrolling it again. A machine removed for good and later reused rejoins under a fresh node id, with a new token.

remove runs on the leader. A follower cannot do it, and the reason is worth stating because it is not a permissions rule: a follower's write to the Raft log comes back as “forward this to the leader” and nothing forwards it. Attempted on a follower, the change would report success having removed nothing, and the node would still be a voter.

The arithmetic, which is the whole safety story

Removing a member shrinks the membership, so the quorum shrinks too. That is usually good and occasionally fatal:

Before	Removing	After	Verdict
3 members, 2 alive	the dead one	2 members, 2 alive, quorum 2	Allowed — and the cluster ends up healthier than it was
3 members, 2 alive	a live one	2 members, 1 alive, quorum 2	Refused — unwritable, and unrepairable
5 members, 2 alive	any	4 members, 2 alive, quorum 3	Refused
5 members, 5 alive	one	4 members, quorum 3	Allowed, with a warning: an even count tolerates no failures
1 member	it	—	Refused — no write could ever succeed again

The second row is the one the refusal exists for, and it is not obvious from inside the operation: three members with one already dead, and an operator removes the healthy spare. Membership falls to two, the quorum stays two, one voter is alive. The cluster can no longer write — and it cannot be fixed by then removing the dead node, because that repair is itself a write. There is no way back from that state without editing the log by hand.

What `--force` does, and what it will never do

--force abandons the workloads the cluster still places on the node. That is a real decision an operator gets to make: the machine may already be gone, and insisting on a clean drain would be insisting on something impossible.

It does not relax any of the quorum rules. The workload rules protect the workloads, and you may knowingly abandon those. The quorum rules protect the cluster's ability to be repaired at all, and no amount of operator intent makes an unwritable cluster writable again — so --force does not reach them, and a test asserts that it does not.

Why a name is not enough

Node names are hostnames and are not required to be unique — the node id is the key. So a name matching two nodes is refused rather than resolved to whichever came first. And the node you most want to remove is usually the one that has stopped gossiping, whose name may no longer resolve at all, so a raw Raft id is accepted too and the error lists the ids the configuration currently names:

no node named "rk-03" and no member with that Raft id. The configuration
names: 1 (10.0.0.1:7400), 2 (10.0.0.2:7400), 17384…  (10.0.0.3:7400)

Other refusals

The node is the leader. openraft can remove a leader, and doing so makes the operation's own commit race its author's demotion. Waiting for leadership to move costs seconds and turns a coin flip into a certainty.
It still carries work. Drain it, or say --force. Removing a node the scheduler still points at orphans those workloads: the units keep running, because they belong to PID 1 rather than to the agent, and the cluster stops accounting for them.
It was never a member. Not an error. An operator who runs this twice, or who removes a node whose join never finished, wanted it gone — and it is gone.

The cordon is deleted with the node. Left behind, one node cordons would list a machine that is no longer in the cluster, forever.

What a removal is not

proved against a real three-node group

A removed node keeps everything it held and stops receiving anything committed after it left. That is a departure, not a wipe: its data directory is intact, so a machine removed by mistake can be inspected, and a machine being decommissioned still needs wiping separately.

Asserted end-to-end rather than argued: three real openraft members, one removed, then a write on the leader that must still commit and reach the remaining follower — and must not reach the removed node. A membership change that returns Ok and leaves an unwritable cluster would pass a weaker test.

What is not here

works today, for one provider

Ordering machines is a separate, confirmed path. It is deliberately absent from the provider interface — it spends money and it is irreversible, so it does not belong to a call that looks like any other. It lives in one cluster create, which orders every machine before configuring the first: a create that cannot order the second machine has otherwise already spent money on a one-node cluster reporting itself healthy.

Scaleway is the wired API. OVH has request signing, DNS reconciliation and the Additional-IP move built and no create path; Hetzner and AWS are capability declarations. All three are refused before anything is ordered, by name, with what is missing and what to do instead — a flag that accepts a provider it cannot act on is a flag that fails after the first machine is billed. Provisioning those yourself and starting at Getting started works exactly as before.

Every proof so far is on one machine. Three agents on one workstation with ephemeral ports is a real cluster in every respect the code can tell apart — real gossip, real Raft, real elections, real partitions when a process is killed. It is not a real network, a real firewalld, or a real provider's routing. The first multi-machine run will find things, and this page will be edited when it does.

Where to go next

Getting started — creating the cluster in the first place, and a SoliDB cluster on top of it.
Consensus — what is allowed into replicated state, and what a quorum round trip costs.
Stateful clusters — membership for SoliDB, SoliES and SoliKV, which is a separate thing from the cluster's own.
Operations — upgrades, backup, and the acceptance run.