Scheduling
Where work goes, which domain resolves to it, and which port it holds.
Placement with capacity, labels, anti-affinity and rack spread; grace periods, cooldowns and a disruption budget; the alias table with guarded flips; cluster-wide port leases; and the routing table the proxy consumes. Fifty unit tests on the pure planner, fifteen against a real Raft member.
It is wired in. Every control node runs the loop and its
lease decides which one actually plans, so a node that loses the lease
cannot keep writing placements it no longer owns. one run
--cluster declares a workload, the scheduler picks the nodes, and
each chosen agent starts it.
Verified on this workstation: two replicas of one workload landed on two different nodes, and each node started only its own. Nothing wrote a file.
Artifact replication runs on a timer, and so does the routing-table push. Both were listed here as decisions nobody called. The lease holder now places artifact copies across racks every twenty seconds and pushes the routing table to each configured proxy; every node fetches the copies it is assigned and releases the ones it is not.
The lease, not Raft leadership, is what says which node owns those decisions — the same lease the scheduler holds, so the three cannot disagree about who is in charge.
Nothing on this page is undriven any more. Cross-region routing was the last of them: the WAN pool carries the region summaries it reads now, and suspension has the proxy's per-app last-request time. The overview has what each policy does with its input.
Three consumers, not three mechanisms
None of this adds a coordination primitive. The scheduler is a named lease plus a watch; an alias flip is a guarded transaction on a version; a port claim is a compare-and-swap on an absent key. Consensus already provides all three, so what is left here is decisions rather than machinery — which is exactly where the thinking should be.
The planner is a pure function
plan(workloads, nodes, allocations, policy, now) -> decisions.
No I/O, no clock read, no randomness. Every rule below is therefore provable
at the desk, which matters more here than anywhere else: scheduler bugs
present as "the site was down for four minutes and now everything has
moved", and that is not a thing to debug in production.
It also has to be deterministic, not merely usually so. Every tie ends on the node id and every iteration order is sorted. A tie broken by hash order places on A, then prefers B for no expressible reason, and the workload moves forever while every dashboard shows it healthy.
Nothing is written by plan. The caller applies the decisions,
which is what makes a dry run a real dry run rather than a second code path
that drifts.
Choosing a node
The tie-break order is the algorithm:
- Spread. Fewest instances of this job already in the same rack or datacenter. Availability first — two replicas in one rack is one rack's worth of availability, sold as two.
- Best fit. Among nodes that still fit, the one with the least room left. Packing tight is what makes unlimited preview environments affordable; spreading by default leaves every node half full and doubles the fleet the price model has to carry.
- Node id. So the answer is a function of the inputs.
Capacity committed earlier in the same pass counts. Without that, two workloads sized for one slot both land on it and the node OOMs — the classic single-pass scheduler bug.
When a node goes away
| State | Keeps its work | Takes new work |
|---|---|---|
alive | yes | yes |
suspect | yes | no |
down | until the grace period lapses | no |
left | no — a drain is a decision, not a failure | no |
| cordoned | yes | no |
Suspect is the row that matters. SWIM marks a node suspect on a dropped packet far more often than on a real failure. Relocating on suspicion restarts production every time the network hiccups, and the restart is felt by users while the node was never actually gone.
Two more brakes on the same instinct. A cooldown leaves a recently-placed instance alone regardless of what the scores say, so a flapping node cannot bounce one workload around the cluster. And a disruption budget caps moves per pass: ten workloads losing their node do not stampede onto the survivors together, each pulling artifacts and starting and health-checking at the same instant. Deferred moves are not failures and are not reported as such.
Two refusals
Almost no usable nodes visible → refuse the whole pass. When the scheduler can see no usable node, the overwhelmingly likely explanation is that it lost sight of the cluster, not that every machine died at once. Acting on that view relocates the entire fleet onto whatever it can still see. Refusing costs one pass; being wrong costs the cluster.
Placing new work is exempt. The guard exists to protect what is running; a cluster that cannot schedule anything because it once saw a degraded view is a different outage.
A key this build cannot decode → pause, do not act. A spec written by a newer node is not a spec that was deleted, and the two call for opposite reactions. Every read returns what it could parse and what it could not, and a pass with anything in the second list does nothing at all. One version-skewed key costs a paused scheduler; collapsing the two cases means a rolling upgrade stops production.
Pinned workloads are never moved
solidb holds a node-local volume. Starting a second copy
elsewhere is not failover — it is two databases writing to two disks and a
reconciliation nobody wants. When its node dies, the instance is reported
stranded, with the reason, and left alone.
That is the honest answer, and it is surfaced rather than buried: one
ps carries a RELOC column and one node drain
refuses to empty a node holding pinned work without naming what would be
interrupted. Every system that hides this burns its operators exactly once.
Why "no node available" is never the message
It is true and useless. The difference between everything is cordoned and nothing has 48 GiB free is the difference between a one-command fix and a purchase order, and the scheduler is the only thing that knows which it was.
cannot place demo[0]: every usable node is cordoned, suspect,
or has not reported its capacity
cannot place huge[0]: of 3 matching node(s): 3 lack 49152 MiB of free memory
cannot place logs[0]: of 3 matching node(s): 2 would fall below their free-disk
floor (10% of the disk, at most 10 GiB) after reserving 20480 MiB;
1 lack 4096 MiB of free memory
cannot place eu[0]: no node carries the required labels [("dc", "par-1")]When every matching node was turned away for a resource, the message counts them per resource. One word for all of them — the old message said memory whatever the cause — sends an operator to buy the wrong part.
No workload fills a disk
Every node keeps a free-disk floor whatever it is given: 10% of its disk, at most 10 GiB, so a 40 GiB disk keeps 4 and a 4 TiB disk keeps 10 rather than 400. It applies to a workload that declares no disk at all, which is most of them. Without it, a workload asking for nothing fits on a disk with nothing left — and on a mixed fleet best-fit prefers the small old box, which is often the one whose disk is full. A full disk takes the journal, the Raft log and every workload on the node down with it.
one run --cluster --name logs --memory-mb 4096 --disk-mb 20480 -- /usr/bin/logs
--memory-mb is the unit's MemoryMax=, and placement
now reserves it; --disk-mb is what the workload needs on top of
the floor. Both are counted against what earlier placements in the same pass
and the work already running there have reserved. --disk-mb is a
reservation, not a quota: nothing caps what the workload writes. A node that
could not read its disk reports zeros, and is given no work until it can.
The floor is checked when work is placed. A disk that fills after
placement moves nothing and raises nothing in the cluster yet —
one df and one metrics show it.
A blocked instance is reported every pass and written never. Recording it would make the next pass believe it was handled, and a workload silently not running is the worst outcome available. It is also reported once, not twice — an instance that is both orphaned and missing is one problem, and two alerts for it is how alert fatigue starts.
Aliases
An alias is a domain pointed at a workload. Repointing one is the rollback primitive: the bodies change, the process ids do not, and it is a single guarded entry rather than a redeploy.
txn(
guards: [VersionIs("alias/x.soli.app", observed)],
on_success: [Put("alias/x.soli.app", "web-b"),
Put("alias/x.soli.app/previous", "web-a")],
on_failure: [],
)The guard is what makes a flip safe to retry: two operators rolling back at once must not produce a state neither of them asked for. Creating an alias guards on absent instead, so two tenants claiming one domain do not silently resolve in favour of whoever wrote second.
The routing table
Computed in the cluster, not in the proxy, so "which backends does this domain have" has exactly one answer — and so a proxy that is behind is behind by a whole table rather than by half of one.
x.soli.app → [ http://10.0.0.11:20001, http://10.0.0.12:20001 ]
Full URLs, not ports, from the first day. On one node these
read http://127.0.0.1:20001; on a cluster they carry the node's
address, and nothing on the proxy side has to change to go multi-node.
Defining this as a port is the decision that costs a migration later.
Two entries are deliberately absent from the table:
- An instance placed but not yet holding a port. It is starting, not serving; routing to it produces connection-refused for real users on every deploy.
- A domain with no reachable backend. Omitted entirely, never emitted with an empty target list — an empty list makes the proxy answer 502 for a domain it could have left to its static configuration, turning "one workload is down" into "this domain is broken".
Port leases
Ports are unique per node, not per cluster. Two workloads on different machines may both hold 20001; pretending otherwise would exhaust a ten-thousand-port range at a few hundred nodes.
The preferred port is derived from the job and instance index, so a workload lands on the same port every deploy on every node. That is worth more than it looks — firewall rules, log greps and an operator's muscle memory all key on the port, and one that moves on every restart invalidates all three quietly. Replicas derive different ports, because they may share a node.
From there it is a compare-and-swap on an absent key: N agents racing produce exactly one winner and the losers are told who took it, without a second round trip. The probe is bounded — ten thousand failed swaps is ten thousand quorum round trips, and the useful outcome is an error, not a stalled pass.
Only the holder may release. A node back from a partition would otherwise drop the claim of whoever took the port after it, and two workloads would then bind the same port on the same machine.
The agent already allocates ports locally, with a real
TcpListener bind as the veto. That answers "is this port free
on this machine right now" — necessary, and not sufficient: it forgets
across agent restarts and cannot see a claim made from another node. The
Raft lease makes the claim durable and arbitrated; the local bind makes it
true. Both, in that order.
One scheduler, and only one
Two schedulers on one cluster is the worst failure this design has: both read
the same view, both decide to relocate, and every workload ends up started
twice. The guarantee is the scheduler lease — so it comes from
consensus rather than from a lock file or a convention.
The holder renews at a third of the TTL. Renewing at the deadline means already having lost it once the write's quorum round trip is counted. On a clean shutdown the lease is handed back rather than left to lapse, so the successor starts in milliseconds instead of after a full TTL.
Removals are written before placements. A move written before its removal leaves both allocations live for the width of one write, and the agent on the old node has every reason to keep running its copy during that window — which is the double-run the whole layer exists to prevent.
And the assertion that guards all of it: a second pass over an unchanged cluster writes nothing. A scheduler that rewrites allocations every pass makes every agent see a change every pass, and the fleet restarts on a timer while looking perfectly healthy.