1,100+ tests · rocky 10 · selinux enforcing gossip :7300 raft :7302 api :7304 rocky 10 / systemd 257

Proxy integration

Around thirty apps are in production behind soli-proxy. Migration is per app, reversible, and additive on the proxy side.

works today

one import reads the proxy's state read-only and shows what the cluster would make of it. Run against production it finds 31 apps, 28 of them symlinks — plus the things worth knowing before a migration.

And the seam is joined. The proxy has an external_routes table, PUT /api/v1/routing-table, and resolution that falls through to it; the node holding the scheduler lease builds the table from replicated state and pushes it, every few seconds.

Verified end to end against a real proxy on this workstation: one run --cluster declared a workload, the scheduler placed it, the agent started it and published the port it actually bound, and one alias set made a domain resolve to it. A request for that domain came back 200 with the workload's own body; an unknown host got 421 rather than being routed at random; and one alias rm cut the route within one push.

One pusher, not one per control node. The proxy rejects an out-of-order table, so several pushers would be safe — and would spend their time issuing 409s at each other while the table lagged behind whichever one last won. The index is the Raft log index: a local counter restarts at zero on a new leader and a wall clock can go backwards between two machines, and the proxy needs it to increase across exactly that event.

Reading the proxy without touching it

$ one import --proxy-root /home/rocky
APP                              SLOT      PORT  START
agents.solisoft.net              green    20027  soli serve . --port $PORT …  (symlink)
crm.solisoft.net                 blue     20000  soli serve . --port $PORT …  (symlink)
…
31 app(s), 28 via symlink
  unrecognised: .claude — no app.infos start_script, no luaonbeans.org, no app/models
  port 9022 is leased to openworld.solisoft.net, which was not found
  …
nothing was written: this reads the proxy's state and stops there

It exits 3 when the view is degraded, on the same contract as one nodes: a check that cannot tell "all understood" from "I could not read part of it" is worse than no check — and this one runs before a migration.

The two categories it reports are the ones nobody would think to look for. A directory matching no detection rule is not silently dropped: the proxy would not start it either, so the import is right, but an app the importer cannot explain is exactly the one to look at first. And a port leased to an app that no longer exists means either a lease was never released or the scan is pointed at the wrong root — both worth knowing before trusting a single line of the output.

The seam

AppManager::get_running_app_domains() is the single source feeding route resolution, ACME registration and static-rule syncing. One added field makes the cluster able to contribute to it: 

external_routes: Arc<ArcSwap<HashMap<String, Vec<Target>>>>

get_running_app_domains() returns own ∪ external, and a new PUT /api/v1/routing-table endpoint swaps the map, guarded by a monotonic index so an out-of-order push is ignored. Roughly forty lines, purely additive, and routing, ACME and rule syncing all keep working unchanged.

get this right on day one

Define the table as Host → Vec<Target { url, weight }> — full URLs, not ports. On a single node the URLs are http://localhost:<port>; multi-node they become http://10.0.0.12:<port>. Getting this shape right now means the move to multi-node needs no further proxy change at all.

Most apps have no config file

Only two of roughly thirty-six apps have an app.infos. The rest rely on the proxy's auto-detection. An importer that only reads app.infos would miss almost every app in production, so it has to replicate the detection logic exactly.

Ports must not move

There is an inconsistency worth knowing about: the app config default says 20000-30000 while the allocator's own constants say 30000-40000. The allocation path passes the app's config, so production is on 20000-30000.

The cluster allocator adopts that range, and the importer reads existing leases verbatim. Zero port churn at cutover — which matters when thirty-six apps sit behind ACME certificates and firewall rules keyed to those ports.

Migration, one app at a time

  1. Shadow. The agent observes and supervises nothing. one ps shows what it would have done, diffed against the proxy's own view for a couple of weeks.
  2. Adopt without restarting. A transient scope with an explicit PID list takes a running app into a cgroup without touching it — visible in one top, still supervised by the proxy.
  3. Cut over. Per app, never in bulk. The current port is pinned as a static lease so the route does not move; start on the other slot, health-gate, flip. Stopping cleanly through the proxy is mandatory — removing a directory leaves an orphan process holding its ports.
  4. Demote the proxy. Supervision, port allocation and the sites watcher come out. Routing, TLS, ACME, Lua, circuit breaking and metrics stay.

Rollback is per app and immediate: clear the flag and the proxy rediscovers it.

the one step left, and it needs a human

No app has been cut over yet. Every mechanism above exists and is tested; what does not exist is a rehearsal. That gap cannot be closed by code or by this page, and writing the runbook from the design rather than from a run is exactly how a runbook comes to describe a procedure nobody has performed.

The next action, in order, on the lowest-traffic app:

  1. one import --proxy-root /home/rocky and diff against GET /api/v1/apps until the two agree. They should already.
  2. Declare it for the cluster with its current port pinned, and check one ps and the pushed routing table name the same port.
  3. Stop it through the proxy — not by removing a directory, which leaves an orphan holding its ports — and flip the alias.
  4. Watch it for a day. Then write this section again, from what happened, including whatever went wrong. Especially whatever went wrong.

Until that is done, treat the four steps above this note as a design, not as a procedure. They are the difference between a migration that is understood and one that has been performed.

The hidden blocker: ACME across several proxies

one piece left, and it is the one with a cost

Challenge distribution works. A proxy reports the HTTP-01 tokens it needs proven on GET /api/v1/acme-challenges and accepts a set on PUT, and the node holding the scheduler lease now collects every proxy's tokens and gives the union to all of them. Let's Encrypt validates over the cluster's public address and lands wherever routing sends it, so every node holding every token is the only arrangement where the request can land anywhere.

No private key crosses that. A challenge token is served to anyone who asks for the challenge URL — that is what HTTP-01 is — so distributing one grants nothing that was not already public. Which means each proxy can keep ordering its own certificate with its own key: the duplicate-certificate limit is five per week for an identical domain set, and three nodes renewing on a sixty-day cycle order three times per cycle. The rate limit was the reason to share certificates, and it does not bite at this size.

Verified against two real proxies: a token known only to one was provable on both after a distribution pass, and stopped being provable on either when it was abandoned. The pass also refuses to push when no proxy could be read, because the push replaces rather than merges and an empty union would take every live token out of service at once — a lab caught the first version doing exactly that, by reading a response shape it did not recognise as an empty set.

What is left is the shared certificate store, and it is the piece with a cost rather than a piece of plumbing: putting certificates in replicated state puts a private TLS key on every control node. It is not needed at three nodes. It becomes needed at the size where per-node ordering hits the weekly duplicate limit, and that is a security posture to choose deliberately rather than acquire by growing.

The elected orderer that store would need — one ACME account per region, a Raft lease choosing who orders, a rate-limit ledger — is written and tested as soli-one-acme, but it is a library only: no binary links it yet, and nothing above depends on it.

What the proxy side actually does

Three additions, each doing nothing until a table arrives — which is how they were able to ship beside thirty-one running apps:

  • PUT /api/v1/routing-table takes the complete set — not a delta. A missed push then self-corrects on the next one, and removing a route needs no separate call.
  • A monotonic index. Pushes can arrive out of order — a retry overtaking the write that superseded it. Without the index, a late old table silently reinstates routes that were deliberately removed, and it looks exactly like a rollback nobody asked for. A stale push answers 409, so the pusher can tell refused from applied without parsing a body.
  • Resolution falls through, own apps first. During a migration a domain may briefly exist on both sides, and the node that actually holds the process has to win. Preferring the pushed table would hand traffic to a workload that may not have started yet.

The URL is parsed at push time, not at request time. A malformed URL rejected on push is one error message for the pusher; the same URL rejected on a request is a 502 for a real user with no explanation.

sync_routes and ACME registration now work from own ∪ external. A static rule left in proxy.conf for a cluster-managed domain would shadow the pushed route, and the symptom — one domain still reaching the old backend after a migration — is the hardest kind to spot because everything else works.

The proxies' API keys

The agent names each proxy with --proxy <admin URL>. The key the proxy's admin API expects is not passed on that line: an argument is readable in /proc/<pid>/cmdline and in ps by every account on the host, workloads included. The old --proxy <url>,<key> form is refused at start rather than accepted with a warning, because the warning would be read after the key had been in ps for as long as the agent ran.

The keys live in one file, one <admin-url> <key> per line, # for comments. Each --proxy is matched to its key by URL rather than by position, so a reordered flag cannot send one proxy's key to another. The agent reads --proxy-keys-file, or else the systemd credential proxy-keys, which keeps the file readable by root alone on disk: 

install -m 0400 /dev/stdin /etc/soli-one/proxy-keys <<'KEYS'
# admin URL               key
http://10.0.0.9:9090      <key>
http://10.0.0.10:9090     <key>
KEYS
systemctl edit soli-oned
#   [Service]
#   LoadCredential=proxy-keys:/etc/soli-one/proxy-keys
systemctl restart soli-oned

A drop-in rather than a line in the shipped unit, where it sits commented out: systemd refuses to start a unit whose LoadCredential= file is missing, and most nodes push to no proxy at all.

One constraint on the proxy itself

soli-proxy resolves run/logs/, run/app_state.json, run/ports.lock and certs/ relative to its working directory, with no flag or environment override. Two ways forward: make those configurable, or run the proxy as a managed workload whose working directory points at a cluster-managed volume. The second works today with no proxy change and is the right first step.