Workloads

Four kinds of thing to run, one spec, and an honest class for what must never move.

The claim, and its test

The agent does not fork its workloads. It asks PID 1 to. Everything awkward on this page is the price of that one property, so it is tested against the real soli-oned binary rather than a library harness — and with a workload that increments a counter rather than a sleep, because sleep proves presence and a counter proves it is still executing.

1. start the agent; it starts the workload
2. assert the agent is NOT in the workload's parent chain
     ← without this the test could pass for the wrong reason
3. kill -9 the agent, assert it died on signal 9
4. assert the counter keeps advancing
5. assert MainPID is unchanged — not restarted, never interrupted
6. start a replacement agent
7. assert it ADOPTS: same MainPID, same InvocationID

Step 2 is the one people forget. Step 7 checks InvocationID rather than the PID alone: systemd assigns a fresh one to every unit start, so two consecutive runs of the same unit cannot be confused however PIDs are recycled — a class of bug the proxy's PID comparison is exposed to.

Catching why, before systemd forgets

CollectMode=inactive-or-failed frees the unit object as soon as it stops being active. That is not optional — without it a failed unit holds its name forever and the next start fails with UnitExists, which is exactly the state a crash-looping job is in. The cost is that the exit status cannot be read after the fact: by the time a reconcile pass notices the unit is gone, there is nothing left to ask.

So it is captured from the PropertiesChanged stream as it changes. systemd emits that signal with a null property list, which makes the bus include every emits-change property at once — so one signal carries ActiveState, MainPID, ExecMainCode, ExecMainStatus and Result together, with no follow-up calls.

One match rule for the whole agent, on path_namespace, rather than one per unit: at a few hundred workloads that is the difference between a working bus daemon and a wedged one. And the unit's object path is derived from its name rather than looked up, because looking it up races — a workload that fails instantly is gone before GetUnit answers, and those are precisely the crashes worth diagnosing.

The result is a diagnosis rather than a shrug:

web: exited with status 7; retry 1/3 in 1s
api: OOM-killed — the workload exceeded its memory limit; retry 2/3 in 2s
job: quarantined: exited with status 1 — 4 consecutive failures

The OOM line is the one that pays for the machinery. An OOM kill and a kill -9 both arrive as CLD_KILLED with signal 9; only Result=oom-kill separates "raise the memory limit" from "someone killed it", and only one of those recurs deterministically at 3am.

The counter has to outlive the allocation

A restarted job gets a fresh allocation and a fresh unit name. A failure counter living in the allocation therefore resets on every crash, and "3 consecutive failures" could never be reached — a crash loop would spin forever looking healthy. The counter is keyed on the job, and there is a test that drives a job past its budget and asserts it stops being restarted.

Ports are derived, not counted out

A job's port comes from a hash of (job, slot), so it gets the same port every time it is placed. Firewall rules stay valid across a restart, dashboards keep meaning the same thing, and an operator who remembers "billing is on 24107" stays right. Sequential allocation gives none of that and reshuffles on every agent restart.

The range is 20000-29999, matching what the proxy actually uses today — which is what makes a migration cost zero port churn, and that matters when ~36 apps sit behind ACME certificates and firewall rules keyed to those ports. one import takes existing leases verbatim.

The node keeps a veto. The hash says which port should be free; only a real bind says whether it is. Something outside the cluster's model can be holding it, and starting a workload that cannot bind turns a clear allocation error into a confusing health-check failure several seconds later. A port that fails to bind is blacklisted for a day, so the allocator does not pick the same occupied one on every pass forever.

$PORT and $WORKERS are substituted into the command — the same two placeholders the proxy supports, and with no shell involved, so nothing here becomes command injection. One fix along the way: the proxy's naive replace turns $PORTAL into 8080AL; a placeholder only substitutes here when what follows cannot continue an identifier.

Batch and scheduled jobs

[[job]]
id       = "nightly-backup"
command  = ["/usr/bin/backup.sh"]
schedule = "*-*-* 03:00:00"
timeout_secs = 3600

A schedule alone makes it a batch job — nobody writes a cron expression for something meant to stay up. The unit becomes Type=oneshot, and that difference is not cosmetic: under Type=exec a process that exits is a workload that died, so a batch job declared as a service looks like a crash to the restart policy on every successful run.

The timeout is not optional. A batch job with none that hangs holds its slot forever and every scheduled run piles up behind it — which is how a nightly job becomes an outage three days later.

The timer and the service it drives are created in one call, with the service riding in StartTransientUnit's aux argument, so there is never a moment where one exists without the other. And the allocation id is derived from the spec rather than drawn at random — unlike a service, where two starts are two different runs. A random id would make every reconcile pass create a new timer beside the old one, and within an hour the node would hold hundreds of timers for one job, all firing.

Secrets

Never the environment — that leaks through podman inspect and /proc/<pid>/environ, both readable by anything that can see the process. Never baked into the artifact — every backup of that artifact would then contain the secret. systemd does the delivery, which means no cleanup path of ours can leak it.

Suspend and resume

The economics of the product live here. Soli Cloud sells unlimited preview environments, and a Soli deployment is a long-lived process — an idle preview holds its memory twenty-four hours a day. Without a way to reclaim that, either previews get capped or the margin goes.

Only Stop is scale-to-zero. Freeze holds every page exactly where it was; it is a latency optimisation for a paid warm tier, and treating it as a way to fit more previews on a node makes a capacity plan wrong by however much the frozen tenants were using.

The frozen state is read back from FreezerState, not assumed from the request. Freezing can fail, and an agent that believes its own request has a workload it thinks is asleep and is not.

Why not reuse the proxy's app config

soli-proxy already supervises processes well — blue/green slots, health-gated promotion, quarantine after three crashes, port allocation verified by a real bind. But its AppConfig is shaped for web apps: domain, health_check, port_range_start. There is no free-form command, no restart policy, no notion of a job that finishes.

So the workload spec is a strict superset of it. That is not aesthetic: if it were not a superset, the roughly thirty apps in production could not be migrated without rewriting each one.

The shape

WorkloadSpec {
  kind:        SoliApp | Process | Container | Batch
  replicas:    Fixed(n) | PerNode | Singleton
  resources:   cpu_millis, memory_mb, disk_mb
  placement:   constraints, affinity, dc/rack spread
  restart:     Always{backoff} | OnFailure{max} | Never
  update:      BlueGreen{health_gate} | Recreate | Never
  health:      Http{path} | Tcp | Exec | None
  ports:       Dynamic | Static(u16) | Host
  mounts:      Artifact{digest} | Tmpfs{secrets} | Pinned{path}
  relocatable: bool
}

Supervision through systemd

The agent does not fork its workloads. It asks systemd to, over D-Bus, as transient units named soli-one-<workload>-<alloc>.service. That buys several things outright:

• cgroups v2 — MemoryMax, CPUQuota, TasksMax. The per-tenant quotas the PaaS needs are expressible directly.
• journald — log capture and rotation, with indexed per-allocation lookup.
• Per-workload hardening — ProtectSystem=strict, PrivateTmp, NoNewPrivileges, a dropped capability set.
• Workloads outlive the agent. systemd owns the units, so restarting or upgrading soli-oned does not touch running work. On start the agent lists units by prefix and re-adopts them.

That last point is the strongest argument for the whole approach, and it is testable: kill the agent with -9, and the workload keeps serving.

Who a workload runs as

The class that must not move

soli-sfu holds long-lived UDP media sessions. Blue/green deploying it or migrating it drops calls, and no amount of orchestration cleverness fixes that. So it is expressible directly:

relocatable = false
update      = Never
ports       = [ Host(3478/udp) ]

The scheduler never relocates it on its own. On drain it stops being advertised for new sessions while the existing ones finish — and the refusal is enforced in the execution layer, not only in the scheduler, so one buggy reconcile cannot issue a stop.

Naming, and the bug it prevents

Job ids contain dots (acme.soli.app) and systemd slice names use - as their hierarchy separator. Both matter more than they look.

Under the obvious slug rule — lowercase, map everything outside [a-z0-9] to _ — these three collapse to the same string:

acme.soli.app  ┐
acme-soli-app  ├─>  acme_soli_app
acme_soli_app  ┘

Three tenants, one unit name. Deploying the first stops the second's workload, and nothing in the logs says why. So a hash of the original is appended whenever the transformation lost information, which makes the function injective while keeping simple names typeable:

acme           -> acme
acme.soli.app  -> acme_soli_app_5d2e1f04
acme-prod      -> acme_prod_77c0e9f1

Stripping - is the other half: without it, a tenant named acme-prod would land inside acme's slice subtree and inherit its memory limit.

The slot is in the unit name deliberately. The reconciler keys on (job, slot), so it has to recover the slot from a unit it finds running — and the case that matters is exactly when the local state file is gone, which is when the two halves of a cutover would otherwise be indistinguishable.

Twelve jobs capped at 512 MiB each are also collectively capped by their tenant slice, enforced by the kernel's cgroup hierarchy, with no accounting code in the agent at all.

Six D-Bus traps, written down once

Each of these has a test asserting the D-Bus signature the builder produces. Without them the failure is a systemd error naming neither the property nor the type.

Restart and quarantine

The proxy quarantines after three consecutive unexpected exits, and its counter clears only on an explicit deploy. Two consequences, both real: an app that crashes once a week eventually quarantines itself and nobody can reconstruct why; and systemctl restart soli-proxy silently launders every quarantine, so a crash loop survives an incident review by being invisible.

Here the counter resets after a window of stability, and quarantine persists across an agent restart. Two further distinctions the proxy does not make:

• A host reboot is never a failure. Otherwise a node rebooted four times during maintenance quarantines every job on it.
• A failed start job quarantines immediately. With Type=exec it means execve() failed — a missing or unexecutable binary. Retrying three times spends a minute reaching the same answer.

Mobility, stated plainly

This column is in one ps because it is the single fact that determines what happens when a machine dies. Every system that fudges it burns its operators exactly once.

Using every core for one build

one secret put sccache-scheduler-toml --from-file scheduler.toml
one secret put sccache-server-toml    --from-file server.toml

one run --cluster --name sccache-scheduler --secret sccache-scheduler-toml -- \
    /bin/sh -c 'SCCACHE_NO_DAEMON=1 exec sccache-dist scheduler --config "$CREDENTIALS_DIRECTORY/sccache-scheduler-toml"'
one run --cluster --replicas 3 --name sccache-server --secret sccache-server-toml -- \
    /usr/local/bin/sccache-server

# on the laptop
export RUSTC_WRAPPER=sccache
cargo build --release
sccache --show-stats          # "Successful distributed compiles", per server