The Journey So Far

Milestone 1 keel-spec

The foundation: a YAML schema for describing a jail (image, command, network, resources, restart policy) plus the parsing and validation that turns YAML into a typed JailSpec. This is where the core invariants of the whole system were decided: what counts as a valid jail name, how cpu/memory strings get parsed into concrete limits, which fields are allowed to change on a re-apply and which are immutable for the life of the jail, and how CIDR addresses are validated. Thirteen unit tests plus four end-to-end tests, all running on any OS, no FreeBSD required.

Milestone 2 keel-jail, keel-zfs

The first milestone that actually touches FreeBSD. Two crates, each built around the same pattern that carries through the rest of the project: a trait describing what the crate does (JailRuntime, ZfsManager), an in-memory Fake implementation for fast tests on any machine, and a real implementation that shells out to jail(8), jexec(8), rctl(8), and zfs(8) on FreeBSD.

Getting the real implementations right took real hardware: is_running first miscounted zombie processes as "running" until tested against an actual jail; ps invocation syntax that looked right on paper didn't parse the way FreeBSD expected; a zfs snapshot race under parallel tests had to be made tolerant of losing that race rather than erroring.

Milestone 3 keel-net

Adds keel-net and its NetManager trait: creating bridges, attaching a jail to one over an epair(4) pair with a static address, and tearing that down again. This milestone is also where keel-jail::create gained a vnet parameter, since VNET-enabled jails need to be created differently from the start, an early breaking change caught before it could compound. By this point the "verify on the real VM before writing the plan" discipline was fully in place, and Milestone 3 shipped with zero fix rounds across all five of its tasks.

Milestone 4 keel-agentd, the reconciliation core

The milestone that ties everything together. Reconciler<J, Z, N> is generic over the three runtime traits from Milestones 1 through 3, so it can be instantiated against the Fake* implementations for fast, FreeBSD-free testing today, and against the real Process*/Cli* implementations once a later milestone wires it up to an actual daemon.

Its public API is small on purpose: new, apply, delete, reconcile. Underneath, seven tasks built it up in layers: a JailRecord with the naming and path derivation rules (jail names, dataset paths, epair names), a crash-safe state store that writes to a temp file and renames rather than risking a torn write, a per-jail BackoffState (starts at one second, doubles up to a five-minute cap, resets after sixty seconds of stable uptime), the provisioning path with automatic rollback on partial failure, and finally the public reconcile() that runs the whole desired-versus-observed diff for every jail in one pass, returning a list of per-jail failures so one broken jail never blocks the others from being reconciled.

This was also the milestone where the review discipline paid for itself most visibly. Three real bugs were caught, not by the first pass of tests, but by treating every review (per-task, then a final whole-branch pass on top) as a genuine adversarial check rather than a formality:

• delete() assumed that tearing down a jail, its dataset, and its resource limits were all safe to call on something that was never actually created, matching how network detach already behaved. Only the network side turned out to be built that way; the others needed the same tolerance added explicitly, for the real case of deleting a jail that was applied but never got as far as being provisioned.
• A test for crash-loop restart asserted that a jail would restart with zero elapsed time, but the backoff cooldown from the initial provisioning was, correctly, still armed at that instant. The bug was in the test's timing, not the reconciler.
• The final whole-branch review caught a real one: a failed restart attempt never armed the backoff cooldown at all, because the code returned early on error before reaching the line that would have armed it. Successful restarts were protected; failing ones, exactly the crash-loop case backoff exists for, were not. Fixed with a regression test that injects a restart failure and proves the cooldown now engages.

Milestone 4 finished at 71 tests, all passing, all still running without touching FreeBSD.

Milestone 5 local HTTP API + keelctl

The first milestone where keel-agentd actually runs: a binary that wires the real ProcessJailRuntime/CliZfsManager/ProcessNetManager implementations into a Reconciler, drives it on a 5-second timer, and serves apply/get/delete over a local HTTP API on a 0600 Unix socket, plus keelctl, the companion CLI. A single worker thread owns the Reconciler exclusively; the HTTP server and timer only ever reach it through a command channel, so apply/delete take effect immediately (reconciled before the caller's response) without introducing a second concurrent owner. Everything is still hand-rolled and dependency-light, in keeping with the rest of the project: a small HTTP/1.1 parser (httparse) over a blocking UnixListener, YAML wire bodies reusing the spec's existing serde_yaml, no async runtime.

Milestones 1-4 were tested entirely against fakes; Milestone 5's FreeBSD VM verification was the first time the whole stack ran against the real system end-to-end, and it promptly found two real bugs that no fake-backed test could have caught, both in code shipped since Milestone 2:

• keel-jail's destroy only ran jail -r, never reaping the process start_command had spawned into the jail. The kill left a zombie holding a reference into the jail's rootfs mount, so an immediately following dataset teardown failed with "device busy", exactly the sequence Reconciler::delete runs on every deletion of a jail with a running command. Fixed by tracking spawned children per jail name and blocking-waiting on only the destroyed jail's own children.
• Independently, even with that fixed, zfs destroy could still fail with "dataset is busy" for a brief window after jail -r returns, a real kernel-level mount-release timing gap, reproducible from a plain shell with no Rust involved. Fixed with a short bounded retry, the same pattern already used for clone_from_base's snapshot race.

Milestone 5 finished at 96 tests on macOS, plus a clean 5-for-5 real apply → running → delete cycle against actual jails, ZFS clones, VNET networking, and rctl limits on the FreeBSD VM.

Milestone 6 rc.d service + smoke test

The milestone that turns keel-agentd from "a binary you run in a terminal" into a real FreeBSD system service, closing out sub-project 1. No new code was needed for daemonization itself: an rc.d script wraps the unchanged Milestone 5 binary with the base system's daemon(8) utility, using -r for restart-on-crash and -S to route its output to syslog. The only Rust changes were two eprintln! call sites (daemon startup, per-jail reconciliation failures), no logging framework, no daemonize/signal-hook dependency, no custom SIGTERM handler, since the existing "jails outlive the daemon" design already makes the default terminate-on-signal behavior correct. A committed smoke test script (scripts/smoke-test.sh) proves the whole lifecycle end-to-end: install, start, apply a real spec, kill keel-agentd directly to simulate a crash, confirm both the automatic restart and correct state recovery with no duplicate jail, stop the service and confirm the jail keeps running, clean teardown.

One design subtlety mattered before any code was written: daemon(8) takes two different pidfile flags, and combining -r with the wrong one (-p, the child's pid) means service ... stop kills the child directly while daemon(8) is still watching and immediately restarts it, the stop command would silently do nothing. Verified directly on the VM before committing to the design; the rc.d script uses -P (the supervisor's own pid) throughout.

Full VM verification surfaced two more real bugs, neither reachable by any fakes-backed test, since both are genuine OS/supervisor-interaction characteristics that only exist under a real daemon(8)-supervised run:

• keel-jail's start_command spawned the jailed process with Rust's default stdio inheritance, so a long-running jail held keel-agentd's own stdout/stderr open. Under daemon(8) -S, those are the write end of a pipe daemon(8) reads to detect (via EOF) that keel-agentd itself died and needs restarting, an inherited pipe held open by the jailed workload meant daemon(8) silently never restarted a killed keel-agentd whenever any jail with a running command was active. Root-caused on the VM with procstat -f, showing the jailed process's fd 0/1/2 as a pipe shared with daemon(8)'s own relay pipe; fixed by giving the jailed process its own /dev/null stdio.
• The smoke test's own "jails outlive the daemon" check was a permanent false negative: jls's default columns never include the jail's name, only its dataset path, so grep-ing for the name-prefixed string could never match regardless of whether the jail was actually still running. Fixed with a direct jls -j <name> lookup instead.

Milestone 6 finished at 97 tests on macOS (96 plus one new regression test pinned to the exact stdio-leak mechanism via procstat), plus two consecutive clean end-to-end smoke test runs on the FreeBSD VM.

Milestone 7 keel-controlplane, the node registry

The first milestone of sub-project 2 (the multi-node control plane), deliberately scoped to answer only "which nodes exist, and are they alive", no scheduling, no placement, no spec-forwarding, all of which stay separate future sub-projects. A new crate, keel-controlplane, follows the exact same shape keel-agentd already established: a single worker thread exclusively owns an in-memory Registry, reachable only through an mpsc command channel, fronted by the same hand-rolled httparse-based HTTP server keel-agentd uses for its local API, just over a TcpListener instead of a Unix socket, since nodes live on separate hosts. Three endpoints: register a node, heartbeat a node, list all nodes with their liveness computed on read (Alive within 15 seconds of the last heartbeat, Dead after) rather than tracked by a background sweep.

keel-agentd gained three new, entirely optional CLI flags (--node-id, --control-plane-addr, --advertise-addr) and a registration.rs background thread: register once at startup, heartbeat every 5 seconds, and treat any heartbeat failure, a rejected/unknown node or a connection error, identically, by simply re-registering on the next tick. There is deliberately no persistence anywhere in this milestone and no backoff: a control plane that restarts forgets every node, and every node notices and re-registers within one heartbeat interval, the same self-healing-over-durability bet the reconciler already made in Milestone 4, just applied to cluster membership instead of jail state.

One constraint surfaced only once tests were being written, not before: the design assumed an in-process test could simulate "the control plane restarts" by dropping a TcpListener and rebinding a fresh one to the same address. Verified directly that this doesn't work, std::net::TcpListener::bind fails with "Address already in use" as long as the original socket is still alive and listening in another thread, regardless of SO_REUSEADDR. That specific behavior, the one genuinely OS-level part of this milestone, was verified for real instead, against three separate FreeBSD VMs: all three nodes registered and showed Alive within one heartbeat interval; killing keel-agentd on one node flipped only that node to Dead after the 15-second threshold, the other two unaffected; restarting keel-controlplane emptied the registry ([]), and both surviving nodes' own logs showed the expected heartbeat failed: control plane returned status 404 → re-register cycle, landing back at all-Alive within about one heartbeat interval with neither node process ever restarted.

Milestone 7 finished at 122 tests on macOS (96 inherited, 26 new across keel-controlplane's registry/worker/http layers and keel-agentd's new CLI flags and registration client), zero warnings, final whole-branch review clean with no Critical or Important findings.

Milestone 8 routing jail specs to a specific node

The second milestone of sub-project 2, and the one that completes the multi-node control plane: given the node registry from Milestone 7, a client can now apply, get, or delete a JailSpec on a specific, named node by routing the request through keel-controlplane, rather than connecting to that node's keel-agentd directly. The caller always names the exact node; there is still no scheduler and no bin-packing, that stays a separate, not-yet-designed future sub-project.

keel-controlplane gained Registry::resolve and four new HTTP routes (PUT/GET/DELETE /nodes/<id>/jails/...) that forward a request byte-for-byte over a fresh TCP connection to the resolved node's keel-agentd, relaying the response back untouched. It never deserializes the spec body, and gained no new dependency on keel-spec or keel-agentd's wire types. Unknown or Dead nodes are rejected before any connection is attempted, not after a timeout. keel-agentd gained a second, entirely optional TCP listener serving the exact same jails API already served by its Unix socket, bound to --advertise-addr (whose contract changes from Milestone 7's undialed display string to a real, dialable host:port) only when the Milestone 7 control-plane flags are set; the Unix socket itself is completely unchanged. keelctl gained --control-plane-addr/--node flags to route through the control plane; omitting both preserves its exact prior behavior.

VM verification found and fixed a real, if minor, test-only bug: a test helper added for the forwarding routes read an incoming request with a single, non-looping socket read, while the real forwarding code sends it as two separate writes before shutting down its write half. Under load that could leave unread bytes queued when the test helper's stream was dropped, and a BSD-derived TCP stack can turn that into a spurious reset instead of a clean close. Fixed by draining reads to EOF before responding.

The VM run also turned up two real, pre-existing product bugs, both invisible to any fakes-backed test and both the same underlying gap: neither keel-jail's ProcessJailRuntime::destroy nor keel-zfs's CliZfsManager::destroy_dataset ever mapped a "doesn't exist" failure to the NotFound error variant Reconciler::delete's Milestone 4 tolerance already expects, so deleting a jail record whose provisioning failed before it was ever created (a scenario that had simply never come up in five prior milestones of VM testing) failed outright instead of succeeding as a no-op. Fixed in both crates by matching the real command's own "not found" stderr text, the same idiom already used elsewhere in each file, verified with new FreeBSD-only regression tests and a direct end-to-end reproduction of the original failure.

Milestone 8 finished at 139 tests on macOS (122 inherited, 17 new across keel-controlplane's forwarding layer, keel-agentd's TCP listener, and keelctl's routed mode), final whole-branch review clean, and full VM verification across three real nodes: a spec applied to one specific node landed there and nowhere else, routed get/delete worked, unknown-node and dead-node targets were rejected with clear errors before any connection attempt, and plain single-node keelctl usage stayed completely unaffected throughout.

Milestone 9 scheduler, automatic node placement

The first milestone of sub-project 3, and the one that finally answers "which node does an unscheduled jail land on": given the node registry and node-forwarding from Milestones 7-8, a client can now apply, get, or delete a JailSpec without naming a node at all. keel-controlplane gained a new in-memory Placements table (jail_name -> node_id), owned by the same single worker thread that already owns the node Registry, and a pure scheduler::pick_node function: among Alive nodes, pick the one with the fewest jails currently recorded in Placements, ties broken by ascending node id. Placement is by jail count only, not real CPU/memory pressure, since nodes report no capacity today; teaching them to is future work, if it's ever needed.

Three new routes, PUT/GET/DELETE /jails/<name> (no node segment in the path), sit alongside Milestone 8's unchanged /nodes/<id>/jails/<name> routes, which now also update the same Placements table on success, so jail counts and lookups stay consistent no matter which route family placed a given jail. Applying an already-placed jail is sticky, forwarding to the same node it originally landed on rather than re-scheduling; a jail whose recorded node has since gone Dead is never silently rescheduled elsewhere, it surfaces the same Dead error a named-node route already gives, and a human can still explicitly repin it via the named route. keelctl's --node flag became optional: --control-plane-addr alone now means "schedule it," while --control-plane-addr --node <id> keeps Milestone 8's exact-node behavior unchanged.

Milestone 9 finished at 161 tests on macOS (139 inherited, 22 new across keel-controlplane's Placements, scheduler, worker, and http layers), final whole-branch review clean, and full VM verification across three real nodes: applying two differently-named specs with no --node landed them on two different nodes, re-applying one of them kept it on its original node even after a third, emptier node joined, and plain named-node keelctl usage stayed completely unaffected throughout.

Milestone 10 resource-aware bin-packing

The second milestone of sub-project 3: the scheduler introduced in Milestone 9 picked a node by jail count alone; this milestone teaches it to consider actual CPU and memory instead. keel-agentd detects its own capacity once at startup via sysctl -n hw.ncpu/sysctl -n hw.physmem (the same shell-out idiom already used for jail(8)/zfs(8)), reports it once at registration, and reports its currently committed load, the sum of spec.resources.{cpu,memory} across every jail its own Reconciler tracks, on every heartbeat. keel-controlplane's scheduler::pick_node now ranks Alive nodes by min(free_cpu/capacity_cpu, free_memory/capacity_memory), the fraction of headroom in whichever resource is most constrained, rather than by jail count, while remaining completely opaque to JailSpec bodies: it never learns a node's numbers by parsing a spec, only by the node self-reporting its own aggregate totals.

This is a ranking heuristic, not admission control: the scheduler never sees an incoming jail's own resource request, since that would require deserializing the spec, which stays off-limits, so it can't guarantee a chosen node actually has room. A node that turns out unable to fit a jail simply fails that apply normally, the same honest limitation Milestone 9's count-based version already had.

Milestone 10 finished at 171 tests on macOS (161 inherited, 10 new across keel-agentd's new capacity module and Reconciler::committed_resources, and keel-controlplane's wire, registry, scheduler, worker, and http layers), final whole-branch review clean, and full VM verification across three real nodes: detected capacity matched direct sysctl queries exactly; a large jail committing most of one node's capacity landed there first (both nodes tied at registration), and a second, smaller jail then correctly landed on the other node, the one with more headroom fraction rather than fewer jails; sticky re-apply and named-node routing stayed completely unaffected throughout.

Milestone 11 control-plane shared-secret authentication

Milestones 7-10 built the multi-node control plane under a trust model each of those milestones' specs stated explicitly: same-network trust assumed, hardening deferred. Concretely, anyone who could reach keel-controlplane's TCP port or a keel-agentd's opt-in --advertise-addr listener could register a fake node, spoof another node's heartbeat, or apply/delete a jail on any node, with no credential of any kind. This milestone closes that gap for authorization, not yet for confidentiality: a single shared cluster secret, generated once by the operator and distributed as a file to every host, is now required on every request keel-controlplane and keel-agentd's TCP listener serve, checked via a hand-rolled constant-time comparison (no new dependency) at a single choke point in each crate's route(). Every outbound call this project already makes, keel-agentd's registration/heartbeat, and keel-controlplane's forwarding to a node, attaches the same token. keel-agentd's Unix socket keeps its original, unwrapped route() and is completely untouched: its 0600-permission trust boundary was never the gap this milestone exists to close. TLS/wire encryption is deliberately out of scope here, a separate future milestone; this one stops an unauthenticated actor from issuing commands, not a sophisticated on-path eavesdropper from reading the wire.

keel-controlplane, keel-agentd, and keelctl each gained a new --auth-token-file <path> flag: unconditionally required on keel-controlplane (it has no non-networked mode at all), and required alongside the existing control-plane flags on keel-agentd/keelctl (extending the pairing check Milestone 7 already established for --node-id/--advertise-addr), plain single-node usage of either binary is entirely unaffected.

Milestone 11 finished at 208 tests on macOS (191 inherited, 17 new across both crates' auth modules and the HTTP/registration/CLI wiring), and full VM verification across three real nodes: all three registered Alive against an authenticated GET /nodes; a scheduled apply/get/delete round trip succeeded end-to-end with auth in place; a keelctl call with a missing token file failed locally without ever reaching the network, and one with a wrong-but-present token reached the control plane and got a generic 401; a node restarted with a stale, mismatched token repeated registration failed: control plane returned status 401 every heartbeat and never appeared Alive until its token file was corrected; clean teardown confirmed on all three VMs afterward.

Milestone 12 mutual TLS for the control plane

Milestone 11's shared-secret token closed the authorization gap but left two things on record as its own stated limitations: no confidentiality (an eavesdropper on the LAN could still read the token and every spec body in flight), and no per-identity credential (one leaked token compromised the whole cluster, with no way to revoke just one node's or operator's access). This milestone replaces the token entirely with mutual TLS: every connection, client to control plane, node to control plane, control plane to node, is encrypted, and identity is proven by a certificate signed by a private CA the operator generates once, not by a string copied to every host. It is this project's first genuinely new runtime dependency, rustls pinned to its ring crypto backend (default-features = false, features = ["ring", "std"]) rather than the default aws-lc-rs, since the latter is C/assembly and would add a build-toolchain requirement this project has never had; certificate generation itself stays entirely in a new scripts/gen-certs.sh shelling out to openssl, not a new Rust dependency.

Every accepted connection is wrapped in a real TLS handshake requiring and verifying a client certificate before any HTTP request is parsed, which let Milestone 11's auth::check calls, and keel-agentd's route/route_authenticated split, disappear entirely: identity is now proven at the transport layer, so route() itself needs no auth-awareness at all, on either the Unix-socket or TLS-wrapped path. keel-agentd's Unix socket keeps calling that same, now-simpler route() unwrapped, exactly as it always has. One certificate per node (and one for the control plane itself, issued the identical way) doubles as both a TLS server certificate, when it's dialed, and a client certificate, when it dials out, matching the "one identity, no artificial split" principle node-id already embodied everywhere else.

Three real bugs surfaced only once the whole pipeline was exercised together, none caught by any single crate's own test suite in isolation. Twice, a read loop added to tolerate rustls's UnexpectedEof (its documented behavior for a TLS peer that closes without a close_notify alert) was copied without the accompanying Content-Length cross-check that makes the tolerance safe, letting a connection cut short be silently accepted as a complete response instead of erroring; one instance (keel-controlplane's own node-forwarding path) was genuinely exploitable, the other (keelctl's own response to the operator's terminal) was worse, since the truncated body is exactly what a human would see as if it were correct. Third, a cross-crate crypto-provider conflict: keel-controlplane::tls and keel-agentd::tls each guard rustls's one-time crypto-provider install with their own crate-local Once, and linking both into one test binary meant the second crate's first-ever call saw rustls's ordinary "already installed" Err and turned it into a panic instead of ignoring it, the same idiom rustls's own internal helper already uses for this exact case.

Milestone 12 finished at 202 tests on macOS (191 inherited from Milestone 11, minus the deleted auth modules' tests, plus new coverage across three tls modules and the HTTP/registration/CLI TLS wiring), and full VM verification across three real nodes: all three registered Alive against an authenticated GET /nodes over real mTLS; a scheduled apply/get/delete round trip succeeded end-to-end with real client certificates; a keelctl call with a missing certificate file failed locally without ever reaching the network; clean teardown confirmed on all three VMs afterward. VM verification also turned up a real, if mundane, testing artifact: a committed test fixture certificate, generated on a different machine than the one it was later checked against, carried a notBefore field the verifying VM's own clock hadn't reached yet, correctly rejected by rustls for being not-yet-valid rather than for the untrusted-issuer reason the test intended to demonstrate.

Milestone 13 certificate revocation and rotation automation

Milestone 12's mutual TLS closed the confidentiality and per-identity gaps but explicitly deferred two things: no revocation (ejecting one compromised identity meant regenerating the CA and redistributing every certificate in the cluster), and no rotation story beyond a fully manual procedure with no way for a running keel-controlplane or keel-agentd to pick up a replacement certificate without a restart. This milestone closes both: certificate revocation lists (CRLs), checked at the TLS handshake itself using support already built into rustls 0.23 (no new dependency), and a background reload thread in both long-running daemons so a freshly issued certificate or a freshly regenerated CRL takes effect live, with no restart and no coordinated fleet-wide redeploy.

scripts/gen-certs.sh grew a real openssl ca certificate database, since openssl ca -gencrl is the only way to produce a CRL. Reissuing an existing node/client/control-plane identity now rotates it: the new certificate is issued first, and only once that succeeds is the previous certificate under that name revoked and crl.pem regenerated, so a failed reissue never strands an identity with zero valid certificates. New revoke <name> and crl subcommands let an operator eject a compromised identity or refresh the CRL on demand. CRL enforcement was wired symmetrically into all three binaries: every listener verifies an incoming client certificate against the CRL, and every outbound caller verifies the peer's server certificate against the same CRL via a custom WebPkiServerVerifier.

Development surfaced a real gap in the milestone's own design: the spec assumed the test CA's private key was committed to the repository alongside its certificate, so a certificate database could be grown around the existing CA. It never was, only the certificate was. The fix regenerated the CA plus its fixtures from scratch, matching their exact identity (SAN, extended key usage) so every already-passing Milestone 12 test kept working unchanged.

Milestone 13 finished at 215 tests on macOS, clippy clean, a final whole-branch review with no Critical or Important findings, and full VM verification across three real nodes: FreeBSD 15.1's real openssl confirmed to run the new certificate-database workflow identically to the development machine; revoking one node's certificate and redistributing only the refreshed CRL got it rejected within one 30-second reload interval, confirmed by a real CertificateRevoked TLS alert, with no process ever restarting; rotating a different node's certificate produced the exact expected behavior live, a brief revoke-triggered failure followed by automatic recovery once its own reload picked up the new certificate, again with no restart and no manual intervention beyond distributing the files; a third, untouched node kept completing scheduled round trips throughout; clean teardown confirmed on all three VMs afterward.

Milestone 14 cross-node overlay networking

The first milestone of sub-project 5, and the one that closes the last gap left by a multi-node control plane running without any cross-node data-plane networking: through Milestone 13, each node's keel0 bridge and its jails' epair(4) attachments were entirely host-local, with no L2 or L3 connectivity between different nodes' jails. This milestone closes that gap using plain kernel routing rather than a tunnel protocol, no vxlan/gre/WireGuard, no new dependency, no new encapsulation, leaning on the same same-LAN trust this project's VM fleet has always run on.

Each node is assigned a distinct, non-overlapping /24 subnet, deterministically derived from its node-id and an operator-configured --cluster-cidr via a hand-rolled FNV-1a hash. keel-controlplane derives and collision-checks each node's subnet at registration time and returns it in the registration response and every GET /nodes entry. keel-agentd stores its own assigned subnet and, on the same 5-second registration/heartbeat tick that already existed, additionally fetches the full peer list and reconciles its kernel routing table (route(8)) to route directly to every Alive peer's subnet via that peer's real address, no new background thread. A JailSpec applied with a network.address outside its own node's assigned subnet is rejected before any provisioning is attempted.

A final whole-branch review caught a genuine cross-cutting bug before any VM was touched: --advertise-addr has been a host:port string since Milestone 8, and it was flowing unstripped into the route(8) gateway argument, which does not accept a port, so on real hardware no peer route would ever have installed, silently defeating this milestone's entire purpose despite every test passing. Fixed by stripping the port the same way tls.rs's server_name_from_addr already had to. Live VM testing then surfaced a second gap invisible to any amount of code review: the keel0 bridge had never been assigned an IP address in any milestone up to now, so a node had no locally-connected route for its own advertised subnet and its jails had no default route for reply traffic to escape their own /24. Fixed in keel-net alone: the gateway is derived from the jail's own address (network address + 1), idempotently assigned to the bridge, and configured as the jail's default route.

Full VM verification confirmed the whole chain end-to-end: two nodes derived distinct, stable, reproducible subnets; a jail on each could ping the other's jail IP directly with 0% packet loss; killing a node's keel-agentd got its route withdrawn from the surviving node's kernel table within one heartbeat-tick window; an out-of-subnet address was rejected with no jail ever created; clean teardown confirmed on all three VMs afterward.

Milestone 15 service discovery via replica sets

The first milestone of sub-project 6, service discovery and load balancing, scoped to exactly two things: a replica-set concept (N interchangeable copies of a jail template, scheduled across nodes, named deterministically) and discovery (querying which replicas are currently healthy and where). It deliberately stops short of any traffic-distribution mechanism, a proxy, DNS-based round robin, or client-side load balancing; that stays a distinct, later milestone in the same sub-project.

A new kind: Service spec produces N deterministically named jails (<name>-0 through <name>-{N-1}). keel-controlplane took on its first-ever dependency on keel-spec to construct these replica specs itself, plus a new Services registry structurally parallel to the existing Placements/Registry. Replica placement reuses the existing Placements map completely unchanged; a new spreading wrapper filters candidate nodes to exclude any already hosting another replica of the same service before falling back to plain headroom-based bin-packing. Each replica's address is auto-assigned as the first free address in its target node's Milestone-14 pod_cidr, tracked in a new per-node UsedAddresses map. Self-healing reconciliation piggybacks entirely on the existing 5-second heartbeat: on each incoming heartbeat, the control plane diffs each service's desired replica count against how many currently resolve to a placement on an Alive node, scheduling missing replicas and tearing down excess ones, deliberately stopping at "node is Alive," not "jail is running," so a crash-looping replica is left to its own node's existing crash-loop backoff rather than being rescheduled on top of it. GET /services/<name> is the actual discovery surface, returning only Alive+running replicas.

The final whole-branch review, done only after all ten tasks had already passed their own individual reviews, caught one real bug none of those narrower passes could see: ReconcileServices originally treated a crash-looping-but-still-placed replica identically to a genuinely missing one, rescheduling it onto a different node without ever tearing down the original, a bug that would have ping-ponged a persistently crash-looping replica across nodes, leaving an orphan behind at every hop. Fixed by making "present" mean "placed on an Alive node" rather than requiring it to also be currently running.

Milestone 15 finished at 332 tests, clippy clean, and full unit/integration coverage; the design's own Open Questions accept one known, bounded concurrency limitation (two racing reconcile computations can, in a narrow window, schedule the same missing replica onto two different nodes) rather than fixing it in this milestone, the same way earlier milestones accepted "no hard admission guarantee" as a bounded eventual-consistency gap rather than a correctness bug.

Milestone 16 service load balancing via a per-node virtual-IP proxy

The second and closing milestone of sub-project 6: a kind: Service now gets a stable, cluster-wide virtual IP that any jail can connect to and be transparently relayed to a currently-healthy replica, round robin, with zero caller-side awareness of which node actually answers, the same job Kubernetes' Service/kube-proxy mechanism does. kind: Service gains one field, spec.port; keel-controlplane gains a new --service-cidr flag and allocates one VIP per service via a genuinely new hash function, derive_service_vip, operating at host-address granularity, deliberately not a reuse of Milestone 14's derive_pod_cidr, which is hardcoded to whole-/24-block granularity and cannot produce a single host address. The existing 5-second heartbeat's response, previously always empty, now carries every known service's {name, vip, port, replicas}, the replica list computed by a function shared with GET /services/<name>'s own filter so the two can never drift apart. Every node's keel-agentd diffs that table each tick: a new service gets its VIP aliased onto the node's own keel0 bridge (FreeBSD's ifconfig alias, a mechanism this project had never needed before) and a round-robin relay listener bound to <vip>:<port>; a failed connect retries the rotation's next replica exactly once.

A design-review pass before implementation began caught four problems in the spec itself: the VIP-derivation algorithm as originally written was unimplementable (it claimed to reuse derive_pod_cidr, which cannot produce a host address); port, unlike vip, had been left mutable on a routine re-apply; the spec described discovering a VIP via keelctl get services, a command that doesn't exist (keelctl never gained a bare-collection verb for GET /services); and a fail-open heartbeat behavior was attributed to the wrong milestone. All four were corrected before any implementation task began.

A final whole-branch review, after all eleven implementation tasks had already passed their own individual reviews, caught two problems at the FreeBSD-networking boundary no unit test reaches. First: add_alias aliased the VIP with no explicit netmask, and FreeBSD's ifconfig defaults a bare alias to its address's classful mask, /8 for a 10.x VIP, which would install a connected route for the entire 10.0.0.0/8 range on that bridge, shadowing Milestone 14's per-node /24 pod routing on the same node. Fixed by pinning an explicit /32 host netmask. Second: the relay's TcpStream::connect used the OS default timeout (~75 seconds on FreeBSD), which would stall failover for the exact scenario this milestone's failover story depends on, a replica whose node just died but isn't yet marked Dead. Fixed with an explicit 1-second connect_timeout.

Milestone 16 finished at 366 tests, clippy clean, and full 3-node VM verification: the netmask fix was confirmed directly, netstat -rn showed a proper /32 host route via lo0 on both nodes, not a hijacked /8. A 2-replica service's VIP was read back via a direct GET /services call, confirmed aliased on both nodes' bridges, and a real jail connecting to it got relayed and echoed correctly, strictly alternating between both replicas. Killing one replica's node (keel-agentd, not the VM) triggered Milestone 15's existing self-healing reconciliation to reschedule a replacement onto the surviving node, and the VIP kept answering throughout; deleting the service removed the alias from both nodes' bridges and left new connections unanswered. Two environment-level gaps surfaced during the run, neither a bug in this milestone's own code: net.inet.ip.forwarding was off on both nodes, silently black-holing all cross-node jail traffic until re-enabled, and the two nodes' mTLS certificates had drifted onto a different CA than the control plane's own, predating this session, requiring a full re-issue before registration would succeed.

Milestone 17 persistent volumes on a single node

The first milestone of sub-project 7, persistent storage. Every milestone through 16 treated a jail's entire filesystem as disposable: Reconciler::provision clones a jail's rootfs from a base image, and Reconciler::delete destroys that same dataset the instant the jail is deleted. This milestone builds the foundation for data that outlives a jail on one node, deliberately stopping there: whether a stateful service's data eventually follows a rescheduled replica via node-pinning or via zfs send/receive replication is left for a later milestone in this sub-project, the same way Milestone 15 built the replica-set concept before Milestone 16 added cross-node load balancing on top of it.

A kind: Jail spec gains spec.volumes, each entry a name, mount path, and ZFS-quota-style size, validated the same way a jail name and memory quantity already are, and immutable after first apply. keel-zfs gains create_volume, a plain quota-scoped dataset independent of the rootfs-cloning path, and a new ZfsError::Busy variant so an exhausted-retries "dataset is busy" failure is distinguishable, at the type level, from any other failure. keel-jail gains a new MountManager trait, the same real/fake split every OS-interaction trait in this project already uses, for mount -t nullfs/umount/mount -p. Reconciler::provision creates and mounts each declared volume before starting the jail (idempotent, so an agentd restart with the jail already running is a no-op); Reconciler::delete unmounts every declared volume before destroying the rootfs dataset, but never destroys a volume's own dataset, the entire point of this milestone. GetVolume/DeleteVolume act purely against ZFS, never consulting a jail record, since a volume can outlive every jail that ever referenced it; GET/DELETE /volumes/<name> map NotFound/Busy to 404/409. keel-controlplane gained two forwarding-only route arms; keelctl gained a delete-volume <name> verb.

TDD surfaced one real deviation from the initial design before any VM was touched: the design's own std::fs::create_dir_all call inside provision, run unconditionally against an absolute rootfs path, broke every fake-backed reconciler test outright, since a path like /zroot/keel/jails/web-1/data doesn't exist as a real directory in a test environment. Fixed by moving directory creation behind a new MountManager::ensure_mount_point method instead, real for CliMountManager, a no-op for FakeMountManager, the same real/fake abstraction discipline this project already applies everywhere else.

Real 3-node VM verification found two more bugs, both only reachable against the actual mount(8)/umount(8) binaries: is_mounted parsed mount -p's output by splitting on a single literal tab and indexing, but a real FreeBSD 15.1 VM pads its columns with a variable number of tabs for alignment rather than exactly one tab per field, fixed by splitting on any run of whitespace instead; and unmount's "already unmounted, tolerate it" check looked for the stderr text "not currently mounted", but this FreeBSD release's real umount reports both a never-mounted path and an already-unmounted one as "not a file system root directory", a different message than assumed. Both were caught immediately by new real-VM test files added for this milestone and fixed before any end-to-end testing began.

The full chain then passed end-to-end on the real fleet, routed through keel-controlplane: a jail with a declared volume landed on a node, its dataset was created and quota-scoped, and it was nullfs-mounted before the jail started; a file written through the running jail was readable back the same way. Deleting the jail unmounted the volume and destroyed the rootfs dataset, but the volume's own dataset survived, confirmed directly with zfs list. Re-applying the same jail (sticky placement kept it on the same node) remounted the existing dataset without recreating it, and the earlier file was still there. Deleting the volume while the jail still held it mounted returned 409 with the dataset untouched; deleting the jail first and retrying returned 200, and a follow-up read on the same name returned 404. A plain jail with no declared volumes ran with no behavior change throughout.

Milestone 17 finished at 403 tests passing on macOS against fakes, plus new real-VM tests for create_volume, destroy_dataset's busy-detection, and the full CliMountManager mount/unmount cycle, and full 3-node VM verification of the entire volume lifecycle. One unrelated, pre-existing test failure surfaced while running the full suite on the real VM, in keel-net's route-idempotency test, a crate this milestone never touched, left for a future session rather than folded into this one's scope.

Milestone 18 stateful services via node-pinning

The second and closing milestone of sub-project 7. Milestone 17 stopped kind: Service/JailTemplate short of a volumes field entirely, leaving the choice between two candidate mechanisms, scheduler node-pinning or zfs send/receive replication, as an open question for later. This milestone answers it with node-pinning: a kind: Service whose template declares volumes gets each replica pinned to whichever node it was first placed on for the life of that replica, trading automatic failover away in exchange for never silently creating a fresh, empty volume on a different node in place of one that still holds real data. zfs send/receive replication remains a distinct, undesigned, later effort.

JailTemplate gains volumes: Vec<VolumeMount>, the exact type Milestone 17 already defined, validated by the same validate_volumes function, now also called from parse_and_validate_service. JailTemplate::to_jail_spec maps each template volume into that replica's own volume, rewriting only its name to {replica}-{volume} (e.g. replica web-0's template volume data becomes web-0-data); since replica names are already globally unique, the derived per-replica volume names are automatically unique too. Since Services::apply already rejects any template change on an existing service via derived PartialEq, a stateful service's volumes were already immutable once created with no code change needed. The entire runtime mechanism is one conditional in keel-controlplane's Command::ReconcileServices handler: computing present_indices now branches on whether a service's template has any volumes. Stateless services keep today's behavior unchanged, a Dead-node replica drops out and gets rescheduled; a stateful service's every placed index counts as present regardless of whether its node resolves, so a replica pinned to a Dead node is neither torn down nor replaced, it simply waits for keel-agentd's own crash-safe on-disk state to reconcile it back to running once its node returns, with no control-plane involvement. No keel-agentd changes, no new HTTP routes, no new keelctl verbs: everything downstream of present_indices and every Milestone 17 volume mechanism already worked unmodified.

A design-review pass before implementation caught one factual error in the spec itself: it claimed that manually deleting a replica pinned to an unreachable node "forwards to the recorded node and fails (500)." Tracing handle_scheduled_delete showed the request never reaches the forward step at all, resolve_placement fails first against registry.resolve and returns 404, the same response an unplaced name would get. Corrected in the spec before any code was written.

Real 3-node VM verification surfaced one genuine, pre-existing bug in Milestone 17's own code: Reconciler::rollback_provision never unmounted a jail's declared volumes before calling destroy_dataset on its rootfs, unlike delete(), which already gets this ordering right. A provision failure occurring after the volume-mount step, surfaced here by one node's stripped-down test image initially lacking /sbin/route, an environment gap rather than a product bug, left the mount sitting on the rootfs dataset, so destroy_dataset failed silently and every later retry's zfs clone was permanently wedged on "already exists." Fixed by unmounting declared volumes first in rollback_provision too, mirroring delete() exactly, with a new regression test.

Real 3-node VM verification then passed in full: a 2-replica stateful service scheduled its replicas onto two distinct nodes with their own separate, correctly-quota-scoped volume datasets; distinct data written into each was readable back through the jail's own mount point. Killing one replica's node's keel-agentd (not the VM) and waiting past the 15-second Dead threshold left that replica's jail running untouched on its own node, confirmed GET /services/<name> reporting only the survivor as healthy, and confirmed the dead replica's index was not recreated on the cluster's third node; its volume dataset and data were untouched throughout. Restarting that node's keel-agentd found the pinned replica already running, unaffected by the whole episode, with its data intact and no control-plane action taken. Scaling the service down by one tore down the removed replica's jail while its volume survived by default; scaling back up rescheduled it onto the same node and found the old data intact rather than starting empty. A second scale-down/keelctl delete-volume/scale-up cycle confirmed the contrasting explicit-cleanup path: after delete-volume, the next scale-up landed on a genuinely fresh, empty volume instead. Two environment-level gaps surfaced along the way, neither a bug in this milestone's own code: the missing test-image binary above, and two of the three nodes still serving requests with a stale, already-revoked operator client certificate left over from an earlier session, both fixed by redistributing the current files from the control-plane host.

Milestone 18 finished at 412 tests passing on macOS against fakes (8 new for the feature itself, plus 1 for the rollback_provision fix), clippy at 42 warnings (41 inherited plus one new assert_eq! literal-bool in the new test, this project's own established idiom), and full 3-node VM verification of the entire node-pinning lifecycle, closing out sub-project 7.

Roadmap what's next

Sub-project 1: single-node jail reconciliation daemon: complete. Jail spec language, jail lifecycle + ZFS clone provisioning, VNET networking, the reconciliation core, the local HTTP API + CLI, and rc.d service integration with an end-to-end smoke test are all done.

Sub-project 2: multi-node control plane: complete. The node registry (register/heartbeat/list over TCP, self-healing membership) and routing jail specs to a specific node through the control plane are both done.

Sub-project 3: scheduling and cluster growth: complete. Automatic node placement and resource-aware bin-packing (CPU/memory headroom, not just jail count) are both done.

Sub-project 4: control-plane hardening: complete. Shared-secret authentication, mutual TLS replacing it entirely, and certificate revocation/rotation automation are all done.

Sub-project 5: cluster networking: complete. Cross-node overlay networking, deterministic per-node subnets, kernel routing, and apply-time subnet validation are all done.

Sub-project 6: service discovery and load balancing: complete. Service discovery via replica sets and load balancing via a per-node virtual-IP proxy are both done.

Sub-project 7: persistent storage: complete. Persistent volumes on a single node and stateful services via node-pinning are both done. Cross-node volume movement (zfs send/receive replication or another mechanism) and an operator "force re-pin" verb remain undesigned.

Not yet designed (future sub-projects, each will get its own spec): external/outside-cluster reachability (a NodePort/LoadBalancer equivalent), and bhyve VM workloads alongside jails.