Architecture

Crate map

Crate	Responsibility
keel-spec	The JailSpec YAML schema, parsing, and validation. No FreeBSD dependency; runs on any OS.
keel-jail	The JailRuntime trait: create/start/stop/destroy a jail, check if it's running. Also the MountManager trait: mount -t nullfs/umount/mount -p for a jail's persistent volumes, independent of JailRuntime since mounting isn't jail-specific at the OS level. Fake in-memory implementations plus real ones that shell out to jail(8), jexec(8), rctl(8), and mount(8)/umount(8).
keel-zfs	The ZfsManager trait: clone a jail's root filesystem from a base dataset, destroy it again, and (independent of cloning) create a plain quota-scoped dataset for a persistent volume. Fake plus a real implementation over zfs(8).
keel-net	The NetManager trait: create a bridge, attach a jail to it over an epair(4) pair with a static address, add/remove a kernel route, alias/unalias a second address on a bridge (a service VIP), tear a jail's attachment down. Fake plus a real implementation over FreeBSD networking commands.
keel-agentd	The reconciliation daemon: a generic Reconciler<J, Z, N, M> over the four traits above, a crash-safe on-disk state store, crash-loop backoff, a local HTTP API over a Unix socket, its own detected CPU/memory capacity and currently committed load, kernel-route reconciliation to every other Alive node's subnet, a per-service proxy module that aliases a VIP and relays connections round-robin to healthy replicas, and per-jail persistent volume creation/mounting decoupled from the jail's own lifecycle.
keelctl	The CLI: apply, get, delete, delete-volume, talking to keel-agentd's Unix socket directly, or routed through keel-controlplane to a named or scheduled node; sniffs kind: Jail vs kind: Service automatically.
keel-controlplane	The node registry, scheduler, service registry, and request router: register/heartbeat/list nodes over TCP (liveness computed on read, not a background sweep), deterministic per-node subnet (pod_cidr) and per-service virtual-IP (vip) allocation, a resource-aware scheduler, same-service spreading for replica placement, and byte-opaque forwarding of apply/get/delete/volume-lookup/volume-delete to a named or scheduled node's keel-agentd.

The trait + fake + real pattern

Every FreeBSD-facing crate follows the same shape: a trait describing what the crate does (JailRuntime, ZfsManager, NetManager, MountManager), an in-memory Fake implementation for fast tests on any machine, and a real implementation that shells out to the corresponding FreeBSD command. keel-agentd's Reconciler<J, Z, N, M> is generic over all four, so the exact same reconciliation logic runs against Fake* implementations in CI and against Process*/Cli* implementations on a real host.

Reconciliation flow

Reconciler's public API is small on purpose: new, apply, delete, reconcile. apply and delete update the desired state immediately (a caller's request is reconciled before the HTTP response returns); reconcile runs on a 5-second timer and diffs desired versus observed state for every jail in one pass, provisioning what's missing, restarting what crashed (with per-jail backoff starting at one second, doubling to a five-minute cap, and resetting after sixty seconds of stable uptime), and cleaning up what was removed. A failure reconciling one jail is returned in a per-jail failure list rather than aborting the whole pass, so one broken jail never blocks the others.

Single-worker-thread ownership

Both keel-agentd and keel-controlplane use the same concurrency pattern: a single worker thread exclusively owns the mutable state (the Reconciler in one case, an in-memory Registry in the other), reachable only through an mpsc command channel. The HTTP server (a small hand-rolled parser over httparse, no async runtime) and, for keel-agentd, the reconciliation timer, only ever reach that state through the channel, so there is never a second concurrent owner to synchronize against.

Routing a spec to a node

A node is a routing-layer concept only: it never appears in a JailSpec. When a request arrives at keel-controlplane for /nodes/<id>/jails/..., the worker thread first resolves <id> against the Registry, the same liveness computation GET /nodes uses, rejecting an unknown or Dead node immediately, before any network call. Only once a live address comes back does the handler open a fresh TCP connection to that node's keel-agentd, forward the request byte-for-byte, and relay the response back untouched; the body is never deserialized, so keel-controlplane has no dependency on keel-spec.

On the receiving end, keel-agentd's jails API is transport-agnostic: the same route table serves both the always-on Unix socket and a second, entirely optional TCP listener, bound to --advertise-addr only when the node is configured to register with a control plane. A node with no control-plane flags set behaves exactly as it did before this existed.

Scheduling: picking a node automatically

Alongside the node-naming routes, keel-controlplane serves a second route family, /jails/<name> with no node segment. On PUT, if the jail name has no recorded placement yet, a pure scheduler::pick_node function picks the Alive node with the most headroom in whichever of CPU or memory is most constrained (min(free_cpu/capacity_cpu, free_memory/capacity_memory)), ties broken by ascending node id, then forwards there exactly like the named-node path. Resolving is a pure read: the resulting placement (recorded in the same in-memory Placements table, jail_name -> node_id) is only written after a confirmed success response comes back, so a failed apply never leaves a phantom placement.

Re-applying an already-placed jail is sticky: it always resolves to the node already recorded, never a fresh scheduling decision, so a jail is never silently split across two nodes. If that recorded node has since gone Dead, the request fails with the same error a named-node route would give; nothing reschedules it automatically. The two route families share one Placements table: a jail applied by explicit node name is just as visible to a later scheduled GET as one the scheduler placed itself.

The capacity/headroom numbers the scheduler ranks on come entirely from each node's own self-reports, never from keel-controlplane inspecting a JailSpec: keel-agentd detects its host's CPU/memory capacity once at startup via sysctl and sends it at registration, then sums spec.resources.{cpu,memory} across every jail its own Reconciler tracks and sends that committed total on every heartbeat. This keeps the control plane's opacity to JailSpec bodies intact, at the cost of being a ranking heuristic rather than an admission guarantee: the scheduler never learns the incoming jail's own resource request, so a node it picks can still turn out unable to fit that jail, in which case the apply simply fails normally rather than retrying elsewhere.

Cluster networking: per-node subnets and kernel routing

Every node is assigned a distinct, non-overlapping /24 subnet (pod_cidr), deterministically derived from its node-id and an operator-configured --cluster-cidr via a hand-rolled FNV-1a hash, collision-checked and returned at registration. There is no tunnel/overlay protocol: connectivity between different nodes' jails is plain kernel routing, leaning on the same-LAN trust this project's whole design already assumes. On the same 5-second heartbeat tick used everywhere else, keel-agentd fetches the full peer list and reconciles its kernel routing table (route(8)) to route directly to every Alive peer's subnet via that peer's real address; a dead peer's route is withdrawn the same way. A JailSpec applied with a network.address outside its own node's assigned subnet is rejected before any provisioning is attempted.

Each node's keel0 bridge is itself given an address (the subnet's network address plus one), idempotently assigned via keel-net's attach_jail, so the node has a locally-connected route for its own subnet and its jails have a default route for reply traffic to escape their own /24.

Services: replica sets and load-balancing

A kind: Service spec produces N deterministically-named jail replicas (<name>-0 through <name>-{N-1}), reusing the exact same Placements map, scheduler, and auto-addressing every plain kind: Jail already uses, plus a spreading wrapper that prefers a node not already hosting another replica of the same service. A new Services registry (desired replica count, template, port, and the service's own virtual IP), structurally parallel to Placements/Registry, tracks each service; like everything else in keel-controlplane, it is unpersisted, forgotten on a restart.

Self-healing and load-balancing both piggyback on the existing 5-second heartbeat, no new thread, no new polling cadence. On every incoming heartbeat, the control plane diffs each service's desired replica count against how many currently resolve to a placement on an Alive node (deliberately not also requiring the jail to be running, so a crash-looping replica is left to its own node's existing crash-loop backoff rather than rescheduled on top of it), scheduling missing replicas and tearing down excess ones. The same heartbeat's response carries every known service's {name, vip, port, replicas}, the replica list computed by the identical Alive+running filter GET /services/<name> uses, so the two can never drift apart.

A service's virtual IP is allocated once, on first creation, from a cluster-wide --service-cidr pool entirely distinct from any node's pod_cidr, derived deterministically from the service name via a host-address-granularity hash function, derive_service_vip, and never changes afterward, the same immutable-once-created treatment spec.template already gets, extended to spec.port too. A collision during derivation is resolved by linearly probing forward through --service-cidr's host addresses; exhausting the pool entirely is a hard rejection, not a silent partial success.

Every node's keel-agentd diffs the heartbeat's service table against what it's currently proxying: a new service gets its VIP aliased onto the node's own keel0 bridge, as a second address alongside the bridge's existing gateway address (FreeBSD's ifconfig alias, pinned to an explicit /32 host netmask so it can never shadow the node's own pod_cidr routing), plus a round-robin relay listener bound to <vip>:<port>; a known service gets its replica list hot-swapped in place; a disappeared service gets torn down. Each accepted connection picks the next replica in rotation, connects, and relays bytes bidirectionally until either side closes; a failed connect attempt (bounded by a short explicit timeout, not the OS default) retries the rotation's next replica exactly once. A service with zero currently-healthy replicas keeps its alias and listener up but closes every accepted connection immediately, and a node that can't reach the control plane keeps serving its last-known replica list rather than tearing anything down.

The proxy is a pure L4 TCP byte relay: it never inspects, parses, or terminates whatever protocol is inside the connection. There is no session affinity, no rebalancing of already-established connections when the replica list changes, and no external/outside-cluster reachability, the VIP is reachable only from other jails/nodes inside the cluster, the same boundary pod_cidr already has.

Persistent volumes: decoupled lifecycle on a single node

A jail's spec.volumes (see the JailSpec reference) each get their own ZFS dataset under {pool}/keel/volumes/{name}, entirely separate from the jail's own rootfs dataset under {pool}/keel/jails/{name}. Reconciler::provision creates a volume's dataset (via ZfsManager::create_volume, a plain quota-scoped zfs create, not a clone) the first time a jail referencing it is provisioned, and nullfs-mounts it onto the jail's rootfs at the declared mountPath before the jail starts; both steps are idempotent, checking dataset_exists/is_mounted first, so an agentd restart with the jail already running and already mounted is a no-op. Reconciler::delete unmounts every declared volume before destroying the rootfs dataset, exactly mirroring the ordering that already avoids "device busy" for the rootfs mount itself, but it never calls destroy_dataset on a volume's own dataset. That's the whole mechanism: a volume's data simply isn't touched by anything that only ever operates on a jail record.

GetVolume/DeleteVolume, the two operations that do destroy a volume for good, are keyed purely by dataset path and never consult the jail-record map at all, since a volume can (and by design, routinely does) outlive every jail that ever referenced it. ZfsError::Busy, a new variant distinguishing an exhausted-retries "dataset is busy" failure from any other kind, is what lets the HTTP layer tell apart "still mounted by a running jail, delete that first" (409) from "never existed or already deleted" (404) without guessing from a generic error string.

A volume is never itself scheduled or tracked cluster-wide: it simply lives on whatever node the jail that first referenced it was already placed on by the existing scheduler, and keel-controlplane's only involvement is two forwarding-only route arms, structurally identical to its existing jail-forwarding routes. This is a deliberate, bounded gap rather than an oversight: if a jail with volumes is deleted and later re-applied under the same name after its placement record no longer exists, the scheduler may place it on a different node, which has no way to know data exists elsewhere, so it silently creates a fresh, empty volume. Detecting or preventing that needs a cluster-wide volume registry this milestone deliberately doesn't build.

Stateful services: node-pinning

A kind: Service's template can also declare volumes (see the JailSpec reference), which makes every replica of that service stateful. JailTemplate::to_jail_spec maps each template volume into that replica's own volume by rewriting only its name to {replica}-{volume}; since replica names are already globally unique, the derived per-replica volume names are automatically unique too, with no new collision-checking logic. Because Services::apply already rejects any template change on an existing service, a stateful service's volumes were already immutable once created before this feature existed.

The entire node-pinning mechanism is one conditional in the same heartbeat-piggybacked reconciliation described above: computing which replica indices currently "count as present" branches on whether the service's template has any volumes. A stateless service keeps the existing behavior exactly, an index counts as present only if its node currently resolves as reachable, so a Dead-node replica drops out and is rescheduled elsewhere. A stateful service's every placed index counts as present regardless of whether its node resolves, so a replica pinned to a Dead node is neither torn down nor replaced: it simply waits for that node's own keel-agentd to reconcile it back to running from its own crash-safe on-disk state once the node returns, with no control-plane involvement at all. This trades automatic failover away in exchange for the same guarantee plain volumes already give a single jail: never silently starting a stateful replica fresh and empty on a different node while the original data is still sitting untouched on the node that went down. GET /services/<name>'s own Alive+running health filter is intentionally stricter than this pinning logic, so a pinned-but-unreachable replica still correctly disappears from what the load-balancing proxy actually routes traffic to, even while the control plane keeps waiting for it rather than replacing it. Scaling a stateful service down leaves the removed replica's volume dataset untouched, exactly like a plain jail's volume; scaling back up later finds the old data still there, unless it was explicitly cleaned up with keelctl delete-volume in between.

Ingress: automatic HTTPS

A kind: Ingress (see the JailSpec reference) is the first path from the public internet to anything Keel manages; every other primitive lives entirely on private VNET networking. keel-agentd maintains a single, node-wide, system-managed nginx jail (name-prefixed keel-ingress like every other Keel-owned resource) whose configuration is a pure function of every currently-applied Ingress spec together, not one reconciliation per Ingress the way jails and volumes work: one server block per host, each proxying to its own backend Service's VIP:port.

browser --443--> node's public IP --pf rdr--> ingress jail (nginx,
  SNI picks the host's server block, terminates TLS with that host's
  cert) --proxy_pass, internal bridge--> backend Service's VIP:port
  (the existing per-node proxy) --round-robin relay--> a replica jail

Certificate issuance uses the ACME protocol's DNS-01 challenge exclusively, never HTTP-01: proving domain ownership by publishing a DNS TXT record needs no inbound reachability to the node at all, only outbound access to the DNS provider's API and to Let's Encrypt itself, which is why every real-VM verification of this feature was possible without a public IP. A DnsProvider trait (currently one real implementation, OVH's REST API, signed with OVH's application-key/application-secret/consumer-key scheme) and an AcmeClient trait (wrapping the instant-acme crate) are both node-level daemon configuration, not part of any spec: OVH's API credentials and the ACME directory URL (Let's Encrypt staging vs. production) are secrets and environment choices, not something that should ever end up committed to a YAML spec in git.

On every 5-second reconcile tick, for each applied Ingress whose certificate has no expiry yet or is within 30 days of expiring: request a new certificate via AcmeClient::request_certificate, which creates an ACME order, fetches the DNS-01 challenge token, publishes it as a TXT record via DnsProvider::create_txt_record, polls public DNS resolvers directly (not the provider's own API) until it's visible, tells the ACME server the challenge is ready, and downloads the finished certificate once the order is valid, cleaning up the TXT record afterward regardless of outcome. A failed renewal is retried with the same crash-loop backoff pattern used everywhere else in this project, and the previously-issued certificate, if any, keeps serving traffic in the meantime; one Ingress's failure never touches any other's certificate or its own already-working configuration. cert_expires_at_unix alone is never trusted as proof a certificate is actually usable: the reconciler also checks the certificate file still exists in the ingress jail's own rootfs before skipping reissuance, since that rootfs can in principle be recreated independently of the persisted expiry record.

Nginx's configuration is regenerated, validated with nginx -t (via jexec), and only reloaded on success; a bad new Ingress spec, or a backend Service whose VIP isn't known yet, is simply skipped for that pass rather than tearing down every other host that was already working. Host-level pf rules (a redirect from the node's public IP's ports 80 and 443 to the ingress jail's internal bridge address) are applied once, independent of how many Ingress specs exist, on their own reconciliation schedule with the same backoff-and-retry treatment as everything else; port 80 exists only for a plain HTTP-to-HTTPS redirect, since this project's DNS-01-only approach has no use for ACME's HTTP-01 challenge.

Deliberately out of scope for now: DNS providers other than OVH (the DnsProvider trait is the seam a second one would implement), wildcard certificates, multiple domains on one Ingress, path-based routing, and cross-node Ingress routing, an Ingress and every one of its backend Service's replicas must currently live on the same node, since the ingress jail's nginx proxy_pass only ever reaches VIPs local to its own node's bridge.