HTTP API reference

Authentication

Every endpoint reachable over TCP, both of keel-controlplane's routes below and keel-agentd's optional TCP listener, requires mutual TLS: the caller must present a certificate signed by the cluster's private CA, and that certificate is checked in real time against a certificate revocation list (CRL). A connection presenting no certificate, a certificate from a different CA, or a revoked certificate fails the TLS handshake itself, before any HTTP request is parsed or any route below ever runs. keel-agentd's Unix socket is unaffected by any of this: no TLS, no certificate check, gated only by its 0600 filesystem permission.

keel-controlplane (unconditionally) and keel-agentd (whenever its control-plane flags are set) each require four flags to serve this API: --tls-ca-file, --tls-cert-file, --tls-key-file, and --tls-crl-file. Certificates are issued once per identity by an operator via scripts/gen-certs.sh; see the CLI reference for the exact commands. Revoking a certificate, or rotating one to a fresh keypair, takes effect on an already-running keel-controlplane/keel-agentd within 30 seconds, with no restart, since both reload their certificate/key/CA/CRL files from disk on a background timer.

keel-agentd (Unix socket)

Default socket: /var/run/keel-agentd.sock.

Method & path	Body	Success	Errors
PUT /jails/<name>	JailSpec YAML; metadata.name must match <name>	200, reconciled immediately before responding	400 invalid YAML or name/body mismatch, 409 attempted change to an immutable field
GET /jails	none	200, YAML list of every tracked jail	None
GET /jails/<name>	none	200, YAML for that jail	404 unknown jail
DELETE /jails/<name>	none	200, jail/dataset/network/limits torn down before responding	404 unknown jail

When started with the control-plane flags (--node-id, --control-plane-addr, --advertise-addr), keel-agentd serves this exact same set of routes over a second, TCP listener bound to --advertise-addr, so keel-controlplane can reach it from another host. The Unix socket is unaffected either way.

keel-controlplane (TCP)

Nodes live on separate hosts, so this API is served over a TcpListener instead of a Unix socket.

Method & path	Body	Success	Errors
POST /nodes/register	YAML: id, addr, capacity_cpu, capacity_memory	200, YAML {pod_cidr}: the node's deterministically-derived, collision-checked /24 subnet (registered or re-registered; committed load resets to zero until the next heartbeat)	400 invalid or incomplete YAML, 409 the derived pod_cidr collides with an already-registered node
POST /nodes/<id>/heartbeat	YAML: committed_cpu, committed_memory, jails (per-jail running status)	200, YAML list of every known service's {name, vip, port, replicas} (empty list if no services exist); node's last-seen time, committed load, and per-jail health updated	400 invalid or incomplete YAML, 404 unknown node (the node should re-register)
GET /nodes	none	200, YAML list of every node: id, addr, pod_cidr, computed status (Alive within 15 seconds of the last heartbeat, otherwise Dead), last_seen_secs, capacity_cpu, capacity_memory, committed_cpu, committed_memory	None
PUT /nodes/<id>/jails/<name>	JailSpec YAML, forwarded as-is	Whatever the node's own PUT /jails/<name> returns, relayed byte-for-byte; the jail's placement is recorded against <id>	404 unknown or dead node (rejected before any connection is attempted), 500 could not reach the node
GET /nodes/<id>/jails	none	Whatever the node's own GET /jails returns, relayed byte-for-byte	404 unknown or dead node, 500 could not reach the node
GET /nodes/<id>/jails/<name>	none	Whatever the node's own GET /jails/<name> returns, relayed byte-for-byte	404 unknown or dead node, 500 could not reach the node
DELETE /nodes/<id>/jails/<name>	none	Whatever the node's own DELETE /jails/<name> returns, relayed byte-for-byte; the jail's recorded placement is cleared	404 unknown or dead node, 500 could not reach the node
PUT /jails/<name>	JailSpec YAML, forwarded as-is	Whatever the scheduled node's own PUT /jails/<name> returns, relayed byte-for-byte; the placement is recorded (or, if <name> is already placed, reused as-is, sticky)	404 the jail's recorded node is unknown or dead, 503 no alive node to schedule onto, 500 could not reach the node
GET /jails/<name>	none	Whatever the recorded node's own GET /jails/<name> returns, relayed byte-for-byte	404 no recorded placement for <name>, or its node is unknown/dead; 500 could not reach the node
DELETE /jails/<name>	none	Whatever the recorded node's own DELETE /jails/<name> returns, relayed byte-for-byte; the recorded placement is cleared	404 no recorded placement for <name>, or its node is unknown/dead; 500 could not reach the node

Liveness is computed on every read rather than tracked by a background sweep. There is no persistence: a control-plane restart forgets every node, every recorded placement, and every service definition; each node notices on its next heartbeat and re-registers automatically, and a scheduled jail or service would need to be re-applied to get a fresh placement (a service happens to recover the same VIP after a restart, since derivation is deterministic, but this is a byproduct, not a guarantee). pod_cidr is a distinct address pool from --service-cidr below; the two are never validated against each other.

All seven /nodes/<id>/jails/... and /jails/... routes forward the request byte-for-byte over a fresh TCP connection to the resolved node's keel-agentd and relay the response back untouched; the control plane never deserializes the JailSpec body. A node marked Dead or not found in the registry is rejected immediately, before any connection is attempted. The named-node and scheduled route families share one placement table, so a jail placed through either one is visible to a lookup through the other.

capacity_cpu/capacity_memory/committed_cpu/committed_memory have no default value: a registration or heartbeat body missing any of them fails to parse (400) rather than silently defaulting to zero, since a node that appeared to have no committed load would otherwise be scored by the scheduler as having maximal headroom. The scheduled PUT /jails/<name> route uses these numbers only to rank already-registered nodes by headroom; it never learns the incoming jail's own resource request, so it cannot guarantee the node it picks actually has room for it.

Services (replica sets + load balancing)

A kind: Service spec produces N deterministically-named jail replicas (<name>-0 through <name>-{N-1}), spread across nodes, self-healed on the same 5-second heartbeat as everything else, and given one stable, cluster-wide virtual IP that transparently load-balances across whichever replicas are currently healthy. See the JailSpec reference for the spec fields.

Method & path	Body	Success	Errors
PUT /services/<name>	ServiceSpec YAML; metadata.name must match <name>	200; on first creation a VIP is allocated from --service-cidr and both template and port become immutable for the life of the service; a re-apply may only change replicas	400 name already in use by an unmanaged jail or a different service, 409 attempted change to template or port, 503 --service-cidr exhausted (no free VIP left)
GET /services/<name>	none	200, YAML list of that service's currently healthy (Alive node + running jail) replicas: name, node, address	404 unknown service
GET /services	none	200, YAML list of every known service: name, desired_replicas, vip, port. The only human-facing surface that exposes a service's VIP; keelctl has no verb for this bare collection (see the CLI reference), so reaching it means a direct HTTP call.	None
DELETE /services/<name>	none	200, every replica torn down (recorded placements cleared); the VIP alias and relay listener disappear from every node on their next heartbeat	404 unknown service

keelctl apply -f sniffs kind out of the parsed YAML to choose PUT /jails/<name> or PUT /services/<name> automatically; keelctl get/delete try the jail path first and fall back to the service path on a 404, since jail and service names share one flat namespace.

Reconciliation for services piggybacks entirely on the existing heartbeat, the same way node liveness and jail routing already do: on every incoming POST /nodes/<id>/heartbeat, the control plane also diffs each service's desired replica count against how many currently resolve to a placement on an Alive node (deliberately not also requiring the jail to be running, so a crash-looping replica is left to its own node's local backoff rather than rescheduled on top of it), and computes the heartbeat response's per-service {vip, port, replicas} table using the exact same Alive+running filter GET /services/<name> uses, so the two can never drift apart. Every node's keel-agentd diffs that table against what it's currently proxying: a new service gets its VIP aliased onto the node's own bridge and a round-robin relay listener bound to <vip>:<port>; a known service gets its replica list hot-swapped; a disappeared service gets torn down. A failed connect attempt retries the rotation's next replica exactly once; a service with zero currently-healthy replicas keeps its alias and listener up but closes every accepted connection immediately.

The VIP is reachable only from other jails/nodes inside the cluster, the same boundary pod_cidr already has; there is no NodePort/LoadBalancer equivalent yet. The proxy is a pure L4 TCP byte relay: it never inspects, parses, or terminates whatever protocol is inside the connection, and there is no session affinity (each new connection picks a healthy replica independently) and no load-balancing algorithm more sophisticated than round robin.

Volumes

A jail's declared spec.volumes (see the JailSpec reference) are created and mounted automatically as part of PUT /jails/<name>/DELETE /jails/<name> above; the two routes below act purely against ZFS, keyed only by dataset path, and never consult a jail record, since a volume can outlive every jail that ever referenced it.

Method & path	Body	Success	Errors
GET /volumes/<name>	none	200, YAML {name}, regardless of whether any jail currently has it mounted	404 the dataset was never created or has already been destroyed
DELETE /volumes/<name>	none	200, dataset destroyed for good	404 the dataset was never created or has already been destroyed; 409 still mounted by a running jail (ZFS itself refuses to destroy a mounted dataset; delete the jail first)

keel-controlplane forwards GET/DELETE /nodes/<id>/volumes/<name> to the named node's keel-agentd, byte-for-byte, the same handle_forward path /nodes/<id>/jails/... already uses; there is no scheduled (node-less) route for volumes, since a volume is never itself scheduled, it simply lives wherever the jail that first referenced it already landed. keelctl delete-volume <name> is the only keelctl verb that reaches this API; there is no keelctl get-volume, the same "operator surface is deliberately smaller than the full HTTP API" precedent GET /services already set.

Ingress (automatic HTTPS)

Unlike jails, services, and volumes, Ingress is served only by keel-agentd's own local socket, never routed through keel-controlplane: an Ingress and the Service it points at must live on the same node in this milestone (see the JailSpec reference). There is no /nodes/<id>/ingress/... or scheduled /ingress/<name> route on the control plane.

Method & path	Body	Success	Errors
PUT /ingress/<name>	IngressSpec YAML; metadata.name must match <name>	200; the singleton ingress jail is provisioned if it doesn't exist yet, and a certificate is issued on the node's next reconcile pass if one isn't already valid	400 invalid YAML, name/body mismatch, or backend.service doesn't name a currently-known Service
GET /ingress	none	200, YAML list of every applied Ingress spec	None
GET /ingress/<name>	none	200, YAML for that Ingress spec	404 unknown name
DELETE /ingress/<name>	none	200; the host's server block is removed from nginx's configuration on the next reconcile pass	404 unknown name

Certificate issuance and renewal run entirely on the node's own 5-second reconcile tick, the same crash-loop backoff pattern every other reconciliation path in this project uses: a failed DNS-01 challenge or ACME request doesn't block any other Ingress's certificate, and a certificate within 30 days of expiry is renewed automatically with no operator action. The DNS provider (currently OVH only) and the ACME directory URL (Let's Encrypt staging or production) are keel-agentd daemon config, not part of any spec, since they're credentials rather than per-Ingress data; see the automatic HTTPS quickstart for exactly which flags and files that config needs.

Cross-node Ingress routing (an Ingress on one node fronting a Service whose replicas are scheduled elsewhere) is out of scope for this milestone; see the Architecture page.