Architecture
How one WorkerApp becomes one operated fleet, and why the design looks the
way it does. The celld behaviors each guardrail encodes are indexed in the repository's
celld behaviors reference;
this page is the working summary.
The coordination model
celld nodes coordinate through an object-storage bucket (deployments, per-cell SQLite
data, ownership records, node leases) with no consensus service, no leader election, and
no membership list. A node claims a cell with a conditional write, so the bucket decides
who wins, and exactly one node serves a cell at a time. celld does not acknowledge a
write until it survives a failure (RPO=0): either the write is in the bucket
(bucket durability) or follower nodes hold it on disk (fleet
durability). The bucket is the source of truth; nodes are replaceable.
The operator's job is everything around that: turning one WorkerApp into one
fleet of celld pods sharing a bucket prefix, and encoding celld's operational rules so
they cannot be misconfigured.
Components
| Component | Role |
|---|---|
| celld-operator | Watches WorkerApp resources, validates them against its allowlists, reconciles the per-fleet stack, owns rollout sequencing, and (on the leader) polls every fleet's /state to export Prometheus metrics. |
| Deploy pipeline (your CI) | Builds Wrangler projects with celld deploy and publishes deployments to the fleet bucket. celld adopts them in place; the pipeline also bumps appVersion, or lets appVersion: auto follow the bucket. |
| Shared edge | A platform-owned Gateway (Gateway API), an istio-ingressgateway, or an ingress controller. The operator emits one route object per app in the tenant's namespace; TLS belongs to the edge. |
| Object store | Externally operated and qualified. The bucket, its prefix, and its credentials are inputs you supply; the operator neither provisions nor runs the store, and it should live outside the cluster so one incident cannot take out compute and the source of truth together. |
Tenancy: fleet-per-app
Each application gets its own fleet: its own StatefulSet, ServiceAccount, bucket prefix, and credentials. Tenancy lives at the Kubernetes layer, not inside celld.
The isolation argument. celld trusts the application code it runs, so
the design assumes the worst case: a V8 isolate escape hands the attacker the whole node
process. Under fleet-per-app, that node holds only the tenant's own cells, and its
bucket credential (one identity per fleet, scoped to …/apps/<app>/ and
nothing else) reaches only the tenant's own prefix. The blast radius of a full
runtime compromise is the tenant's own app. For higher assurance tiers, pin
fleets to dedicated node pools with spec.scheduling.nodeSelector and
tolerations for kernel-level separation.
The economics. The unit of tenancy is a fleet (two to three pods minimum
for HA), not an isolate. Upstream sizing: one 8 GB node holds about 1,000 resident
cells, and inactive cells cost almost nothing. Fleets can share one physical bucket via
key prefixes (s3://BUCKET/PREFIX), which keeps store administration
centralized while credential scoping keeps authority per fleet.
What gets reconciled per WorkerApp
Every child object is named <app>-celld (the headless peer Service is
<app>-celld-internal) and is owned by the WorkerApp. The operator refuses
to modify an object of that name that it does not control.
| Object | Purpose |
|---|---|
| ServiceAccount | The fleet identity; carries the IRSA role or AKS workload-identity client ID. |
Service <app>-celld-internal | Headless, publishNotReadyAddresses, port 8081: stable per-pod peer DNS that still resolves while a pod drains. |
Service <app>-celld | The serving Service on port 8080; type and annotations from spec.service. |
| NetworkPolicy | :8081 only from fleet pods and the operator's manager pods; :8080 open to the edge. |
| PodDisruptionBudget | maxUnavailable: 1, so node maintenance is as serialized as rollouts. |
| StatefulSet | The fleet itself; its rolling-update partition belongs to the rollout controller. |
| HTTPRoute, VirtualService, or Ingress | Only when spec.hostnames is set and the ingress mode is not none. Obsolete route kinds are removed when the mode or hostnames change. |
| Istio AuthorizationPolicy | When the Istio CRDs exist: :8081 limited to the fleet's and the operator's principals. |
| KEDA ScaledObject | Only with spec.autoscaling.enabled; deleted again when autoscaling is turned off. |
The fleet pod
A StatefulSet rather than a Deployment: pods need a stable advertise identity for the peer protocol. Details the operator sets on every pod:
- Environment:
CELLD_BUCKET,CELLD_INTERNAL_ADDR=0.0.0.0:8081,CELLD_ADVERTISE=$(POD_NAME).<app>-celld-internal.<ns>.svc.<clusterDomain>:8081,CELLD_WATCH=/var/lib/celld,CELLD_SHUTDOWN_TOTAL_MS=40000,CELLD_MAX_RESIDENT_CELLS,CELLD_MAX_RSS_MB(80% of the container memory limit, set explicitly so the ceiling is visible in the pod spec; celld would derive the same value from the cgroup limit),CELLD_DURABILITY, and the telemetry variables. Optional:S3_ENDPOINT,AWS_REGION,AZURE_STORAGE_ACCOUNT_NAME,CELLD_TRUST_FORWARDED_HEADERS=1, and, before celld v0.5 only,CELLD_VARS_FILE. - Probes: a TCP startup probe on :8080 with a ten-minute budget;
readiness is HTTP GET
/.well-known/celld/health, which answers 503 during a graceful drain and pulls the pod from EndpointSlices; liveness is TCP-socket only, because an HTTP liveness probe on the health path would kill nodes mid-drain. - Termination:
terminationGracePeriodSeconds: 60around celld's 40-second total shutdown bound, so the kubelet does not SIGKILL a node that is still handing off and sealing. - Hardening: UID/GID 10001, non-root, read-only root filesystem,
writable
/tmpand/var/lib/celldvolumes only, all capabilities dropped,RuntimeDefaultseccomp, and no automatically mounted service-account token. - Storage:
/var/lib/celldis an emptyDir by default. Withspec.storageit is a PersistentVolumeClaim per ordinal, retained on scale-down and on deletion, and pods require distinct hosts (spreadAcrossZonesadditionally requires distinct zones). Every fleet also gets a soft one-per-host topology spread. - Annotations:
celld-operator.io/config-hashdigests every referenced Secret so a rotation rolls the fleet;celld-operator.io/app-versioncarries the expected deployment under the defaultRestartdeployment policy, so a version bump triggers the gated rollout. - The StatefulSet's
rollingUpdate.partitionis owned by the rollout controller, never by hand and never directly by GitOps.
The celld behaviors the design depends on
Every operator behavior traces to a celld behavior, verified against celld v0.6.2:
| celld behavior | Operator consequence |
|---|---|
Two listeners: public (--listen, serves the Worker, reserves only /.well-known/celld/health) and internal (--internal-listen, peer protocol + an unauthenticated alpha operator API). | Separate Services; the internal listener gets NetworkPolicy + AuthorizationPolicy and is never routed. |
Peers dial the --advertise address; peer requests carry an HMAC, but celld terminates no TLS and peer traffic is plaintext. | Headless-service DNS for advertise; encryption is the cluster's job (ambient mesh or an encrypted CNI). |
| One fleet runs one application. | Fleet-per-app tenancy. |
v0.4+ polls deploy/current.json every 30 seconds and adopts a new deployment in place; running requests finish on the old generation. | appVersion is the expected deployment; completion requires every pod to report it. A gated restart is retained by default and optional with InPlace. |
A replacement's first healthy response waits for fleet recovery; /state.restoring is the activation backlog. | The operator still steps the partition one ordinal at a time and requires a live, complete restoring=0 sweep: a conservative fleet-wide backstop. |
SIGTERM drains: health flips to 503, new requests get 503 and a close, cells hand off. CELLD_SHUTDOWN_TOTAL_MS (default 40s) bounds the whole stop. | Pin 40s, grant 60s of termination grace, retry 503s at the edge where possible, never probe liveness over HTTP. |
| The store must enforce conditional create/overwrite, exact ranged reads, and read-after-write consistency; every node probes it at startup. Bucket credentials are fleet-admin authority. | Qualified-store list, celld diagnose plus hack/cas-hammer for anything else, per-fleet credential scoping, and immutable bucket identity. |
| Several version boundaries are not rolling-safe; security fixes land on the latest release only. | A compatibility table refuses unsafe Rolling changes, requires Recreate, and blocks downgrades below v0.3 outright. |
| Telemetry writes Parquet to the bucket or OTLP to a collector; celld records no metrics. | Telemetry env from spec.telemetry; the operator's /state-derived Prometheus export is the metrics plane. |
Placement is traffic-driven for active cells; v0.6.2 rebalances only hibernated cells. Pressure shedding uses CELLD_MAX_RSS_MB. | Scale up early, expect slow absorption of active load, set memory bounds explicitly from the pod limit. |
| A node whose lease lapses self-fences and exits with code 3; upstream requires an unlimited restarting supervisor with at least one lease lifetime (10s) between attempts. | The kubelet's restart policy and CrashLoopBackOff supply that; celld_self_fenced makes a fence loop visible. |
fleet durability acknowledges a write once follower nodes fsync it; a one-node fleet cannot complete a fleet proof. | The operator defaults to bucket. fleet requires persistent storage, at least three replicas, and required host separation. |
Trust boundaries
| Boundary | Mechanism |
|---|---|
| Internet → Worker | Edge TLS termination and per-host routing. Application-level auth is the tenant's job; celld does not authenticate end users. |
| Tenant ↔ tenant | Namespaces, NetworkPolicy, per-fleet bucket credentials; node-pool separation via spec.scheduling. |
| WorkerApp author → cluster | Operator allowlists for images, IAM roles, Azure identities, and bucket endpoints. Authors remain trusted: they can reference Secrets in their namespace. |
| Anything → operator API (:8081) | Unauthenticated upstream → NetworkPolicy + AuthorizationPolicy; only fleet pods and the operator reach it; never routed. |
| Node ↔ node (peer protocol) | celld's HMAC, clock limit, and replay protection, plus transport encryption from your mesh or CNI. |
| Fleet → bucket | One prefix-scoped identity per fleet. The bucket credential is fleet-admin authority, so scoping is the tenancy enforcement. |
| Operator → fleets | NetworkPolicy admits only the manager pods (by label) in the operator namespace; with Istio, the AuthorizationPolicy narrows it to the operator's principal. |
Repository layout
api/v1alpha1: theWorkerApptypes and CEL validation.internal/controller:fleet_resources.go(resource builders),rollout.go(the gated rollout state machine and version boundaries),fleetstate.go(the/statepoller and metrics),deploytracker.go(appVersion: auto), andvalidation.go(admission and allowlist checks).hack/cas-hammer: the store-qualification concurrency hammer.dist/chart: the Helm chart.docs/: the celld behaviors index, the production guide, and the production audit.