Architecture

How one WorkerApp becomes one operated fleet, and why the design looks the way it does. The celld behaviors each guardrail encodes are indexed in the repository's celld behaviors reference; this page is the working summary.

The coordination model

celld nodes coordinate through an object-storage bucket (deployments, per-cell SQLite state, ownership leases) with no consensus service, no membership protocol, and no failure detector. Object-storage compare-and-swap guarantees exactly one node owns a cell at a time, and celld does not acknowledge a write before it is durable in the bucket (RPO=0). The bucket is the source of truth; nodes are replaceable.

The operator's job is everything around that: turning one WorkerApp into one fleet of celld pods sharing a bucket prefix, and encoding celld's operational rules so they cannot be misconfigured.

Components

Component	Role
celld-operator	Watches `WorkerApp` resources, reconciles the full per-fleet stack, owns rollout sequencing, exports fleet metrics.
Deploy pipeline (your CI)	Builds Wrangler projects with `celld deploy` and publishes deployments to the fleet bucket, then bumps `appVersion` (or lets `appVersion: auto` pick it up).
Shared edge	One platform-owned Gateway (Gateway API); one HTTPRoute per app in the tenant's namespace, attached by `parentRefs` and authorized by the Gateway's own `allowedRoutes`. TLS belongs to the Gateway.
Object store	Externally operated and qualified. The bucket, its prefix, and its credentials are inputs you supply; the operator neither provisions nor runs the store, and it should live outside the cluster so one incident cannot take out compute and the source of truth together.

Tenancy: fleet-per-app

Each application gets its own fleet: its own StatefulSet, namespace, bucket prefix, and credentials. Tenancy lives at the Kubernetes layer, not inside celld.

The isolation argument. celld adds no sandbox beyond V8, so the design assumes the worst case: a V8 isolate escape hands the attacker the whole node process. Under fleet-per-app, that node holds only the tenant's own cells, and its bucket credential (one IAM role per fleet, locked to …/apps/<app>/ and nothing else) reaches only the tenant's own prefix. The blast radius of a full runtime compromise is the tenant's own app. For higher assurance tiers, pin fleets to dedicated node pools for kernel-level separation — which today means a cluster-level mechanism such as a namespace default node selector or an admission policy, since WorkerApp exposes no scheduling fields.

The economics. The unit of tenancy is a fleet (two to three pods minimum for HA), not an isolate. Upstream sizing: one 8 GB node holds ~1,000 resident cells, and inactive cells cost approximately nothing. Fleets can share one physical bucket via key prefixes (celld supports s3://BUCKET/PREFIX), which keeps store administration centralized while IAM scoping keeps authority per fleet.

What gets reconciled per WorkerApp

StatefulSet

A StatefulSet rather than a Deployment: pods need a stable advertise identity for the peer protocol, and stable identity enables the handoff=preserve fast-reload optimization later. Details the operator sets:

A headless Service <app>-celld-internal; each pod advertises $(POD_NAME).<app>-celld-internal.<ns>.svc.cluster.local:8081.
Environment: CELLD_BUCKET, CELLD_INTERNAL_ADDR=0.0.0.0:8081, CELLD_ADVERTISE, CELLD_SHUTDOWN_DRAIN_MS=25000, CELLD_MAX_RESIDENT_CELLS, CELLD_MAX_RSS_MB (~80% of the container memory limit, set explicitly so the ceiling is visible in the pod spec rather than inferred — celld would otherwise derive the same 80% from the cgroup limit itself), CELLD_VARS_FILE from the vars Secret, and CELLD_OTEL=1 when telemetry is enabled.
terminationGracePeriodSeconds: 40: drain bound plus margin, so the kubelet never SIGKILLs a draining node.
Readiness probe: HTTP GET /__celld/health on :8080. celld answers 503 during a graceful drain, which pulls the pod from EndpointSlices, the built-in drain signal.
Liveness probe: TCP-socket only. An HTTP liveness probe on the health path would kill nodes mid-drain, converting every graceful handoff into the abrupt-kill path.
Pod-template annotation celld-operator.io/app-version, the declarative rollout trigger, since nodes load their deployment from the bucket at startup only.
The StatefulSet's rollingUpdate.partition is owned by the rollout controller, never by hand and never directly by GitOps.

Networking and security wrapping

Public Service on :8080; ingress per the configured ingress mode, each carrying 503-retry policy, and WebSocket timeout policy in the modes that can express it.
NetworkPolicy: :8081 ingress from fleet pods and the operator's namespace only. Reinforced by an Istio AuthorizationPolicy when Istio is installed. The internal listener is never routed.
publishNotReadyAddresses on the headless service, so peers can reach draining pods while they hand off cells.
PodDisruptionBudget maxUnavailable: 1, so node maintenance stays as serialized as rollouts.
Per-fleet ServiceAccount; bucket credentials via EKS IRSA (GKE Workload Identity is not wired yet) (celld's credential chain accepts web identity tokens) or a Secret.

The celld facts the design depends on

Every operator behavior traces to a documented celld fact (as of celld v0.2.0):

celld fact	Operator consequence
Two listeners: public (`--listen`, serves the Worker) and internal (`--internal-listen`, peer protocol + unauthenticated operator API).	Separate Services; the internal listener gets NetworkPolicy + AuthorizationPolicy and is never routed.
Peers dial the `--advertise` address; celld terminates no TLS; docs require a trusted network or encrypted overlay.	Headless-service DNS for advertise; ambient-mesh mTLS (or an encrypted CNI) for the pod network.
One fleet runs one application deployment.	Fleet-per-app tenancy.
Nodes load the deployment from the bucket at startup only.	Deploys are bucket-publish plus rolling restart; the template annotation is the trigger.
Rolling-update rule: wait for every node to report `restoring=0` before the next restart; the restore work lands on the peers.	A vanilla rolling update is insufficient; the operator runs a partition-stepped, restoring-gated rollout.
SIGTERM starts a graceful drain bounded by `CELLD_SHUTDOWN_DRAIN_MS` (default 25s); health flips to 503; new requests get 503+close.	Termination grace 40s; 503 retries at the gateway; no HTTP liveness probe.
The bucket must enforce conditional create/overwrite and read-after-write consistency; bucket credentials are fleet-admin authority.	Qualified-store list, qualification procedure, per-fleet credential scoping.
Some celld upgrades are explicitly not rolling-safe (v0.1 ↔ v0.2 mixed fleets break); security fixes land on the latest release only.	The Recreate strategy and a version-compatibility gate; latest-only support policy.
Telemetry writes Parquet traces/logs to the bucket or OTLP; W3C `traceparent` propagates; no metrics signal yet.	Per-tenant observability from the tenant's own prefix; the operator's `/state`-derived Prometheus export is the metrics plane.
No rebalancing on node join; placement is traffic-driven; pressure shedding via `CELLD_MAX_RSS_MB`.	Scale up early, expect slow absorption; memory bounds set explicitly from the pod limit.

Trust boundaries

Boundary	Mechanism
Internet → Worker	Gateway: TLS termination, per-host routing. Application-level auth is the tenant's job; celld does not authenticate end users.
Tenant ↔ tenant	Namespace + NetworkPolicy + fleet-scoped IAM; node-pool separation via cluster-level policy.
Anything → operator API (:8081)	Unauthenticated by design upstream → NetworkPolicy + AuthorizationPolicy; only fleet pods and the operator reach it; never routed.
Node ↔ node (peer protocol)	celld's own HMAC/replay auth plus ambient mTLS for confidentiality.
Fleet → bucket	Per-fleet IAM role, prefix-scoped. The bucket credential is fleet-admin authority, so scoping is the tenancy enforcement.
Operator → fleets	NetworkPolicy admits the operator's namespace; with Istio installed, the AuthorizationPolicy narrows it to the operator's ServiceAccount.

Repository layout

api/v1alpha1: the WorkerApp types.
internal/controller: fleet_resources.go (resource builders), rollout.go (the gated rollout state machine), fleetstate.go (the /state poller and metrics).
hack/cas-hammer: the store-qualification concurrency hammer.
dist/chart: the Helm chart.