celld-operator

Deploying updates

celld v0.4 and newer follow the bucket on their own: every node polls deploy/current.json every 30 seconds and adopts a new deployment in place. The operator's job is to make the resource say what the fleet should serve, verify that every pod actually serves it, and gate everything that does need a restart.

How a deploy reaches the fleet

celld deploy publishes a deployment to the bucket and moves the pointer. Each node adopts it at its next poll: new requests use the new generation, requests already running finish on the old one, and resident Durable Objects move at safe points. No pod restart is needed to serve new code.

spec.appVersion is therefore the expected deployment, not a pin on what celld serves. The operator records it in status.expectedAppVersion, reports each pod's live version, generation, and draining/swapping counts in status.deployments, and sets status.rolledOutAppVersion only when every pod serves the expectation with nothing draining or swapping. The version string is the Version ID that celld deploy prints, matched literally.

Deployment policy

deploymentPolicyWhen appVersion changes
Restart (default)The expected version is stamped on the pod template, so a change also runs the gated rollout. celld does not need the restart to adopt code; it gives the resource an explicit convergence point.
InPlaceNo restart. The fleet adopts through celld's own polling, and the operator waits for the observed deployments to converge.

Every other pod-template change (image, resources, durability, telemetry, Secret rotation, scheduling) rolls under either policy.

Pinned mode: appVersion in git

$ celld deploy . --bucket s3://platform-cells/apps/chat …    # 1. publish
$ kubectl patch workerapp chat -n tenant-acme --type merge \
    -p '{"spec":{"appVersion":"7b21e0c4a9d3f508"}}'           # 2. declare

Or bump appVersion in git and let your GitOps tool apply it. Because nodes adopt the bucket pointer regardless of the resource, give pinned fleets a drift check: with deployTrackingSecretRef set, the operator reads deploy/current.json and raises DeployTrackingReady: False, reason VersionMismatch, when the bucket and the resource disagree. Without it, the operator never reads the bucket and drift shows up only as a rollout that never converges.

Auto mode: follow the bucket

Prefer celld deploy to be the whole story? Set appVersion: auto and a dedicated, read-only tracking credential:

spec:
  appVersion: auto
  deployTrackingSecretRef: chat-deploy-reader   # AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
                                            # optional AWS_SESSION_TOKEN

The operator polls the pointer every --deploy-poll-interval (default 60s) and resolves it into the expected version. Grant the Secret read access to the fleet's deploy/current.json and nothing else. Tracking never falls back to the manager's ambient identity and never assumes the fleet's own role.

On success the condition is DeployTrackingReady: True, reason Tracking, message following <version>.

The gated rollout

Vanilla StatefulSet rolling updates gate each step on the new pod's readiness. celld's restore work lands on the peers that absorbed the drained node's cells, which a stock rolling update cannot see. Running ahead of the fleet's real recovery is not a data-loss risk, but it is a cold-start latency storm: warm capacity leaves faster than the fleet re-warms. celld v0.4+ already holds a replacement's first healthy response until the fleet recovers; the operator adds a conservative fleet-wide backstop on top:

on pod-template change:
  pause KEDA and wait for Paused=True, with no HPA targeting the fleet
  refuse if the celld change crosses a non-rolling boundary
  partition ← live replica count            # freeze: no pod updates yet
  while partition > 0:
    gate 1: the StatefulSet has observed the change and a 10s settle
            interval since the last release has passed
    gate 2: every released ordinal (≥ partition) runs the new revision
            and is Ready
    gate 3: every expected pod exists, is Ready, answers /state, and the
            fleet-wide sum is restoring == 0
    all pass → partition ← partition − 1    # release exactly one more pod
  Ready once every pod serves the expected deployment, nothing draining

There is no timeout. A rollout that cannot pass a gate waits in phase RollingOut, re-checking every 10 seconds, and never proceeds past a churning fleet. status.rollout.waitingOn says why, for example chat-celld-2: not ready, waiting for rollout settle interval, waiting for KEDA pause and HPA removal, or fleet state: restoring=3, …. Alert on a WorkerApp that stays in RollingOut.

A fleet that is not healthy will not roll. Gate 3 needs every pod Ready and reachable, so a fix for a fleet that never became healthy cannot step through a Rolling update. Set celld.updateStrategy: Recreate to replace every pod at once, then return to Rolling.

While the fleet is not Ready the ScaledObject stays paused, so KEDA and the partition controller never fight over replica count. Unmanaged HPAs that target the same StatefulSet also block rollouts. Where the edge retries drain 503s, rollouts are invisible to clients; see Networking & ingress.

celld version upgrades

Bumping spec.celld.image rolls the fleet the same gated way, except across boundaries that upstream flags as not rolling-safe. The operator checks every boundary between the old and new versions, in both directions, against both pod templates and every live pod, so an unfinished rollout cannot hide an older runtime. A Rolling request across a flagged boundary is refused with phase: Degraded and the upstream reason.

ChangeRolling?Why
v0.1 ↔ v0.2NoOwnership records changed address semantics and block objects changed format.
v0.2.1 → v0.3.0YesA v0.3 node that cannot replicate to a v0.2 peer falls back to bucket proofs until the peer upgrades.
v0.3+ → v0.2 or olderBlockedRefused even with Recreate: pod disappearance cannot prove every node sealed its log. Migrate to a new WorkerApp instead.
v0.3 ↔ v0.4Nov0.4's versioned peer tunnel refuses v0.3 peers, and v0.3 cannot read v0.4's large Workers KV value references.
v0.4 ↔ v0.5NoUpstream requires stopping the whole fleet first.
v0.5 ↔ v0.6Bucket onlyv0.6 changes fleet replication streams. Rolling is allowed only when both pod templates use bucket durability.
v0.6.0 → v0.6.1 → v0.6.2YesFinish the runtime upgrade before enabling features that need the newer release.

To cross a non-rolling boundary, the resource has to ask for it:

celld:
  image: ghcr.io/denoland/celld:v0.6.2
  updateStrategy: Recreate

Recreate pauses autoscaling, scales the fleet to zero, records the transition on the StatefulSet, waits until an uncached census finds no old pods (terminating ones included), then starts the new version. That is an availability event by design, and status surfaces it as phase: Recreating. An image whose tag the operator cannot parse also requires Recreate; image tags must carry a numeric major.minor.patch version.

Rollbacks have their own rules. Do not downgrade to v0.4.0 after paged restore has activated on v0.4.1+; the older runtime cannot restore paged cells. Before rolling back from v0.6.1+, remove Python deployments and the settings that need it, including epoch GC. See the upstream upgrade notes. celld ships security fixes for its latest release only, so plan to track head and keep the operator current alongside it.

Rotating Secrets

The operator digests every Secret the pod template references (the bucket credential secretRef, and vars.secretRef on pre-v0.5 images) into the celld-operator.io/config-hash annotation. A rotation changes the template and rolls the fleet through the same gated rollout. The tracking Secret is read directly by the operator and needs no rollout.

Worker variables on celld v0.5+ live in the deployment's Wrangler vars, so a change is a new celld deploy. spec.vars exists only for older images, and the operator rejects it on v0.5+; remove it before upgrading.

Changes that need a new WorkerApp

Bucket identity (name, endpoint, storage account) and spec.storage are immutable; an existing ephemeral StatefulSet cannot gain volume claims in place. To change them, plan a maintenance migration: stop writes, verify the runtime has uploaded acknowledged data, stop the old fleet, create a new WorkerApp, then resume traffic. Never run two fleets on one bucket prefix at the same time.

CI validation

  1. Pull request: pinned esbuild + celld deploy . --dry-run. Unsupported config keys fail here with a clean named-key error, not at runtime.
  2. Merge: celld deploy . against the fleet's bucket prefix. Nodes adopt the new deployment within about 30 seconds.
  3. Declare: bump spec.appVersion (GitOps), or let appVersion: auto pick it up, and watch for Ready with the new version in the APP column.
Edit this page on GitHubUpdated 2026-10-18