Deploying updates

celld nodes load their application deployment from the bucket at startup only, so publishing a new version serves nothing until the fleet restarts. Every deploy is therefore two steps: publish to the bucket, then roll the fleet. The operator owns the second step and makes it safe.

Pinned mode: appVersion in git

$ celld deploy . --bucket s3://platform-cells/apps/chat …    # 1. publish
$ kubectl patch workerapp chat -n tenant-acme --type merge \
    -p '{"spec":{"appVersion":"7b21e0c4a9d3f508"}}'           # 2. roll

Or bump appVersion in git and let your GitOps tool apply it. The version string is the Version ID celld deploy prints — a 16-character hex digest of the deployment, not a git SHA — and the operator matches it literally against the bucket pointer.

In pinned mode, a fleet whose credentials come from bucket.credentialsFrom.secretRef also gets a drift check: the operator reads deploy/current.json and raises DeployTrackingReady: VersionMismatch when the bucket and the resource disagree, because nodes load the bucket's version regardless of what the resource says. Fleets authenticating with iamRole get no such check — the operator does not read the bucket for them, so drift stays silent.

Auto mode: follow the bucket

Prefer celld deploy to be the whole story? Set appVersion: auto and the operator follows the bucket's deploy/current.json itself. A new publish rolls the fleet within one poll interval (--deploy-poll-interval, default 60s), with no resource edit, and status.rolledOutAppVersion reports the concrete version being served.

Tracking reads use the fleet's secretRef credentials when set, or the operator's ambient AWS identity otherwise, and are cached per fleet for one poll interval. Two limits are worth knowing before you rely on auto mode:

• s3:// buckets only. A gs:// fleet reports DeployTrackingReady: BucketUnreachable permanently; pin appVersion instead.
• An unreachable bucket never rolls the fleet anywhere. With a cached value the operator holds it (BucketUnreachable, message holding last known version …); with no value ever read it falls back to the version the live fleet already runs. A tracking outage freezes the version, it does not change it.

On success the condition is DeployTrackingReady: True, reason Tracking, message following <version>.

The gated rollout

Vanilla StatefulSet rolling updates gate each step on the new pod's readiness. celld's documented rule is different: after restarting a node, wait until every node reports restoring=0 before restarting the next, and that restore work lands on the peers that absorbed the drained node's cells, which a stock rolling update cannot see. Running ahead of the fleet's real recovery is not a data-loss risk (the bucket is authoritative, RPO=0), but it is a cold-start latency storm: warm capacity leaves faster than the fleet re-warms.

The operator implements the documented procedure with the StatefulSet partition:

on template change (appVersion, image, or secret rotation):
  partition ← live replica count          # freeze: no pod updates yet
  while partition > 0:
    gate 1: every already-released ordinal (≥ partition) runs the
            new revision and is Ready
    gate 2: the fleet reports /state restoring == 0
    both pass → partition ← partition − 1  # release exactly one more pod
  # status.rolledOutAppVersion is set later, once the StatefulSet
  # itself reports fully updated and ready.

Both gates are evaluated before each step, so a fleet must already be at restoring=0 before the first pod is released. Gate 2 polls every running pod, with one deliberate exception: a pod that is both unreachable and not Ready is skipped, because it holds no cells — otherwise a fleet that was never healthy could never roll out the fix for whatever broke it. A pod that is Ready but unreachable does hold the gate.

There is no timeout. A rollout that cannot pass a gate waits indefinitely in phase RollingOut, re-checking every 10–15 seconds, and never proceeds past a churning fleet. Watch status.rollout.waitingOn to see why — for example chat-celld-2: not ready, fleet: restoring=3, or state unreachable: pod chat-celld-1: …. Alert on a rollout that stays in RollingOut, not on Degraded: the operator reserves Degraded for a refused breaking upgrade.

Whenever the fleet is not Ready — rolling out, recreating, converging, or degraded — the autoscaler is paused and replicas are pinned, so KEDA and the partition controller never fight over replica count. Where the edge retries drain 503s, rollouts are invisible to clients; see Networking & ingress.

celld version upgrades

Bumping spec.celld.image rolls the fleet the same gated way, except across boundaries upstream flags as not rolling-safe. celld v0.1.0 → v0.2.0 was one: advertise records and block-object formats changed, and mixed fleets are forbidden.

The operator maintains a version-compatibility table from celld release notes and refuses a Rolling strategy across a flagged boundary with phase: Degraded, so a GitOps diff alone can never create a mixed fleet. To cross the boundary, the resource must explicitly set:

celld:
  image: ghcr.io/denoland/celld:v0.2.0
  updateStrategy: Recreate

Recreate scales the fleet to zero, waits for every node to drain, then starts the new version. That is an availability event by design; the resource has to ask for it, and status surfaces it as phase: Recreating.

Rotating variables and secrets

Worker vars come from a Kubernetes Secret (key vars.env, NAME=value lines) mounted and passed via CELLD_VARS_FILE. The operator digests every referenced Secret into a pod-template annotation, so a rotation changes the template and rolls the fleet through the same gated rollout. Values are never baked into bundles.

CI validation

The recommended pipeline:

1. Pull request: pinned esbuild + celld deploy . --dry-run. Unsupported config keys fail here with a clean named-key error, not at runtime.
2. Merge: celld deploy . against the fleet's bucket prefix. The bucket now holds the new deployment; nothing is serving it yet.
3. Roll out: bump spec.appVersion (GitOps), or let appVersion: auto pick it up.