Deploying updates
celld v0.4 and newer follow the bucket on their own: every node polls
deploy/current.json every 30 seconds and adopts a new deployment in place.
The operator's job is to make the resource say what the fleet should serve, verify that
every pod actually serves it, and gate everything that does need a restart.
How a deploy reaches the fleet
celld deploy publishes a deployment to the bucket and moves the pointer. Each
node adopts it at its next poll: new requests use the new generation, requests already
running finish on the old one, and resident Durable Objects move at safe points. No pod
restart is needed to serve new code.
spec.appVersion is therefore the expected deployment, not a pin on
what celld serves. The operator records it in status.expectedAppVersion,
reports each pod's live version, generation, and draining/swapping counts in
status.deployments, and sets status.rolledOutAppVersion only when
every pod serves the expectation with nothing draining or swapping. The version string is
the Version ID that celld deploy prints, matched literally.
Deployment policy
deploymentPolicy | When appVersion changes |
|---|---|
Restart (default) | The expected version is stamped on the pod template, so a change also runs the gated rollout. celld does not need the restart to adopt code; it gives the resource an explicit convergence point. |
InPlace | No restart. The fleet adopts through celld's own polling, and the operator waits for the observed deployments to converge. |
Every other pod-template change (image, resources, durability, telemetry, Secret rotation, scheduling) rolls under either policy.
Pinned mode: appVersion in git
$ celld deploy . --bucket s3://platform-cells/apps/chat … # 1. publish
$ kubectl patch workerapp chat -n tenant-acme --type merge \
-p '{"spec":{"appVersion":"7b21e0c4a9d3f508"}}' # 2. declare
Or bump appVersion in git and let your GitOps tool apply it. Because nodes
adopt the bucket pointer regardless of the resource, give pinned fleets a drift check:
with deployTrackingSecretRef set, the operator reads
deploy/current.json and raises DeployTrackingReady: False, reason
VersionMismatch, when the bucket and the resource disagree. Without it, the
operator never reads the bucket and drift shows up only as a rollout that never
converges.
Auto mode: follow the bucket
Prefer celld deploy to be the whole story? Set appVersion: auto
and a dedicated, read-only tracking credential:
spec:
appVersion: auto
deployTrackingSecretRef: chat-deploy-reader # AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
# optional AWS_SESSION_TOKEN
The operator polls the pointer every --deploy-poll-interval (default 60s)
and resolves it into the expected version. Grant the Secret read access to the fleet's
deploy/current.json and nothing else. Tracking never falls back to the
manager's ambient identity and never assumes the fleet's own role.
s3://buckets only. The CRD rejectsautowithout ans3://bucket and a tracking Secret; ags://oraz://fleet admitted by an older CRD reportsDeployTrackingReady: UnsupportedStore. PinappVersionthere.- An unreachable bucket never moves the expectation. With a cached
value the operator holds it (
BucketUnreachable, message holding last known version …); with none it keeps the version the live pod template already declares. Failures back off, and the pointer is capped at 64 KiB.
On success the condition is DeployTrackingReady: True, reason
Tracking, message following <version>.
The gated rollout
Vanilla StatefulSet rolling updates gate each step on the new pod's readiness. celld's restore work lands on the peers that absorbed the drained node's cells, which a stock rolling update cannot see. Running ahead of the fleet's real recovery is not a data-loss risk, but it is a cold-start latency storm: warm capacity leaves faster than the fleet re-warms. celld v0.4+ already holds a replacement's first healthy response until the fleet recovers; the operator adds a conservative fleet-wide backstop on top:
on pod-template change:
pause KEDA and wait for Paused=True, with no HPA targeting the fleet
refuse if the celld change crosses a non-rolling boundary
partition ← live replica count # freeze: no pod updates yet
while partition > 0:
gate 1: the StatefulSet has observed the change and a 10s settle
interval since the last release has passed
gate 2: every released ordinal (≥ partition) runs the new revision
and is Ready
gate 3: every expected pod exists, is Ready, answers /state, and the
fleet-wide sum is restoring == 0
all pass → partition ← partition − 1 # release exactly one more pod
Ready once every pod serves the expected deployment, nothing draining
There is no timeout. A rollout that cannot pass a gate waits in phase
RollingOut, re-checking every 10 seconds, and never proceeds past a churning
fleet. status.rollout.waitingOn says why, for example
chat-celld-2: not ready, waiting for rollout settle interval,
waiting for KEDA pause and HPA removal, or
fleet state: restoring=3, …. Alert on a WorkerApp that stays in
RollingOut.
A fleet that is not healthy will not roll. Gate 3 needs every pod
Ready and reachable, so a fix for a fleet that never became healthy cannot step through
a Rolling update. Set celld.updateStrategy: Recreate to replace every pod at
once, then return to Rolling.
While the fleet is not Ready the ScaledObject stays paused, so KEDA and the
partition controller never fight over replica count. Unmanaged HPAs that target the same
StatefulSet also block rollouts. Where the edge retries drain 503s, rollouts are
invisible to clients; see Networking & ingress.
celld version upgrades
Bumping spec.celld.image rolls the fleet the same gated way, except across
boundaries that upstream flags as not rolling-safe. The operator checks
every boundary between the old and new versions, in both directions, against both pod
templates and every live pod, so an unfinished rollout cannot hide an older runtime. A
Rolling request across a flagged boundary is refused with phase: Degraded and
the upstream reason.
| Change | Rolling? | Why |
|---|---|---|
| v0.1 ↔ v0.2 | No | Ownership records changed address semantics and block objects changed format. |
| v0.2.1 → v0.3.0 | Yes | A v0.3 node that cannot replicate to a v0.2 peer falls back to bucket proofs until the peer upgrades. |
| v0.3+ → v0.2 or older | Blocked | Refused even with Recreate: pod disappearance cannot prove every node sealed its log. Migrate to a new WorkerApp instead. |
| v0.3 ↔ v0.4 | No | v0.4's versioned peer tunnel refuses v0.3 peers, and v0.3 cannot read v0.4's large Workers KV value references. |
| v0.4 ↔ v0.5 | No | Upstream requires stopping the whole fleet first. |
| v0.5 ↔ v0.6 | Bucket only | v0.6 changes fleet replication streams. Rolling is allowed only when both pod templates use bucket durability. |
| v0.6.0 → v0.6.1 → v0.6.2 | Yes | Finish the runtime upgrade before enabling features that need the newer release. |
To cross a non-rolling boundary, the resource has to ask for it:
celld:
image: ghcr.io/denoland/celld:v0.6.2
updateStrategy: Recreate
Recreate pauses autoscaling, scales the fleet to zero, records the transition on the
StatefulSet, waits until an uncached census finds no old pods (terminating ones
included), then starts the new version. That is an availability event by
design, and status surfaces it as phase: Recreating. An image whose tag
the operator cannot parse also requires Recreate; image tags must carry a numeric
major.minor.patch version.
Rollbacks have their own rules. Do not downgrade to v0.4.0 after paged restore has activated on v0.4.1+; the older runtime cannot restore paged cells. Before rolling back from v0.6.1+, remove Python deployments and the settings that need it, including epoch GC. See the upstream upgrade notes. celld ships security fixes for its latest release only, so plan to track head and keep the operator current alongside it.
Rotating Secrets
The operator digests every Secret the pod template references (the bucket credential
secretRef, and vars.secretRef on pre-v0.5 images) into the
celld-operator.io/config-hash annotation. A rotation changes the template
and rolls the fleet through the same gated rollout. The tracking Secret is read
directly by the operator and needs no rollout.
Worker variables on celld v0.5+ live in the deployment's Wrangler vars, so a
change is a new celld deploy. spec.vars exists only for older
images, and the operator rejects it on v0.5+; remove it before upgrading.
Changes that need a new WorkerApp
Bucket identity (name, endpoint, storage account) and spec.storage are
immutable; an existing ephemeral StatefulSet cannot gain volume claims in place. To
change them, plan a maintenance migration: stop writes, verify the runtime has uploaded
acknowledged data, stop the old fleet, create a new WorkerApp, then resume traffic.
Never run two fleets on one bucket prefix at the same time.
CI validation
- Pull request: pinned esbuild +
celld deploy . --dry-run. Unsupported config keys fail here with a clean named-key error, not at runtime. - Merge:
celld deploy .against the fleet's bucket prefix. Nodes adopt the new deployment within about 30 seconds. - Declare: bump
spec.appVersion(GitOps), or letappVersion: autopick it up, and watch forReadywith the new version in theAPPcolumn.