Object stores
The bucket is celld's control plane and durability root. It must enforce conditional writes atomically, return exact byte ranges, and read a write back immediately. A store that merely accepts the conditional headers without enforcing them fails late and silently, as two nodes owning one cell.
The storage contract
celld requires four properties from the store:
- Conditional create: a write that fails when the object already
exists (
If-None-Match: *on S3-compatible stores). - Conditional overwrite: a write that fails when the object changed
since it was read (ETag compare-and-swap via
If-Match; on GCS, thex-goog-if-generation-matchprecondition). - Read-after-write consistency: a read after a successful write returns that write.
- Ranged reads: a read returns exactly the requested byte range.
Epoch GC (CELLD_LTX_RETENTION_SECS, off by default) additionally needs
list-after-write consistency. The operator exposes no field for it.
How celld uses it
Each cell has one ownership record naming the owner node's session and a
fencing epoch. A node acquires a cell with a conditional create, or a
compare-and-swap on an existing record, so the store admits exactly one winner, and
every activation advances the epoch. The replicator copies each cell's SQLite data to
cells/<cell>/ltx/e<epoch>/ with plain writes: the epoch in the key
is the fence, so a node that lost ownership can only write into a superseded prefix.
The acknowledgement rule holds every response until a durability proof covers the writes it can reveal. After a bucket proof, celld re-reads the ownership record and acknowledges only if it still names this node at this epoch; because it is a read, not a clock comparison, a paused process or a skewed clock cannot pass it. A fleet proof instead waits for follower nodes to fsync the write, and a takeover seals the prior owner's node-log session before it restores, so a stale owner cannot complete another fleet proof. A node that cannot renew its lease fences itself: node failure is a normal input, not a recovery procedure.
Qualified and disqualified stores
| Status | Stores |
|---|---|
| Qualified | Amazon S3, Cloudflare R2, Tigris, Google Cloud Storage, and Azure Blob Storage. celld's release tests run against R2; the S3 path uses the same client and headers. Azure was qualified under an account key, a VM managed identity, and an AKS workload identity. |
| Not for production | MinIO community edition: it passes the storage test but is not qualified for production. Avoid RELEASE.2025-09-06T17-38-46Z, which rejects the first deploy. Azurite is for local development only. |
| Not qualified | Backblaze B2, Hetzner Object Storage, DigitalOcean Spaces. They do not implement the required conditional writes: two nodes can own one cell. |
| Unproven | Anything else. Treat it as disqualified until you qualify it yourself with the checks below. |
Dialects and credentials
celld speaks three dialects, so spec.bucket.name is always
s3://, gs://, or az://, followed by the bucket and an
optional prefix. The CRD rejects anything else, and the bucket name, endpoint, and
storage account are immutable once set: moving a fleet to a new bucket means a new
WorkerApp.
| Dialect | Notes | Credentials |
|---|---|---|
s3:// | AWS S3, R2, Tigris, and other S3-compatible stores. endpoint (HTTPS, and on the operator's allowedBucketEndpoints list) for anything but AWS; region when it cannot be inferred (auto for R2). | iamRole (EKS IRSA); EKS Pod Identity with credentialsFrom left empty; or secretRef with AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. |
gs:// | Cloud Storage XML API with generation preconditions. celld rejects an endpoint and S3 credentials, and ignores the region. | Google OAuth credentials. The operator does not wire GKE Workload Identity yet. |
az:// | The name is a Blob Storage container; storageAccount is required. celld rejects an endpoint. Public Azure cloud only. | azureClientID (AKS workload identity, needs Storage Blob Data Contributor); secretRef with AZURE_STORAGE_ACCOUNT_KEY; or a node managed identity with only storageAccount set. |
appVersion: auto reads the deploy pointer over the S3 API, so it works only
for s3:// buckets; gs:// and az:// fleets pin
appVersion.
Qualifying a store yourself
Before trusting any store not on the qualified list, and again after every store upgrade, run both checks:
# 1. celld's own sequential contract test: create, reject-create, update,
# reject-stale, against the real bucket (s3://, gs://, or az://).
$ celld diagnose --bucket <bucket> [--endpoint <url>] [--region <region>]
# 2. The operator repo's concurrency hammer: N racers, exactly one winner
# per round. S3 dialect only.
$ go run ./hack/cas-hammer --bucket <bucket> --endpoint <url> \
--writers 8 --rounds 32
The sequential test proves the condition is applied at all; every celld node repeats it, plus a ranged-read check, before it serves, and stops if the store violates the contract. Only the hammer probes atomicity under race.
Read both failing hammer exit codes as disqualifying. Exit 1 means a
round admitted more than one winner: the store cannot fence celld cells. Exit 2 means
the run could not complete, which includes a conditional PUT answering with something
other than a precondition failure. Only exit 0 qualifies. Both tests write: the probe
uses the reserved probe/ prefix, the hammer its own --prefix.
Prefixes and credentials
A fleet's bucket may carry a key prefix, such as s3://platform-cells/apps/chat,
and every fleet object lives under it. celld reserves probe/,
cells/, nodes/, node-cells/, fleet/,
deploy/, deploy-blobs/, log/, wake/, and
telemetry/ under the prefix and deletes objects under some of them, so
applications must not write there. Many fleets can share one physical bucket while
credential scoping keeps authority per fleet.
Bucket credentials are fleet-admin authority. The bucket holds
deployments, cell state, leases, and the peer-authentication secret, so whoever holds
the credential controls the fleet. Scope one identity per fleet to that fleet's prefix
and nothing else, grant put and delete there (the startup probe writes and
removes an object under probe/), prefer workload identity over static keys,
and rotate immediately after suspected disclosure.
Placement and failure domains
- The operator never runs the store. Use a managed store, or operate your own outside the fleet's failure domain.
- Keep the store out of the workload cluster. A cluster incident must not take out compute and the source of truth together.
- RPO=0 is inherited from the store's durability. That is the argument for a managed store even in otherwise self-hosted deployments.