Object stores

The bucket is celld's control plane and durability root. It must enforce conditional writes atomically and read a write back immediately. A store that merely accepts the conditional headers without enforcing them fails late and silently, as two nodes owning one cell.

The fencing contract

celld requires three properties from the store:

1. Conditional create: a write that fails when the object already exists (If-None-Match: * on S3-compatible stores).
2. Conditional overwrite: a write that fails when the object changed since it was read (ETag compare-and-swap via If-Match; on GCS, the x-goog-if-generation-match precondition).
3. Read-after-write consistency: a completed write is visible to the next read.

How celld uses them: each cell has one ownership record naming the owner node and a fencing epoch. Acquiring a cell is a conditional write, so the store accepts exactly one winner. The replication stream itself uses plain writes — the epoch in the key is the fence, so a fenced node's writes land in a superseded prefix.

Two further mechanisms turn that into the RPO=0 promise. The acknowledgement rule: a write is held until the replicator proves it reached the bucket, and only then does celld re-read the ownership record and acknowledge — and only if the record still names this node at this epoch. Because it is a read, not a clock comparison, a paused process or skewed clock cannot pass it. The epoch seal: the first activation to restore from an epoch writes e<epoch>.seal.json with a conditional create, fixing the highest transaction any later restore of that epoch may read; the first restorer wins, and if the seal write fails the activation fails. Without it, a fenced node's post-takeover appends could resurface in a later restore. A node that cannot reach the bucket cannot renew its lease, stops writing, and releases its residency: node failure is a normal input, not a recovery procedure.

Qualified and disqualified stores

celld speaks exactly two dialects — S3 and Google Cloud Storage — so a bucket spec is either s3://… or gs://… and nothing else. Stores that expose no S3-compatible API (Azure Blob Storage among them) cannot back a fleet, whatever their consistency guarantees; the operator's CRD rejects the spec outright.

For gs:// buckets, celld uses the Cloud Storage XML API with generation preconditions and Application Default Credentials. It rejects an S3 --endpoint for a gs:// bucket, rejects S3 static credentials for one (so credentialsFrom.secretRef is a startup failure, not a fallback), and ignores the storage region.

Qualifying a store yourself

Before trusting any store not on the qualified list, and again after every store upgrade, run both checks:

# 1. celld's own sequential CAS contract test (from the celld repo).
#    S3-compatible store:
$ CELLD_CAS_LIVE=1 CELLD_CAS_BUCKET=<bucket> CELLD_CAS_ENDPOINT=<url> \
    cargo test -p celld put_cas_contract -- --nocapture
#    GCS (no endpoint — celld rejects one for gs://):
$ CELLD_CAS_LIVE=1 CELLD_CAS_BUCKET=gs://<bucket> \
    cargo test -p celld put_cas_contract -- --nocapture

# 2. The operator repo's concurrency hammer (N racers, exactly one winner
#    per round). S3 dialect only:
$ go run ./hack/cas-hammer --bucket <bucket> --endpoint <url> \
    --writers 8 --rounds 32

The sequential test proves the condition is applied at all; only the hammer probes atomicity under race. The hammer speaks the S3 dialect, so a gs:// bucket gets the contract test alone.

Read both failing exit codes as disqualifying. Exit 1 means a round admitted more than one winner — the store cannot fence celld cells. Exit 2 means the run could not complete, and one cause is a conditional PUT answering with something other than a precondition failure, which is equally a store that does not speak the protocol celld depends on. Only exit 0 qualifies.

Placement and failure domains

• The operator never runs the store. Run a managed store, or operate your own outside the fleet's failure domain.
• Keep the store out of the workload cluster. A cluster incident must not take out compute and the source of truth together.
• RPO=0 is inherited from the store's durability. That is the argument for a managed store even in otherwise self-hosted deployments.

Prefixes and credentials

A fleet's bucket may carry a key prefix, such as s3://platform-cells/apps/chat, and all fleet objects (deployments, SQLite replicas, ownership records, node leases, the peer-auth secret, telemetry) live under it. Many fleets can share one physical bucket while IAM scoping keeps authority per-fleet.