Observability

Two planes, both per-tenant by construction: celld's built-in tracing (spans and logs, written to the tenant's own bucket prefix or your OTLP collector) and the operator's Prometheus metrics (capacity and health, derived from each pod's /state).

Traces and logs

One field enables celld's telemetry for the fleet:

telemetry:
  enabled: true       # sets CELLD_OTEL=1 (the WorkerApp default)
  retention: 30d      # bucket sink only; "none" defers to bucket lifecycle rules
  # sink: otlp        # or send OTLP/HTTP to a collector instead:
  # otlpEndpoint: http://otel-collector.monitoring.svc:4318

Sink	Behavior
`bucket` (default)	Parquet traces and logs under the fleet bucket's `telemetry/` prefix, partitioned by node and hour. DuckDB queries them directly, with no services to run. celld deletes files older than `retention`.
`otlp`	OTLP/HTTP protobuf to the collector at `otlpEndpoint` (sets `OTEL_EXPORTER_OTLP_ENDPOINT`). Setting `otlpEndpoint` without a sink selects `otlp` automatically. Every fleet reports as its own service: the operator sets `OTEL_SERVICE_NAME` to the WorkerApp name, so a shared collector does not flatten all fleets into celld's default `celld` service.

What celld emits:

A span per stateless-Worker request, per cell event (fetch, alarm, RPC, WebSocket message), per outbound fetch(), and per cell start, with queue waits and durability facts.
Every console.log line as a log record carrying the trace and span ID of the handler that wrote it; the correlation survives await.
W3C traceparent read on incoming requests and sent on outbound fetches, so your gateway's OpenTelemetry tracing joins edge spans to Worker and Durable Object spans with zero app code.

Under load, telemetry sheds before requests do, and celld counts what it sheds. The trace schema is v0-unstable; column names can change before a stable release.

Querying the bucket sink

INSTALL httpfs; LOAD httpfs;
CREATE SECRET r2 (TYPE s3, KEY_ID '…', SECRET '…',
                  ENDPOINT 'ACCOUNT.r2.cloudflarestorage.com', URL_STYLE 'path');
SELECT name, count(*), avg(duration_us)/1000 AS avg_ms
FROM 's3://platform-cells/apps/chat/telemetry/traces/*/*/*/*/*/*.parquet'
GROUP BY name ORDER BY 2 DESC;

celld flushes a Parquet file every 5 minutes by default (or sooner at 5 MiB of buffered events), which bounds how fresh these queries can be; the operator exposes no field for that interval, so treat the bucket sink as minutes-behind rather than live. Run the hourly compaction job or queries grow slow within hours. The job is a DuckDB COPY that rewrites one past hour of small files into one zstd Parquet; run it on a maintenance node, because celld does not compact its own files.

Fleet metrics

celld has no metrics endpoint yet, so the operator polls each pod's internal /state (from the leader only, every --state-poll-interval, default 15s) and exports Prometheus series per WorkerApp and per pod:

Metric	Meaning
`celld_resident_cells`	Occupied (resident) cells per pod
`celld_resident_cell_utilization`	occupied / maxResidentCells (0..1), the primary capacity signal
`celld_restoring`	Cold activations in flight
`celld_evicting`	Cells being evicted
`celld_shedding`	1 while pressure-shedding; the hard out-of-capacity signal
`celld_state_up`	1 if `/state` answered the last poll

The same series drive autoscaling, dashboards, and alerting. Request rates and latencies are derived from spans (or from your gateway's metrics) until upstream ships a native metrics signal.

At-a-glance status

$ kubectl get workerapps -A
NAMESPACE     NAME    PHASE        APP                READY   RESTORING   AGE
tenant-acme   chat    Ready        7b21e0c4a9d3f508   3       0           4d
tenant-beta   docs    RollingOut   sha-11aa22   2       1

PHASE, the rolled-out version, ready pods, and fleet-wide restoring count are printer columns on the CRD; conditions carry the detail. See the WorkerApp status reference.