Observability
Two planes, both per-tenant by construction: celld's built-in tracing (spans and logs,
written to the tenant's own bucket prefix or your OTLP collector) and the operator's
Prometheus metrics (capacity, memory, and health, derived from each pod's
/state).
Traces and logs
One block configures celld's telemetry for the fleet; it is on by default:
telemetry:
enabled: true # the default; false sets CELLD_OTEL=0
retention: 30d # bucket sink only; "none" defers to bucket lifecycle rules
# sink: otlp # or send OTLP/HTTP to a collector instead:
# otlpEndpoint: http://otel-collector.monitoring.svc:4318
| Sink | Behavior |
|---|---|
bucket (default) | CELLD_OTEL=1: Parquet traces and logs under the fleet bucket's telemetry/ prefix, partitioned by node and hour. DuckDB queries them directly, with no services to run. celld deletes files older than retention (CELLD_OTEL_RETENTION). |
otlp | OTLP/HTTP protobuf to a collector. On celld v0.5+ the operator puts the collector base URL in CELLD_OTEL (default http://localhost:4318 when otlpEndpoint is omitted); older images get the legacy CELLD_OTEL_SINK and OTEL_EXPORTER_OTLP_ENDPOINT. Setting otlpEndpoint without a sink selects otlp. |
In both sinks the operator sets OTEL_SERVICE_NAME to the WorkerApp name, so a
shared collector or a cross-fleet query does not flatten every fleet into celld's
default celld service.
What celld emits:
- A span per stateless-Worker request, per cell event (fetch, alarm, RPC, WebSocket
message), per outbound
fetch(), and per cell start, carrying the queue wait and the known durability facts. - Every console line as a log record with the trace and span ID of the handler that
wrote it, across
await, with severity from the method:debug→ DEBUG,log/info→ INFO,warn→ WARN,error→ ERROR. - W3C
traceparentread on incoming requests and sent on outbound fetches, so your edge's OpenTelemetry tracing joins edge spans to Worker and Durable Object spans with no app code.
Under load, telemetry sheds before requests do, and celld counts what it sheds. The
schema is v0-unstable: column names can change before a stable release.
Querying the bucket sink
INSTALL httpfs; LOAD httpfs;
CREATE SECRET r2 (TYPE s3, KEY_ID '…', SECRET '…',
ENDPOINT 'ACCOUNT.r2.cloudflarestorage.com', URL_STYLE 'path');
CREATE VIEW traces AS SELECT * FROM
read_parquet('s3://platform-cells/apps/chat/telemetry/traces/*/*/*/*/*/*.parquet');
CREATE VIEW logs AS SELECT * FROM
read_parquet('s3://platform-cells/apps/chat/telemetry/logs/*/*/*/*/*/*.parquet');
-- The slowest requests, then every error line.
SELECT name, duration_us, trace_id FROM traces ORDER BY duration_us DESC LIMIT 20;
SELECT time_unix_us, body FROM logs WHERE severity_number >= 17;
celld writes a Parquet file every 5 minutes or at about 5 MiB of buffered events,
whichever comes first, so treat the bucket sink as minutes behind rather than live; the
operator exposes no field for the flush interval. For a near-live view, use the OTLP
sink. celld does not compact its own files: run an hourly DuckDB COPY on a
maintenance node that rewrites the previous hour into one zstd Parquet file, or queries
slow down within hours.
Fleet metrics
celld records no metrics, so the operator's leader polls each pod's internal
/state every --state-poll-interval (default 15s) and exports
gauges labelled namespace, workerapp, and pod.
Responses with missing counters, negative values, runtime errors, or trailing data are
rejected rather than exported.
| Metric | Meaning |
|---|---|
celld_resident_cells | Occupied (resident) cells per pod |
celld_resident_cell_utilization | Occupied / maxResidentCells (0..1), the primary capacity signal |
celld_restoring | Cold activations in flight; the rollout controller's step gate |
celld_evicting | Cells being evicted |
celld_shedding | 1 while pressure-shedding: the hard out-of-capacity signal |
celld_rss_bytes / celld_in_use_bytes | Process RSS, and RSS minus allocator retention |
celld_cgroup_working_set_bytes / celld_cgroup_current_bytes | Active cgroup memory (used by pressure shedding) and the full cgroup charge (used by the absolute cap); absent outside a readable Linux cgroup |
celld_container_restarts | The kubelet's restart count for the celld container |
celld_self_fenced | 1 if the last exit was a celld self-fence (exit code 3) |
celld_state_up | 1 if /state answered the last poll |
celld_fleet_complete | Per app: 1 when the last sweep reached every expected pod |
celld_fleet_sample_timestamp_seconds | Per app: when the last sweep ran |
What to alert on
- A fence loop:
celld_self_fenced == 1with a risingcelld_container_restartsmeans the store or the bucket credential is broken. The kubelet keeps restarting the node, as celld requires, but it will not recover by itself. - Stale or partial samples:
celld_fleet_complete == 0, an oldcelld_fleet_sample_timestamp_seconds, orcelld_state_up == 0. Autoscaling stops making decisions on such samples. - Capacity: sustained
celld_shedding, or utilization near 1 atmaxReplicas. - Stuck changes: a WorkerApp that stays in
RollingOut, orAvailable: False.
At-a-glance status
$ kubectl get workerapps -A
NAMESPACE NAME PHASE APP READY RESTORING AGE
tenant-acme chat Ready 7b21e0c4a9d3f508 3 0 4d
tenant-beta docs RollingOut 2 1 9d
PHASE, the fully rolled-out version, ready pods, and the fleet-wide restoring
count are printer columns; APP is empty until every pod serves the expected
deployment. status.deployments lists each pod's live version and generation,
and conditions carry the detail. See the
WorkerApp status reference.