celld-operator

Observability

Two planes, both per-tenant by construction: celld's built-in tracing (spans and logs, written to the tenant's own bucket prefix or your OTLP collector) and the operator's Prometheus metrics (capacity, memory, and health, derived from each pod's /state).

Traces and logs

One block configures celld's telemetry for the fleet; it is on by default:

telemetry:
  enabled: true       # the default; false sets CELLD_OTEL=0
  retention: 30d      # bucket sink only; "none" defers to bucket lifecycle rules
  # sink: otlp        # or send OTLP/HTTP to a collector instead:
  # otlpEndpoint: http://otel-collector.monitoring.svc:4318
SinkBehavior
bucket (default)CELLD_OTEL=1: Parquet traces and logs under the fleet bucket's telemetry/ prefix, partitioned by node and hour. DuckDB queries them directly, with no services to run. celld deletes files older than retention (CELLD_OTEL_RETENTION).
otlpOTLP/HTTP protobuf to a collector. On celld v0.5+ the operator puts the collector base URL in CELLD_OTEL (default http://localhost:4318 when otlpEndpoint is omitted); older images get the legacy CELLD_OTEL_SINK and OTEL_EXPORTER_OTLP_ENDPOINT. Setting otlpEndpoint without a sink selects otlp.

In both sinks the operator sets OTEL_SERVICE_NAME to the WorkerApp name, so a shared collector or a cross-fleet query does not flatten every fleet into celld's default celld service.

What celld emits:

Under load, telemetry sheds before requests do, and celld counts what it sheds. The schema is v0-unstable: column names can change before a stable release.

Querying the bucket sink

INSTALL httpfs; LOAD httpfs;
CREATE SECRET r2 (TYPE s3, KEY_ID '…', SECRET '…',
                  ENDPOINT 'ACCOUNT.r2.cloudflarestorage.com', URL_STYLE 'path');
CREATE VIEW traces AS SELECT * FROM
  read_parquet('s3://platform-cells/apps/chat/telemetry/traces/*/*/*/*/*/*.parquet');
CREATE VIEW logs AS SELECT * FROM
  read_parquet('s3://platform-cells/apps/chat/telemetry/logs/*/*/*/*/*/*.parquet');

-- The slowest requests, then every error line.
SELECT name, duration_us, trace_id FROM traces ORDER BY duration_us DESC LIMIT 20;
SELECT time_unix_us, body FROM logs WHERE severity_number >= 17;

celld writes a Parquet file every 5 minutes or at about 5 MiB of buffered events, whichever comes first, so treat the bucket sink as minutes behind rather than live; the operator exposes no field for the flush interval. For a near-live view, use the OTLP sink. celld does not compact its own files: run an hourly DuckDB COPY on a maintenance node that rewrites the previous hour into one zstd Parquet file, or queries slow down within hours.

Fleet metrics

celld records no metrics, so the operator's leader polls each pod's internal /state every --state-poll-interval (default 15s) and exports gauges labelled namespace, workerapp, and pod. Responses with missing counters, negative values, runtime errors, or trailing data are rejected rather than exported.

MetricMeaning
celld_resident_cellsOccupied (resident) cells per pod
celld_resident_cell_utilizationOccupied / maxResidentCells (0..1), the primary capacity signal
celld_restoringCold activations in flight; the rollout controller's step gate
celld_evictingCells being evicted
celld_shedding1 while pressure-shedding: the hard out-of-capacity signal
celld_rss_bytes / celld_in_use_bytesProcess RSS, and RSS minus allocator retention
celld_cgroup_working_set_bytes / celld_cgroup_current_bytesActive cgroup memory (used by pressure shedding) and the full cgroup charge (used by the absolute cap); absent outside a readable Linux cgroup
celld_container_restartsThe kubelet's restart count for the celld container
celld_self_fenced1 if the last exit was a celld self-fence (exit code 3)
celld_state_up1 if /state answered the last poll
celld_fleet_completePer app: 1 when the last sweep reached every expected pod
celld_fleet_sample_timestamp_secondsPer app: when the last sweep ran

What to alert on

At-a-glance status

$ kubectl get workerapps -A
NAMESPACE     NAME   PHASE        APP                READY   RESTORING   AGE
tenant-acme   chat   Ready        7b21e0c4a9d3f508   3       0           4d
tenant-beta   docs   RollingOut                      2       1           9d

PHASE, the fully rolled-out version, ready pods, and the fleet-wide restoring count are printer columns; APP is empty until every pod serves the expected deployment. status.deployments lists each pod's live version and generation, and conditions carry the detail. See the WorkerApp status reference.

Edit this page on GitHubUpdated 2026-10-18