celld-operator

Autoscaling

CPU is the wrong primary signal for celld: fleets are capacity-bound by resident cells and memory, and celld records no metrics of its own. The operator closes both gaps: it exports the metrics and encodes the scaling policy.

The metrics source

The leader polls every fleet pod's internal /state endpoint every --state-poll-interval (default 15s), with 16 polls in flight and a 30-second budget per sweep, and publishes each sweep atomically as Prometheus series labelled namespace, workerapp, and (per pod) pod. The full list is in Observability; scaling uses these:

MetricRole in scaling
celld_resident_cell_utilizationOccupied cells / maxResidentCells (0..1) per pod. The primary signal, averaged across the fleet.
celld_shedding1 while a pod is pressure-shedding: the hard out-of-capacity signal.
celld_fleet_complete1 when the last sweep reached every expected pod. Every scaling query requires it.
celld_fleet_sample_timestamp_secondsWhen the sweep ran. Every scaling query requires a sample younger than 60 seconds.

Scrape the operator (the chart serves metrics on :8443 and can create a ServiceMonitor with prometheus.enable), and tell the operator where KEDA can query them with --prometheus-url.

What the operator materializes

With spec.autoscaling.enabled: true, the operator creates a KEDA ScaledObject targeting the fleet's StatefulSet, with Prometheus triggers:

Each query is wrapped in the completeness and freshness guard, and empty results are not treated as zero: partial data or an unreachable Prometheus makes the scaler error out instead of reporting low utilization. Triggers use KEDA's Value metric type because the queries are already fleet aggregates.

autoscaling:
  enabled: true
  minReplicas: 3     # keep ≥ 2 for HA; ≥ 3 with fleet durability
  maxReplicas: 10    # the tenant's cost ceiling
  targets:
    residentCellUtilization: 70
    p95LatencyMs: 250  # optional; needs Istio telemetry

celld-aware policy, encoded

Requirements and degradation

Autoscaled WorkerApps need KEDA v2.12.0 or newer (for the Paused=True acknowledgement) and a Prometheus scraping the operator. Without the KEDA CRDs, the WorkerApp reports AutoscalingReady: False, reason KEDAUnavailable, and spec.autoscaling has no effect. Non-autoscaled WorkerApps need neither.

A missing or broken Prometheus is not a condition: KEDA owns that failure. Alert on KEDA scaler errors, on celld_fleet_complete == 0, and on stale celld_fleet_sample_timestamp_seconds, and keep the poll interval well below 60 seconds. Large or unhealthy fleets can fail the sweep's completeness check rather than extend its deadline.

Who owns the replica count. With autoscaling enabled, KEDA owns it and spec.replicas is only the initial size. Turn autoscaling off and the operator deletes the ScaledObject and reconciles spec.replicas again.

Sizing rule of thumb (upstream): about 1,000 resident cells per 8 GiB node. Set resources.maxResidentCells and resources.memoryGi together; the operator derives the container limit, CELLD_MAX_RSS_MB (80% of the limit), and the admission cap from them.

Edit this page on GitHubUpdated 2026-10-18