Autoscaling
CPU is the wrong primary signal for celld: fleets are capacity-bound by resident cells and memory, and celld records no metrics of its own. The operator closes both gaps: it exports the metrics and encodes the scaling policy.
The metrics source
The leader polls every fleet pod's internal /state endpoint every
--state-poll-interval (default 15s), with 16 polls in flight and a 30-second
budget per sweep, and publishes each sweep atomically as Prometheus series labelled
namespace, workerapp, and (per pod) pod. The full
list is in Observability; scaling uses these:
| Metric | Role in scaling |
|---|---|
celld_resident_cell_utilization | Occupied cells / maxResidentCells (0..1) per pod. The primary signal, averaged across the fleet. |
celld_shedding | 1 while a pod is pressure-shedding: the hard out-of-capacity signal. |
celld_fleet_complete | 1 when the last sweep reached every expected pod. Every scaling query requires it. |
celld_fleet_sample_timestamp_seconds | When the sweep ran. Every scaling query requires a sample younger than 60 seconds. |
Scrape the operator (the chart serves metrics on :8443 and can create a
ServiceMonitor with prometheus.enable), and tell the operator where KEDA can
query them with --prometheus-url.
What the operator materializes
With spec.autoscaling.enabled: true, the operator creates a KEDA
ScaledObject targeting the fleet's StatefulSet, with Prometheus triggers:
- Utilization:
avg(celld_resident_cell_utilization)for the app againsttargets.residentCellUtilization(default 70%). - Shedding:
max(celld_shedding)above 0.5, so any shedding pod scales the fleet up regardless of the average. - Latency (optional): the p95 of Istio's
istio_request_duration_millisecondsat the fleet's Service againsttargets.p95LatencyMs, so traffic-bound, stateless-heavy apps scale even at low cell counts. It needs Istio metrics in the same Prometheus.
Each query is wrapped in the completeness and freshness guard, and empty results are not
treated as zero: partial data or an unreachable Prometheus makes the scaler error out
instead of reporting low utilization. Triggers use KEDA's Value metric type
because the queries are already fleet aggregates.
autoscaling:
enabled: true
minReplicas: 3 # keep ≥ 2 for HA; ≥ 3 with fleet durability
maxReplicas: 10 # the tenant's cost ceiling
targets:
residentCellUtilization: 70
p95LatencyMs: 250 # optional; needs Istio telemetry
celld-aware policy, encoded
- Scale up early; expect slow absorption. Active-cell placement is traffic-driven: a new pod takes active cells only as traffic activates unowned cells or shedding releases them. celld v0.6.2 does rebalance hibernated cells by node weight, but busy cells stay put. Hence the conservative 70% default and the shedding fast path.
- Scale down slowly, one pod at a time. Removing a pod is a graceful
drain that hands its cells to peers (cold restores) and closes its WebSockets. The
operator allows one pod per 5 minutes after a stabilization window of 10 minutes, or 30
for
websockets: truefleets. - Autoscaling and rollouts never fight. Whenever the phase is not
Ready, the ScaledObject is paused. Before changing the fleet, the operator also waits for KEDA to acknowledge the pause (Paused=True) and for any HPA targeting the StatefulSet to disappear; a stalled pause blocks the rollout and shows instatus.rollout.waitingOn.
Requirements and degradation
Autoscaled WorkerApps need KEDA
v2.12.0 or newer (for the Paused=True acknowledgement) and a
Prometheus scraping the operator. Without the KEDA CRDs, the WorkerApp reports
AutoscalingReady: False, reason KEDAUnavailable, and
spec.autoscaling has no effect. Non-autoscaled WorkerApps need neither.
A missing or broken Prometheus is not a condition: KEDA owns that failure. Alert on
KEDA scaler errors, on celld_fleet_complete == 0, and on stale
celld_fleet_sample_timestamp_seconds, and keep the poll interval well below
60 seconds. Large or unhealthy fleets can fail the sweep's completeness check rather than
extend its deadline.
Who owns the replica count. With autoscaling enabled, KEDA owns it
and spec.replicas is only the initial size. Turn autoscaling off and the
operator deletes the ScaledObject and reconciles spec.replicas again.
Sizing rule of thumb (upstream): about 1,000 resident cells per
8 GiB node. Set resources.maxResidentCells and
resources.memoryGi together; the operator derives the container limit,
CELLD_MAX_RSS_MB (80% of the limit), and the admission cap from them.