Autoscaling

CPU is the wrong primary signal for celld: fleets are capacity-bound by resident cells and memory, and upstream ships no metrics endpoint yet. The operator closes both gaps: it exports the metrics and encodes the scaling policy.

The metrics source

The operator already polls each fleet pod's internal /state endpoint for rollout gating (it is the one authorized cross-namespace caller). It re-exports what it sees as Prometheus metrics, per WorkerApp and per pod, on the leader every --state-poll-interval (default 15s):

Metric	Meaning
`celld_resident_cells`	Occupied (resident) cells per pod.
`celld_resident_cell_utilization`	occupied / `maxResidentCells` (0..1), the primary scale signal.
`celld_restoring`	Cold activations in flight, cold activations in flight; a health signal, and the rollout controller's step gate — not an autoscaling input.
`celld_evicting`	Cells being evicted.
`celld_shedding`	1 while pressure-shedding; the hard out-of-capacity signal.
`celld_state_up`	1 if `/state` answered the last poll.

Point Prometheus at the operator (the chart exposes metrics on :8443), and tell the operator where KEDA can query them with --prometheus-url.

What the operator materializes

With spec.autoscaling.enabled: true, the operator creates a KEDA ScaledObject targeting the fleet's StatefulSet:

A Prometheus scaler on celld_resident_cell_utilization against your targets.residentCellUtilization (default 70%).
Immediate scale-up when any pod sheds, regardless of the fleet average.
Optionally, a gateway-side scaler on targets.p95LatencyMs, so traffic-bound, stateless-Worker-heavy apps scale even at low cell counts.

autoscaling:
  enabled: true
  minReplicas: 3     # keep ≥ 2 for HA
  maxReplicas: 10    # the tenant's cost ceiling
  targets:
    residentCellUtilization: 70
    p95LatencyMs: 250  # optional; needs Istio telemetry

celld-aware policy, encoded

These behaviors are set by the operator rather than left to tuning:

Scale up early; expect slow absorption. celld has no rebalancer: a new pod is a spare that fills only as traffic activates unowned cells or as pressure shedding releases them. Hence the conservative 70% default target and the shedding fast-path: by the time shedding starts, the fleet is already rebalancing the hard way.
Scale down slowly, and never during churn. Removing a pod is a graceful drain, but it hands that node's cells to peers (cold restores) and closes its WebSockets (clients reconnect). Default: one pod per 5 minutes after a long stabilization window — 10 minutes normally, 30 for websockets: true fleets.
Autoscaling and rollouts never fight. Whenever the phase is not Ready — RollingOut, Recreating, but also Pending and Degraded — the operator pauses the ScaledObject (KEDA pause annotation) and pins replicas; the partition arithmetic assumes a stable count. Scaling resumes when the fleet is Ready again, which means the StatefulSet reports fully updated and ready. A fleet parked in Degraded therefore has autoscaling pinned until you resolve it.

Requirements and degradation

Autoscaling needs KEDA installed and a Prometheus scraping the operator. When the KEDA CRDs are absent the WorkerApp reports AutoscalingReady: False with reason KEDAUnavailable and spec.autoscaling has no effect. (A missing or unreachable Prometheus is not detected: KEDA owns that failure, and the condition stays True.)

Enabling autoscaling hands replica count to KEDA permanently. Once spec.autoscaling.enabled is true the operator stops reconciling replicas altogether, so later edits to spec.replicas are ignored — and if KEDA is absent the fleet simply stays at whatever count it already had. Set the floor and ceiling with minReplicas/maxReplicas, and to hand control back, disable autoscaling first.

Sizing rule of thumb (upstream): ~1,000 resident cells per 8 GiB node. Set resources.maxResidentCells and resources.memoryGi together; the operator derives the container limit, CELLD_MAX_RSS_MB (~80% of the limit), and the admission cap from these two numbers.