celld-operator

Networking & ingress

Every fleet has two listeners with very different audiences: the public Worker listener (:8080), which your users reach through ingress, and the internal listener (:8081), which nothing outside the fleet and the operator may ever reach. The operator wires both.

Ingress modes

How spec.hostnames get routed is an operator-level choice (--ingress-mode, or operator.ingressMode in Helm). Each app gets one route object carrying every hostname.

ModeMechanismDrain-503 retry
httproute (default)A Gateway API HTTPRoute attached to a shared Gateway (--gateway-name/--gateway-namespace, default edge/infra). Cross-namespace attachment is authorized by that Gateway's allowedRoutes; TLS is the Gateway's own.Opt-in: operator.httpRouteRetries: true adds 2 attempts on 503. Needs experimental-channel CRDs.
virtualserviceA classic Istio VirtualService bound to pre-existing networking.istio.io Gateways (--istio-gateways, comma-separated namespace/name).Always: 2 attempts on connect-failure,refused-stream,503.
ingressA networking.k8s.io/v1 Ingress; --ingress-class selects the class and --cluster-issuer requests per-app cert-manager TLS.Always, as ingress-nginx annotations (proxy-next-upstream: error timeout http_503, 3 tries). Other controllers ignore them.
noneNo routes; hostnames has no effect.n/a

Changing the mode, or removing every hostname, deletes the obsolete route objects the WorkerApp owns.

What IngressReady means

StatusReasonMeaning
TrueRouteObservedHTTPRoute: fresh Accepted and ResolvedRefs conditions from the Gateway, and the Gateway reports Programmed.
UnknownIngressReconciled / VirtualServiceReconciledThe object exists; creation alone does not prove traffic flows, so check the controller.
FalseGatewayAPIUnavailable / IstioUnavailableThe CRDs for the selected mode are not installed; hostnames are not routed.
FalseRouteReconciledRetryDroppedRetries were requested but standard-channel CRDs pruned the field. The operator stops rewriting it and re-checks after a persisted five-minute delay, so upgrading the CRDs needs no manual reset.
FalseRouteErrorAny other create or update error; the message carries it.

With watchNamespace set and the Gateway in another namespace, the manager needs get on that Gateway (a Role and RoleBinding there); otherwise the condition stays Unknown.

Internal consumers: no ingress at all

Internal-only apps need no hostnames. Consumers in the cluster reach the serving Service directly at <app>-celld.<namespace>.svc:8080. The spec.service block shapes that Service:

service:
  type: LoadBalancer       # ClusterIP (default) | LoadBalancer | NodePort
  annotations:             # cloud LB configuration, e.g. an internal LB
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Assigned node ports survive reconciliation, and annotations the operator added are removed again when you delete them from the spec.

Drain-aware edge policy

A draining celld node answers new requests with 503 and a closed connection, expecting the client to retry against a healthy node. At the platform edge, the gateway can be that client: with a retry on 503, drains, and therefore rollouts, are invisible to end users. In httproute mode retries are off by default because HTTPRouteRule.retry exists only in the Gateway API experimental channel. Without them, clients must tolerate rollout 503s. WebSocket connections can still close during node replacement in every mode.

Related wiring the operator takes care of:

The WebSocket profile

spec.websockets: true switches the app's routes and scaling to a socket-friendly profile:

There is no session affinity. celld's peer tunnel serves any cell from any node, so the operator configures none; add it at your edge if you want requests to land on a cell's owner for latency. A WebSocket cannot follow a cell to a new owner, so clients must reconnect after a handoff.

Keeping proxies happy without waking cells: have clients ping periodically and answer with setWebSocketAutoResponse. celld replies to a matched message without waking the hibernated cell, and the proxy's idle clock resets.

Forwarded headers and request.url

celld ignores X-Forwarded-Host and X-Forwarded-Proto by default, so behind a TLS-terminating edge a Worker sees the pod's address and scheme in request.url. spec.trustForwardedHeaders: true sets CELLD_TRUST_FORWARDED_HEADERS=1. celld takes the last value of each header, so enable it only when every hop in front of the fleet replaces both headers. ingress-nginx does; Envoy-based gateways set X-Forwarded-Proto but not always X-Forwarded-Host.

Cluster DNS and IP families

Peers advertise $(POD_NAME).<app>-celld-internal.<ns>.svc.<clusterDomain>:8081. Set spec.clusterDomain when your cluster's DNS suffix is not cluster.local, and set the operator's Istio principal separately when the trust domain differs. celld's listeners bind IPv4, so IPv6-only pod networking is not supported.

Where TLS lives

celld terminates no TLS anywhere. Public TLS belongs to the Gateway (or Ingress + cert-manager). Peer traffic is plaintext HTTP: the fleet HMAC authenticates tunnel establishment and control requests but encrypts nothing, so encrypt the pod network (Istio ambient mode, or a CNI with WireGuard). The internal listener's protections are covered in Security.

Edit this page on GitHubUpdated 2026-10-18