Networking & ingress
Every fleet has two listeners with very different audiences: the public Worker listener (:8080), which your users reach through ingress, and the internal listener (:8081), which nothing outside the fleet and the operator may ever reach. The operator wires both.
Ingress modes
How spec.hostnames get routed is an operator-level choice
(--ingress-mode, or operator.ingressMode in Helm). Each app gets
one route object carrying every hostname.
| Mode | Mechanism | Drain-503 retry |
|---|---|---|
httproute (default) | A Gateway API HTTPRoute attached to a shared Gateway (--gateway-name/--gateway-namespace, default edge/infra). Cross-namespace attachment is authorized by that Gateway's allowedRoutes; TLS is the Gateway's own. | Opt-in: operator.httpRouteRetries: true adds 2 attempts on 503. Needs experimental-channel CRDs. |
virtualservice | A classic Istio VirtualService bound to pre-existing networking.istio.io Gateways (--istio-gateways, comma-separated namespace/name). | Always: 2 attempts on connect-failure,refused-stream,503. |
ingress | A networking.k8s.io/v1 Ingress; --ingress-class selects the class and --cluster-issuer requests per-app cert-manager TLS. | Always, as ingress-nginx annotations (proxy-next-upstream: error timeout http_503, 3 tries). Other controllers ignore them. |
none | No routes; hostnames has no effect. | n/a |
Changing the mode, or removing every hostname, deletes the obsolete route objects the WorkerApp owns.
What IngressReady means
| Status | Reason | Meaning |
|---|---|---|
True | RouteObserved | HTTPRoute: fresh Accepted and ResolvedRefs conditions from the Gateway, and the Gateway reports Programmed. |
Unknown | IngressReconciled / VirtualServiceReconciled | The object exists; creation alone does not prove traffic flows, so check the controller. |
False | GatewayAPIUnavailable / IstioUnavailable | The CRDs for the selected mode are not installed; hostnames are not routed. |
False | RouteReconciledRetryDropped | Retries were requested but standard-channel CRDs pruned the field. The operator stops rewriting it and re-checks after a persisted five-minute delay, so upgrading the CRDs needs no manual reset. |
False | RouteError | Any other create or update error; the message carries it. |
With watchNamespace set and the Gateway in another namespace, the manager
needs get on that Gateway (a Role and RoleBinding there); otherwise the
condition stays Unknown.
Internal consumers: no ingress at all
Internal-only apps need no hostnames. Consumers in the cluster reach the serving Service
directly at <app>-celld.<namespace>.svc:8080. The
spec.service block shapes that Service:
service:
type: LoadBalancer # ClusterIP (default) | LoadBalancer | NodePort
annotations: # cloud LB configuration, e.g. an internal LB
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
Assigned node ports survive reconciliation, and annotations the operator added are removed again when you delete them from the spec.
Drain-aware edge policy
A draining celld node answers new requests with 503 and a closed connection, expecting
the client to retry against a healthy node. At the platform edge, the gateway can be
that client: with a retry on 503, drains, and therefore rollouts, are invisible to
end users. In httproute mode retries are off by default because
HTTPRouteRule.retry exists only in the Gateway API experimental channel.
Without them, clients must tolerate rollout 503s. WebSocket connections can still close
during node replacement in every mode.
Related wiring the operator takes care of:
- Readiness is celld's own
/.well-known/celld/health; a draining pod flips to 503 and leaves the EndpointSlice. - The headless internal Service sets
publishNotReadyAddresses, so peers can still reach a draining pod to take its cells. - Liveness is TCP only, because an HTTP liveness probe on the health path would kill nodes mid-handoff.
The WebSocket profile
spec.websockets: true switches the app's routes and scaling to a
socket-friendly profile:
- Long timeouts where the mode can express them:
httproutedisables the route's request timeout, andingresssetsproxy-read-timeout/proxy-send-timeoutto 3600s. Invirtualservicemode the operator emits no timeout override, so set one in your own mesh defaults if you serve long-lived sockets that way. - Conservative scale-down: a removed pod closes its sockets, so the stabilization window stretches to 30 minutes (from 10). See autoscaling policy.
There is no session affinity. celld's peer tunnel serves any cell from any node, so the operator configures none; add it at your edge if you want requests to land on a cell's owner for latency. A WebSocket cannot follow a cell to a new owner, so clients must reconnect after a handoff.
Keeping proxies happy without waking cells: have clients ping
periodically and answer with setWebSocketAutoResponse. celld replies to a
matched message without waking the hibernated cell, and the proxy's idle clock
resets.
Forwarded headers and request.url
celld ignores X-Forwarded-Host and X-Forwarded-Proto by
default, so behind a TLS-terminating edge a Worker sees the pod's address and scheme in
request.url. spec.trustForwardedHeaders: true sets
CELLD_TRUST_FORWARDED_HEADERS=1. celld takes the last value of each
header, so enable it only when every hop in front of the fleet replaces both headers.
ingress-nginx does; Envoy-based gateways set X-Forwarded-Proto but not always
X-Forwarded-Host.
Cluster DNS and IP families
Peers advertise $(POD_NAME).<app>-celld-internal.<ns>.svc.<clusterDomain>:8081.
Set spec.clusterDomain when your cluster's DNS suffix is not
cluster.local, and set the operator's Istio principal separately when the
trust domain differs. celld's listeners bind IPv4, so IPv6-only pod networking is not
supported.
Where TLS lives
celld terminates no TLS anywhere. Public TLS belongs to the Gateway (or Ingress + cert-manager). Peer traffic is plaintext HTTP: the fleet HMAC authenticates tunnel establishment and control requests but encrypts nothing, so encrypt the pod network (Istio ambient mode, or a CNI with WireGuard). The internal listener's protections are covered in Security.