Skip to content

Use LoadBalancer Services and Envoy Backend TLS on OpenShift - #1383

Merged
ArnobKumarSaha merged 5 commits into
masterfrom
arnob-ocp-gw-lb
Oct 29, 2026
Merged

ArnobKumarSaha merged 5 commits into
masterfrom
arnob-ocp-gw-lb

Conversation

@ArnobKumarSaha

@ArnobKumarSaha ArnobKumarSaha commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

OpenShift 4.19+ owns the gateway.networking.k8s.io group. The ingress operator goes Degraded (GatewayAPICRDsDegraded) when unmanaged CRDs exist. TCPRoute isn't managed in any OpenShift release, and BackendTLSPolicy is only managed from 4.22.

OpenShift only (detected with the existing distro.openshift helper):

  • ace:

    • Drop the nats-tcp/s3proxy-tcp Gateway listeners and the NATS TCPRoute.
    • Add the <fullname>-nats-lb LoadBalancer Service (4222).
    • Point the /nats HTTPRoute at an Envoy Gateway Backend with spec.tls, instead of using BackendTLSPolicy.
  • s3proxy:

    • Drop the TCPRoute.
    • Add the <fullname>-lb LoadBalancer Service (4224).
    • Point the HTTPRoute at a Backend with spec.tls, instead of using BackendTLSPolicy.
  • acaas: set TLS on the existing Backend instead of using BackendTLSPolicy.

  • New value gateway.loadBalancer.annotations (ace, s3proxy): set the same IP-sharing annotations as the Envoy Service (service-gateway-presets envoy.service.annotations) so all three LoadBalancer Services share one IP. Clients still use platform.host:4222/4224. Examples:

    • MetalLB: metallb.universe.tf/loadBalancerIPs: <ip> + metallb.universe.tf/allow-shared-ip: ace
    • Azure/ARO: service.beta.kubernetes.io/azure-load-balancer-ipv4: <ip>
    • spec.loadBalancerIP is deliberately not exposed (deprecated since Kubernetes 1.24).
    • AWS NLB can't share an IP across Services; not handled here.
  • ace pgpool:

    • Run pgpool with runAsGroup: 0. Pod and container runAsGroup are no longer the namespace UID.
    • Set the exporter securityContext to the namespace UID range (restricted SCC).

All platforms:

  • enableBackend: true under config.envoyGateway.extensionApis, set for:

    • opscenter-features voyager-gateway (the cluster gateway)
    • service-gateway and service-gateway-presets (the per-namespace gateway.voyagermesh.com/ace gateway that serves the ace/s3proxy/acaas routes). Without it, /nats fails with ResolvedRefs=False: Backend is disabled in Envoy Gateway configuration.
  • catalog: regenerated by make refresh (editor charts v0.38.0 → v0.39.0).

Depends on:

  • voyagermesh/crd-manager#30: installs the backends.gateway.envoyproxy.io CRD.
  • voyagermesh/gateway#458: envoy-gateway starts without the TCPRoute, TLSRoute, ListenerSet and BackendTLSPolicy CRDs.

Verification (helm template):

  • Non-OpenShift ace/acaas: output is identical to the base branch, apart from randomly generated secrets.
  • OpenShift: no TCPRoute/BackendTLSPolicy is rendered.

@ArnobKumarSaha
ArnobKumarSaha force-pushed the arnob-ocp-gw-lb branch 2 times, most recently from 0a14707 to bc39009 Compare October 29, 2026 16:12
OpenShift's ingress operator marks itself Degraded when unmanaged
gateway.networking.k8s.io CRDs (TCPRoute, BackendTLSPolicy) exist.
On OpenShift, expose NATS and s3proxy via dedicated LoadBalancer
Services and use Envoy Gateway Backend TLS instead. Always enable
the Envoy Gateway Backend extension API.

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
The pgpool image grants write access to /opt/pgpool-II/etc via group 0.
With runAsGroup set to the namespace UID, the entrypoint cannot write its
config files and the container crash-loops.

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
The ace /nats and s3proxy HTTPRoutes on OpenShift reference an Envoy
Backend, but they are served by the service-gateway (gateway.voyagermesh.com/ace),
whose EnvoyGateway config lacked extensionApis.enableBackend, so the route
failed with ResolvedRefs=False.

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
@ArnobKumarSaha
ArnobKumarSaha changed the base branch from release-v2026.10.12 to master October 29, 2026 18:09
@ArnobKumarSaha
ArnobKumarSaha merged commit 2bcac57 into master Oct 29, 2026
3 of 4 checks passed
@ArnobKumarSaha
ArnobKumarSaha deleted the arnob-ocp-gw-lb branch October 29, 2026 18:11
ArnobKumarSaha added a commit that referenced this pull request Oct 29, 2026
Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant