istio(1.30.3): Enhanced Ambient Mesh Stability, Control Plane Performance, and Critical Fixes

📋 Recommended Actions

⚠️ Action Required
Immediate action is required for existing auto-registered WorkloadEntries to ensure HBONE capability. Review updates to better support your users.

📝 Summary

Istio 1.30.3 delivers crucial stability and performance boosts, especially for ambient mesh environments. This release resolves a deadlock in the CNI node agent, fixing a critical issue that could block the ZDS server. A significant performance enhancement comes from suppressing unnecessary XDS pushes for metadata-only VirtualService changes, reducing control plane load. For ambient users, HBONE auto-registration is now more reliable, ensuring non-Kubernetes workloads correctly advertise HBONE, though existing WorkloadEntries may need a manual label update. Operations engineers will appreciate the fix for file-mounted certificate reloads, resolving issues with Kubernetes secret rotations. Additionally, the node untaint controller is now configurable, offering greater flexibility. Upgrade to benefit from these vital bug fixes and improved mesh reliability.

🛡️ Ambient Mesh Stability & RBAC Fixes

This release significantly improves the stability and correctness of Istio’s ambient mesh, addressing several critical issues that could impact cross-network traffic, policy enforcement, and overall control plane responsiveness. These updates ensure a more robust and predictable ambient experience for your workloads.

Waypoint RBAC for Cross-Network Ambient Traffic Improved

Cross-network traffic flowing through an east-west gateway will no longer be blocked by spurious deny-all RBAC filters, even when destination services have L7 AuthorizationPolicies. This fix ensures seamless communication across network boundaries in ambient mode.

• Source: pilot/pkg/networking/core/listener_waypoint.go (lines 482-485), releasenotes/notes/60806.yaml

Enhanced HBONE Auto-Registration for WorkloadEntries

Non-Kubernetes workloads that are auto-registered via WorkloadEntries will now correctly advertise their HBONE (HTTP-based Overlay Network Environment) capability. Previously, these workloads might default to plaintext communication. During auto-registration, Istiod will now add the networking.istio.io/tunnel: http label for HBONE-compatible proxies, ensuring proper tunnel establishment.

Action Required: For WorkloadEntries auto-registered before this upgrade, you’ll need to either re-register them (by reconnecting a fresh instance) or manually add the networking.istio.io/tunnel: http label to their existing WorkloadEntry to enable HBONE.

apiVersion: networking.istio.io/v1beta1
kind: WorkloadEntry
metadata:
  name: my-legacy-workload-entry
  namespace: default
  labels:
    networking.istio.io/tunnel: http # Add this label for HBONE
spec:
  address: 192.168.1.100
  serviceAccount: default

• Source: pilot/pkg/autoregistration/controller.go (lines 699-709), pilot/pkg/autoregistration/controller_test.go (lines 476-541), releasenotes/notes/60788.yaml

Source:

• pilot/pkg/networking/core/listener_waypoint.go (482-485)
• releasenotes/notes/60806.yaml (1-9)
• pilot/pkg/autoregistration/controller.go (699-709)
• pilot/pkg/autoregistration/controller_test.go (476-541)
• releasenotes/notes/60788.yaml (1-18)

🚀 Control Plane Performance & Configuration

This release focuses on optimizing the Istio control plane, addressing performance regressions, improving multicluster credential handling, and enhancing configuration flexibility for node management. These changes aim to make Istiod more efficient and resilient, particularly in large-scale and dynamic environments.

VirtualService Push Optimization to Prevent Excessive XDS Pushes

An issue introduced in Istio 1.30 that caused metadata-only changes to VirtualService objects to trigger unnecessary XDS pushes has been fixed. This regression led to increased control plane CPU usage and push latency, especially for clusters using GitOps tooling. Istio now restores the pre-1.30 behavior, where only spec changes or istio.io label/annotation changes will trigger an XDS push.

• Source: pilot/pkg/model/config_compare.go (lines 1-66), pilot/pkg/model/virtualservice_controller.go (lines 96-107), pilot/pkg/model/virtualservice_controller_test.go (lines 771-884), releasenotes/notes/60629-vs-push-suppression.yaml

Multicluster Secret Rotation Deadlock Resolved

A deadlock preventing Istiod from picking up updated remote cluster secrets (e.g., during credential or token rotation) has been fixed. Previously, a new cluster registry could deadlock while waiting to sync, leaving the service registry stale for the affected remote cluster. This ensures seamless and robust multicluster operations during credential updates.

• Source: pilot/pkg/serviceregistry/aggregate/controller.go (lines 149-179), pilot/pkg/serviceregistry/aggregate/controller_test.go (lines 500-543), pilot/pkg/serviceregistry/kube/controller/multicluster.go (lines 221-226), releasenotes/notes/60612.yaml

Configurable Node Untaint Controller for CNI

Operators now have more flexibility in managing node taints with the introduction of a configurable taint name for the Pilot node untaint controller. This feature allows you to customize the taint key that the controller watches and removes from nodes once CNI pods are ready. The default taint name remains cni.istio.io/not-ready.

You can configure this using the PILOT_NODE_UNTAINT_CONTROLLERS_TAINT_NAME environment variable in the Istiod deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istiod
  namespace: istio-system
spec:
  template:
    spec:
      containers:
      - name: discovery
        env:
        - name: PILOT_ENABLE_NODE_UNTAINT_CONTROLLERS
          value: "true"
        - name: PILOT_NODE_UNTAINT_CONTROLLERS_TAINT_NAME
          value: "my-custom-taint/not-ready"

• Source: pilot/pkg/controllers/untaint/nodeuntainter.go (lines 33-205), pilot/pkg/controllers/untaint/nodeuntainter_test.go (lines 100-112), pilot/pkg/features/pilot.go (lines 170-176), manifests/charts/istio-control/istio-discovery/templates/deployment.yaml (lines 195-197), manifests/charts/istio-control/istio-discovery/values.yaml (lines 47-49), operator/pkg/apis/values_types.pb.go (lines 3474-3523), operator/pkg/apis/values_types.proto (lines 1018-1022), releasenotes/notes/59499.yaml

Improved Default ExportTo Logic

An issue where additional namespaces specified in meshConfig.defaultServiceExportTo and meshConfig.defaultVirtualServiceExportTo were not being honored has been resolved. This ensures that services and virtual services without explicit exportTo settings correctly apply the mesh-wide defaults, including export to other namespaces when configured.

Source:

• pilot/pkg/model/config_compare.go (1-66)
• pilot/pkg/model/virtualservice_controller.go (96-107)
• pilot/pkg/model/virtualservice_controller_test.go (771-884)
• releasenotes/notes/60629-vs-push-suppression.yaml (1-14)
• pilot/pkg/serviceregistry/aggregate/controller.go (149-179)
• pilot/pkg/serviceregistry/aggregate/controller_test.go (500-543)
• pilot/pkg/serviceregistry/kube/controller/multicluster.go (221-226)
• releasenotes/notes/60612.yaml (1-10)
• pilot/pkg/controllers/untaint/nodeuntainter.go (33-205)
• pilot/pkg/controllers/untaint/nodeuntainter_test.go (100-112)
• pilot/pkg/features/pilot.go (170-176)
• manifests/charts/istio-control/istio-discovery/templates/deployment.yaml (195-197)
• manifests/charts/istio-control/istio-discovery/values.yaml (47-49)
• operator/pkg/apis/values_types.pb.go (3474-3523)
• operator/pkg/apis/values_types.proto (1018-1022)
• releasenotes/notes/59499.yaml (1-9)
• pilot/pkg/model/push_context.go (1083-1092)
• pilot/pkg/model/push_context_test.go (1544-1699)
• releasenotes/notes/60560.yaml (1-11)

🔒 Security & Policy Enforcement Updates

This release brings important fixes and improvements to security and policy enforcement, including reliable certificate reloads and correct application of Telemetry policies for waypoints. These updates are crucial for maintaining a secure and observable mesh.

Reliable File-Mounted Certificate Reloads

A critical bug where pilot-agent missed certificate reloads on subsequent Kubernetes secret rotations for file-mounted certificates has been fixed. This ensures that Envoy proxies correctly receive updated certificates, preventing disruptions and maintaining security. The fix specifically addresses MOVED_TO events on ..data symlinks, which are used by kubelet to atomically publish new secret files.

• Source: security/pkg/nodeagent/cache/secretcache.go (lines 955-961, 979-982, 1079-1084), security/pkg/nodeagent/cache/secretcache_test.go (lines 1256-1282), releasenotes/notes/59912.yaml, tests/integration/security/filebased_tls_origination/destination_rule_tls_test.go (lines 69-146)

Cross-Namespace Telemetry Policy for Waypoints Corrected

An issue where Telemetry resources from other namespaces were not correctly included in a waypoint proxy’s configuration has been resolved. This fix ensures that waypoint proxies properly apply namespace-wide Telemetry policies from the service’s namespace, even if the waypoint itself is in a different namespace, enhancing observability for cross-namespace traffic.

• Source: pilot/pkg/model/telemetry.go (lines 432-441), pilot/pkg/model/telemetry_test.go (lines 1099-1155), releasenotes/notes/60665.yaml

Warning for Private JWKS Keys in RequestAuthentication

To enhance security posture, Istio now issues a warning if inline JSON Web Key Sets (JWKS) provided in RequestAuthentication resources contain private key material. Envoy only requires public keys for token verification, so including private keys is unnecessary and can pose a security risk if exposed. Configurations with private keys will still be accepted but with a clear warning.

• Source: pkg/config/validation/validation.go (lines 1611-1631), pkg/config/validation/validation_test.go (lines 7245-7260)

Source:

• security/pkg/nodeagent/cache/secretcache.go (955-961, 979-982, 1079-1084)
• security/pkg/nodeagent/cache/secretcache_test.go (1256-1282)
• releasenotes/notes/59912.yaml (1-8)
• tests/integration/security/filebased_tls_origination/destination_rule_tls_test.go (69-146)
• pilot/pkg/model/telemetry.go (432-441)
• pilot/pkg/model/telemetry_test.go (1099-1155)
• releasenotes/notes/60665.yaml (1-8)
• pkg/config/validation/validation.go (1611-1631)
• pkg/config/validation/validation_test.go (7245-7260)

🐛 Core Bug Fixes & Resiliency Improvements

This release includes critical bug fixes that enhance the overall resiliency and correctness of the Istio system, addressing potential deadlocks, memory leaks, and issues with Envoy drain processes.

Ztunnel CNI Deadlock Resolved

A deadlock in the ambient CNI node agent, which could permanently block the ZDS server when a pod deletion event occurred concurrently with a ztunnel (re)connection, has been fixed. This improves the stability of ambient mesh deployments, especially under high churn.

• Source: cni/pkg/nodeagent/ztunnelserver.go (lines 84-89), cni/pkg/nodeagent/ztunnelserver_linux.go (lines 32-35), cni/pkg/nodeagent/ztunnelserver_linux_test.go (lines 567-609), releasenotes/notes/cni-ztunnel-poddeleted-deadlock.yaml

Memory Leak in Istiod’s needResync Fixed

Istiod no longer leaks needResync entries for failed pod IPs that were not being cleaned up. This fix improves memory management and long-term stability of the control plane, particularly in environments with frequent pod failures or updates.

• Source: pilot/pkg/serviceregistry/kube/controller/controller_test.go (lines 2216-2226), pilot/pkg/serviceregistry/kube/controller/endpointslice.go (lines 85-108, 233-253), releasenotes/notes/needresync-leak.yaml

Envoy Internal Listeners Excluded from Drain Check

The EXIT_ON_ZERO_ACTIVE_CONNECTIONS drain check in pilot-agent will now correctly fire for ambient ingress gateways and waypoints. Previously, it never reached zero due to internal HBONE listener connections (envoy_internal_*) being counted, forcing proxies to wait for terminationGracePeriodSeconds. These internal listeners are now correctly excluded from the active connection count, enabling faster and more efficient proxy draining.

• Source: pkg/envoy/agent.go (lines 253-259), pkg/envoy/agent_test.go (lines 38-60, 132-138), releasenotes/notes/60731.yaml

Waypoint Inbound Routes Treat Like Sidecar Outbound

Default HTTP retries for inbound routes of waypoints are now correctly applied. The inbound route of a waypoint is now treated more like a sidecar’s outbound route, ensuring that the mesh config’s defaultHttpRetryPolicy correctly applies to local services attached to waypoints. This fixes an issue where HTTP retries were not being properly configured for these routes.

• Source: pilot/pkg/networking/core/httproute.go (lines 113-131), pilot/pkg/networking/core/httproute_test.go (lines 2038-2089), releasenotes/notes/60682.yaml

WasmPlugin Cross-Namespace Waypoint Fix

Resolved a bug where a WasmPlugin in an application namespace targeting a Service via targetRefs would cause a waypoint proxy to crash-loop on startup. The LDS path correctly included the plugin for the waypoint, but the ECDS lookup path rejected it as cross-namespace, leaving Envoy waiting for a resource that would never arrive. This fix ensures WasmPlugins can be correctly applied in cross-namespace waypoint scenarios.

• Source: releasenotes/notes/wasmplugin-cross-namespace-waypoint.yaml

Source:

• cni/pkg/nodeagent/ztunnelserver.go (84-89)
• cni/pkg/nodeagent/ztunnelserver_linux.go (32-35)
• cni/pkg/nodeagent/ztunnelserver_linux_test.go (567-609)
• releasenotes/notes/cni-ztunnel-poddeleted-deadlock.yaml (1-9)
• pilot/pkg/serviceregistry/kube/controller/controller_test.go (2216-2226)
• pilot/pkg/serviceregistry/kube/controller/endpointslice.go (85-108, 233-253)
• releasenotes/notes/needresync-leak.yaml (1-6)
• pkg/envoy/agent.go (253-259)
• pkg/envoy/agent_test.go (38-60, 132-138)
• releasenotes/notes/60731.yaml (1-8)
• pilot/pkg/networking/core/httproute.go (113-131)
• pilot/pkg/networking/core/httproute_test.go (2038-2089)
• releasenotes/notes/60682.yaml (1-9)
• releasenotes/notes/wasmplugin-cross-namespace-waypoint.yaml (1-13)

Minor Updates & Housekeeping

This release includes several dependency updates, such as bumps to ztunnel, prometheus, cel-go, go-oidc, go-proxyproto, docker/cli, moby/buildkit, and various golang.org/x modules. Testing infrastructure also saw improvements, including the addition of UserNamespace support for echo tests and refactoring JWT test data to use embedded JWKS.