📋 Recommended Actions

⚠️ Action Required
Immediate action is required for existing auto-registered WorkloadEntries if HBONE functionality is desired. These require re-registration or manual labeling. Review all updates for ambient mesh deployments to ensure proper traffic flow and resource management.

📝 Summary

Istio 1.29.6 delivers critical stability and reliability enhancements, particularly for ambient mesh users. This release addresses a vital issue where cross-network ambient traffic through east-west gateways could be spuriously blocked by L7 AuthorizationPolicies. A deadlock in the ambient CNI node agent, which could occur during concurrent pod deletion and ztunnel reconnections, has also been resolved, improving robustness. Importantly, HBONE compatibility for non-Kubernetes auto-registered WorkloadEntries is now correctly propagated, though existing entries may need manual updates. Furthermore, a significant memory leak in Istiod related to needResync entries for failed pods has been fixed, enhancing controller stability and reducing resource consumption. These updates collectively bolster Istio’s ambient capabilities and overall operational efficiency.

🚀 Enhanced Ambient Mesh Reliability and Interoperability

This release significantly bolsters the reliability and interoperability of Istio’s ambient mesh, resolving several critical issues that could impact traffic flow, proxy draining, and workload registration. We’ve tackled an issue where cross-network traffic through east-west gateways was sometimes blocked unnecessarily by L7 AuthorizationPolicies. Additionally, a tricky deadlock in the ambient CNI node agent, preventing proper ZDS server operation during concurrent pod deletions and ztunnel reconnections, has been eliminated. These fixes ensure a smoother, more predictable experience for ambient mesh deployments, especially in multi-network and dynamic environments.

Previously, the east-west gateway’s listener configuration could inadvertently apply deny-all RBAC filters to cross-network traffic, even when L7 AuthorizationPolicies were present for the destination service. This has been addressed by ensuring the policy service is explicitly set to nil for east-west gateways, preventing unintended policy enforcement:

// In pilot/pkg/networking/core/listener_waypoint.go
if isEastWestGateway {
    cc.policyService = nil
}

For the CNI node agent, a potential deadlock during ztunnel disconnection or pod deletion events has been resolved. The connection manager (connMgr) now safely closes the connection’s Done() channel and uses a snapshot of connections for iteration, avoiding race conditions and ensuring that PodDeleted calls can complete even if a ztunnel disconnects concurrently:

// In cni/pkg/nodeagent/ztunnelserver.go
func (c *connMgr) deleteConn(conn ZtunnelConnection) {
    log.Debug("ztunnel disconnected")
    close(conn.Done()) // Close done channel BEFORE locking
    c.mu.Lock()
    defer c.mu.Unlock()
    // ... rest of delete logic
}

// And for PodDeleted iteration:
for _, conn := range z.conns.snapshotConns() {
    // ... send message
}

Source:

  • pilot/pkg/networking/core/listener_waypoint.go (385-387)
  • cni/pkg/nodeagent/ztunnelserver.go (84-100)
  • cni/pkg/nodeagent/ztunnelserver_linux.go (32-35)
  • cni/pkg/nodeagent/ztunnelserver_linux_test.go (567-598)

✨ Correct HBONE Propagation for Auto-Registered WorkloadEntries

For users leveraging Istio’s ambient mesh with auto-registered WorkloadEntries for non-Kubernetes workloads, this update ensures that the HBONE (HTTP-based Overlay Network) capability is correctly propagated. Previously, the system might not have consistently applied the necessary tunnel label during auto-registration, potentially leading to plaintext traffic instead of the expected secure HBONE tunnel. This fix guarantees that proxies compatible with HBONE will have their WorkloadEntries appropriately labeled, enabling seamless and secure communication.

The WorkloadEntry generation logic has been updated to automatically include the networking.istio.io/tunnel: http label if the associated proxy is HBONE-capable (proxy.EnableHBONEListen() returns true) and the label isn’t already explicitly set. This ensures that Istiod correctly identifies these workloads as HBONE-compatible:

// In pilot/pkg/autoregistration/controller.go
// If the proxy accepts HBONE, set the tunnel label so istiod marks the workload HBONE-capable
if proxy.EnableHBONEListen() {
    if entry.Labels == nil {
        entry.Labels = map[string]string{}
    }
    if _, ok := entry.Labels[model.TunnelLabel]; !ok {
        entry.Labels[model.TunnelLabel] = model.TunnelHTTP
    }
}

Action Required: Existing auto-registered WorkloadEntries created before this update will not automatically gain the HBONE label. To ensure these workloads use HBONE, you must either re-register them (e.g., by restarting the workload) or manually add the networking.istio.io/tunnel: http label to their corresponding WorkloadEntry resource.

Source:

  • pilot/pkg/autoregistration/controller.go (699-708)
  • pilot/pkg/autoregistration/controller_test.go (474-547)
  • pilot/pkg/serviceregistry/kube/controller/ambient/workloads_test.go (914-954)

🩹 Istiod Stability: Fixed Memory Leak for Failed Pods

We’ve addressed a persistent memory leak in Istiod related to how it manages needResync entries for failed pods. Previously, when an EndpointSlice update removed an IP address corresponding to a failed pod, the associated needResync entry was not correctly cleaned up. Over time, this could lead to increased memory consumption and degraded performance for the Istiod control plane. This fix ensures that such entries are properly evicted, contributing to a more stable and resource-efficient Istiod.

The endpointSliceController now includes a cleanupRemovedEndpoints function. This function is invoked during an EndpointSlice UPDATE event to compare the old and new slices. If an IP address is present in the old slice but no longer in the current one, endpointDeleted is explicitly called for that IP, ensuring that any pending needResync entries are cleared. This prevents indefinite accumulation of stale entries, especially after pods fail and are filtered from the pod cache.

// In pilot/pkg/serviceregistry/kube/controller/endpointslice.go
func (esc *endpointSliceController) cleanupRemovedEndpoints(old, cur *v1.EndpointSlice) {
    curAddrs := sets.New[string]()
    for _, e := range cur.Endpoints {
        for _, a := range e.Addresses {
            curAddrs.Insert(a)
        }
    }
    key := config.NamespacedName(cur)
    for _, e := range old.Endpoints {
        for _, a := range e.Addresses {
            if !curAddrs.Contains(a) {
                esc.c.pods.endpointDeleted(key, a)
            }
        }
    }
}

Source:

  • pilot/pkg/serviceregistry/kube/controller/controller_test.go (2216-2224)
  • pilot/pkg/serviceregistry/kube/controller/endpointslice.go (85-115, 233-255)

🧹 Refined Envoy Drain Logic for Ambient Gateways

A subtle but significant issue affecting the graceful shutdown of Envoy proxies in ambient ingress gateways and waypoints has been resolved. Previously, the EXIT_ON_ZERO_ACTIVE_CONNECTIONS drain mechanism would sometimes fail to trigger, forcing the proxy to wait for the full terminationGracePeriodSeconds to expire. This was due to Envoy’s internal HBONE (HTTP-based Overlay Network) plumbing listeners being incorrectly counted as active connections. By excluding these internal listeners, proxies can now drain and exit promptly, improving resource utilization and deployment efficiency.

The activeProxyConnections calculation in pkg/envoy/agent.go has been updated to explicitly ignore internal Envoy listeners prefixed with listener.envoy_internal_. These listeners (connect_originate, connect_terminate, main_internal, etc.) are used for in-process communication and do not represent client-facing connections. Excluding them ensures that the active connection count accurately reflects external traffic, allowing the EXIT_ON_ZERO_ACTIVE_CONNECTIONS logic to function as intended.

// In pkg/envoy/agent.go
// Skip Envoy internal listeners (envoy.bootstrap.internal_listener). These are
// in-process plumbing — e.g. ambient HBONE connect_originate / connect_terminate /
// main_internal — with no client-facing semantics. Their downstream_cx_active counts
// pipe connections from other filter chains and stays positive as long as the
// upstream HBONE pool holds any tunnel open, which would otherwise prevent
// EXIT_ON_ZERO_ACTIVE_CONNECTIONS from ever firing on ambient gateways and waypoints.
if strings.HasPrefix(parts[0], "listener.envoy_internal_") {
    continue
}

Source:

Minor Updates & Housekeeping

This release also includes various dependency updates, such as bumping github.com/google/cel-go to v0.29.2, github.com/prometheus/prometheus to v0.311.3, github.com/pires/go-proxyproto to v0.15.0, and updating internal proxy, ztunnel, istio/api, and istio/client-go components, alongside refreshed build-tools image versions.