istio(1.29.5): Critical Bug Fixes for Ambient Mode, Gateway API, and Multicluster Stability

📋 Recommended Actions

⚠️ Action Required
Immediate patching required to address critical stability issues in Ambient Mode, Kubernetes Gateway API, and Multicluster deployments.

📝 Summary

Istio 1.29.5 lands with crucial bug fixes that significantly bolster the stability of your mesh. This release tackles a critical issue in Ambient Mode where enrolled pods could drop from the host health-probe ipset, leading to broken health checks after node restarts. We’ve also resolved a major traffic outage scenario during Kubernetes Gateway API canary upgrades, ensuring smoother transitions when changing istio.io/rev labels. For multicluster users, a panic (close of closed channel) from rapid kubeconfig secret updates is now fixed, enhancing resilience. Additionally, a memory leak within the KRT controller framework has been patched, preventing gradual memory growth. These targeted fixes are essential for maintaining robust and predictable service mesh operations. Upgrade now to secure these vital stability improvements.

🐛 Ambient Mode: Host Probe IPSet Reliability Fix

A critical bug in Ambient Mode could previously lead to pods being excluded from the host health-probe ipset, especially after a node or kubelet restart. This often resulted in kubelet probes being misdirected to ztunnel and subsequently rejected, causing service disruption. This release addresses that by implementing a self-healing mechanism, ensuring that once an enrolled pod’s IP reappears in the cache, its host probe ipset entry is immediately re-asserted, maintaining continuous health check functionality.

The MeshDataplane interface now includes a SyncHostProbeIPSet method, used by the CNI’s node agent informer. This method is triggered during pod reconciliation if an enrolled pod’s IP changes. Crucially, the syncHostAddrSets function, called during startup, now defers destructive pruning if the snapshot is incomplete (i.e., some enrolled pods don’t yet have observable IPs), preventing the accidental eviction of live pods.

// cni/pkg/nodeagent/informers.go
// ...

		if isEnrolled && shouldBeEnabled && !isTerminated {
			podIPs := util.GetPodIPsIfPresent(currentPod)
			if len(podIPs) > 0 && !slices.EqualUnordered(podIPs, util.GetPodIPsIfPresent(oldPod)) {
				if err := s.dataplane.SyncHostProbeIPSet(currentPod, podIPs); err != nil {
					log.Warnf("failed to sync host probe ipset for enrolled pod, will retry: %v", err)
					return err
				}
			}
		}
// ...

Source:

• cni/pkg/nodeagent/helpers_test.go (56-60)
• cni/pkg/nodeagent/informers.go (299-311)
• cni/pkg/nodeagent/informers_test.go (1240-1349)
• cni/pkg/nodeagent/meshdataplane_linux.go (237-279)
• cni/pkg/nodeagent/net_linux.go (87-92)
• cni/pkg/nodeagent/server.go (47-53)
• cni/pkg/nodeagent/server_linux_test.go (612-658)
• cni/pkg/nodeagent/server_unspecified.go (47-50)
• releasenotes/notes/ambient-probe-ipset-reconcile.yaml (1-10)

🚀 Kubernetes Gateway API: Enhanced Canary Upgrade Stability

When performing canary upgrades of Kubernetes Gateways, a change in the istio.io/rev label could previously trigger a brief traffic outage. The issue arose because the prior owning control plane would immediately drop the resource and push empty xDS configuration to gateway pods still running on the old revision. This release eliminates this instability, allowing the previous control plane to continue emitting configuration until all pods have transitioned, ensuring a seamless upgrade experience without traffic disruption.

The filtering based on tagWatcher.IsMine() has been intentionally removed from the configuration emission layer in GatewayCollection and ListenerSetCollection. This means that a control plane will continue to generate configuration for a Gateway or ListenerSet even if its istio.io/rev label points to a different revision. Status writes for non-owning revisions and Deployment management are still appropriately filtered in other components, preventing conflicts while enabling robust canary upgrade flows. This is crucial for avoiding a temporary outage during controlled rollouts.

// pilot/pkg/config/kube/gateway/gateway_collection.go
// ...
	// Note: tagWatcher.IsMine() is intentionally not filtered at this config-emission layer. Filtering
	// here caused a temporary outage when a Gateway's or ListenerSet's istio.io/rev label was changed:
	// the prior owning control plane immediately dropped the resource and pushed empty xDS config to
	// pods still running on the old revision (see https://github.com/istio/istio/issues/59959).
	// Status writes are filtered to the owning revision via RegisterStatus in
	// pilot/pkg/status/collections.go, and Deployment management is filtered in deploymentcontroller.go,
	// so emitting config from non-owning revisions is safe and matches how core Istio CRDs behave.
// ...

Source:

• pilot/pkg/config/kube/gateway/controller_test.go (139-165)
• pilot/pkg/config/kube/gateway/gateway_collection.go (87-99, 244-250)
• releasenotes/notes/59959.yaml (1-14)

🔮 Multicluster Stability: Idempotent KubeController Close

Operators managing multicluster Istio deployments faced potential panics, specifically a “close of closed channel” error, when a remote cluster’s kubeconfig secret was updated twice in rapid succession. This issue, occurring due to non-idempotent Close calls on the kubeController, could lead to instability. This release now ensures that the kubeController.Close() method is idempotent, preventing these panics and significantly improving the robustness of multicluster secret management.

A sync.Once primitive has been added to the kubeController.Close() method. This ensures that the cleanup and channel closing operations are executed only a single time, even if Close() is called multiple times concurrently. This change effectively guards against the race condition that previously led to panics during rapid credential rotations.

// pilot/pkg/serviceregistry/kube/controller/multicluster.go
// ...
func (k *kubeController) Close() {
	k.closeOnce.Do(func() {
		close(k.stop)
		clusterID := k.Controller.clusterID
		k.MeshServiceController.UnRegisterHandlersForCluster(clusterID)
		k.MeshServiceController.DeleteRegistry(clusterID, provider.Kubernetes)
		if k.workloadEntryController != nil {
			k.MeshServiceController.DeleteRegistry(clusterID, provider.External)
		}
		if err := k.Controller.Cleanup(); err != nil {
			log.Warnf("failed cleaning up services in %s: %v", clusterID, err)
		}
		if k.opts.XDSUpdater != nil {
			k.opts.XDSUpdater.ConfigUpdate(&model.PushRequest{Full: true, Reason: model.NewReasonStats(model.ClusterUpdate), Forced: true})
		}
	})
}
// ...

Source:

• pilot/pkg/serviceregistry/kube/controller/multicluster.go (53-77)
• releasenotes/notes/multicluster-kubecontroller-double-close.yaml (1-9)

🩹 KRT Framework: Reverse-Index Memory Leak Fixed

A subtle memory leak was discovered in the Kubernetes Resource Table (KRT) controller framework. This occurred when the FilterKey used in a Fetch operation changed—for instance, if a pod was relabeled to point to a different waypoint. The old reverse-index entries were not properly cleaned up, leading to a gradual increase in memory usage and potentially unnecessary recomputations over time. This release includes a fix to correctly prune these stale entries, ensuring efficient memory management and stable performance for KRT-based controllers.

The dependencyState.update function has been modified to explicitly call delete on existing dependencies before adding new ones. This ensures that when an object’s dependencies change (e.g., due to a label modification altering the FilterKey), any stale entries in the indexedDependencies map are properly removed. A new test, TestCollectionChangingFilterKeyDependency, has also been added to prevent future regressions of this memory leak.

// pkg/kube/krt/collection.go
// ...
func (i dependencyState[I]) update(key Key[I], deps []*dependency) {
	// We will override the current dependencies with the new ones, so
	// we first remove the existing dependencies from the reverse index
	// (indexedDependencies). Otherwise, when "I" goes away, a delete
	// operation might leave some reverse index behind.
	i.delete(key)

	// Update the I -> Dependency mapping. Notice that we're overriding
	// all dependencies for the key. That means that as the delete call
	// above removes the key from objectDependencies, that doesn't matter
	// as we reassign the key here once again.
	i.objectDependencies[key] = deps
// ...

Source:

• pkg/kube/krt/collection.go (54-68)
• pkg/kube/krt/dependency_state_test.go (1-137)
• releasenotes/notes/krt-dependency-reverse-index-leak.yaml (1-8)

⚙️ Sidecar Compatibility: Correct FactoryKey for Older Proxies

Prior to Istio 1.29.2, sidecar proxies expected a different factoryKey (envoy.string) for certain filter state values, specifically in the context of HBONE and multi-network features. An oversight could lead to incorrect configuration generation for these older sidecars, potentially breaking traffic or functionality upon upgrade. This release ensures backward compatibility by dynamically selecting the correct factoryKey based on the proxy’s version, preventing breakage for users still on earlier 1.29.x sidecar versions.

Conditional logic has been introduced in buildInnerConnectOriginateListener and buildHCMConnectTerminateChain to use "envoy.string" as the FactoryKey for proxies older than 1.29.2, and "istio.hashable_string" for newer versions. Similarly, the specific ConnectAuthorityFilter and RequestSourceFilter implementations are chosen based on the proxy’s version. This allows seamless operation across different proxy versions within a mesh during upgrades.

// pilot/pkg/networking/core/listener.go
// ...
func buildInnerConnectOriginateListener(push *model.PushContext, proxy *model.Proxy) *listener.Listener {
	// TODO: Remove in 1.32
	factoryKey := "istio.hashable_string"
	if !proxy.VersionGreaterOrEqual(&model.IstioVersion{Major: 1, Minor: 29, Patch: 2}) {
		factoryKey = "envoy.string"
	}
// ...

Source:

• pilot/pkg/networking/core/listener.go (1401-1437)
• pilot/pkg/networking/core/listener_test.go (377-446)
• pilot/pkg/networking/core/listener_waypoint.go (186-215)
• pilot/pkg/xds/filters/filters.go (348-468)
• releasenotes/notes/istio-hashable-string.yaml (1-6)

Minor Updates & Housekeeping

This release includes a robustification of the build system with enhanced retry logic for go module and tool fetches, improving reliability in transient network conditions. We’ve also bumped the Go version to 1.25.9 and updated several dependencies, contributing to overall stability and security. Additionally, a minor fix addresses parsing for authorization policy dry-run annotations when the value is empty.