📋 Recommended Actions

⚠️ Action Required
Upgrade to Istio 1.30.1 as soon as possible to benefit from critical stability fixes and improved Gateway API diagnostics. Review the sections on ambient mode, multicluster, and Gateway API headers for specific impacts.

📝 Summary

Istio 1.30.1 delivers crucial stability enhancements and significant Gateway API improvements. This release addresses critical issues like a concurrent map writes panic in the CNI agent and a deadlock in the multicluster ClusterStore, bolstering overall mesh reliability. Traffic management sees a vital fix for consistentHash load balancing, ensuring new endpoints receive traffic reliably, resolving an Envoy regression. Ambient mode users will find better traffic distribution with a fix for mutating shared load balancer presets, preventing misconfigurations where traffic was routed to not-ready endpoints. The Gateway API ecosystem gains an important istioctl analyze check (IST0176) that warns about outdated CRD versions, preventing silent failures. External SDS provider logic is refined for more flexible credential management, and invalid HTTP/GRPCRoute header values are now properly reported, enhancing configuration validation. Kiali also receives an update to version 2.26.0. Upgrade promptly to secure these stability and diagnostic benefits.

⚙️ CNI Stability: Concurrency Fix and NFTables JSON Probe

This release significantly improves the stability of the Istio CNI, particularly for Ambient mode deployments. We’ve resolved a critical concurrent map writes panic that could occur when multiple pods were simultaneously added to the ambient mesh on the same node. To further enhance robustness, a new initialization check has been introduced that verifies the nft binary’s support for JSON output, which is essential for certain configurations. If JSON support is absent, the CNI agent will now gracefully fall back to the iptables backend, preventing indefinite retry loops and improving startup reliability.

The fix for the concurrent map writes panic involved re-scoping the NftablesRuleBuilder to be per-call rather than a shared field on the NftablesConfigurator struct. This prevents race conditions during concurrent pod-add operations. The new JSON probe is implemented with detectNftJSONSupport, which executes nft --json list tables at startup. If the output indicates “JSON support not compiled-in,” Istio CNI logs a warning and switches to iptables.

// NftablesConfigurator handles nftables rule management for Ambient mode.
// Shared across concurrent pod-add goroutines; rule builders are scoped
// per-call and passed through, not held on the struct.
type NftablesConfigurator struct {
	// ... existing fields ...
	// ruleBuilder *builder.NftablesRuleBuilder // Removed
}

func (cfg *NftablesConfigurator) AppendInpodRules(podOverrides config.PodLevelOverrides) (*knftables.Transaction, error) {
	rb := builder.NewNftablesRuleBuilder(config.GetConfig(cfg.cfg))
	// ... rules are appended to rb ...
	return cfg.executeCommands(rb)
}

Source:

  • cni/pkg/nftables/nftables.go (60-64, 145-145, 558-558)
  • cni/pkg/nftables/nftables_test.go (189-216)
  • cni/pkg/nodeagent/detect_nft_json_linux.go (1-51)
  • cni/pkg/nodeagent/server_linux.go (73-89)
  • releasenotes/notes/60328.yaml (1-17)

📈 Ambient Mode: Load Balancer Preset Mutation Fix

A subtle but critical bug in ambient mode’s traffic distribution has been squashed. Previously, when a service combined publishNotReadyAddresses: true with a PreferSameZone or PreferSameNode traffic distribution, it would inadvertently mutate a shared package-level load balancer preset. This meant that every subsequent service utilizing the same traffic distribution preset would inherit healthPolicy: AllowAll, leading to traffic being incorrectly routed to not-ready endpoints cluster-wide. This fix ensures that each service correctly receives its own isolated load balancing configuration.

The preferSameZoneLoadBalancer and preferSameNodeLoadBalancer variables, which were previously global pointers, have been refactored into functions that return a fresh *workloadapi.LoadBalancing object on each call. This change ensures that any mutations applied by a specific service (like setting HealthPolicy for PublishNotReadyAddresses) are confined to that service’s configuration and do not impact other services sharing the same traffic distribution preset.

// preferSameZoneLoadBalancer returns a fresh LoadBalancing config for PreferSameZone
// traffic distribution. It must return a new pointer on every call because callers
// (notably constructService) may mutate the returned object in place (for example,
// setting HealthPolicy when PublishNotReadyAddresses is true). Returning a shared
// package-level pointer here previously caused those mutations to leak across
// services, poisoning the preset for every other service in the cluster.
func preferSameZoneLoadBalancer() *workloadapi.LoadBalancing {
	return &workloadapi.LoadBalancing{
		// ... configuration ...
	}
}

Source:

  • pilot/pkg/serviceregistry/ambient/services.go (699-732)
  • pilot/pkg/serviceregistry/ambient/services_test.go (1620-1667)
  • releasenotes/notes/60423.yaml (1-8)

🚨 Traffic Management: Consistent Hash Load Balancing Fix

A critical issue affecting consistentHash load balancing in DestinationRule has been resolved. Previously, after scaling up a service, traffic might not have been correctly directed to the newly added endpoints. This stemmed from an Envoy regression (envoyproxy/envoy#45212) where the RING_HASH ring, crucial for consistent hashing, was not rebuilt on endpoint changes during batched updates. This fix restores proper functionality, ensuring that all endpoints, including new ones, are considered for traffic distribution.

To mitigate the Envoy regression, Istio now explicitly disables the envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update flag within Envoy’s bootstrap configuration. By setting this flag to false, we ensure that the load balancer’s ring hash is rebuilt reliably when endpoint changes occur, thus guaranteeing that consistentHash routing behaves as expected and new endpoints promptly receive traffic.

{
  "name": "global config",
  "static_layer": {
    "envoy.deprecated_features:envoy.config.listener.v3.Listener.hidden_envoy_deprecated_use_original_dst": true,
    "envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update": false,
    "envoy.reloadable_features.fixed_heap_use_allocated": true,
    "envoy.reloadable_features.http_reject_path_with_fragment": false,
    "re2.max_program_size.error_level": "32768"
  }
}

Source:

  • pkg/bootstrap/config.go (393-395)
  • pkg/bootstrap/testdata/all_golden.json (15-18)
  • releasenotes/notes/60312.yaml (1-9)

🌐 Multicluster Stability: ClusterStore Deadlock Fix

A subtle but impactful deadlock within the multicluster ClusterStore has been resolved. This deadlock could occur under specific conditions when AllReady attempted to recursively acquire a read lock via triggerRecomputeOnSync and GetByID, while a concurrent write operation was waiting on the same RWMutex. This fix eliminates the potential for the ClusterStore to become unresponsive, improving the reliability and performance of multicluster deployments.

The triggerRecomputeOnSync function has been refactored to directly receive the *Cluster object rather than looking it up by its ID. This prevents the recursive acquisition of the read lock (RLock) that was the root cause of the deadlock, especially when AllReady was already holding an RLock. The modification ensures that access patterns to the ClusterStore’s internal mutex are strictly sequential, preventing contention.

// triggerRecomputeOnSync sets up a goroutine to wait for the cluster to be synced,
// and then triggers a recompute when it is. Callers must pass the cluster directly
// rather than its ID: AllReady holds the store RLock when calling this, and looking
// the cluster up here would attempt a recursive RLock that can deadlock against a
// concurrent writer waiting on the same RWMutex.
func (c *ClusterStore) triggerRecomputeOnSync(cl *Cluster) {
	c.casMu.Lock()
	defer c.casMu.Unlock()
	id := cl.ID
	// ... existing logic ...
	// if cluster.WaitUntilSynced(cluster.stop) && !cluster.Closed() && !cluster.SyncDidTimeout() && c.GetByID(id) != nil {
	if cl.WaitUntilSynced(cl.stop) && !cl.Closed() && !cl.SyncDidTimeout() && c.GetByID(id) != nil {
		// ... trigger recomputation ...
	}
}

Source:

  • pkg/kube/multicluster/clusterstore.go (232-261)
  • releasenotes/notes/multicluster-clusterstore-rlock-deadlock.yaml (1-7)

🛡️ Gateway API: Ignored Policy Attachment Annotation

Istio now offers enhanced flexibility in Gateway API policy management with the introduction of the istio.io/ignore-policy-attachment annotation. This powerful feature allows operators to explicitly exclude specific BackendTLSPolicy or XBackendTrafficPolicy objects from being translated into Istio configuration. This is particularly useful in multi-controller environments where certain policies might be intended for a different gateway controller, preventing unintended interactions or conflicts and streamlining policy application.

To leverage this, simply add the istio.io/ignore-policy-attachment: "true" annotation to any BackendTLSPolicy or XBackendTrafficPolicy resource. When this annotation is detected, Istio will ignore the policy during its translation process and report an IgnoredByIstio status reason, clearly indicating that the policy was deliberately excluded. This ensures clear communication about policy intent across different controllers in your Kubernetes cluster.

apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: tls-ignored
  namespace: default
  annotations:
    istio.io/ignore-policy-attachment: "true"
spec:
  targetRefs:
  - kind: Service
    name: echo
    group: ""
  validation:
    wellKnownCACertificates: System
    hostname: echo.example.com

Source:

  • pilot/pkg/config/kube/gateway/backend_policies.go (48-52, 287-291, 521-525, 606-633)
  • pilot/pkg/config/kube/gateway/testdata/backend-tls-policy-ignored.yaml (1-46)
  • pilot/pkg/config/kube/gateway/testdata/backend-tls-policy-ignored.status.yaml.golden (1-58)
  • pilot/pkg/config/kube/gateway/testdata/backend-traffic-policy-ignored.yaml (1-52)
  • pilot/pkg/config/kube/gateway/testdata/backend-traffic-policy-ignored.status.yaml.golden (1-53)
  • releasenotes/notes/60122.yaml (1-21)

⚠️ Gateway API: Invalid Header Value Reporting

Improving the robustness and user feedback for Gateway API configurations, Istio now correctly identifies and reports InvalidFilter status for HTTPRoute and GRPCRoute filters that contain invalid characters in header values. Previously, such misconfigurations were silently dropped from the Envoy configuration, making debugging difficult. This enhancement ensures that any headers with control characters (excluding tab) are flagged immediately, providing clear feedback to operators and preventing unexpected traffic behavior.

The createHeadersFilter function, responsible for translating HTTPHeaderFilter configurations, has been enhanced to include a isValidHeaderValue check. This validation now explicitly disallows header values containing control characters (e.g., newline, carriage return, null byte) which are generally forbidden in HTTP headers. If an invalid character is detected, a ConfigError with the InvalidFilter reason is returned, prompting the Gateway API status to reflect the issue. This makes it much easier to pinpoint and correct malformed header configurations.

func isValidHeaderValue(v string) bool {
	for _, c := range []byte(v) {
		if c == 0x09 || c >= 0x20 && c != 0x7F {
			continue
		}
		return false
	}
	return true
}

func createHeadersFilter(filter *k8s.HTTPHeaderFilter) (*istio.Headers_HeaderOperations, *ConfigError) {
	// ... filter for nil ...
	for _, h := range append(filter.Set, filter.Add...) {
		if !isValidHeaderValue(h.Value) {
			return nil, &ConfigError{
				Reason:  InvalidFilter,
				Message: fmt.Sprintf("header %q value contains invalid characters", h.Name),
			}
		}
	}
	return &istio.Headers_HeaderOperations{
		// ... header operations ...
	}, nil
}

Source:

🔒 Gateway API: Enhanced External SDS Provider Logic

Istio’s handling of external SDS (Secret Discovery Service) providers for Gateways has been significantly improved, offering more flexible and robust certificate management. This update refines the credential resolution logic to prioritize UDS sockets, then MeshConfig-defined extension providers, and finally falls back to ADS (Kubernetes Secrets) if no explicit provider is found. This ensures that credential requests are handled correctly across various deployment scenarios, while also fixing an issue where credential names were not consistently passed to external SDS servers, preventing multiple gateways from requesting distinct certificates.

The ConstructSdsSecretConfigForCredential function now uses a clear priority order. For sds:// prefixed credential names, it first checks for UDS socket availability (for local SDS agents). If none, it scans MeshConfig.ExtensionProviders for a matching SDS provider. If still no provider is found, it automatically falls back to requesting the secret via ADS, treating the resource name (after stripping sds://) as a Kubernetes Secret reference. This allows for diverse SDS deployments without silent failures and correctly derives the CA certificate resource name as <credential-name>-cacert for Mutual TLS.

func ConstructSdsSecretConfigForCredential(name string, credentialSocketExist bool, push *model.PushContext) *tls.SdsSecretConfig {
	// ... existing logic ...
	if strings.HasPrefix(name, security.SDSExternalCredentialPrefix) {
		resourceName, _ := strings.CutPrefix(name, security.SDSExternalCredentialPrefix)
		if credentialSocketExist {
			// Preserve full name (including sds:// prefix) for backward compatibility
			return ConstructSdsSecretConfigForCredentialSocket(name, security.SDSExternalClusterName)
		}
		if push != nil {
			for _, provider := range push.Mesh.ExtensionProviders {
				// ... logic to find and use SDS extension provider ...
			}
		}
		// No UDS socket or extension provider 
		// Fall back to ADS (Kubernetes Secret via istiod)
		name = resourceName
	}
	return &tls.SdsSecretConfig{ /* ... */ }
}

Source:

  • pilot/pkg/security/model/authentication.go (74-106)
  • pilot/pkg/security/model/authentication_test.go (713-847, 931-936)
  • releasenotes/notes/external-sds-gateway-fix.yaml (1-14)

🔎 Istioctl Analyze: Gateway API CRD Version Checks

To prevent silent failures and streamline debugging for Gateway API users, istioctl analyze now includes a new check, IST0176. This analyzer flags Gateway API Custom Resource Definitions (CRDs) that are installed at a version below the minimum required by the current Istio release. Outdated CRDs can lead to resources (like TLSRoute) being silently filtered by istiod, causing unexpected behavior such as TLS passthrough not working. This proactive diagnostic helps ensure your Gateway API setup is compliant and fully functional.

The new CRDVersionAnalyzer compares the gateway.networking.k8s.io/bundle-version annotation on installed Gateway API CRDs (e.g., tlsroutes.gateway.networking.k8s.io, grpcroutes.gateway.networking.k8s.io, backendtlspolicies.gateway.networking.k8s.io) against a map of MinimumCRDVersions maintained by Istio. If an installed CRD version (stripped of any pre-release suffix) is found to be older than the required minimum, istioctl analyze will report a warning message, advising users to upgrade their Gateway API CRDs. This makes it much easier to diagnose and fix configuration issues related to CRD compatibility.

// MinimumCRDVersions is the minimum Gateway API bundle version (the value of the
// `gateway.networking.k8s.io/bundle-version` annotation on the CRD) required by
// this Istio binary, keyed by CRD name. CRDs below the listed version are not
// processed by istiod and will not be watched.
var MinimumCRDVersions = map[string]*semver.Version{
	"grpcroutes.gateway.networking.k8s.io":         semver.New(1, 1, 0, "", ""),
	"backendtlspolicies.gateway.networking.k8s.io": semver.New(1, 4, 0, "", ""),
	"tlsroutes.gateway.networking.k8s.io":          semver.New(1, 5, 0, "", ""),
}

Source:

  • pkg/config/analysis/analyzers/all.go (62-62)
  • pkg/config/analysis/analyzers/k8sgateway/crdversion.go (1-78)
  • pkg/config/analysis/analyzers/testdata/gateway-api-crd-version-old.yaml (1-45)
  • pkg/config/analysis/msg/messages.gen.go (271-274, 342-345, 993-1002)
  • pkg/config/analysis/msg/messages.yaml (711-725)
  • pkg/config/schema/gatewayapi/minimumversions.go (1-32)
  • pkg/kube/kclient/crdwatcher.go (177-177)
  • releasenotes/notes/gateway-api-crd-version-analyzer.yaml (1-9)

Minor Updates & Housekeeping

This release also updates the Kiali addon to version 2.26.0, includes several dependency bumps across various components, and migrates the internal h2c wrapper to use standard library http.Protocols for improved platform compatibility and performance.