📋 Recommended Actions

⚠️ Action Required
Immediate upgrade to Istio 1.28.5 is strongly recommended for all users due to critical security patches addressing JWT forgery, XDS debug endpoint authentication bypasses, and WasmPlugin SSRF vulnerabilities. Review changes to XDS debug endpoints if you rely on unauthenticated plaintext access, as this behavior now requires explicit configuration or authentication.

📝 Summary

Istio 1.28.5 lands with crucial security updates and significant enhancements across the mesh. This release patches a critical vulnerability where Istio’s JWT authentication fallback mechanism could leak a private key, enabling attackers to forge tokens. A high-severity fix now secures XDS debug endpoints (like syncz and config_dump), preventing unauthenticated access on plaintext ports. Additionally, WasmPlugin image fetching is fortified with SSRF protection, closing another potential attack vector. Beyond security, the Gateway API sees improvements, specifically addressing issues where InferencePool configurations were lost during VirtualService merges. Ambient Mesh deployments get smarter port discovery for native sidecars, ensuring correct inbound listener configuration, and gain new flexibility with a ZtunnelNamespace flag. These updates combine critical fixes with valuable operational improvements, making 1.28.5 a vital upgrade for a more secure and robust service mesh.

🔒 Critical Security Fix: Preventing JWT Forgery via JWKS Private Key Leak

A critical security vulnerability has been identified and patched where Istio’s JWT authentication fallback mechanism could inadvertently leak an RSA private key. Previously, if fetching a JSON Web Key Set (JWKS) failed due to network issues or an invalid URL, Istio would fall back to a hardcoded FakeJwks constant that, alarmingly, contained a complete RSA keypair including the private key. This flaw meant that attackers could extract this private key from Istio’s source code or Envoy configuration dumps and then forge arbitrary JWT tokens. These forged tokens would be accepted as valid during the JWKS fetch failure window, allowing attackers to bypass authentication and gain unauthorized access. This fix ensures that such a critical authentication bypass is no longer possible.

The vulnerability has been addressed by replacing the insecure FakeJwks constant with a new PublicOnlyJwks constant. This new fallback JWKS contains only a public RSA key, where the corresponding private key was generated once and immediately discarded. Consequently, it is now cryptographically impossible for anyone to forge JWTs that would validate against this public key. This change implements a ‘fail-closed’ security posture: if JWKS fetching fails, JWT authentication will also fail, preventing unauthorized access rather than enabling it.

// PublicOnlyJwks is a JWKS containing a public RSA key where the private key was generated and immediately discarded.
// This is used as a fail-closed fallback when JWKS fetch fails - since nobody has the private key, no valid JWTs can be forged.
const PublicOnlyJwks = `{
  "keys": [
    {
      "e": "AQAB",
      "kty": "RSA",
      "n": "0xObjM0UvS_oaazjpEYlAbwctEJ4L8pH3OuTb7qth7gUwqet-EzQB4dgFdvSdMgrLnSncQGRjpEYz3F3viIbH-3EN3TxSlPNviHxeOdyiBVfumN8dMxbLvVJUpfNOnvmMxJcl-8NNjAwcOjk4otSALaYgYYyOPyvKtgVdrQr-FoubWX4yrjxW-MJ2-7OBeepUUNOsVwGV23YX03sVkkyvY3otRflkBcY3_HKpBxJl9wk2GyOShN4_PNUF9-vwfnvOXMbCDX-w4PTef4geeb_GiT40YCKHTKMSPVanGRn5GExIWmki-mmqh94-mPTJyBR74ShspL3BxPZitDL574T1w"
    }
  ]
}`

Source:

  • pilot/pkg/model/jwks_resolver.go (295-310)
  • pilot/pkg/model/jwks_resolver_test.go (790-819)
  • releasenotes/notes/jwks-private-key-leak.yaml (1-22)

🔒⚠️ Hardened Security: XDS Debug Endpoints Now Require Authentication

To enhance the overall security posture of Istio, XDS debug endpoints, including syncz and config_dump, now explicitly require authentication. Previously, these sensitive endpoints were accessible without authentication on plaintext XDS port 15010. This allowed potential information disclosure to any entity with network access to the Istiod control plane. This update ensures that only authorized and authenticated clients can query these endpoints, significantly reducing the attack surface and protecting critical configuration and status information.

The change introduces an authentication check within the StatusGen and DebugGen components. When the features.EnableDebugEndpointAuth flag is set to true (which is the default behavior in this release), any attempt to access /debug/syncz or /debug/config_dump via a plaintext connection without a verified identity will be rejected with an Unauthenticated gRPC status error. Additionally, the debug endpoint now correctly passes the caller’s namespace context, enabling fine-grained authorization checks for cross-namespace access scenarios.

If your existing tooling or istioctl commands rely on unauthenticated plaintext access (e.g., using --plaintext), they might fail after this upgrade. To restore the previous behavior, you can explicitly set the ENABLE_DEBUG_ENDPOINT_AUTH environment variable to false in your Istiod deployment, though this is generally not recommended for production environments.

func (sg *StatusGen) handleInternalRequest(proxy *model.Proxy, w *model.WatchedResource, _ *model.PushRequest) (model.Resources, model.XdsLogDetails, error) {
	if features.EnableDebugEndpointAuth && proxy.VerifiedIdentity == nil {
		log.Warnf("proxy %s is not authorized to receive debug info. Ensure you are connecting over TLS port and are authenticated.", proxy.ID)
		return nil, model.DefaultXdsLogDetails, grpcstatus.Error(codes.Unauthenticated, "authentication required")
	}

	// ... existing logic ...
}

func processDebugRequest(dg *DebugGen, resourceName string, callerNamespace string) bytes.Buffer {
	var buffer bytes.Buffer
	debugURL := "/debug/" + resourceName
	ctx := context.WithValue(context.Background(), CallerNamespaceKey{}, callerNamespace)
	hreq, _ := http.NewRequestWithContext(ctx, http.MethodGet, debugURL, nil)
	handler, _ := dg.DebugMux.Handler(hreq)
	response := NewResponseCapture()
	handler.ServeHTTP(response, hreq)
	return buffer
}

Source:

  • pilot/pkg/xds/statusgen.go (76-81)
  • pilot/pkg/xds/debuggen.go (155-163)
  • pilot/pkg/xds/debug_test.go (378-438)
  • releasenotes/notes/statusgen-xds-auth.yaml (1-21)
  • releasenotes/notes/xds-debug-namespace-auth.yaml (1-6)

🔒 Enhanced Security: WasmPlugin Image Fetching Gains SSRF Protection

A potential Server-Side Request Forgery (SSRF) vulnerability has been mitigated in the WasmPlugin image fetching mechanism. This vulnerability could arise if a malicious WWW-Authenticate header with a crafted realm URL was returned during image pull, potentially tricking the fetching client into making requests to internal or forbidden endpoints. This fix introduces robust validation for these realm URLs, ensuring that WasmPlugin image pulls cannot be exploited to facilitate SSRF attacks against internal services or cloud metadata APIs.

The ImageFetcher now uses an ssrfProtectionTransport to wrap its underlying HTTP client. This custom transport intercepts 401 Unauthorized responses and inspects the WWW-Authenticate header. It validates all realm parameters within this header against a strict set of rules:

  • Only http and https schemes are allowed.
  • Hosts must be present and cannot be localhost.
  • Known cloud metadata DNS names (e.g., metadata.google.internal) are blocked.
  • Private, loopback, and link-local IP ranges (e.g., 10.0.0.0/8, 127.0.0.1/8, 169.254.0.0/16) are blocked.

Any realm URL failing these validations will cause the image fetch to be rejected, preventing the SSRF attempt.

// ssrfProtectionTransport wraps http.RoundTripper to block SSRF via bearer realm
type ssrfProtectionTransport struct {
	inner http.RoundTripper
}

func (t *ssrfProtectionTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	resp, err := t.inner.RoundTrip(req)
	if err != nil || resp == nil {
		return resp, err
	}

	// check 401 responses for malicious WWW-Authenticate realm
	if resp.StatusCode == http.StatusUnauthorized {
		for _, auth := range resp.Header.Values("WWW-Authenticate") {
			if err := validateAllRealms(auth); err != nil {
				resp.Body.Close()
				return nil, fmt.Errorf("rejected unsafe bearer realm: %w", err)
			}
		}
	}

	return resp, nil
}

Source:

  • pkg/wasm/imagefetcher.go (64-154)
  • pkg/wasm/imagefetcher_test.go (629-843)
  • releasenotes/notes/wasm-ssrf-bearer-realm.yaml (1-11)

✨ Improved Gateway API: Seamless VirtualService Merging for InferencePools

Istio’s Gateway API integration now handles VirtualService merging more robustly, specifically fixing an issue where InferencePool configurations were lost when multiple HTTPRoutes referencing different InferencePools were attached to the same Gateway. This enhancement ensures that all specialized configurations, crucial for AI/ML workloads using InferencePools, are correctly preserved and aggregated, providing a more reliable and predictable experience for Gateway API users. You can now define complex routing with InferencePools across multiple HTTPRoutes without fear of configuration loss.

The fix specifically targets the mergeHTTPRoutes function, which is responsible for consolidating multiple HTTPRoutes into a single VirtualService representation. Previously, the Extra field within the VirtualService (where InferencePool configurations are stored) was not deep-copied or merged correctly across different HTTPRoute objects. This led to only the first HTTPRoute’s InferencePool configuration being retained, with subsequent ones being overwritten or discarded. The updated logic now ensures that the ConfigExtraPerRouteRuleInferencePoolConfigs map within the Extra field is properly deep-copied and then merged, aggregating all unique InferencePool route configurations from all associated HTTPRoutes. This guarantees that your AI/ML traffic routing policies remain consistent and complete.

// Deep copy the InferencePool configs map to avoid race conditions
// The default DeepCopy() only does shallow copy of Extra field
if base.Extra != nil {
	if ipConfigs, ok := base.Extra[constants.ConfigExtraPerRouteRuleInferencePoolConfigs].(map[string]kube.InferencePoolRouteRuleConfig); ok {
		// Create a new map to avoid modifying the shared underlying map
		newIPConfigs := make(map[string]kube.InferencePoolRouteRuleConfig, len(ipConfigs))
		for k, v := range ipConfigs {
			newIPConfigs[k] = v
		}
		base.Extra[constants.ConfigExtraPerRouteRuleInferencePoolConfigs] = newIPConfigs
	}
}
for i, config := range configs[1:] {
	// ... existing logic ...
	// Merge Extra field (especially for InferencePool configs)
	if config.Extra != nil {
		for k, v := range config.Extra {
			// For InferencePool configs, merge the maps
			// ... (other logic omitted for brevity) ...
				for routeName, routeConfig := range configMap {
					baseMap[routeName] = routeConfig
				}
			// ...
		}
	}
}

Source:

  • pilot/pkg/config/kube/gateway/route_collections.go (813-860)
  • releasenotes/notes/58393.yaml (1-9)

⚙️ Ambient Mesh: Smarter Native Sidecar Port Discovery

For Ambient Mesh deployments, Istio now more accurately identifies and accounts for ports exposed by native sidecar init containers. Previously, the system might have overlooked ports defined in these specialized init containers, leading to incomplete inbound listener configurations for workloads running in ambient mode. This enhancement ensures that all relevant ports, including those used by native sidecar init containers with a restartPolicy=Always, are correctly discovered and considered when configuring inbound traffic, improving the reliability and functionality of your Ambient Mesh applications.

The port discovery logic within Pilot has been updated across multiple components to include ports from native sidecar init containers. Specifically, the FindPortName, getPortMap, and FindPort utility functions now iterate through pod.Spec.InitContainers and include any ports from containers that have a restartPolicy set to Always. This ensures that even long-running init containers, often used for specialized native sidecar functionalities, have their ports correctly mapped for Istio’s traffic management. Additionally, StripPodUnusedFields now preserves ports and restart policy for these specific init containers to ensure they are considered during processing.

func FindPortName(pod *v1.Pod, name string) (int32, bool) {
	// ... existing logic for regular containers ...
	// Also search native sidecar init containers (restartPolicy=Always).
	for _, container := range pod.Spec.InitContainers {
		if container.RestartPolicy == nil || *container.RestartPolicy != v1.ContainerRestartPolicyAlways {
			continue
		}
		for _, port := range container.Ports {
			if port.Name == name && port.Protocol == v1.ProtocolTCP {
				return port.ContainerPort, true
			}
		}
	}
	return 0, false
}

// Similar changes applied to getPortMap, FindPort, and StripPodUnusedFields.

Source:

  • pilot/pkg/serviceregistry/kube/controller/ambient/helpers.go (170-181)
  • pilot/pkg/serviceregistry/kube/controller/pod.go (235-249)
  • pilot/pkg/serviceregistry/kube/controller/util.go (62-73)
  • pkg/kube/util.go (405-420)

🚀 Ztunnel Deployment Flexibility with New ZtunnelNamespace Flag

Istio 1.28.5 introduces enhanced flexibility for Ambient Mesh deployments by allowing users to specify a custom namespace for the Ztunnel daemonset. Previously, Ztunnel deployments were tightly coupled to the istio-system namespace. This new ZtunnelNamespace flag empowers operators to deploy Ztunnel in a location of their choosing, aligning with organizational policies, namespace segregation strategies, or custom deployment workflows, without compromising Ambient Mesh functionality.

A new ZtunnelNamespace field has been added to the Istio Config structure, which can be configured via the --istio.test.kube.ztunnelNamespace command-line flag in test environments or similar configuration mechanisms in production deployments. This flag dictates where the Ztunnel daemonset is expected to reside. The Istio test framework and Ambient Mesh components now dynamically reference this configured namespace when interacting with Ztunnel pods for operations like restarts, secure metrics collection, and istioctl zc commands. This change provides greater modularity and control over the placement of core Istio components.

type Config struct {
	// ... existing fields ...
	// The namespace where the ztunnel daemonset resides (default: "istio-system").
	ZtunnelNamespace string
	// ... remaining fields ...
}

// Example of usage in tests:
// t.Clusters().Default().Kube().AppsV1().DaemonSets(i.Settings().ZtunnelNamespace)

Source:

  • pkg/test/framework/components/istio/config.go (93-117)
  • pkg/test/framework/components/istio/flags.go (24-28)
  • tests/integration/ambient/baseline_test.go (3568-3591)
  • tests/integration/ambient/istioctl_test.go (41-43)

Minor Updates & Housekeeping

This release also includes general housekeeping, such as dependency updates for various golang.org/x modules (x/net, x/sync, x/sys, x/crypto, x/mod, x/term, x/text, x/tools), updates to the BASE_VERSION, and improvements to the test framework to ensure more reliable pod rollout checks during restarts.