Advanced Topics

Service Mesh with Istio

● Advanced ⏱ 20 min read

Istio is a service mesh that adds traffic management, mTLS, and observability to Kubernetes workloads without requiring application code changes. It does this by injecting an Envoy sidecar proxy into every pod and intercepting all traffic at the network layer. This guide covers the Istio resource model and the traffic management features that justify the operational overhead.

Istio Architecture

Istio — control plane vs data plane

CONTROL PLANE — Istiod

• Pilot — service discovery, xDS config push

• Citadel — certificate authority (SPIFFE)

• Galley — config validation

Pushes routing rules to every sidecar

DATA PLANE — Envoy sidecars

• One per pod (injected automatically)

• Intercepts all inbound + outbound TCP

• Enforces routing, retries, timeouts

App sees plain HTTP — mTLS in proxy

xDS API: Istiod pushes Listener/Route/Cluster/Endpoint config to Envoy proxies in real time. No restart required when routing rules change.

Istiod (control plane) computes routing rules and pushes them to Envoy sidecars via xDS. The app process is unaware of the proxy — it makes plain TCP connections.

Installation

install Istio with istioctl

# Download istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH="$PATH:$PWD/istio-1.22.0/bin"

# Install with the default profile (good for production)
istioctl install --set profile=default -y

# Verify components
istioctl verify-install

# Enable sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled

# Check sidecar is injected (2/2 containers)
kubectl get pods -n production

VirtualService

A VirtualService defines routing rules for traffic destined to a Kubernetes Service. It is evaluated by the Envoy proxy on the client side — the proxy that's making the outbound call applies the rules before sending the request.

VirtualService — route by header and path

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp
  namespace: production
spec:
  hosts:
  - myapp                           # matches the K8s Service name
  http:
  - name: "canary-users"
    match:
    - headers:
        x-user-group:
          exact: "beta"             # route beta users to v2
    route:
    - destination:
        host: myapp
        subset: v2
  - name: "api-v1"
    match:
    - uri:
        prefix: /api/v1
    timeout: 3s                     # per-route timeout
    retries:
      attempts: 3
      perTryTimeout: 1s
      retryOn: gateway-error,connect-failure,retriable-4xx
    route:
    - destination:
        host: myapp
        subset: v1
  - name: "default"
    route:
    - destination:
        host: myapp
        subset: v1

DestinationRule

A DestinationRule defines policies applied to traffic after routing — load balancing algorithm, connection pool settings, subsets (versions), and outlier detection (circuit breaking).

DestinationRule — define subsets and load balancing

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: myapp
  namespace: production
spec:
  host: myapp
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN            # ROUND_ROBIN, RANDOM, LEAST_CONN, PASSTHROUGH
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http2MaxRequests: 1000
        pendingRequests: 50
  subsets:
  - name: v1
    labels:
      version: v1                   # matches pod labels
  - name: v2
    labels:
      version: v2
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN         # override for this subset

Canary Deployments

weighted traffic split — 90% v1, 10% v2

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp-canary
  namespace: production
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 90
    - destination:
        host: myapp
        subset: v2
      weight: 10                    # shift: 80/20 → 50/50 → 0/100 → delete VS

Adjust the weights progressively. When v2 proves stable (via Prometheus error rate metrics), shift to 100% and delete the VirtualService. No pod counts to manage — traffic split is independent of replica count.

Circuit Breaking

Outlier detection automatically ejects unhealthy endpoints from the load balancing pool. When a host returns consecutive 5xx errors, Envoy stops sending it traffic for an ejection interval.

DestinationRule — outlier detection

spec:
  host: myapp
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5       # eject after 5 consecutive 5xx
      interval: 10s                 # check interval
      baseEjectionTime: 30s         # first ejection duration
      maxEjectionPercent: 50        # never eject more than 50% of hosts
      minHealthPercent: 30          # stop ejecting if < 30% of hosts are healthy

Fault Injection

Fault injection lets you test resilience by injecting delays or errors into traffic — without touching the application code. Use this to verify timeout and retry configurations work correctly.

VirtualService — inject 500ms delay for 10% of requests

spec:
  hosts:
  - downstream-service
  http:
  - fault:
      delay:
        percentage:
          value: 10.0               # 10% of requests get the delay
        fixedDelay: 500ms
      abort:
        percentage:
          value: 5.0                # 5% of requests get a 503
        httpStatus: 503
    route:
    - destination:
        host: downstream-service

Built-in Observability

Istio automatically generates the three pillars of observability for every service in the mesh — no SDK changes required:

Signal	Generated by	Backend
Metrics	Envoy exposes `istio_requests_total`, latency histograms, connection counts per service pair	Prometheus → Grafana
Traces	Envoy propagates B3/W3C trace headers, generates spans per hop	Jaeger / Tempo
Access logs	Envoy logs every request with status, latency, upstream cluster	Fluent Bit → Loki

useful istioctl commands

# Check proxy config for a pod (routes, clusters, listeners)
istioctl proxy-config routes deploy/myapp -n production
istioctl proxy-config clusters deploy/myapp -n production

# Check that mTLS is working between two services
istioctl authn tls-check myapp-pod.production myapp.production.svc.cluster.local

# Analyse configuration for issues
istioctl analyze -n production

# Dashboard — open Kiali (service topology UI)
istioctl dashboard kiali