Service Mesh Master Class

Engineering-grade reference manual for Istio, Linkerd, Consul, traffic management, mTLS, observability, and production service-mesh operations.

Overview

A service mesh adds a dedicated infrastructure layer for service-to-service communication. Instead of embedding retries, TLS, telemetry, and traffic splitting in every application, the mesh provides those features through sidecars, proxies, gateways, policies, or node-level agents.

Use a service mesh when you need consistent cross-service security, progressive delivery, traffic control, and deep observability across many services. Do not add one just because Kubernetes is present; a mesh adds operational complexity.

When to Use a Service Mesh

Good fit:

many microservices with service-to-service traffic
strict mTLS requirements
canary or blue-green releases across services
standardized retries, timeouts, circuit breaking, and telemetry
multi-cluster or multi-tenant networking controls

Poor fit:

a small monolith or a few services
teams that cannot operate Kubernetes basics yet
clusters without good observability foundations
latency-sensitive workloads where proxy overhead is unacceptable
organizations without ownership for mesh upgrades and policy drift

Mesh Comparison

Mesh	Strengths	Tradeoffs
Istio	powerful traffic management, gateways, policy ecosystem	larger operational surface
Linkerd	simple, lightweight, strong defaults	fewer advanced traffic primitives
Consul	service discovery, multi-platform, VM + Kubernetes support	separate HashiCorp operational model

Istio Installation

Download and install Istio CLI:

curl -L https://istio.io/downloadIstio | sh -

Add istioctl to your path:

export PATH="$PWD/istio-*/bin:$PATH"

Install the default profile:

istioctl install --set profile=default -y

Verify control plane pods:

kubectl get pods -n istio-system

Check mesh status:

istioctl proxy-status

Istio Sidecar Injection

Enable namespace injection:

kubectl label namespace production istio-injection=enabled

Restart deployments to inject sidecars:

kubectl rollout restart deployment -n production

Confirm sidecars are present:

kubectl get pods -n production -o jsonpath='{range .items[*]}{.metadata.name}{" containers="}{range .spec.containers[*]}{.name}{","}{end}{"\n"}{end}'

Disable injection for a namespace:

kubectl label namespace production istio-injection-

Istio Gateway and VirtualService

Expose HTTP traffic through an Istio gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: web-gateway
  namespace: production
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - web.example.com

Route traffic to a service:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: web
  namespace: production
spec:
  hosts:
    - web.example.com
  gateways:
    - web-gateway
  http:
    - route:
        - destination:
            host: web.production.svc.cluster.local
            port:
              number: 8080

Apply manifests:

kubectl apply -f gateway.yaml
kubectl apply -f virtualservice.yaml

Inspect routes:

istioctl proxy-config routes deploy/web -n production

Canary Traffic Splitting

Split traffic between stable and canary versions:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: web
  namespace: production
spec:
  host: web.production.svc.cluster.local
  subsets:
    - name: stable
      labels:
        version: stable
    - name: canary
      labels:
        version: canary
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: web
  namespace: production
spec:
  hosts:
    - web.production.svc.cluster.local
  http:
    - route:
        - destination:
            host: web.production.svc.cluster.local
            subset: stable
          weight: 90
        - destination:
            host: web.production.svc.cluster.local
            subset: canary
          weight: 10

Increase canary weight after validation:

kubectl patch virtualservice web -n production --type merge -p '{"spec":{"http":[{"route":[{"destination":{"host":"web.production.svc.cluster.local","subset":"stable"},"weight":50},{"destination":{"host":"web.production.svc.cluster.local","subset":"canary"},"weight":50}]}]}}'

Timeouts and Retries

Add timeout and retry policy:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api
  namespace: production
spec:
  hosts:
    - api.production.svc.cluster.local
  http:
    - timeout: 3s
      retries:
        attempts: 2
        perTryTimeout: 1s
        retryOn: gateway-error,connect-failure,refused-stream
      route:
        - destination:
            host: api.production.svc.cluster.local

Verify Envoy cluster settings:

istioctl proxy-config clusters deploy/api -n production

Circuit Breaking

Set connection pool and outlier detection:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: api-circuit-breaker
  namespace: production
spec:
  host: api.production.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 2m
      maxEjectionPercent: 50

Istio mTLS

Check mesh mTLS status:

istioctl authn tls-check

Enable strict mTLS for a namespace:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT

Allow only a specific service account to call a workload:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend
  namespace: production
spec:
  selector:
    matchLabels:
      app: api
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/production/sa/frontend

Istio Observability

Open the Kiali dashboard:

istioctl dashboard kiali

Open Grafana if the add-on is installed:

istioctl dashboard grafana

Open Jaeger tracing:

istioctl dashboard jaeger

Inspect listener configuration:

istioctl proxy-config listeners deploy/web -n production

Inspect endpoints:

istioctl proxy-config endpoints deploy/web -n production

Linkerd Installation

Check cluster readiness:

linkerd check --pre

Install Linkerd CRDs and control plane:

linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

Validate install:

linkerd check

Install dashboard extension:

linkerd viz install | kubectl apply -f -

Open dashboard:

linkerd viz dashboard

Linkerd Injection

Inject Linkerd into manifests:

kubectl get deploy -n production -o yaml | linkerd inject - | kubectl apply -f -

Annotate a namespace for automatic injection:

kubectl annotate namespace production linkerd.io/inject=enabled

Restart workloads:

kubectl rollout restart deployment -n production

Check meshed workloads:

linkerd viz stat deploy -n production

Linkerd Traffic Split

Install SMI extension if needed:

linkerd smi install | kubectl apply -f -

Example traffic split:

apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
  name: web-split
  namespace: production
spec:
  service: web
  backends:
    - service: web-stable
      weight: 900m
    - service: web-canary
      weight: 100m

Watch live request stats:

linkerd viz stat trafficsplit -n production

Consul Service Mesh

Install Consul on Kubernetes with Helm:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install consul hashicorp/consul --namespace consul --create-namespace --set global.name=consul

Check Consul pods:

kubectl get pods -n consul

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: api
spec:
  protocol: http

Create an intentions policy:

consul intention create -allow web api

List intentions:

consul intention list

Observability Patterns

Key mesh metrics to monitor:

request rate by service
success/error rate by route
p50/p95/p99 latency
retry volume
mTLS handshake failures
proxy CPU and memory
control-plane health

Prometheus query examples:

sum(rate(istio_requests_total[5m])) by (destination_service)

histogram_quantile(0.95, sum(rate(istio_request_duration_milliseconds_bucket[5m])) by (le, destination_service))

sum(rate(istio_requests_total{response_code=~"5.."}[5m])) by (destination_service)

Debugging Service Mesh Issues

Check whether sidecars are injected:

kubectl get pod web-abc123 -n production -o jsonpath='{.spec.containers[*].name}'

Run Istio analyzer:

istioctl analyze -A

Inspect Envoy bootstrap:

istioctl proxy-config bootstrap deploy/web -n production

Inspect Envoy logs:

kubectl logs deploy/web -n production -c istio-proxy --tail=100

Check Linkerd proxy logs:

kubectl logs deploy/web -n production -c linkerd-proxy --tail=100

Port-forward a service for direct testing:

kubectl port-forward svc/web 8080:80 -n production

Run a temporary curl pod:

kubectl run curl -n production --rm -it --image=curlimages/curl -- sh

Production Rollout Checklist

Define the service-mesh use case and owner.
Install mesh in a non-production cluster.
Add observability before enforcing policy.
Mesh one low-risk namespace first.
Measure latency and resource overhead.
Enable mTLS in permissive mode, then strict mode.
Add traffic policies gradually.
Document rollback commands before rollout.
Add dashboards and alerts for control-plane health.
Schedule regular mesh upgrades.

Rollback Commands

Remove Istio injection from a namespace:

kubectl label namespace production istio-injection-
kubectl rollout restart deployment -n production

Uninstall Istio control plane:

istioctl uninstall --purge -y
kubectl delete namespace istio-system

Remove Linkerd from manifests:

kubectl get deploy -n production -o yaml | linkerd uninject - | kubectl apply -f -

Uninstall Linkerd:

linkerd uninstall | kubectl delete -f -

Uninstall Consul Helm release:

helm uninstall consul -n consul
kubectl delete namespace consul

Common Pitfalls

Installing a mesh before teams understand Kubernetes networking.
Enforcing strict mTLS before every workload has a sidecar.
Adding retries without timeouts, causing retry storms.
Forgetting that sidecars consume CPU and memory.
Treating dashboards as proof that policy is correct.
Allowing traffic rules to drift without code review.
Running multiple meshes in the same namespace without a migration plan.

Quick Reference

# Istio install and status
istioctl install --set profile=default -y
istioctl proxy-status
istioctl analyze -A

# Istio dashboards
istioctl dashboard kiali
istioctl dashboard grafana
istioctl dashboard jaeger

# Linkerd install and status
linkerd check --pre
linkerd install | kubectl apply -f -
linkerd check
linkerd viz stat deploy -n production

# Consul install and intentions
helm install consul hashicorp/consul --namespace consul --create-namespace
consul intention list

# Kubernetes mesh verification
kubectl get pods -n istio-system
kubectl get pods -n linkerd
kubectl get networkpolicy -A

Track this guide

Service Mesh Master Class #

Overview #

When to Use a Service Mesh #

Mesh Comparison #

Istio Installation #

Istio Sidecar Injection #

Istio Gateway and VirtualService #

Canary Traffic Splitting #

Timeouts and Retries #

Circuit Breaking #

Istio mTLS #

Istio Observability #

Linkerd Installation #

Linkerd Injection #

Linkerd Traffic Split #

Consul Service Mesh #

Observability Patterns #

Debugging Service Mesh Issues #

Production Rollout Checklist #

Rollback Commands #

Common Pitfalls #

Quick Reference #