Implementing the Circuit Breaker Pattern in Kubernetes

Introduction

In a microservices architecture, services communicate with each other over the network, which introduces latency, failures, and timeouts. If one service fails, it can cause cascading failures, leading to a complete system outage. The Circuit Breaker Pattern helps prevent these failures from propagating, ensuring system resilience.

In this blog, we’ll set up circuit breaking in Kubernetes using Istio, implement failure handling, and demonstrate how to recover from failures gracefully.

Why Circuit Breakers?

The Problem

• Unstable services: If a dependent service is slow or failing, all requests pile up, increasing resource consumption.
• Cascading failures: A single failing service can bring down the entire system.
• Poor user experience: Without intelligent request handling, users experience timeouts and failures.

The Solution: Circuit Breaker Pattern

• Detects when a service is slow or failing and temporarily stops sending requests.
• Prevents resource exhaustion by limiting concurrent requests.
• Automatically recovers once the service is stable.

Step 1: Setting Up Circuit Breaking in Kubernetes

Prerequisites

• A running Kubernetes cluster (Minikube, kind, or any managed K8s).
• Istio Service Mesh installed.

Step 2: Deploying Sample Microservices

We will deploy two microservices:

• Product Service (simulates a reliable service).
• Order Service (calls the Product Service, sometimes failing).

Deploy the Product Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: product-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: product-service
  template:
    metadata:
      labels:
        app: product-service
    spec:
      containers:
      - name: product-service
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: product-service
spec:
  selector:
    app: product-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Deploy the Order Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: httpd
        ports:
        - containerPort: 80
        env:
        - name: PRODUCT_SERVICE_URL
          value: "http://product-service"
---
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Step 3: Enabling Circuit Breaking with Istio

Now, let’s limit requests to the Product Service using Istio’s DestinationRule.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: product-service-circuit-breaker
spec:
  host: product-service
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 2
      interval: 5s
      baseEjectionTime: 30s

What’s Happening Here?

• Limits max requests per connection to prevent overload.
• If Istio detects 2 consecutive 5xx errors, it ejects the service for 30 seconds.
• Prevents an unhealthy service from taking down dependent services.

Step 4: Testing the Circuit Breaker

To test the circuit breaker, simulate failures in the Product Service by sending multiple requests:

kubectl exec -it $(kubectl get pod -l app=order-service -o jsonpath='{.items[0].metadata.name}') -- curl -X GET http://product-service

Now, if the Product Service fails multiple times, Istio stops sending requests temporarily, preventing further failures.

Conclusion

Circuit Breakers are essential for building fault-tolerant microservices.
Prevents cascading failures by intelligently rejecting requests.
Enhances system resilience by allowing only healthy services to process traffic.
Automatically recovers when services are back online.

Using Istio’s built-in circuit breaking, Kubernetes workloads can self-heal and prevent system-wide outages!

Would you use Circuit Breakers in your production environment? Let’s discuss!