{
"$type": "site.standard.document",
"bskyPostRef": {
"cid": "bafyreifilfuqkrgytryupkkiq6ks2oeewth25dev5jhyazqn5cfqbsprz4",
"uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpubk7cajju2"
},
"coverImage": {
"$type": "blob",
"ref": {
"$link": "bafkreiatrd6go55ixdscqxyuuelkk64y7c7oj7flqbr5y6i3wwzi4feqsa"
},
"mimeType": "image/webp",
"size": 254144
},
"path": "/samson_tanimawo/why-your-microservices-need-circuit-breakers-and-how-to-add-them-1ee8",
"publishedAt": "2026-07-04T23:02:12.000Z",
"site": "https://dev.to",
"tags": [
"microservices",
"reliability",
"sre",
"devops",
"Nova AI Ops",
"https://novaaiops.com"
],
"textContent": "## The Cascading Failure That Took Down Everything\n\nOur payment service went down for 3 minutes. No big deal, right? Except every service that called payments kept retrying. The retry storms consumed all available connections. Within 10 minutes, all 12 services were down.\n\n3 minutes of one service failing became 45 minutes of total outage.\n\nCircuit breakers prevent this.\n\n## How Circuit Breakers Work\n\n\n State Machine:\n\n CLOSED ──(failures exceed threshold)──→ OPEN\n ↑ │\n │ │\n └──(success)──← HALF-OPEN ←──(timeout)──┘\n\n CLOSED: Normal operation. Requests pass through.\n Track failure rate.\n\n OPEN: Requests fail immediately (fast failure).\n No traffic to the struggling service.\n Wait for timeout period.\n\n HALF-OPEN: Allow one test request through.\n If it succeeds → CLOSED\n If it fails → OPEN\n\n\n## Implementation in Python\n\n\n import time\n from enum import Enum\n from threading import Lock\n\n class CircuitState(Enum):\n CLOSED = \"closed\"\n OPEN = \"open\"\n HALF_OPEN = \"half_open\"\n\n class CircuitBreaker:\n def __init__(self, failure_threshold=5, recovery_timeout=30,\n success_threshold=3):\n self.failure_threshold = failure_threshold\n self.recovery_timeout = recovery_timeout\n self.success_threshold = success_threshold\n self.state = CircuitState.CLOSED\n self.failure_count = 0\n self.success_count = 0\n self.last_failure_time = None\n self.lock = Lock()\n\n def call(self, func, *args, **kwargs):\n with self.lock:\n if self.state == CircuitState.OPEN:\n if time.time() - self.last_failure_time > self.recovery_timeout:\n self.state = CircuitState.HALF_OPEN\n self.success_count = 0\n else:\n raise CircuitBreakerOpenError(\n f\"Circuit breaker is OPEN. Retry after \"\n f\"{self.recovery_timeout}s\"\n )\n\n try:\n result = func(*args, **kwargs)\n self._on_success()\n return result\n except Exception as e:\n self._on_failure()\n raise\n\n def _on_success(self):\n with self.lock:\n if self.state == CircuitState.HALF_OPEN:\n self.success_count += 1\n if self.success_count >= self.success_threshold:\n self.state = CircuitState.CLOSED\n self.failure_count = 0\n self.failure_count = 0\n\n def _on_failure(self):\n with self.lock:\n self.failure_count += 1\n self.last_failure_time = time.time()\n if self.failure_count >= self.failure_threshold:\n self.state = CircuitState.OPEN\n if self.state == CircuitState.HALF_OPEN:\n self.state = CircuitState.OPEN\n\n # Usage\n payment_breaker = CircuitBreaker(failure_threshold=5, recovery_timeout=30)\n\n def process_payment(order):\n try:\n return payment_breaker.call(payment_service.charge, order)\n except CircuitBreakerOpenError:\n return queue_for_retry(order) # Graceful degradation\n\n\n## What to Do When the Circuit Opens\n\nThe circuit breaker buys you time. Use it wisely:\n\n\n\n def handle_open_circuit(service_name, request):\n strategies = {\n 'payment': lambda r: queue_for_retry(r), # Retry later\n 'recommendations': lambda r: return_cached(r), # Serve stale data\n 'analytics': lambda r: drop_silently(r), # Non-critical, skip\n 'auth': lambda r: allow_with_cached_token(r), # Cached auth\n 'search': lambda r: return_popular_results(r), # Fallback results\n }\n return strategies.get(service_name, lambda r: return_error(r))(request)\n\n\n## Monitoring Circuit Breakers\n\n\n circuit_breaker_metrics:\n - name: circuit_breaker_state\n type: gauge\n labels: [service, target]\n # 0=closed, 1=open, 2=half_open\n\n - name: circuit_breaker_failures_total\n type: counter\n labels: [service, target]\n\n - name: circuit_breaker_rejected_total\n type: counter\n labels: [service, target]\n # Requests rejected while circuit is open\n\n alerts:\n - alert: CircuitBreakerOpen\n expr: circuit_breaker_state == 1\n for: 1m\n severity: warning\n message: \"Circuit breaker for {{ $labels.target }} is OPEN\"\n\n\n## The Configuration That Matters\n\n\n circuit_breakers:\n payment-service:\n failure_threshold: 5\n recovery_timeout: 30s\n success_threshold: 3\n timeout_per_request: 5s\n\n search-service:\n failure_threshold: 10 # More tolerant\n recovery_timeout: 15s # Recover faster\n success_threshold: 2\n timeout_per_request: 2s\n\n auth-service:\n failure_threshold: 3 # Less tolerant (critical)\n recovery_timeout: 10s # Recover very fast\n success_threshold: 1\n timeout_per_request: 1s\n\n\nCritical services get lower thresholds (less tolerance) and faster recovery.\n\nIf you want AI-powered circuit breaker tuning and cascading failure prevention, check out what we're building at Nova AI Ops.\n\n**Written by Dr. Samson Tanimawo**\nBSc · MSc · MBA · PhD\nFounder & CEO, Nova AI Ops. https://novaaiops.com",
"title": "Why Your Microservices Need Circuit Breakers (And How to Add Them)"
}