设计与 UI
error-recovery-retry
Design robust error recovery, retry logic, and fallback strategies for production AI agents. Covers transient failure handling, circuit breakers, exponential backoff, state recovery, graceful degradation, and dead-letter queues for agent systems.
cosmicstack-labs · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add cosmicstack-labs/mercury-agent-skills --skill error-recovery-retry技能说明
name: error-recovery-retry description: 'Design robust error recovery, retry logic, and fallback strategies for production AI agents. Covers transient failure handling, circuit breakers, exponential backoff, state recovery, graceful degradation, and dead-letter queues for agent systems.' metadata: author: cosmicstack-labs version: 1.0.0 category: ai-ml tags: - error-recovery - retry-logic - circuit-breaker - fallback-strategies - fault-tolerance - graceful-degradation
Error Recovery & Retry Logic for Agents
Overview
Agents fail. APIs time out. Models return garbage. Tools throw exceptions. The difference between a production-grade system and a prototype is how gracefully it fails. This skill covers comprehensive error recovery patterns — from simple retries to circuit breakers, stateful recovery, and human escalation paths.
Core Concepts
Failure Taxonomy
| Failure Type | Example | Frequency | Recoverable? |
|---|---|---|---|
| Transient | API timeout, network glitch | Common | ✅ Yes — retry |
| Rate Limited | 429 Too Many Requests | Common | ✅ Yes — backoff |
| Validation | Invalid tool parameters | Occasional | ✅ Yes — fix and retry |
| Model Error | LLM returns nonsense | Occasional | ⚠️ Maybe — retry with different prompt |
| Context Overflow | Token limit exceeded | Rare | ✅ Yes — compress and retry |
| Permission | Agent lacks access | Rare | ❌ No — escalate |
| Security | Injection attempt detected | Rare | ❌ No — alert and block |
| Permanent | Tool deleted, endpoint gone | Rare | ❌ No — escalate to human |
Recovery Strategy Decision Tree
┌──────────────┐
│ Agent Error │
└──────┬───────┘
│
┌───────────┴───────────┐
│ │
Transient? Permanent?
│ │
┌────┴────┐ ┌──────┴──────┐
│ │ │ │
Retry Circuit Fallback Escalate
+backoff Breaker Agent to Human
Step-by-Step Implementation
Step 1: Retry with Exponential Backoff
import asyncio
import random
from functools import wraps
from typing import Callable, Any
async def retry_with_backoff(
fn: Callable,
max_retries: int = 3,
base_delay: float = 1.0,
max_delay: float = 60.0,
backoff_factor: float = 2.0,
jitter: bool = True,
retryable_exceptions: tuple = (TimeoutError, ConnectionError,
RateLimitError)
) -> Any:
"""Execute a function with exponential backoff retry logic."""
last_exception = None
for attempt in range(max_retries + 1):
try:
return await fn()
except retryable_exceptions as e:
last_exception = e
if attempt == max_retries:
raise # Exhausted retries
# Calculate delay with exponential backoff
delay = min(base_delay * (backoff_factor ** attempt), max_delay)
# Add jitter to prevent thundering herd
if jitter:
delay = delay * (0.5 + random.random() * 0.5)
logger.warning(
f"Attempt {attempt + 1}/{max_retries + 1} failed: {e}. "
f"Retrying in {delay:.1f}s..."
)
await asyncio.sleep(delay)
raise last_exception # Shouldn't reach here, but safety
class RetryPolicy:
"""Configurable retry policy for agent operations."""
def __init__(self, name: str, max_retries: int = 3,
base_delay: float = 1.0, max_delay: float = 60.0,
backoff_factor: float = 2.0):
self.name = name
self.max_retries = max_retries
self.base_delay = base_delay
self.max_delay = max_delay
self.backoff_factor = backoff_factor
self.consecutive_failures = 0
async def execute(self, fn: Callable) -> Any:
"""Execute with this policy's retry configuration."""
try:
result = await retry_with_backoff(
fn,
max_retries=self.max_retries,
base_delay=self.base_delay,
max_delay=self.max_delay,
backoff_factor=self.backoff_factor
)
self.consecutive_failures = 0
return result
except Exception as e:
self.consecutive_failures += 1
raise
def is_circuit_breaking(self, threshold: int = 5) -> bool:
"""Check if consecutive failures exceed threshold."""
return self.consecutive_failures >= threshold
# Predefined policies
RETRY_POLICIES = {
"tool_call": RetryPolicy("tool_call", max_retries=3, base_delay=0.5),
"api_request": RetryPolicy("api_request", max_retries=5, base_delay=1.0),
"llm_generation": RetryPolicy("llm_generation", max_retries=2, base_delay=2.0),
"database_query": RetryPolicy("database_query", max_retries=3, base_delay=0.1),
}
Step 2: Circuit Breaker Pattern
class CircuitBreaker:
"""Prevent repeated calls to failing services."""
STATES = ["CLOSED", "OPEN", "HALF_OPEN"]
def __init__(self, name: str, failure_threshold: int = 5,
recovery_timeout: float = 30.0,
half_open_max_calls: int = 3):
self.name = name
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.half_open_max_calls = half_open_max_calls
self.state = "CLOSED"
self.failure_count = 0
self.last_failure_time = None
self.half_open_calls = 0
async def call(self, fn: Callable, fallback: Callable = None) -> Any:
"""Exe
阅读完整来源 ↗