浏览器与自动化
agent-health-monitoring
Monitor AI agent health, detect anomalies, set up alerting, and maintain observability dashboards for production multi-agent systems. Covers liveness checks, performance metrics, drift detection, and incident response.
cosmicstack-labs · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add cosmicstack-labs/mercury-agent-skills --skill agent-health-monitoring技能说明
name: agent-health-monitoring description: 'Monitor AI agent health, detect anomalies, set up alerting, and maintain observability dashboards for production multi-agent systems. Covers liveness checks, performance metrics, drift detection, and incident response.' metadata: author: cosmicstack-labs version: 1.0.0 category: ai-ml tags: - agent-monitoring - observability - alerting - health-checks - incident-response - production-agents
Agent Health Monitoring & Alerting
Overview
Production multi-agent systems fail silently. An agent that stops responding, returns empty results, or enters an infinite loop can degrade an entire workflow without triggering traditional infrastructure alerts. This skill covers how to build comprehensive health monitoring, metrics collection, and alerting for AI agent fleets.
Core Concepts
Agent Vital Signs
| Metric | What It Measures | Why It Matters |
|---|---|---|
| Response Rate | % of agent invocations that return a result | Dropping rate indicates crashes or context overflows |
| Latency (P50/P95/P99) | Time from invocation to response | Spikes indicate context bloat or degraded model performance |
| Error Rate | % of invocations with errors/tool failures | Rising rate indicates systemic issues |
| Step Count | Number of reasoning steps per task | Unbounded growth indicates looping behavior |
| Tool Call Success Rate | % of tool calls that succeed | Drop indicates broken integrations or rate limiting |
| Token Consumption | Tokens used per agent run | Budget anomalies indicate runaway agents |
| Context Utilization | % of context window used | High utilization risks truncation and quality loss |
| Hallucination Score | Confidence calibration or factuality checks | Degrading accuracy undermines trust |
Alert Severity Levels
| Level | Color | Response Time | Examples |
|---|---|---|---|
| P0 (Critical) | 🔴 Red | < 5 min | Agent completely down, data loss, security breach |
| P1 (High) | 🟠 Orange | < 15 min | Error rate > 20%, latency 5x baseline |
| P2 (Medium) | 🟡 Yellow | < 1 hour | Error rate > 5%, slow degradation |
| P3 (Low) | 🔵 Blue | < 24 hours | Single agent underperforming, minor drift |
Step-by-Step Implementation
Step 1: Instrument Every Agent
Wrap every agent invocation with telemetry:
class MonitoredAgent:
"""Agent wrapper that collects metrics on every invocation."""
def __init__(self, agent, agent_name: str, metrics_client):
self.agent = agent
self.agent_name = agent_name
self.metrics = metrics_client
async def run(self, task: str) -> str:
start_time = time.time()
step_count = 0
token_usage = 0
try:
result = await self.agent.run(task)
# Collect metrics
duration = time.time() - start_time
self.metrics.timing(f"agent.{self.agent_name}.latency", duration)
self.metrics.increment(f"agent.{self.agent_name}.invocations")
self.metrics.increment(f"agent.{self.agent_name}.success")
self.metrics.gauge(f"agent.{self.agent_name}.steps", step_count)
return result
except Exception as e:
duration = time.time() - start_time
self.metrics.increment(f"agent.{self.agent_name}.errors")
self.metrics.timing(f"agent.{self.agent_name}.error_latency", duration)
raise
Step 2: Implement Liveness & Readiness Probes
class AgentHealthProbe:
"""Kubernetes-style health probes for AI agents."""
async def liveness_check(self, agent) -> bool:
"""Is the agent process alive and responding?"""
try:
result = await asyncio.wait_for(
agent.run("Respond with: OK"),
timeout=5.0
)
return "OK" in result
except (asyncio.TimeoutError, Exception):
return False
async def readiness_check(self, agent) -> dict:
"""Is the agent ready to accept tasks?"""
checks = {
"model_available": await self._check_model(agent),
"tools_available": await self._check_tools(agent),
"memory_available": await self._check_memory(agent),
"context_capacity": await self._check_context(agent),
}
return {
"ready": all(checks.values()),
"checks": checks
}
async def deep_check(self, agent) -> dict:
"""Full diagnostic: run a test task and validate output."""
test_task = agent.config.test_prompt
result = await agent.run(test_task)
return {
"passed": self._validate_output(result),
"output_preview": result[:200],
"latency_ms": self._last_latency
}
Step 3: Set Up Anomaly Detection
class AnomalyDetector:
"""Detect unusual agent behavior using statistical methods."""
def __init__(self, window_size: int = 100):
self.window_size = window_size
self.metrics_history = defaultdict(list)
def record(self, agent_name: str, metric: str, value: float):
self.metrics_history[f"{agent_name}:{metric}"].append(value)
# Keep rolling window
history = self.metrics_history[f"{agent_name}:{metric}"]
if len(history) > self.window_size:
history.pop(0)
def is_anomalous(self, agent_name: str, metric: str, value: float,
z_threshold: float = 3.0) -> tuple[bool, float]:
"""Check if a value is anomalous using z-score."""
history = self.metrics_history.get(f"{agent_name}:{metric}", [])
if len(history) < 10:
return False, 0.0 # Not enough data
mean = statistics.mean(his
阅读完整来源 ↗