跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

agent-health-monitoring

Monitor AI agent health, detect anomalies, set up alerting, and maintain observability dashboards for production multi-agent systems. Covers liveness checks, performance metrics, drift detection, and incident response.

cosmicstack-labs · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add cosmicstack-labs/mercury-agent-skills --skill agent-health-monitoring

技能说明


name: agent-health-monitoring description: 'Monitor AI agent health, detect anomalies, set up alerting, and maintain observability dashboards for production multi-agent systems. Covers liveness checks, performance metrics, drift detection, and incident response.' metadata: author: cosmicstack-labs version: 1.0.0 category: ai-ml tags: - agent-monitoring - observability - alerting - health-checks - incident-response - production-agents

Agent Health Monitoring & Alerting

Overview

Production multi-agent systems fail silently. An agent that stops responding, returns empty results, or enters an infinite loop can degrade an entire workflow without triggering traditional infrastructure alerts. This skill covers how to build comprehensive health monitoring, metrics collection, and alerting for AI agent fleets.


Core Concepts

Agent Vital Signs

MetricWhat It MeasuresWhy It Matters
Response Rate% of agent invocations that return a resultDropping rate indicates crashes or context overflows
Latency (P50/P95/P99)Time from invocation to responseSpikes indicate context bloat or degraded model performance
Error Rate% of invocations with errors/tool failuresRising rate indicates systemic issues
Step CountNumber of reasoning steps per taskUnbounded growth indicates looping behavior
Tool Call Success Rate% of tool calls that succeedDrop indicates broken integrations or rate limiting
Token ConsumptionTokens used per agent runBudget anomalies indicate runaway agents
Context Utilization% of context window usedHigh utilization risks truncation and quality loss
Hallucination ScoreConfidence calibration or factuality checksDegrading accuracy undermines trust

Alert Severity Levels

LevelColorResponse TimeExamples
P0 (Critical)🔴 Red< 5 minAgent completely down, data loss, security breach
P1 (High)🟠 Orange< 15 minError rate > 20%, latency 5x baseline
P2 (Medium)🟡 Yellow< 1 hourError rate > 5%, slow degradation
P3 (Low)🔵 Blue< 24 hoursSingle agent underperforming, minor drift

Step-by-Step Implementation

Step 1: Instrument Every Agent

Wrap every agent invocation with telemetry:

class MonitoredAgent:
    """Agent wrapper that collects metrics on every invocation."""
    
    def __init__(self, agent, agent_name: str, metrics_client):
        self.agent = agent
        self.agent_name = agent_name
        self.metrics = metrics_client
    
    async def run(self, task: str) -> str:
        start_time = time.time()
        step_count = 0
        token_usage = 0
        
        try:
            result = await self.agent.run(task)
            
            # Collect metrics
            duration = time.time() - start_time
            self.metrics.timing(f"agent.{self.agent_name}.latency", duration)
            self.metrics.increment(f"agent.{self.agent_name}.invocations")
            self.metrics.increment(f"agent.{self.agent_name}.success")
            self.metrics.gauge(f"agent.{self.agent_name}.steps", step_count)
            
            return result
            
        except Exception as e:
            duration = time.time() - start_time
            self.metrics.increment(f"agent.{self.agent_name}.errors")
            self.metrics.timing(f"agent.{self.agent_name}.error_latency", duration)
            raise

Step 2: Implement Liveness & Readiness Probes

class AgentHealthProbe:
    """Kubernetes-style health probes for AI agents."""
    
    async def liveness_check(self, agent) -> bool:
        """Is the agent process alive and responding?"""
        try:
            result = await asyncio.wait_for(
                agent.run("Respond with: OK"),
                timeout=5.0
            )
            return "OK" in result
        except (asyncio.TimeoutError, Exception):
            return False
    
    async def readiness_check(self, agent) -> dict:
        """Is the agent ready to accept tasks?"""
        checks = {
            "model_available": await self._check_model(agent),
            "tools_available": await self._check_tools(agent),
            "memory_available": await self._check_memory(agent),
            "context_capacity": await self._check_context(agent),
        }
        return {
            "ready": all(checks.values()),
            "checks": checks
        }
    
    async def deep_check(self, agent) -> dict:
        """Full diagnostic: run a test task and validate output."""
        test_task = agent.config.test_prompt
        result = await agent.run(test_task)
        return {
            "passed": self._validate_output(result),
            "output_preview": result[:200],
            "latency_ms": self._last_latency
        }

Step 3: Set Up Anomaly Detection

class AnomalyDetector:
    """Detect unusual agent behavior using statistical methods."""

    def __init__(self, window_size: int = 100):
        self.window_size = window_size
        self.metrics_history = defaultdict(list)

    def record(self, agent_name: str, metric: str, value: float):
        self.metrics_history[f"{agent_name}:{metric}"].append(value)
        
        # Keep rolling window
        history = self.metrics_history[f"{agent_name}:{metric}"]
        if len(history) > self.window_size:
            history.pop(0)

    def is_anomalous(self, agent_name: str, metric: str, value: float, 
                     z_threshold: float = 3.0) -> tuple[bool, float]:
        """Check if a value is anomalous using z-score."""
        history = self.metrics_history.get(f"{agent_name}:{metric}", [])
        if len(history) < 10:
            return False, 0.0  # Not enough data
        
        mean = statistics.mean(his
阅读完整来源 ↗