跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

langgraph-testing-evaluation

Use this skill when you need to test or evaluate LangGraph/LangChain agents: writing unit or integration tests, generating test scaffolds, mocking LLM/tool behavior, running trajectory evaluation (match or LLM-as-judge), running LangSmith dataset evaluations, and comparing two agent versions with A/B-style offline analysis. Use it for Python and JavaScript/TypeScript workflows, evaluator design, experiment setup, regression gates, and debugging flaky/incorrect evaluation results.

soba-labs · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation

技能说明


name: langgraph-testing-evaluation description: "Use this skill when you need to test or evaluate LangGraph/LangChain agents: writing unit or integration tests, generating test scaffolds, mocking LLM/tool behavior, running trajectory evaluation (match or LLM-as-judge), running LangSmith dataset evaluations, and comparing two agent versions with A/B-style offline analysis. Use it for Python and JavaScript/TypeScript workflows, evaluator design, experiment setup, regression gates, and debugging flaky/incorrect evaluation results."

LangGraph Testing & Evaluation

Practical workflows for validating agent quality with:

  • Unit/integration tests
  • Trajectory evaluation
  • LangSmith dataset evaluations
  • A/B-style comparisons between versions

Use this file for high-level flow. Load references/* for detailed implementation.

Start Here

Choose the smallest approach that answers your question:

GoalPrimary methodLoad first
Validate node logic quicklyUnit tests with mocksreferences/unit-testing-patterns.md
Validate multi-step agent behaviorTrajectory evaluationreferences/trajectory-evaluation.md
Track quality over datasets over timeLangSmith evaluationreferences/langsmith-evaluation.md
Compare old vs new agent versionsA/B comparisonreferences/ab-testing.md

Recommended order:

  1. Unit tests
  2. Integration/trajectory checks
  3. Dataset evaluation in LangSmith
  4. A/B comparison before deployment

Quick Commands

Run from repo root.

Generate test scaffolding

# Python (preferred)
uv run skills/langgraph-testing-evaluation/scripts/generate_test_cases.py my_agent:graph --output tests/ --framework pytest

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/generate_test_cases.js ./my-agent.ts:graph --output tests/ --framework vitest

Run trajectory evaluation

# Python: LLM-as-judge
uv run skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.py my_agent:run_agent my_dataset --method llm-judge --model openai:o3-mini

# Python: trajectory match
uv run skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.py my_agent:run_agent dataset.json --method match --trajectory-match-mode strict --reference-trajectory reference.json

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.js ./agent.ts:runAgent my_dataset --method llm-judge --model openai:o3-mini --max-concurrency 4

Run LangSmith dataset evaluation

# Python
uv run skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.py my_agent:run_agent my_dataset --evaluators accuracy,latency --max-concurrency 4

# Python (do not upload experiment results)
uv run skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.py my_agent:run_agent my_dataset --evaluators accuracy --no-upload

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.js ./agent.ts:runAgent my_dataset --evaluators accuracy,latency --max-concurrency 4

Compare two agent versions

# Python
uv run skills/langgraph-testing-evaluation/scripts/compare_agents.py my_agent:v1 my_agent:v2 dataset.json --output comparison_report.json

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/compare_agents.js ./v1.ts:run ./v2.ts:run dataset.json --output comparison_report.json

# JavaScript/TypeScript (force local dataset file only)
node skills/langgraph-testing-evaluation/scripts/compare_agents.js ./v1.ts:run ./v2.ts:run dataset.json --no-langsmith

Create mock response configs

# Python
uv run skills/langgraph-testing-evaluation/scripts/mock_llm_responses.py create --type sequence --output mock_config.json

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/mock_llm_responses.js create --type sequence --output mock_config.json

Core Workflow

  1. Define test scope.
  • Unit: deterministic logic in one node/function.
  • Integration: node interactions and routing.
  • End-to-end: complete response quality on realistic inputs.
  1. Start from deterministic checks.
  • Mock LLM/tool IO for speed and repeatability.
  • Keep real-model tests as a smaller, explicit suite.
  1. Build/curate dataset examples.
  • Use stable inputs and expected outputs.
  • Keep schema simple: inputs and outputs objects (optional metadata).
  • Compatibility note: scripts also accept singular keys (input, output) for legacy datasets.
  1. Run evaluation with explicit gates.
  • Use evaluator keys that map to deployment decisions.
  • Set thresholds in CI for regression prevention.
  1. Compare versions before rollout.
  • Run same dataset on both versions.
  • Check both quality and latency.
  1. Diagnose failures from traces/experiments.
  • Inspect low-scoring examples.
  • Split failures by pattern (routing, tool usage, hallucination, latency spikes).

Current References (Load On Demand)

references/unit-testing-patterns.md

Load when:

  • You need node-level and routing test patterns.
  • You need pytest/vitest/Jest integration patterns.
  • You need robust mocking and flaky-test reduction.

references/trajectory-evaluation.md

Load when:

  • You need trajectory match evaluation (strict, unordered, subset, superset).
  • You need LLM-as-judge trajectory scoring.
  • You need LangSmith experiment comparison for trajectory results.

references/langsmith-evaluation.md

Load when:

  • You need dataset creation/management in LangSmith.
  • You need evaluator signatures and experiment runs in Python/TS.
  • You need CI-friendly workflows with quality thresholds.

references/ab-testing.md

Load when:

  • You need offline A/B comparison methodology.
  • You need significance testing and interpretation.
  • You need production traffic split strategy and guardrails.

Assets

assets/templates/test_template.py

  • Runnable Python pytest template aligned with current LangGraph testing patt
阅读完整来源 ↗