Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

analyze-test-run

Analyze a GitHub Actions integration test run and produce a skill invocation report with failure root-cause issues. TRIGGERS: analyze test run, skill invocation rate, test run report, compare test runs, skill invocation summary, test failure analysis, run report, test results, action run report

microsoft · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add microsoft/GitHub-Copilot-for-Azure --skill analyze-test-run

About this skill


name: analyze-test-run description: "Analyze a GitHub Actions integration test run and produce a skill invocation report with failure root-cause issues. TRIGGERS: analyze test run, skill invocation rate, test run report, compare test runs, skill invocation summary, test failure analysis, run report, test results, action run report" license: MIT metadata: author: Microsoft version: "1.0.6"

Analyze Test Run

Downloads artifacts from a GitHub Actions integration test run, generates a summarized skill invocation report, and files GitHub issues for each test failure with root-cause analysis.

When to Use

  • Summarize results of a GitHub Actions integration test run
  • Calculate skill invocation rates for the skill under test
  • For azure-deploy tests: track the full deployment chain (azure-prepare → azure-validate → azure-deploy)
  • Compare skill invocation across two runs
  • File issues for test failures with root-cause context

Input

ParameterRequiredDescription
Run ID or URLYesGitHub Actions run ID (e.g. 22373768875) or full URL
Comparison RunNoSecond run ID/URL for side-by-side comparison

MCP Tools

All tools use owner: "microsoft" and repo: "GitHub-Copilot-for-Azure" as fixed parameters. method selects the operation within the tool.

ToolmethodKey ParameterPurpose
actions_getget_workflow_runresource_id: run IDFetch run status and metadata
actions_listlist_workflow_run_artifactsresource_id: run IDList all artifacts for a run
actions_getdownload_workflow_run_artifactresource_id: artifact IDGet a temporary download URL for an artifact ZIP
get_job_logs—run_id + failed_only: trueRetrieve job logs when artifact content is inaccessible
search_issues—query: search stringFind existing open issues before creating new ones
create_issue—title, body, labels, assigneesFile a new GitHub issue for a test failure

Workflow

Phase 1 — Download & Parse

  1. Extract the numeric run ID from the input (strip URL prefix if needed)

  2. Fetch run metadata using the MCP actions_get tool:

    actions_get({ method: "get_workflow_run", owner: "microsoft", repo: "GitHub-Copilot-for-Azure", resource_id: "<run-id>" })
    
  3. List artifacts using the MCP actions_list tool, then download each relevant artifact:

    // List artifacts
    actions_list({ method: "list_workflow_run_artifacts", owner: "microsoft", repo: "GitHub-Copilot-for-Azure", resource_id: "<run-id>" })
    // Download individual artifacts by ID
    actions_get({ method: "download_workflow_run_artifact", owner: "microsoft", repo: "GitHub-Copilot-for-Azure", resource_id: "<artifact-id>" })
    

    The download returns a temporary URL. Fetch the ZIP archive from that URL and extract it locally. If the environment restricts outbound HTTP (e.g. AWF sandbox), record in the analysis report that artifact content was unavailable and fall back to job logs via the get_job_logs MCP tool.

  4. Locate these files in the downloaded artifacts:

    • junit.xml — test pass/fail/skip/error results
    • *-SKILL-REPORT.md — generated skill report with per-test details
    • agent-metadata-*.md files — raw agent session logs per test

    ⚠️ Note: If artifact ZIP files cannot be downloaded due to network restrictions, or if downloaded files cannot be extracted, use the get_job_logs MCP tool to identify test failures and produce a best-effort analysis from whatever data is accessible.

Phase 2 — Build Summary Report

Produce a markdown report with four sections. See report-format.md for the exact template.

Section 1 — Test Results Overview

Parse junit.xml to build:

MetricValue
Total testscount from <testsuites tests=…>
Executedtotal − skipped
Skippedcount of <skipped/> elements
Passedexecuted − failures − errors
Failedcount of <failure> elements
Test Pass Ratepassed / executed as %

Include a per-test table with name, duration (from time attribute, convert seconds to Xm Ys), and Pass/Fail result.

Section 2 — Skill Invocation Rate

Read the SKILL-REPORT.md "Per-Test Case Results" sections. For each executed test determine whether the skill under test was invoked.

The skills to track depend on which integration test suite the run belongs to:

azure-deploy integration tests — track the full deployment chain:

SkillHow to detect
azure-prepareMentioned as invoked in the narrative or agent-metadata
azure-validateMentioned as invoked in the narrative or agent-metadata
azure-deployMentioned as invoked in the narrative or agent-metadata

Build a per-test invocation matrix (Yes/No for each skill) and compute rates:

SkillInvocation Rate
azure-deployX% (n/total)
azure-prepareX% (n/total)
azure-validateX% (n/total)
Full skill chain (P→V→D)X% (n/total)

The azure-deploy integration tests exercise the full deployment workflow where the agent is expected to invoke azure-prepare, azure-validate, and azure-deploy in sequence. This three-skill chain tracking is specific to azure-deploy tests only.

All other integration tests — track only the skill under test:

SkillInvocation Rate
{skill-under-test}X% (n/total)

For non-deploy tests (e.g. azure-prepare, azure-ai, azure-kusto), only track whether the primary skill under test was invoked. Do not include azure-prepare/azure-validate/azure-deploy chain columns.

Section 3 — Report Confidence & Pass Rate

Extract from SKILL-REPORT.md:

  • Skill Invocation Success Rate (from the report's statistics section)
  • Overal
Read the complete source ↗