Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

datahub-evals

Use this skill to run DataHub's saved evals and report answers for judging. Triggers on: "run our evals", "run the eval suite", "run eval urn:li:eval:...", "how are our evals doing", "check for eval regressions", "upload this answer as an eval result", "score this answer with the DataHub judge", "compare two agents on the same eval". Answers each eval in a fresh agent with the DataHub tools attached, reports the answer through the DataHub Cloud CLI, and reads back the verdict DataHub's own judge produced.

datahub-project · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add datahub-project/datahub-skills --skill datahub-evals

About this skill


name: datahub-evals description: | Use this skill to run DataHub's saved evals and report answers for judging. Triggers on: "run our evals", "run the eval suite", "run eval urn:li:eval:...", "how are our evals doing", "check for eval regressions", "upload this answer as an eval result", "score this answer with the DataHub judge", "compare two agents on the same eval". Answers each eval in a fresh agent with the DataHub tools attached, reports the answer through the DataHub Cloud CLI, and reads back the verdict DataHub's own judge produced. user-invocable: true allowed-tools: Bash(acryl-datahub-cloud *), Bash(claude *), Bash(pip install acryl-datahub-cloud), Bash(python3 -m venv *), Task

DataHub Evals

Run DataHub's saved evals, and report answers — yours or another agent's — for DataHub to judge.

You are the runner. There is no script: you fetch the evals, answer each one in a fresh agent, and report the answers. Every call to DataHub is one evals subcommand, so the queries and the payload live in the CLI.

acryl-datahub-cloud evals --agent-context   # the CLI's own guide to its commands

Use acryl-datahub-cloud evals. A datahub evals form exists in the CLI's own help text and in some notes, but the group is not wired into the datahub CLI in any shipped release — that name answers No such command 'evals', and this skill does not use it.


Never simulate the judge

If DataHub's judge cannot be reached, say so and stop. Do not score the answer yourself, do not ask a subagent to render a verdict "the way DataHub would", and do not present any locally-produced score as a verdict.

A simulated verdict written in the house style reads as authoritative, gets pasted into a comparison table, and is comparable with nothing. A model is also not a fair judge of an answer it or a sibling produced.

That is why --type is never passed to evals report: omitting it routes the answer through the same judge a native run gets, which is the only thing that makes two runs comparable.


Before you run anything

The CLI is installed.

acryl-datahub-cloud evals --help

If that does not resolve, install the cloud CLI with its evals extra. It pins its own acryl-datahub, so give it its own environment:

python3 -m venv .venv && source .venv/bin/activate
pip install 'acryl-datahub-cloud[datahub-evals]==2.1.4rc1'

Pin a release that has the commands. The eval commands are still pre-release: the latest stable (2.1.3) carries neither the datahub-evals extra nor the cli module, so an unpinned install resolves it, warns that the extra does not exist, and leaves you with no evals at all. Pin the version, or pass --pre.

Quote the extra — an unquoted [...] is a glob in zsh — and take it rather than the bare package: it carries graphql-core, without which every eval query is sent unadapted and the CLI's schema-compatibility checks silently do nothing.

The CLI can reach DataHub.

acryl-datahub-cloud evals list --limit 1

Judge success by the exit code, not by a clean stream: the CLI logs warnings to stderr while returning its result on stdout, so a warning about adapting the GraphQL query for schema compatibility is not a failed call.

If that fails, stop and fix the connection — a bad token or URL surfaces here, before anything is spent. It does not prove the MANAGE_AGENTS privilege that reporting requires; there is no privilege query, so a token without it fails at report time instead.

The answering agent has the SQL workflow skill. A SQL eval is scored on catalog-grounded SQL — the right tables, joins and metric definitions, found through the DataHub tools rather than guessed — which is what datahub-sql-workflow instructs. Without it the answering agent writes plausible SQL against invented columns and fails LLM_JUDGE for a reason that says nothing about the catalog.

Install it where a fresh agent will load it — user level, or the plugin. Not project level: the run happens in an empty working directory, so a skill sitting in some repo's .claude/ is not on the answering agent's path.

ls ~/.claude/skills/datahub-sql-workflow/SKILL.md

A DataHub MCP server the answering agent can reach. The evals measure the DataHub tools, so an agent without them answers from memory and fails for a reason unrelated to the catalog.

claude mcp list

Look for a DataHub server that is Connected, and check two things that are silent when wrong:

  • It points at the same instance the results go to. Answering against one catalog and reporting into another produces verdicts about a catalog the agent never saw.
  • It is not disabled. A disabled server still loads and serves no tools.

MCP servers are scoped per project directory, so where you run from decides what exists. If there is no DataHub server, stop and say so rather than running evals that will all fail the same way.


The tool surface is DataHub-only

An eval question is untrusted text fetched from DataHub, about to be handed to an agent whose tools run without a prompt. Narrow the surface to the DataHub server and nothing else:

--strict-mcp-config --mcp-config <config.json> --allowedTools mcp__<datahub-server>

This is not only the safe surface, it is the one that runs unattended. A wider surface needs tools nobody pre-authorised, and --dangerously-skip-permissions is not a way out — the permission classifier refuses it, so the run stalls or dies rather than answering. Treat "everything configured" as an interactive measurement a person drives, not something this skill produces.

Tool surface
claude --printenforced — `-
Read the complete source ↗