datahub-evals
Use this skill to run DataHub's saved evals and report answers for judging. Triggers on: "run our evals", "run the eval suite", "run eval urn:li:eval:...", "how are our evals doing", "check for eval regressions", "upload this answer as an eval result", "score this answer with the DataHub judge", "compare two agents on the same eval". Answers each eval in a fresh agent with the DataHub tools attached, reports the answer through the DataHub Cloud CLI, and reads back the verdict DataHub's own judge produced.
datahub-project · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add datahub-project/datahub-skills --skill datahub-evals技能说明
name: datahub-evals description: | Use this skill to run DataHub's saved evals and report answers for judging. Triggers on: "run our evals", "run the eval suite", "run eval urn:li:eval:...", "how are our evals doing", "check for eval regressions", "upload this answer as an eval result", "score this answer with the DataHub judge", "compare two agents on the same eval". Answers each eval in a fresh agent with the DataHub tools attached, reports the answer through the DataHub Cloud CLI, and reads back the verdict DataHub's own judge produced. user-invocable: true allowed-tools: Bash(acryl-datahub-cloud *), Bash(claude *), Bash(pip install acryl-datahub-cloud), Bash(python3 -m venv *), Task
DataHub Evals
Run DataHub's saved evals, and report answers — yours or another agent's — for DataHub to judge.
You are the runner. There is no script: you fetch the evals, answer each one in a fresh
agent, and report the answers. Every call to DataHub is one evals subcommand, so the queries
and the payload live in the CLI.
acryl-datahub-cloud evals --agent-context # the CLI's own guide to its commands
Use acryl-datahub-cloud evals. A datahub evals form exists in the CLI's own help text and
in some notes, but the group is not wired into the datahub CLI in any shipped release — that
name answers No such command 'evals', and this skill does not use it.
Never simulate the judge
If DataHub's judge cannot be reached, say so and stop. Do not score the answer yourself, do not ask a subagent to render a verdict "the way DataHub would", and do not present any locally-produced score as a verdict.
A simulated verdict written in the house style reads as authoritative, gets pasted into a comparison table, and is comparable with nothing. A model is also not a fair judge of an answer it or a sibling produced.
That is why --type is never passed to evals report: omitting it routes the answer
through the same judge a native run gets, which is the only thing that makes two runs
comparable.
Before you run anything
The CLI is installed.
acryl-datahub-cloud evals --help
If that does not resolve, install the cloud CLI with its evals extra. It pins its own
acryl-datahub, so give it its own environment:
python3 -m venv .venv && source .venv/bin/activate
pip install 'acryl-datahub-cloud[datahub-evals]==2.1.4rc1'
Pin a release that has the commands. The eval commands are still pre-release: the latest
stable (2.1.3) carries neither the datahub-evals extra nor the cli module, so an unpinned
install resolves it, warns that the extra does not exist, and leaves you with no evals at
all. Pin the version, or pass --pre.
Quote the extra — an unquoted [...] is a glob in zsh — and take it rather than the bare
package: it carries graphql-core, without which every eval query is sent unadapted and the
CLI's schema-compatibility checks silently do nothing.
The CLI can reach DataHub.
acryl-datahub-cloud evals list --limit 1
Judge success by the exit code, not by a clean stream: the CLI logs warnings to stderr while returning its result on stdout, so a warning about adapting the GraphQL query for schema compatibility is not a failed call.
If that fails, stop and fix the connection — a bad token or URL surfaces here, before
anything is spent. It does not prove the MANAGE_AGENTS privilege that reporting requires;
there is no privilege query, so a token without it fails at report time instead.
The answering agent has the SQL workflow skill. A SQL eval is scored on catalog-grounded
SQL — the right tables, joins and metric definitions, found through the DataHub tools rather
than guessed — which is what
datahub-sql-workflow
instructs. Without it the answering agent writes plausible SQL against invented columns and
fails LLM_JUDGE for a reason that says nothing about the catalog.
Install it where a fresh agent will load it — user level, or the plugin. Not project level:
the run happens in an empty working directory, so a skill sitting in some repo's .claude/
is not on the answering agent's path.
ls ~/.claude/skills/datahub-sql-workflow/SKILL.md
A DataHub MCP server the answering agent can reach. The evals measure the DataHub tools, so an agent without them answers from memory and fails for a reason unrelated to the catalog.
claude mcp list
Look for a DataHub server that is Connected, and check two things that are silent when wrong:
- It points at the same instance the results go to. Answering against one catalog and reporting into another produces verdicts about a catalog the agent never saw.
- It is not disabled. A disabled server still loads and serves no tools.
MCP servers are scoped per project directory, so where you run from decides what exists. If there is no DataHub server, stop and say so rather than running evals that will all fail the same way.
The tool surface is DataHub-only
An eval question is untrusted text fetched from DataHub, about to be handed to an agent whose tools run without a prompt. Narrow the surface to the DataHub server and nothing else:
--strict-mcp-config --mcp-config <config.json> --allowedTools mcp__<datahub-server>
This is not only the safe surface, it is the one that runs unattended. A wider surface needs
tools nobody pre-authorised, and --dangerously-skip-permissions is not a way out — the
permission classifier refuses it, so the run stalls or dies rather than answering. Treat
"everything configured" as an interactive measurement a person drives, not something this
skill produces.
| Tool surface | |
|---|---|
claude --print | enforced — `- |