data-labeling
Set up and manage data labeling workflows using manual annotation tools, semi-automated pipelines, active learning, and programmatic weak supervision.
AI Agent Skills · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add h4vzz/awesome-ai-agent-skills --skill data-labeling技能说明
name: data-labeling description: Set up and manage data labeling workflows using manual annotation tools, semi-automated pipelines, active learning, and programmatic weak supervision. license: MIT metadata: author: AI Agent Skills version: 1.0.0
Data Labeling
This skill enables an AI agent to design and execute data labeling workflows for machine learning projects. It covers manual annotation with tools like Label Studio, semi-automated labeling with model-assisted pre-annotation, active learning loops that prioritize the most informative samples, and programmatic weak supervision using labeling functions. The agent handles label schema design, annotator guidelines, quality control through inter-annotator agreement, and export to ML-ready formats.
Workflow
-
Define the labeling schema and guidelines: Design the label taxonomy — classes for classification, entity types for NER, bounding box categories for object detection, or segment labels for semantic segmentation. Write clear annotator guidelines with positive and negative examples for each label, covering boundary cases and ambiguous scenarios.
-
Set up the labeling environment: Configure a labeling tool (Label Studio, Labelbox, or Prodigy) with the schema, import the raw data, and set up user accounts with appropriate permissions. Define the labeling interface template that matches the task type — text classification, span annotation, image bounding boxes, or multi-turn dialogue tagging.
-
Pre-annotate with model predictions: Use existing models or heuristic rules to generate preliminary labels for the dataset. Annotators then review and correct these predictions rather than labeling from scratch, which can reduce annotation time by 40-60%. This is especially valuable for tasks where a decent baseline model already exists.
-
Execute labeling with quality control: Assign labeling tasks to annotators with built-in redundancy — have 2-3 annotators label the same items to measure inter-annotator agreement (Cohen's kappa or Fleiss' kappa). Flag items with low agreement for review by a senior annotator. Track annotator accuracy against a gold-standard set embedded in the task queue.
-
Run active learning iterations: After an initial labeled set is created, train a model and use uncertainty sampling or query-by-committee to select the most informative unlabeled examples for the next round of annotation. This maximizes model improvement per labeled sample and is critical when labeling budgets are limited.
-
Export and validate: Export labeled data in the format required by the training pipeline (JSONL, COCO, CoNLL, CSV). Run validation checks to ensure label consistency, check for missing annotations, and verify that the class distribution meets requirements. Document the labeling process and dataset statistics for reproducibility.
Supported Technologies
- Annotation tools: Label Studio, Labelbox, Prodigy (spaCy), Amazon SageMaker Ground Truth, CVAT
- Weak supervision: Snorkel, Flyingsquid, Skweak
- Active learning: modAL, ALiPy, Prodigy active learning recipes
- Agreement metrics: Cohen's kappa, Fleiss' kappa, Krippendorff's alpha
- Export formats: COCO JSON, Pascal VOC XML, CoNLL, JSONL, Hugging Face Datasets
Usage
Provide the agent with the raw dataset, the task type (classification, NER, object detection, etc.), and the label categories. Optionally specify the labeling tool preference and quality requirements (minimum inter-annotator agreement). The agent will configure the labeling environment, set up quality control, and manage the annotation workflow.
Examples
Example 1: Label Studio Pipeline for Text Classification
Label Studio labeling interface configuration (config.xml):
<View>
<Header value="Classify the customer review sentiment:" />
<Text name="text" value="$text" />
<Choices name="sentiment" toName="text" choice="single-column" showInline="true">
<Choice value="positive" />
<Choice value="negative" />
<Choice value="neutral" />
</Choices>
<Textarea name="notes" toName="text" placeholder="Optional: explain ambiguous cases"
maxSubmissions="1" editable="true" />
</View>
Python script to set up the project and import data:
from label_studio_sdk import Client
ls = Client(url="http://localhost:8080", api_key="your-api-key")
project = ls.start_project(
title="Customer Review Sentiment",
label_config=open("config.xml").read(),
description="Label customer reviews as positive, negative, or neutral.",
)
# Import tasks from a CSV file
import csv
tasks = []
with open("reviews.csv") as f:
for row in csv.DictReader(f):
tasks.append({"data": {"text": row["review_text"]}, "meta": {"source_id": row["id"]}})
project.import_tasks(tasks)
# Configure inter-annotator overlap: each task gets 2 annotators
project.set_params(maximum_annotations=2, overlap_cohort_percentage=100)
print(f"Created project with {len(tasks)} tasks, 2 annotators per task")
# After annotation, export results
annotations = project.export_tasks(export_type="JSON")
# Compute agreement
from sklearn.metrics import cohen_kappa_score
labels_a1 = [a["annotations"][0]["result"][0]["value"]["choices"][0] for a in annotations if len(a["annotations"]) >= 2]
labels_a2 = [a["annotations"][1]["result"][0]["value"]["choices"][0] for a in annotations if len(a["annotations"]) >= 2]
print(f"Cohen's kappa: {cohen_kappa_score(labels_a1, labels_a2):.3f}")
Example 2: Weak Supervision with Snorkel Labeling Functions
import pandas as pd
import numpy as np
from snorkel.labeling import labeling_function, PandasLFApplier, LFAnalysis
from snorkel.labeling.model import LabelModel
SPAM = 1
HAM = 0
ABSTAIN = -1
df = pd.DataFrame({
"text": [
"Congratulations! You've won a free iPhone!", "Meeting at 3pm tomorrow",
"URGENT: claim your prize now!!!", "Can you review the Q3 report?
阅读完整来源 ↗