{"slug":"h4vzz-data-labeling","name":"data-labeling","description":"Set up and manage data labeling workflows using manual annotation tools, semi-automated pipelines, active learning, and programmatic weak supervision.","long_description":"---\nname: data-labeling\ndescription: Set up and manage data labeling workflows using manual annotation tools, semi-automated pipelines, active learning, and programmatic weak supervision.\nlicense: MIT\nmetadata:\n author: AI Agent Skills\n version: 1.0.0\n---\n\n# Data Labeling\n\nThis skill enables an AI agent to design and execute data labeling workflows for machine learning projects. It covers manual annotation with tools like Label Studio, semi-automated labeling with model-assisted pre-annotation, active learning loops that prioritize the most informative samples, and programmatic weak supervision using labeling functions. The agent handles label schema design, annotator guidelines, quality control through inter-annotator agreement, and export to ML-ready formats.\n\n## Workflow\n\n1. **Define the labeling schema and guidelines:** Design the label taxonomy — classes for classification, entity types for NER, bounding box categories for object detection, or segment labels for semantic segmentation. Write clear annotator guidelines with positive and negative examples for each label, covering boundary cases and ambiguous scenarios.\n\n2. **Set up the labeling environment:** Configure a labeling tool (Label Studio, Labelbox, or Prodigy) with the schema, import the raw data, and set up user accounts with appropriate permissions. Define the labeling interface template that matches the task type — text classification, span annotation, image bounding boxes, or multi-turn dialogue tagging.\n\n3. **Pre-annotate with model predictions:** Use existing models or heuristic rules to generate preliminary labels for the dataset. Annotators then review and correct these predictions rather than labeling from scratch, which can reduce annotation time by 40-60%. This is especially valuable for tasks where a decent baseline model already exists.\n\n4. **Execute labeling with quality control:** Assign labeling tasks to annotators with built-in redundancy — have 2-3 annotators label the same items to measure inter-annotator agreement (Cohen's kappa or Fleiss' kappa). Flag items with low agreement for review by a senior annotator. Track annotator accuracy against a gold-standard set embedded in the task queue.\n\n5. **Run active learning iterations:** After an initial labeled set is created, train a model and use uncertainty sampling or query-by-committee to select the most informative unlabeled examples for the next round of annotation. This maximizes model improvement per labeled sample and is critical when labeling budgets are limited.\n\n6. **Export and validate:** Export labeled data in the format required by the training pipeline (JSONL, COCO, CoNLL, CSV). Run validation checks to ensure label consistency, check for missing annotations, and verify that the class distribution meets requirements. Document the labeling process and dataset statistics for reproducibility.\n\n## Supported Technologies\n\n- **Annotation tools:** Label Studio, Labelbox, Prodigy (spaCy), Amazon SageMaker Ground Truth, CVAT\n- **Weak supervision:** Snorkel, Flyingsquid, Skweak\n- **Active learning:** modAL, ALiPy, Prodigy active learning recipes\n- **Agreement metrics:** Cohen's kappa, Fleiss' kappa, Krippendorff's alpha\n- **Export formats:** COCO JSON, Pascal VOC XML, CoNLL, JSONL, Hugging Face Datasets\n\n## Usage\n\nProvide the agent with the raw dataset, the task type (classification, NER, object detection, etc.), and the label categories. Optionally specify the labeling tool preference and quality requirements (minimum inter-annotator agreement). The agent will configure the labeling environment, set up quality control, and manage the annotation workflow.\n\n## Examples\n\n### Example 1: Label Studio Pipeline for Text Classification\n\n**Label Studio labeling interface configuration (`config.xml`):**\n\n```xml\n\n
\n \n \n \n \n \n \n