跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
文档处理

managed-ingestion-service

Ingest documents at scale into Amazon OpenSearch using OpenSearch Ingestion Service (OSIS) pipelines. Upload pre-generated JSONL chunks to S3 and OSIS indexes them — optionally using semantic_enrichment in the sink to create the index with ASE automatically. Cloud ingestion — uploading raw PDF/DOCX and letting document_extractor parse them in the cloud — is available via private beta. Use this skill when the user wants to ingest documents into a cloud OpenSearch domain or collection, process documents at full volume beyond local limits, or set up an OSIS pipeline. Activate even if the user says OSIS, ingestion pipeline, document extraction, S3 ingestion, managed ingestion, or cloud processing.

opensearch-project · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add opensearch-project/opensearch-agent-skills --skill managed-ingestion-service

技能说明


name: managed-ingestion-service description: > Ingest documents at scale into Amazon OpenSearch using OpenSearch Ingestion Service (OSIS) pipelines. Upload pre-generated JSONL chunks to S3 and OSIS indexes them — optionally using semantic_enrichment in the sink to create the index with ASE automatically. Cloud ingestion — uploading raw PDF/DOCX and letting document_extractor parse them in the cloud — is available via private beta. Use this skill when the user wants to ingest documents into a cloud OpenSearch domain or collection, process documents at full volume beyond local limits, or set up an OSIS pipeline. Activate even if the user says OSIS, ingestion pipeline, document extraction, S3 ingestion, managed ingestion, or cloud processing. compatibility: > Requires AWS credentials, an ACTIVE Amazon OpenSearch Service domain (v2.19+) or Serverless collection endpoint, and an S3 bucket for source documents or chunks. metadata: author: opensearch-project version: "3.0"

Managed Ingestion Service

Overview

Ingest documents into Amazon OpenSearch at scale via OSIS pipelines. Supports both Amazon OpenSearch Service (AOS) domains and Amazon OpenSearch Serverless (AOSS) collections.

When to Use

  • User has an OpenSearch domain or AOSS collection and wants to ingest documents at scale
  • User wants to move from local development to cloud ingestion
  • User mentions OSIS, S3 ingestion, cloud processing, managed ingestion

Ingestion Paths

PathUpload to S3OSIS ProcessorStatusBest for
Local processingJSONL chunks (from local Docling)parse_json (pass-through)Available nowValidated locally, deploy at scale
Cloud ingestionRaw PDF/DOCX/PPTX/XLSXdocument_extractorPrivate betaFull cloud processing, no local compute

Both paths support the same sink options — the only difference is source content and processor.

Key Constraints

  • AWS credentials must be configured; check first and stop if missing
  • AOS domains: must be running OpenSearch 2.19+ for ASE; public domains only (no VPC)
  • AOSS collections: must be in ACTIVE state before creating the pipeline
  • The IAM role (sts_role_arn) must have S3 read + OpenSearch write permissions
  • When using semantic_enrichment: it creates the index automatically — do NOT create it manually
  • parse_json is the OSIS processor (parses JSONL lines); semantic_enrichment is in the sink (applies embeddings when writing) — they are independent
  • Save generated YAML to .opensearch/pipelines/<pipeline-name>.yaml

Entry Point

Step 1: Validate credentials

aws sts get-caller-identity

If this fails, stop and ask the user to configure credentials.

Step 2: Determine the deploy target

Ask: "Are you ingesting into an Amazon OpenSearch Service (AOS) domain or an OpenSearch Serverless (AOSS) collection?"

AOS domain:

  • Must be running OpenSearch 2.19+ for semantic_enrichment
  • Verify the domain exists and is in ACTIVE state:
    aws opensearch describe-domain --domain-name <domain-name> --region <region>
    

AOSS collection:

  • Verify it's ACTIVE:
    aws opensearchserverless batch-get-collection \
      --ids <collection-id> --region <region>
    

If the user needs a new domain/collection, provision one using the aws-setup skill. Wait until ACTIVE, then continue here.

Step 3: Determine the data source

Ask: "Do you already have JSONL chunks, or do you have raw documents (PDF/DOCX) that need processing?"

If the user already has JSONL chunks:

  • Validate the JSONL format: each line must be valid JSON with text fields that can be enriched.
  • Proceed to Step 4.

If the user has raw documents (PDF/DOCX/etc.):

Present both options to the user:

I see you have raw PDF documents. There are two ways to process them for ingestion:

Option A — Cloud ingestion (recommended for scale) Upload your PDFs directly to S3 and let the cloud handle parsing, chunking, and indexing with ASE (automatic sparse embeddings) — no local compute needed. This is currently in private beta.

👉 To get access, email opensearch-agent-skills-interests@amazon.com with your AWS account ID and region.

Option B — Local processing (available now) Process your PDFs locally with Docling to produce JSONL chunks, then upload those to S3 for cloud ingestion. Gives you full control over chunk quality.

Which would you prefer? (Or if you'd like to get started immediately, I'll go with Option B while you wait for beta access.)

  • If the user chooses Option A: confirm they should email opensearch-agent-skills-interests@amazon.com, and offer to proceed with Option B in the meantime.
  • If the user chooses Option B: proceed with document_processing_guide.md. Once chunks are ready at .opensearch/chunks/<index>/, return here and continue to Step 4.

Step 4: Gather parameters

Collect from the user (one at a time):

  1. Deploy target — AOS domain endpoint or AOSS collection endpoint
  2. S3 bucket name — where chunks will be uploaded
  3. S3 prefix — key prefix for source files (e.g., chunks/input)
  4. Target index name — will be created by semantic_enrichment if new; or an existing index
  5. IAM role ARN — for OSIS pipeline (or offer to create one)
  6. Network policy name — (AOSS only) network policy allowing OSIS access
  7. Region — AWS region

Step 5: Determine the sink mode

Ask: "Do you already have an existing index to ingest into, or should I create a new one with automatic semantic enrichment (ASE)?"

Path A — Existing index (no semantic_enrichment):

  • The user already has an index with mappings configured.
  • Use the plain opensearch sink (no semantic_enrichment block).
  • Documen
阅读完整来源 ↗