managed-ingestion-service
Ingest documents at scale into Amazon OpenSearch using OpenSearch Ingestion Service (OSIS) pipelines. Upload pre-generated JSONL chunks to S3 and OSIS indexes them — optionally using semantic_enrichment in the sink to create the index with ASE automatically. Cloud ingestion — uploading raw PDF/DOCX and letting document_extractor parse them in the cloud — is available via private beta. Use this skill when the user wants to ingest documents into a cloud OpenSearch domain or collection, process documents at full volume beyond local limits, or set up an OSIS pipeline. Activate even if the user says OSIS, ingestion pipeline, document extraction, S3 ingestion, managed ingestion, or cloud processing.
opensearch-project · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add opensearch-project/opensearch-agent-skills --skill managed-ingestion-service技能说明
name: managed-ingestion-service description: > Ingest documents at scale into Amazon OpenSearch using OpenSearch Ingestion Service (OSIS) pipelines. Upload pre-generated JSONL chunks to S3 and OSIS indexes them — optionally using semantic_enrichment in the sink to create the index with ASE automatically. Cloud ingestion — uploading raw PDF/DOCX and letting document_extractor parse them in the cloud — is available via private beta. Use this skill when the user wants to ingest documents into a cloud OpenSearch domain or collection, process documents at full volume beyond local limits, or set up an OSIS pipeline. Activate even if the user says OSIS, ingestion pipeline, document extraction, S3 ingestion, managed ingestion, or cloud processing. compatibility: > Requires AWS credentials, an ACTIVE Amazon OpenSearch Service domain (v2.19+) or Serverless collection endpoint, and an S3 bucket for source documents or chunks. metadata: author: opensearch-project version: "3.0"
Managed Ingestion Service
Overview
Ingest documents into Amazon OpenSearch at scale via OSIS pipelines. Supports both Amazon OpenSearch Service (AOS) domains and Amazon OpenSearch Serverless (AOSS) collections.
When to Use
- User has an OpenSearch domain or AOSS collection and wants to ingest documents at scale
- User wants to move from local development to cloud ingestion
- User mentions OSIS, S3 ingestion, cloud processing, managed ingestion
Ingestion Paths
| Path | Upload to S3 | OSIS Processor | Status | Best for |
|---|---|---|---|---|
| Local processing | JSONL chunks (from local Docling) | parse_json (pass-through) | Available now | Validated locally, deploy at scale |
| Cloud ingestion | Raw PDF/DOCX/PPTX/XLSX | document_extractor | Private beta | Full cloud processing, no local compute |
Both paths support the same sink options — the only difference is source content and processor.
Key Constraints
- AWS credentials must be configured; check first and stop if missing
- AOS domains: must be running OpenSearch 2.19+ for ASE; public domains only (no VPC)
- AOSS collections: must be in ACTIVE state before creating the pipeline
- The IAM role (
sts_role_arn) must have S3 read + OpenSearch write permissions - When using
semantic_enrichment: it creates the index automatically — do NOT create it manually parse_jsonis the OSIS processor (parses JSONL lines);semantic_enrichmentis in the sink (applies embeddings when writing) — they are independent- Save generated YAML to
.opensearch/pipelines/<pipeline-name>.yaml
Entry Point
Step 1: Validate credentials
aws sts get-caller-identity
If this fails, stop and ask the user to configure credentials.
Step 2: Determine the deploy target
Ask: "Are you ingesting into an Amazon OpenSearch Service (AOS) domain or an OpenSearch Serverless (AOSS) collection?"
AOS domain:
- Must be running OpenSearch 2.19+ for semantic_enrichment
- Verify the domain exists and is in ACTIVE state:
aws opensearch describe-domain --domain-name <domain-name> --region <region>
AOSS collection:
- Verify it's ACTIVE:
aws opensearchserverless batch-get-collection \ --ids <collection-id> --region <region>
If the user needs a new domain/collection, provision one using the aws-setup skill. Wait until ACTIVE, then continue here.
Step 3: Determine the data source
Ask: "Do you already have JSONL chunks, or do you have raw documents (PDF/DOCX) that need processing?"
If the user already has JSONL chunks:
- Validate the JSONL format: each line must be valid JSON with text fields that can be enriched.
- Proceed to Step 4.
If the user has raw documents (PDF/DOCX/etc.):
Present both options to the user:
I see you have raw PDF documents. There are two ways to process them for ingestion:
Option A — Cloud ingestion (recommended for scale) Upload your PDFs directly to S3 and let the cloud handle parsing, chunking, and indexing with ASE (automatic sparse embeddings) — no local compute needed. This is currently in private beta.
👉 To get access, email opensearch-agent-skills-interests@amazon.com with your AWS account ID and region.
Option B — Local processing (available now) Process your PDFs locally with Docling to produce JSONL chunks, then upload those to S3 for cloud ingestion. Gives you full control over chunk quality.
Which would you prefer? (Or if you'd like to get started immediately, I'll go with Option B while you wait for beta access.)
- If the user chooses Option A: confirm they should email opensearch-agent-skills-interests@amazon.com, and offer to proceed with Option B in the meantime.
- If the user chooses Option B: proceed with document_processing_guide.md. Once chunks are ready at
.opensearch/chunks/<index>/, return here and continue to Step 4.
Step 4: Gather parameters
Collect from the user (one at a time):
- Deploy target — AOS domain endpoint or AOSS collection endpoint
- S3 bucket name — where chunks will be uploaded
- S3 prefix — key prefix for source files (e.g.,
chunks/input) - Target index name — will be created by
semantic_enrichmentif new; or an existing index - IAM role ARN — for OSIS pipeline (or offer to create one)
- Network policy name — (AOSS only) network policy allowing OSIS access
- Region — AWS region
Step 5: Determine the sink mode
Ask: "Do you already have an existing index to ingest into, or should I create a new one with automatic semantic enrichment (ASE)?"
Path A — Existing index (no semantic_enrichment):
- The user already has an index with mappings configured.
- Use the plain opensearch sink (no
semantic_enrichmentblock). - Documen