跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
数据与分析

vector-database-operations-pinecone-weaviate-milvus

Guides day-to-day operation of vector databases — index configuration, sharding/replication for scale and availability, and upsert/query performance tuning — with concrete equivalents across Pinecone, Weaviate, and Milvus. Use when a user asks to "configure a vector index," "scale our vector DB for more vectors/QPS," "tune HNSW parameters," "set up sharding or replication for Pinecone/Weaviate/Milvus," "our vector queries got slow," or "upserts are timing out/backing up."

selvarajmurugesan90 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add selvarajmurugesan90/ops-engineering-skills --skill vector-database-operations-pinecone-weaviate-milvus

技能说明


name: vector-database-operations-pinecone-weaviate-milvus description: > Guides day-to-day operation of vector databases — index configuration, sharding/replication for scale and availability, and upsert/query performance tuning — with concrete equivalents across Pinecone, Weaviate, and Milvus. Use when a user asks to "configure a vector index," "scale our vector DB for more vectors/QPS," "tune HNSW parameters," "set up sharding or replication for Pinecone/Weaviate/Milvus," "our vector queries got slow," or "upserts are timing out/backing up." license: Apache-2.0 compatibility: "Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini CLI" metadata: domain: ai-agent maturity: stable

Vector Database Operations (Pinecone, Weaviate, Milvus)

Purpose

A vector database's default configuration works fine for a demo and starts showing real operational pain exactly when it matters most: at production scale, under real query load, with a corpus that keeps growing. This skill covers operating a vector database day to day — configuring the index correctly for its workload, scaling it horizontally (sharding) and for availability (replication), and tuning upsert and query performance — with concrete, comparable guidance across the three most common choices (Pinecone as a managed service, Weaviate and Milvus as commonly self-hosted or managed alternatives). It assumes the index schema and dimension are already correct and validated (see vector-database-configuration-validation) and that data is already flowing in via an ingestion pipeline (see vector-database-ingestion-pipeline-for-rag); this skill is specifically the operate-and-tune layer underneath a RAG system's retrieval stage (see rag-pipeline-design for the retrieval pattern itself, which this skill doesn't repeat).

When to use

  • Standing up a new vector index/collection in Pinecone, Weaviate, or Milvus and choosing its core configuration.
  • Query latency has degraded as the corpus or query volume has grown, and it needs concrete tuning, not just "add more hardware."
  • Deciding how to shard or partition a large corpus (by tenant, by data recency, by content type) across index namespaces/collections.
  • Setting up replication for read throughput or availability during upgrades/maintenance.
  • Upserts are slow, timing out, or backing up during a bulk load or a re-indexing run.
  • Capacity planning before a corpus grows significantly (more documents, more tenants, higher QPS).

Prerequisites & environment

  • A known embedding dimension and distance metric already fixed for the corpus (changing either requires a full re-embed and a new index, not a config tweak — see vector-database-configuration-validation).
  • Estimated corpus size (vector count), expected query QPS, and expected write (upsert) rate — sizing decisions below depend on having real numbers, not guesses.
  • For Pinecone: an account with pod-based or serverless index access (capacity/scaling mechanics differ between the two — check current Pinecone documentation for which applies to your plan, since this has changed over time).
  • For Weaviate/Milvus: a self-hosted or managed cluster with enough nodes to support the replication/sharding plan you choose — these are self-operated systems, so cluster sizing is your responsibility in a way it isn't with a fully managed Pinecone index.
  • Monitoring for index-level metrics (query latency, upsert throughput, index/memory fullness) wired to a dashboard — see prometheus-and-grafana-monitoring-stack for the metrics-pipeline mechanics if self-hosting Weaviate/Milvus.

Step-by-step guidance

  1. Configure the index's core parameters deliberately, not with defaults. All three support HNSW as the common ANN (approximate nearest neighbor) index type; Milvus additionally supports IVF-family indexes, which trade some recall for lower memory at very large scale. Confirm the distance metric matches what the embedding model expects (cosine similarity is the common default; some models are tuned for dot product) — a mismatch here degrades relevance silently, not with an error (see vector-database-configuration-validation for the validation gate on this specifically).

  2. Tune HNSW's core knobs deliberately — they trade recall, latency, and memory against each other:

    • M (max connections per node): higher improves recall, increases memory and index build time. Common starting range: 16-32.
    • ef_construction (candidate list size during index build): higher improves recall at index-build-time cost only (a one-time or per-upsert cost, not a per-query cost). Common starting range: 100-200.
    • ef_search (candidate list size at query time): higher improves recall at direct query-latency cost, tunable per query without rebuilding the index. This is usually the first knob to adjust when tuning the recall/latency trade-off after launch.
    # Weaviate collection config (illustrative — verify field names against
    # your Weaviate version's current API)
    vectorIndexConfig:
      distance: cosine
      ef: 128            # ef_search, tunable post-launch
      efConstruction: 128
      maxConnections: 32  # M
    
    # Milvus index params (illustrative)
    index_params = {
        "index_type": "HNSW",
        "metric_type": "COSINE",
        "params": {"M": 24, "efConstruction": 200},
    }
    search_params = {"ef": 128}  # tuned separately at query time
    
    # Pinecone: HNSW internals are managed for you on pod
    
阅读完整来源 ↗