vector-database-operations-pinecone-weaviate-milvus
Guides day-to-day operation of vector databases — index configuration, sharding/replication for scale and availability, and upsert/query performance tuning — with concrete equivalents across Pinecone, Weaviate, and Milvus. Use when a user asks to "configure a vector index," "scale our vector DB for more vectors/QPS," "tune HNSW parameters," "set up sharding or replication for Pinecone/Weaviate/Milvus," "our vector queries got slow," or "upserts are timing out/backing up."
selvarajmurugesan90 · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add selvarajmurugesan90/ops-engineering-skills --skill vector-database-operations-pinecone-weaviate-milvus技能说明
name: vector-database-operations-pinecone-weaviate-milvus description: > Guides day-to-day operation of vector databases — index configuration, sharding/replication for scale and availability, and upsert/query performance tuning — with concrete equivalents across Pinecone, Weaviate, and Milvus. Use when a user asks to "configure a vector index," "scale our vector DB for more vectors/QPS," "tune HNSW parameters," "set up sharding or replication for Pinecone/Weaviate/Milvus," "our vector queries got slow," or "upserts are timing out/backing up." license: Apache-2.0 compatibility: "Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini CLI" metadata: domain: ai-agent maturity: stable
Vector Database Operations (Pinecone, Weaviate, Milvus)
Purpose
A vector database's default configuration works fine for a demo and starts showing real operational pain exactly when it matters most: at production scale, under real query load, with a corpus that keeps growing. This skill covers operating a vector database day to day — configuring the index correctly for its workload, scaling it horizontally (sharding) and for availability (replication), and tuning upsert and query performance — with concrete, comparable guidance across the three most common choices (Pinecone as a managed service, Weaviate and Milvus as commonly self-hosted or managed alternatives). It assumes the index schema and dimension are already correct and validated (see vector-database-configuration-validation) and that data is already flowing in via an ingestion pipeline (see vector-database-ingestion-pipeline-for-rag); this skill is specifically the operate-and-tune layer underneath a RAG system's retrieval stage (see rag-pipeline-design for the retrieval pattern itself, which this skill doesn't repeat).
When to use
- Standing up a new vector index/collection in Pinecone, Weaviate, or Milvus and choosing its core configuration.
- Query latency has degraded as the corpus or query volume has grown, and it needs concrete tuning, not just "add more hardware."
- Deciding how to shard or partition a large corpus (by tenant, by data recency, by content type) across index namespaces/collections.
- Setting up replication for read throughput or availability during upgrades/maintenance.
- Upserts are slow, timing out, or backing up during a bulk load or a re-indexing run.
- Capacity planning before a corpus grows significantly (more documents, more tenants, higher QPS).
Prerequisites & environment
- A known embedding dimension and distance metric already fixed for the corpus (changing either requires a full re-embed and a new index, not a config tweak — see vector-database-configuration-validation).
- Estimated corpus size (vector count), expected query QPS, and expected write (upsert) rate — sizing decisions below depend on having real numbers, not guesses.
- For Pinecone: an account with pod-based or serverless index access (capacity/scaling mechanics differ between the two — check current Pinecone documentation for which applies to your plan, since this has changed over time).
- For Weaviate/Milvus: a self-hosted or managed cluster with enough nodes to support the replication/sharding plan you choose — these are self-operated systems, so cluster sizing is your responsibility in a way it isn't with a fully managed Pinecone index.
- Monitoring for index-level metrics (query latency, upsert throughput, index/memory fullness) wired to a dashboard — see prometheus-and-grafana-monitoring-stack for the metrics-pipeline mechanics if self-hosting Weaviate/Milvus.
Step-by-step guidance
-
Configure the index's core parameters deliberately, not with defaults. All three support HNSW as the common ANN (approximate nearest neighbor) index type; Milvus additionally supports IVF-family indexes, which trade some recall for lower memory at very large scale. Confirm the distance metric matches what the embedding model expects (cosine similarity is the common default; some models are tuned for dot product) — a mismatch here degrades relevance silently, not with an error (see vector-database-configuration-validation for the validation gate on this specifically).
-
Tune HNSW's core knobs deliberately — they trade recall, latency, and memory against each other:
M(max connections per node): higher improves recall, increases memory and index build time. Common starting range: 16-32.ef_construction(candidate list size during index build): higher improves recall at index-build-time cost only (a one-time or per-upsert cost, not a per-query cost). Common starting range: 100-200.ef_search(candidate list size at query time): higher improves recall at direct query-latency cost, tunable per query without rebuilding the index. This is usually the first knob to adjust when tuning the recall/latency trade-off after launch.
# Weaviate collection config (illustrative — verify field names against # your Weaviate version's current API) vectorIndexConfig: distance: cosine ef: 128 # ef_search, tunable post-launch efConstruction: 128 maxConnections: 32 # M# Milvus index params (illustrative) index_params = { "index_type": "HNSW", "metric_type": "COSINE", "params": {"M": 24, "efConstruction": 200}, } search_params = {"ef": 128} # tuned separately at query time# Pinecone: HNSW internals are managed for you on pod