跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
视频与音频

blob-store

This skill should be used when the user wants a "blob store" or "object storage", names "S3" or an S3-compatible store, needs to "store images / video / files", asks about "multipart upload" or "resumable upload", "signed / presigned URLs", "media storage", "unstructured data at scale", object "versioning", storage "tiering" (hot/cold/archive), or "erasure coding" vs replication for durability. Use it whenever a design must hold large unstructured objects (photos, video, backups, logs, ML datasets) and serve them cheaply and durably, even if the user just says "where do we put the files".

proyecto26 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add proyecto26/system-design-skills --skill blob-store

技能说明


name: blob-store description: This skill should be used when the user wants a "blob store" or "object storage", names "S3" or an S3-compatible store, needs to "store images / video / files", asks about "multipart upload" or "resumable upload", "signed / presigned URLs", "media storage", "unstructured data at scale", object "versioning", storage "tiering" (hot/cold/archive), or "erasure coding" vs replication for durability. Use it whenever a design must hold large unstructured objects (photos, video, backups, logs, ML datasets) and serve them cheaply and durably, even if the user just says "where do we put the files".

Blob store

Store large, immutable, unstructured objects — images, video, backups, model weights, document blobs — in a flat namespace keyed by a string, replicated for durability and served by direct download. Getting it wrong means stuffing multi-megabyte blobs into a row-oriented database (where they bloat the working set, wreck cache locality, and cap throughput) or hand-rolling a file server that loses data on the first disk failure.

When to reach for this

Objects are large (KB to GB), written once and read many times, and you only ever fetch them whole by key — never query inside them. Photo/video stores, user uploads, backups, data-lake/ML datasets, static-site assets, log archives. The access pattern is PUT key → GET key, durability matters, and the total volume is too large or too cold to sit in a primary database.

When NOT to

Small structured records you query, filter, sort, or join — that is data-storage. Data that needs transactions, secondary indexes, or partial updates (blobs are replace-whole, not edit-in-place). Low-latency reads of tiny values (a KV cache or caching wins). A few files on one box that never grow — the local filesystem is fine; a blob store is operational overhead you do not need yet (YAGNI). Naming "object storage" for a workload that is really a database is failure mode #2.

Clarify first

  • Object size distribution — average and p99 size? (Decides chunking, multipart thresholds, and whether reads stream or buffer.) → back-of-the-envelope.
  • Read:write ratio and access recency — write-once/read-many? How fast does data go cold? (Drives tiering and CDN fronting.)
  • Durability and availability target — how many nines of durability? Can a read briefly fail or must it always succeed? (Replication vs erasure coding, multi-region.)
  • Access control — public, private, or time-limited per-object grants? (Signed URLs.)
  • Mutability and history — do objects change? Must old versions be retained (compliance, undo)? (Versioning + lifecycle.)
  • Egress profile — who reads, from where, how often? (CDN offload, egress cost.)

The options

Durability scheme (how many copies, what shape)

  • N-way replication — store N full copies on different nodes/racks/AZs. Use when objects are small, hot, and latency matters; simplest to reason about.
  • Erasure coding (EC) — split an object into k data + m parity shards; any k reconstruct it. Use for large/cold data at scale — same durability as replication at ~1.4x overhead instead of 3x. (Mechanics in references/deep-dive.md.)

Storage tier (price/latency/retrieval trade)

  • Hot/standard — millisecond reads, highest $/GB. Use for actively served objects.
  • Cool/infrequent — cheaper storage, retrieval fee/slightly higher latency. Use for backups and data read a few times a month.
  • Archive/cold — cheapest storage, minutes-to-hours retrieval. Use for compliance retention and rarely-touched data; never for anything on a request path.

Upload path

  • Single PUT — one request. Use for small objects (under the multipart threshold).
  • Multipart / resumable — split into parts, upload in parallel, retry per-part, commit on completion. Use for large objects and flaky networks; the default above the threshold.

Mutation model

  • Immutable + versioning — each write is a new version; deletes are tombstones. Use when history, undo, or accidental-overwrite protection matters.
  • Overwrite-in-place (last-writer-wins) — simplest; no history. Use when only the latest object matters and storage of old copies is waste.

Trade-offs

OptionWhat it solvesWhat it worsensChange it when
N-way replicationSimple, fast reads, fast rebuild3x+ storage costData is large/cold and cost dominates → erasure coding
Erasure codingSame durability at ~1.4x storageCPU + multi-node read on every fetch; slow small-object reads; costly rebuildObjects are small/hot and latency matters → replication
Hot tierLow-latency servingHighest $/GBData goes cold and is rarely read → cool/archive
Archive tierCheapest at-rest storageMinutes–hours to first byte; retrieval feesAnything ends up on a latency-sensitive path → hot/cool
Multipart/resumable uploadLarge files survive flaky links; parallel throughputMore client logic; orphaned parts cost moneyObjects are small → single PUT
VersioningUndo, history, overwrite protectionStorage grows silently; needs lifecycle expiryOnly latest matters → overwrite, last-writer-wins
Signed URLsOffload transfer off your app; scoped accessLeaked/over-broad URLs; clock-skew expiry bugsContent is fully public → CDN + public read

Behavior under stress

A blob store rarely "falls over" the way a database does, but it amplifies trouble in specific ways.

  • Hot object / hot prefix: a viral file or a key scheme where many writes share a prefix concentrates load on one partition. Mitigate: front hot reads with a CDN (content-delivery), randomize/hash key prefixes, replicate the hot object.
  • Metadata-index bottleneck: the index that maps key → shard locations is the real SPOF and the throughput ceiling (millions of tiny objects hurt far more than a few huge ones). *Mitig
阅读完整来源 ↗