Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Other skills

messaging-streaming

This skill should be used when the user designs a "message queue", reaches for "Kafka", "RabbitMQ", "SQS", "Kinesis", "pub/sub", or "event-driven" architecture, asks about "async processing", "background jobs", "stream processing", or wrestles with "exactly-once vs at-least-once", "delivery guarantees", "message ordering", "duplicate handling / dedup", "dead letter queue", "backpressure", "saga orchestration", or "durable workflow". Use it whenever a slow or spiky operation should move off the request path, or two services must be decoupled, even if the user doesn't say "queue".

proyecto26 · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add proyecto26/system-design-skills --skill messaging-streaming

About this skill


name: messaging-streaming description: This skill should be used when the user designs a "message queue", reaches for "Kafka", "RabbitMQ", "SQS", "Kinesis", "pub/sub", or "event-driven" architecture, asks about "async processing", "background jobs", "stream processing", or wrestles with "exactly-once vs at-least-once", "delivery guarantees", "message ordering", "duplicate handling / dedup", "dead letter queue", "backpressure", "saga orchestration", or "durable workflow". Use it whenever a slow or spiky operation should move off the request path, or two services must be decoupled, even if the user doesn't say "queue".

Messaging & Streaming

Move work off the synchronous request path and decouple producers from consumers, so a slow, spiky, or failure-prone operation doesn't block the caller. Getting this wrong is subtle: a queue silently changes the delivery and ordering guarantees, and under load it can absorb a spike gracefully or become the thing that hides a meltdown until the backlog is unrecoverable.

When to reach for this

A step is too slow to do inline (image transcode, fan-out, third-party call); the write path is spiky and needs a buffer to smooth bursts (→ back-of-the-envelope for the spike factor); two services must be decoupled so one can fail or deploy independently; or many consumers need the same event stream. The async hand-off buys responsiveness, isolation, and elasticity (scale producers and consumers separately).

When NOT to

The caller needs the result now to proceed (a synchronous read, a balance check before confirming) — a queue only adds latency and a place for work to get lost. Strong read-after-write within one request. Trivial in-process work that a function call handles. Don't add a broker before a number or a coupling problem justifies it (YAGNI): it's a new stateful system to operate, monitor, and reason about under failure. "We'll need Kafka eventually" is name-dropping, not a requirement.

Clarify first

  • Sync or async? Does the caller need the result inline, or is fire-and-react acceptable? This decides whether a queue belongs here at all.
  • Delivery guarantee needed — is a dropped message acceptable (at-most-once), or must every message be processed (at-least-once + idempotent consumers)?
  • Ordering — must messages be processed in order, globally or per-key (per user, per account)? Global ordering is expensive; per-key usually suffices.
  • Throughput and retention — messages/sec at peak, and how long must they be replayable? (→ back-of-the-envelope.) One-shot work vs. a replayable log.
  • Consumer count and pattern — one worker pool draining a job, or many independent subscribers each reading every event?
  • Failure handling — what happens to a message that keeps failing? Where does it go, and who looks at it?

The options

Sync vs. async — settle this before picking a tool. Stay synchronous when the caller needs the result to continue and the call is fast and reliable; a direct request is simpler to build, trace, and reason about. Go async when the work is slow, spiky, fan-out-heavy, or the caller can react to the result later — this trades immediate consistency and an easy stack trace for responsiveness and isolation. Only after choosing async do the options below apply. Building request/reply over a queue to fake a synchronous answer is a smell — a direct call is the better design.

Queue (work/task queue) — one logical consumer group competes to drain messages; a message is delivered to one worker and removed when acked. Use when there are background jobs or commands to process exactly once-ish, and workers should scale to drain a backlog.

Pub/sub (fan-out) — each subscriber gets its own copy of every message; producers don't know subscribers. Use when multiple independent consumers react to the same event (notify, index, audit) and loose coupling matters.

Stream (durable, replayable log) — an append-only, partitioned, retained log; consumers track their own offset and can replay history. Use when the design needs ordering per partition, multiple consumers at different positions, event sourcing, or reprocessing (→ data-storage for event sourcing/outbox).

Durable workflow (orchestration engine) — code that survives process crashes; the engine persists each step and resumes where it left off, with built-in retries, timers, and compensation. Use when a multi-step process with retries, human delays, and rollback (a saga) would otherwise become a fragile hand-rolled mesh of queues, state flags, and cron jobs.

Delivery semantics cut across all of these: at-most-once (fire and forget, may drop), at-least-once (retries until acked, may duplicate — the practical default), exactly-once (no loss, no dup). True end-to-end exactly-once is impractical: a broker's "EOS" (e.g. Kafka) is intra-cluster only, so across systems you always implement it as at-least-once + idempotent/deduped consumers (→ api-design idempotency keys). See references/deep-dive.md for the mechanics.

Trade-offs

OptionWhat it solvesWhat it worsensChange it when
Queue (work queue)Decouples + buffers; scale workers to drain backlogAt-least-once means duplicates; ordering not guaranteed across workersReplay or many independent consumers are needed → stream/pub-sub
Pub/sub (fan-out)One event, N decoupled reactions; add consumers freelyNo replay (transient); slow subscriber can lag or drop; fan-out amplifies loadHistory/replay or per-key ordering is needed → stream
Stream (log)Ordering per partition, replay, multi-consumer, event sourcingOperationally heavier; partition key is a hot-shard risk; consumers must manage offsetsSimple one-shot jobs don't need a log → queue
Durable workflowCrash-safe long-running sagas; retries/compensation built inNew runtime + programming model; latency overhead; lo
Read the complete source ↗