跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

distributed-logging

This skill should be used when the user designs "distributed logging", "log aggregation", "centralized logs", an "ELK" or "EFK" stack, "log shipping", "structured logging", a "correlation ID" or "trace ID" in logs, "log retention", or "high-volume log ingest". It gives the collect → buffer → ship → index → store → retain pipeline, sampling, ordering, and cold-storage tiering. Use it whenever many services emit logs that must be searched in one place under load, even if the user doesn't say "logging pipeline".

proyecto26 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add proyecto26/system-design-skills --skill distributed-logging

技能说明


name: distributed-logging description: This skill should be used when the user designs "distributed logging", "log aggregation", "centralized logs", an "ELK" or "EFK" stack, "log shipping", "structured logging", a "correlation ID" or "trace ID" in logs, "log retention", or "high-volume log ingest". It gives the collect → buffer → ship → index → store → retain pipeline, sampling, ordering, and cold-storage tiering. Use it whenever many services emit logs that must be searched in one place under load, even if the user doesn't say "logging pipeline".

Distributed logging

Move logs from thousands of processes into one searchable place, fast enough to debug a live incident and cheap enough to keep for months. Getting it wrong is a classic "ignore failure" miss: the logging pipeline is itself a distributed system that buckles under the exact traffic spike you most need it during, and a naive design either drops the evidence or takes down the app it instruments.

When to reach for this

More than one process emits logs and someone needs to search them together; an incident requires correlating a request across services; log volume has outgrown grep on a box; or compliance demands retention. The pipeline buys central search, cross-service correlation, and a durable record decoupled from any single host.

When NOT to

A single service on one host where journald + log rotation is enough — a full pipeline is pure operational overhead (YAGNI). Numeric time-series questions ("what is p99 latency", "is error rate up") belong to metrics, not log scans — that is observability's job; logs answer "what exactly happened to this request". Don't ship every debug line at full volume before a number shows the volume justifies the cost; sample first.

Clarify first

  • Volume and peak — lines/sec and bytes/sec, average and peak (→ back-of-the-envelope). This sizes every stage.
  • Structured or free-text — can producers emit JSON now, or is there legacy text to parse?
  • Query latency need — interactive search in seconds (hot index) vs. occasional forensic/audit reads (cold archive)?
  • Retention + compliance — how long hot, how long cold, any legal hold or PII redaction requirement?
  • Loss tolerance — may logs be dropped/sampled under overload, or is every line evidence (audit/financial)?

The options

Collection (agent on the host)

  • Sidecar/node agent (Fluentd, Fluent Bit, Vector, Filebeat): tails files or reads stdout, adds metadata, ships out. Use when apps log to files/stdout and you want app code untouched — the default.
  • Direct-to-bus SDK: the app writes structured events straight to a transport. Use when you control the code and want exact structure, accepting tighter coupling.

Transport / buffer (the shock absorber)

  • Agent-side buffer + direct ship to indexer: simplest; agent disk-buffers and retries. Use at low-to-moderate volume with one consumer.
  • Durable log bus (messaging-streaming, e.g. Kafka): producers write to a partitioned bus; indexers consume at their own pace. Use at high volume or when multiple sinks (search, archive, analytics) read the same stream.

Index / store (the search backend)

  • Full-text index (Elasticsearch/OpenSearch — the "E" in ELK/EFK): rich queries, expensive RAM/disk. Use when interactive field search matters.
  • Label-indexed store (Loki): indexes only labels, stores log bodies compressed; much cheaper, grep-style queries. Use for high volume where you mostly filter by service/label then scan.

Retention / tiering

  • Hot index → cold object store (blob-store): keep days of searchable data hot, roll older data to compressed objects. Use whenever retention exceeds the hot window economically (almost always).

Trade-offs

OptionWhat it solvesWhat it worsensChange it when
Node agent (Fluent Bit/Vector)No app changes; central metadata + routingOne more daemon per host; parsing CPU; agent can lagYou need exact structure → emit structured events from the app
Direct-to-bus SDKClean structured events, no file parseCouples app to transport; app blocks/loses logs if bus is downCoupling/availability hurts → go back to agent + local buffer
Agent buffer → direct indexerFewest moving partsIndexer backpressure hits producers; no replay; one sinkVolume spikes or you need >1 sink → add a log bus
Durable log bus (Kafka)Absorbs spikes, decouples, replay, fan-outExtra system to run; ordering only per-partition; costVolume is low and single-sink → drop the bus
Full-text index (ES/OpenSearch)Fast rich field searchRAM/disk hungry; mapping explosions; costly at scaleCost dominates and queries are label-filtered → Loki
Label-indexed (Loki)Cheap storage at high volumeWeak full-text; slow on high-cardinality scansYou truly need arbitrary field search → full-text index
Hot index + cold archiveCheap long retentionCold reads are slow/manual to rehydrateForensic reads on old data must be fast → widen hot window

Behavior under stress

The pipeline's failure mode is that incidents generate log spikes — an outage emits floods of errors and stack traces exactly when the pipeline is busiest, so it must degrade without amplifying the outage.

  • Producer backpressure: if the indexer slows and agents ship synchronously, log calls can block the app. Mitigate: bounded local buffer with drop-newest / sample on overflow, never block the request path. A durable bus moves the backlog off the hosts (backpressure + DLQ semantics are owned by messaging-streaming).
  • Volume amplification: one bad deploy logging at debug can 100× volume and blow the index. Mitigate: per-service rate caps at the agent (rate limiting is owned by resilience-failure), dynamic sampling, alerts on ingest bytes/sec.
  • Index hot-shard / mapping explosion: high-cardinal
阅读完整来源 ↗