跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
数据与分析

cx-data-pipeline

Use this skill when the user asks to "set up parsing", "create parsing rule", "extract fields from logs", "regex extraction", "log parsing", "enrich logs", "add context to logs", "custom enrichment table", "lookup table", "geo enrichment", "create metric from logs", "events to metrics", "convert logs to metrics", "generate metrics from events", "recording rule", "precomputed metrics", "PromQL recording", "configure data pipeline", "transform log data", "data processing rules", "rule group", "enrichment settings", "E2M definition", "labels cardinality", "bulk delete rules", "enrichment limits", "search enrichment table", "what should I convert to metrics", "E2M not producing metrics", "E2M no series", "reduce log cost with E2M", "logs to metrics aggregation", "spans to metrics", or wants to configure how Coralogix processes, enriches, or transforms ingested data.

coralogix · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add coralogix/cx-cli --skill cx-data-pipeline

技能说明


name: cx-data-pipeline description: > Use this skill when the user asks to "set up parsing", "create parsing rule", "extract fields from logs", "regex extraction", "log parsing", "enrich logs", "add context to logs", "custom enrichment table", "lookup table", "geo enrichment", "create metric from logs", "events to metrics", "convert logs to metrics", "generate metrics from events", "recording rule", "precomputed metrics", "PromQL recording", "configure data pipeline", "transform log data", "data processing rules", "rule group", "enrichment settings", "E2M definition", "labels cardinality", "bulk delete rules", "enrichment limits", "search enrichment table", "what should I convert to metrics", "E2M not producing metrics", "E2M no series", "reduce log cost with E2M", "logs to metrics aggregation", "spans to metrics", or wants to configure how Coralogix processes, enriches, or transforms ingested data. metadata: version: "0.1.0"

Data Pipeline Skill

Use this skill when configuring how Coralogix processes, enriches, and transforms data. It covers parsing rules (extract structured fields from raw logs), enrichments (add context from lookup tables), Events2Metrics (derive metrics from log/span events), and recording rules (precompute PromQL expressions).


CLI Commands

CommandSubcommandsPurpose
cx parsing-ruleslist, get, create, update, delete, bulk-delete, usage-limitsManage log parsing rules
cx enrichmentslist, add, remove, overwrite, limit, settingsManage enrichment rules
cx enrichments customlist, get, create, update, delete, searchManage custom enrichment tables
cx e2mlist, get, create, update, delete, labels-cardinality, limitsManage Events2Metrics definitions
cx recording-ruleslist, get, create, update, deleteManage Prometheus recording rule groups

Key flags:

  • All create/update operations use --from-file <path> (or - for stdin)
  • All commands support -o json for structured output and -p <profile> for profile selection
  • cx parsing-rules update and cx recording-rules update require both --from-file and the rule group ID
  • cx enrichments custom search requires --id <table-id> and --query <text>
  • cx parsing-rules bulk-delete requires --ids <id1> <id2> ...

Working with JSON Payloads

These commands use complex JSON structures. Always template from an existing resource to avoid format errors:

# 1. Get an existing resource as a template
cx parsing-rules get <rule-group-id> -o json > template.json

# 2. Modify the template (change fields, remove the ID for create operations)

# 3. Create or update
cx parsing-rules create --from-file template.json
cx parsing-rules update --from-file template.json <rule-group-id>

This pattern applies to all create/update operations across all 4 commands. It prevents payload format errors that are the #1 cause of failed attempts.


Parsing Rules Workflow

1. List Existing Rules

cx parsing-rules list -o json
cx parsing-rules list -o json | jq '[.[] | {id, name, enabled, rule_count: (.rules | length)}]'

2. Get a Template

cx parsing-rules get <existing-rule-group-id> -o json > rule-template.json

3. Create New Rule Group

Edit the template for your new service, then:

cx parsing-rules create --from-file rule-template.json

4. Verify Parsing

Query recent logs to confirm fields are extracted (load cx-telemetry-querying for log querying):

cx logs 'source logs | filter $d.subsystem == "my-service" | limit 10' -o json

5. Check Usage Limits

cx parsing-rules usage-limits -o json

Enrichment Workflow

1. List Enrichment Rules

cx enrichments list -o json
cx enrichments settings -o json
cx enrichments limit -o json

2. Create Custom Enrichment Table (if needed)

cx enrichments custom list -o json
cx enrichments custom create --from-file table-definition.json

table-definition.json must use the v5 JSON shape (inline file content, not multipart file=@...):

{
  "name": "IP Lookup",
  "description": "Maps IPs to locations",
  "file": {
    "textual": "ip,city\n1.2.3.4,London",
    "extension": "csv",
    "name": "lookup.csv",
    "size": 24
  }
}

For updates, include customEnrichmentId (number) plus the same fields.

3. Add Enrichment Rules

cx enrichments add --from-file enrichment-rules.json

enrichment-rules.json must use requestEnrichments (not enrichments from list output). Each enrichmentType is an object, not a string:

{
  "requestEnrichments": [
    {
      "fieldName": "sourceIPs",
      "enrichmentType": { "geoIp": { "withAsn": true } }
    }
  ]
}

Other types: {"aws": {"resourceType": "ec2"}}, {"suspiciousIp": {}}, {"customEnrichment": {"id": 1}}.

4. Search Custom Table Data

cx enrichments custom search --id <table-id> --query "search term"

5. Verify Enriched Fields

Query logs on hot storage (FrequentSearch tier) to confirm enriched fields appear. Avoid querying archive for verification - ingestion delays can cause false negatives.

cx logs 'source logs | filter $d.enriched_field != null | limit 5' -o json

Events2Metrics Workflow

E2M derives Prometheus metrics from log/span events. See references/e2m-schemas.md for the full JSON wire format, enums, and cardinality rules.

How E2M is computed (read this first)

E2M aggregates events as they stream through the real-time ingestion pipeline into metric series (~1-min resolution). It is forward-only — metrics start from the moment the E2M is created; there is no backfill.

All ingested data flows through the pipeline; a TCO policy routes each stream into a tier, and the tier decides what

阅读完整来源 ↗