cx-data-pipeline
Use this skill when the user asks to "set up parsing", "create parsing rule", "extract fields from logs", "regex extraction", "log parsing", "enrich logs", "add context to logs", "custom enrichment table", "lookup table", "geo enrichment", "create metric from logs", "events to metrics", "convert logs to metrics", "generate metrics from events", "recording rule", "precomputed metrics", "PromQL recording", "configure data pipeline", "transform log data", "data processing rules", "rule group", "enrichment settings", "E2M definition", "labels cardinality", "bulk delete rules", "enrichment limits", "search enrichment table", "what should I convert to metrics", "E2M not producing metrics", "E2M no series", "reduce log cost with E2M", "logs to metrics aggregation", "spans to metrics", or wants to configure how Coralogix processes, enriches, or transforms ingested data.
coralogix · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add coralogix/cx-cli --skill cx-data-pipelineAbout this skill
name: cx-data-pipeline description: > Use this skill when the user asks to "set up parsing", "create parsing rule", "extract fields from logs", "regex extraction", "log parsing", "enrich logs", "add context to logs", "custom enrichment table", "lookup table", "geo enrichment", "create metric from logs", "events to metrics", "convert logs to metrics", "generate metrics from events", "recording rule", "precomputed metrics", "PromQL recording", "configure data pipeline", "transform log data", "data processing rules", "rule group", "enrichment settings", "E2M definition", "labels cardinality", "bulk delete rules", "enrichment limits", "search enrichment table", "what should I convert to metrics", "E2M not producing metrics", "E2M no series", "reduce log cost with E2M", "logs to metrics aggregation", "spans to metrics", or wants to configure how Coralogix processes, enriches, or transforms ingested data. metadata: version: "0.1.0"
Data Pipeline Skill
Use this skill when configuring how Coralogix processes, enriches, and transforms data. It covers parsing rules (extract structured fields from raw logs), enrichments (add context from lookup tables), Events2Metrics (derive metrics from log/span events), and recording rules (precompute PromQL expressions).
CLI Commands
| Command | Subcommands | Purpose |
|---|---|---|
cx parsing-rules | list, get, create, update, delete, bulk-delete, usage-limits | Manage log parsing rules |
cx enrichments | list, add, remove, overwrite, limit, settings | Manage enrichment rules |
cx enrichments custom | list, get, create, update, delete, search | Manage custom enrichment tables |
cx e2m | list, get, create, update, delete, labels-cardinality, limits | Manage Events2Metrics definitions |
cx recording-rules | list, get, create, update, delete | Manage Prometheus recording rule groups |
Key flags:
- All create/update operations use
--from-file <path>(or-for stdin) - All commands support
-o jsonfor structured output and-p <profile>for profile selection cx parsing-rules updateandcx recording-rules updaterequire both--from-fileand the rule group IDcx enrichments custom searchrequires--id <table-id>and--query <text>cx parsing-rules bulk-deleterequires--ids <id1> <id2> ...
Working with JSON Payloads
These commands use complex JSON structures. Always template from an existing resource to avoid format errors:
# 1. Get an existing resource as a template
cx parsing-rules get <rule-group-id> -o json > template.json
# 2. Modify the template (change fields, remove the ID for create operations)
# 3. Create or update
cx parsing-rules create --from-file template.json
cx parsing-rules update --from-file template.json <rule-group-id>
This pattern applies to all create/update operations across all 4 commands. It prevents payload format errors that are the #1 cause of failed attempts.
Parsing Rules Workflow
1. List Existing Rules
cx parsing-rules list -o json
cx parsing-rules list -o json | jq '[.[] | {id, name, enabled, rule_count: (.rules | length)}]'
2. Get a Template
cx parsing-rules get <existing-rule-group-id> -o json > rule-template.json
3. Create New Rule Group
Edit the template for your new service, then:
cx parsing-rules create --from-file rule-template.json
4. Verify Parsing
Query recent logs to confirm fields are extracted (load cx-telemetry-querying for log querying):
cx logs 'source logs | filter $d.subsystem == "my-service" | limit 10' -o json
5. Check Usage Limits
cx parsing-rules usage-limits -o json
Enrichment Workflow
1. List Enrichment Rules
cx enrichments list -o json
cx enrichments settings -o json
cx enrichments limit -o json
2. Create Custom Enrichment Table (if needed)
cx enrichments custom list -o json
cx enrichments custom create --from-file table-definition.json
table-definition.json must use the v5 JSON shape (inline file content, not multipart file=@...):
{
"name": "IP Lookup",
"description": "Maps IPs to locations",
"file": {
"textual": "ip,city\n1.2.3.4,London",
"extension": "csv",
"name": "lookup.csv",
"size": 24
}
}
For updates, include customEnrichmentId (number) plus the same fields.
3. Add Enrichment Rules
cx enrichments add --from-file enrichment-rules.json
enrichment-rules.json must use requestEnrichments (not enrichments from list output). Each enrichmentType is an object, not a string:
{
"requestEnrichments": [
{
"fieldName": "sourceIPs",
"enrichmentType": { "geoIp": { "withAsn": true } }
}
]
}
Other types: {"aws": {"resourceType": "ec2"}}, {"suspiciousIp": {}}, {"customEnrichment": {"id": 1}}.
4. Search Custom Table Data
cx enrichments custom search --id <table-id> --query "search term"
5. Verify Enriched Fields
Query logs on hot storage (FrequentSearch tier) to confirm enriched fields appear. Avoid querying archive for verification - ingestion delays can cause false negatives.
cx logs 'source logs | filter $d.enriched_field != null | limit 5' -o json
Events2Metrics Workflow
E2M derives Prometheus metrics from log/span events. See references/e2m-schemas.md for the full JSON wire format, enums, and cardinality rules.
How E2M is computed (read this first)
E2M aggregates events as they stream through the real-time ingestion pipeline into metric series (~1-min resolution). It is forward-only — metrics start from the moment the E2M is created; there is no backfill.
All ingested data flows through the pipeline; a TCO policy routes each stream into a tier, and the tier decides what
Read the complete source ↗