+
+
+
+
+
+
+
+ 04 · Pipeline
+ Parse logs in the OpenTelemetry Collector
+
+ The OpenTelemetry Collector is a program you run next to your services. It receives logs, metrics and traces, processes them and exports them to a backend. Its otelcol-contrib build, also published as the Docker image otel/opentelemetry-collector-contrib, includes the file_log receiver, which tails files and turns each line into a log record.
+
+
+
+
+ The receiver takes a list of operators that run on every record, top to bottom. Parsers write their fields to the record's attributes and leave the raw line in the body. Processors such as transform run after the receiver, and an exporter sends the result on:
+
+
+ {pipeline.map((p) => (
+ - {p.name}{p.detail}
+ ))}
+
+
+
+
+ | Operator | Does |
|---|
+
+
+ regex_parser | Matches a regex; each named group becomes an attribute |
+ json_parser, key_value_parser, syslog_parser | Parse JSON, key=value pairs and syslog |
+ time_parser, severity_parser | Set the record's timestamp and severity from a field. Every parser also accepts them as inline timestamp and severity blocks. |
+ move, remove | Rename or delete a field |
+ recombine | Joins records that were split before they reached the receiver, such as long container log lines |
+
+
+
+ When an operator fails on a record, its on_error setting decides what happens next. The default, send, logs the error and passes the record on unchanged. send_quiet does the same at debug level, and drop discards the record.
+
+
+
filelog is now file_log
+
+ Collector 0.149 renamed the receiver from filelog to file_log. Newer versions still accept filelog and log a deprecation warning; older ones only know filelog. The operators kept their names.
+
+
+ This config parses the nginx line from the top of the page into OpenTelemetry attribute names and exports the records to Maple. Save it as config.yaml:
+
+
+
+ -
+ 01
+
Parse, timestamp and severity in one operator
regex_parser captures the groups. Its timestamp block reads 19/Oct/2026:14:02:11 +0000 with the layout %d/%b/%Y:%H:%M:%S %z. The severity block maps 100 to 399 to info, then the 4xx and 5xx placeholders to warn and error. Placeholders only exist from 2xx up, so a 101 WebSocket upgrade needs the numeric range. overwrite_text: true stores INFO as the severity text instead of the raw 200.
+ -
+ 02
+
Rename to OpenTelemetry names
Regex group names can't contain dots, so each move renames a group to its semantic-convention name, such as status to http.response.status_code. remove drops time_local, which is now the record's timestamp.
+ -
+ 03
+
Let unmatched lines through
A line the regex can't match is logged once by regex_parser and sent on with its raw body. The later operators use on_error: send_quiet, so that line doesn't log another error for every field it never had.
+ -
+ 04
+
Fix types and name the service
The transform processor drops the body size when it is -, which Apache writes for empty responses, converts status and size to integers with Int(), and sets service.name to nginx. The exporter block is the one from Maple's Hosts setup.
+
+
+ file_log starts at the end of the file, so only new lines are read; set start_at: beginning to backfill. The Collector needs read access to /var/log/nginx, which on Debian and Ubuntu means the adm group. To keep file offsets across restarts, add the file_storage extension and point the receiver's storage setting at it.
+
+
+
+
+ The debug exporter prints each record instead of sending it. The Collector merges every --config file it is given, so a second file can swap the exporter without editing config.yaml:
+
+
+
+ The 502 request from the sample log comes out as:
+
+
+
+
+ Every other format follows the same steps with a different parser. Each tab shows a sample line, the receiver config and the record the Collector printed for it:
+
+ ({ id, label }))}>
+ {formatTabs.map((t) => (
+
+ Line
+ {t.line}
+ Receiver
+
+ Record
+
+
+
+ ))}
+
+
+
+
+ Operators only run inside receivers that read text, such as file_log. Logs that reach the Collector as OTLP, from an SDK, a Kubernetes agent or another Collector, skip them. For those, the transform processor runs OpenTelemetry Transformation Language (OTTL) statements, and OTTL has its own parsing functions:
+
+
+
+ ParseJSON, ParseKeyValue and ExtractGrokPatterns each return a map, and merge_maps copies it into the attributes. The where clause picks the records each statement applies to, and error_mode: ignore skips a statement that fails on one record. The grok statement uses go-grok's field names, such as source.address and url.original, and already types the status and size as integers.
+
+
+
+
+
+
+
+
+ Sources and further reading
+
+
+