Is your feature request related to a problem? Please describe.
When reading container logs via the file input plugin (Docker JSON log format, e.g. /var/log/containers/*.log), long log lines are split into multiple file records (~16KB per chunk). Each chunk becomes a separate event with a partial log field.
This breaks downstream processing: json_decode / decode fail on incomplete JSON, and fields like msg_body or logs_type are never parsed correctly.
The k8s input plugin already handles this via built-in k8s-multiline, but there is no equivalent for users who read the same files with the file input plugin.
Describe the solution you'd like
Add a new action plugin file_multiline that:
Joins sequential chunks from the same file stream into one event
Works on a configurable field (default: log)
Treats a chunk as complete when the field value ends with a real newline character \n
Supports split_event_size for very long messages (similar to k8s-multiline)
Is intended to run before json_decode / decode in the actions pipeline
Example:
actions:
- type: file_multiline
field: log
split_event_size: 1000000
- type: json_decode
field: log
Describe alternatives you've considered
join action with regex — not suitable: split chunks have no predictable start/continue pattern; content is arbitrary JSON/text inside log.
k8s input instead of file — works, but requires k8s-specific setup and metadata; not ideal when only file tailing is needed.
Pipeline event_timeout only — does not solve joining; only limits wait time for incomplete buffers.
Fixing only in application — avoids splitting at source, but does not help when logs are already written by Docker/CRI.
Is your feature request related to a problem? Please describe.
When reading container logs via the file input plugin (Docker JSON log format, e.g. /var/log/containers/*.log), long log lines are split into multiple file records (~16KB per chunk). Each chunk becomes a separate event with a partial log field.
This breaks downstream processing: json_decode / decode fail on incomplete JSON, and fields like msg_body or logs_type are never parsed correctly.
The k8s input plugin already handles this via built-in k8s-multiline, but there is no equivalent for users who read the same files with the file input plugin.
Describe the solution you'd like
Add a new action plugin file_multiline that:
Joins sequential chunks from the same file stream into one event
Works on a configurable field (default: log)
Treats a chunk as complete when the field value ends with a real newline character \n
Supports split_event_size for very long messages (similar to k8s-multiline)
Is intended to run before json_decode / decode in the actions pipeline
Example:
actions:
field: log
split_event_size: 1000000
field: log
Describe alternatives you've considered
join action with regex — not suitable: split chunks have no predictable start/continue pattern; content is arbitrary JSON/text inside log.
k8s input instead of file — works, but requires k8s-specific setup and metadata; not ideal when only file tailing is needed.
Pipeline event_timeout only — does not solve joining; only limits wait time for incomplete buffers.
Fixing only in application — avoids splitting at source, but does not help when logs are already written by Docker/CRI.