-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Closed
Labels
:Data Management/Ingest NodeExecution or management of Ingest Pipelines including GeoIPExecution or management of Ingest Pipelines including GeoIP>enhancementTeam:Data ManagementMeta label for data/management teamMeta label for data/management team
Description
Description
It's common for the message in a log event to actually be a JSON itself. Specifically, there are shippers that produce ECS-JSON.
We would like to handle this automatically through the normalize_for_stream ingest processor. The idea is to add an additional step, so that it now does the following:
- If it's OTel data: just use as is
- If it's not OTel:
- apply a cheap check whether
messageis a JSON-encoded string (e.g. if it starts with{and ends with}) - If yes:
- Parse the
messageas JSON* - Apply a cheap check whether the resulting object is ECS (e.g. if contains a
@timestampkey) - If it's ECS: merge the resulting object back into the root of the document*
- If not: add the resulting object as is as the value of the
body.structuredfield
- Parse the
- Proceed with namespacing/normalization as before
- apply a cheap check whether
* JSON parsing and merge should behave exactly exactly as defined in logs@json-pipeline
@dakrone is it a problem to have dependencies between one processor and another? More specifically, is it possible for the NormalizeForStreamProcessor to use JsonProcessor#apply?
Metadata
Metadata
Assignees
Labels
:Data Management/Ingest NodeExecution or management of Ingest Pipelines including GeoIPExecution or management of Ingest Pipelines including GeoIP>enhancementTeam:Data ManagementMeta label for data/management teamMeta label for data/management team