Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support to exclude fields from concatenate_all_fields #43

Open
ypid-geberit opened this issue Oct 8, 2018 · 0 comments
Open

Support to exclude fields from concatenate_all_fields #43

ypid-geberit opened this issue Oct 8, 2018 · 0 comments

Comments

@ypid-geberit
Copy link

I would like to use all fields in an event for fingerprint calculation except a few of them which are non-deterministic when re-importing logs using the file or unix socket input for example. This is useful when the list of fields varies and is not fully known. Example use case:

fingerprint {
  concatenate_all_fields => true
  exclude => [
    "#logstash_timestamp",

    "@version",
    "path",
    "port",
    "input_type",
    "type",
    "source",

    "host",
    "@timestamp"
  ]
  target => "[@metadata][_id]"
  method => "SHA512"
  key => "XXX"
  base64encode => true
}

Hint: The @timestamp is excluded because when you re-import logs, Logstash sets the @timestamp at input stage and it is only later overwritten by the timestamp extracted from the log line. Fingerprint calculation is done as early as possible to still generate the same fingerprint even when the processing/transformation of the event is changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant