Skip to content

Commit

Permalink
value-pairs related formatting fixes and deleting duplicate files
Browse files Browse the repository at this point in the history
  • Loading branch information
@eldarnash
eldarnash committed Jul 25, 2023
1 parent 89ce835 commit de02a8e
Show file tree
Hide file tree
Showing 6 changed files with 67 additions and 394 deletions.
185 changes: 65 additions & 120 deletions content/chapter-concepts/concepts-value-pairs/option-value-pairs/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,31 +4,16 @@ weight: 300
---
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

<table>
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<tbody>
<tr class="odd">
<td>Type:</td>
<td>parameter list of the `value-pairs()` option</td>
</tr>
<tr class="even">
<td>Default:</td>
<td>```shell
empty string
```</td>
</tr>
</tbody>
</table>
| | |
| ---------------- | ---------------- |
| Type: | parameter list of the `value-pairs()` option |
| Default: | empty string |

*Description:* The `value-pairs()` option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a `mongodb()` destination.

## Example: Using the value-pairs() option

The following example selects every available information about the log message, except for the date-related macros (`R_\*` and `S_\*`), selects the `.SDATA.meta.sequenceId` macro, and defines a new value-pair called `MSGHDR` that contains the program name and PID of the application that sent the log message.
The following example selects every available information about the log message, except for the date-related macros (`R_*` and `S_*`), selects the `.SDATA.meta.sequenceId` macro, and defines a new value-pair called `MSGHDR` that contains the program name and PID of the application that sent the log message.

```shell
value-pairs(
Expand All @@ -48,36 +33,27 @@ The following example selects the same information as the previous example, but
--pair MSGHDR="$PROGRAM[$PID]: ")
```


{{% alert title="Note" color="info" %}}

Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection).

{{% /alert %}}

The `value-pairs()` option has the following parameters. The parameters are evaluated in the following order:

1. [`scope()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})
## `value-pairs()` parameters {#value-pairs-parameters}

2. [`exclude()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})

3. [`key()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})

4. [`pair()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})

*exclude()*

Type:

Space-separated list of macros to remove from the selection created using the `scope()` option.
The `value-pairs()` option has the following parameters. The parameters are evaluated in the following order:

Default:
1. [`scope()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})
2. [`exclude()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})
3. [`key()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})
4. [`pair()`]({{< relref "/chapter-concepts/concepts-value-pairs/_index.md" >}})

empty string
### `exclude()` {#exclude}

*Description:* This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the `scope()` parameter.
| | |
| ---------------- | ---------------- |
| Type: | Space-separated list of macros to remove from the selection created using the `scope()` option. |
| Default: | empty string |

For example, the following example removes the SDATA macros from the selection.
*Description:* This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the `scope()` parameter. For example, the following example removes the SDATA macros from the selection.

```shell
value-pairs(
Expand All @@ -88,15 +64,12 @@ For example, the following example removes the SDATA macros from the selection.

The name of the macro to remove can include wildcards `(\*, ?)`. Regular expressions are not supported.

*key()*
### `key()` {#key}

Type:

Space-separated list of macros to be included in selection

Default:

empty string
| | |
| ---------------- | ---------------- |
| Type: | Space-separated list of macros to be included in the selection. |
| Default: | empty string |

*Description:* This option selects the specified macros. The selected macros will be included as `MACRONAME = MACROVALUE`, that is using `key("HOST")` will result in `HOST = $HOST`. You can use wildcards `(\*, ?)` to select multiple macros. For example:

Expand All @@ -114,17 +87,12 @@ empty string
)
```

*omit-empty-values()*



Type:

flag
### `omit-empty-values()` {#omit-empty-values}

Default:

N/A
| | |
| ---------------- | ---------------- |
| Type: | flag |
| Default: | N/A |

*Description:* If this option is specified, {{% param "product.abbrev" %}} does not include value-pairs with empty values in the output. For example: `$(format-json --scope none --omit-empty-values)` or

Expand All @@ -137,17 +105,12 @@ N/A

Available in {{% param "product.abbrev" %}} version 3.21 and later.



*pair()*

Type:

name value pairs in `"<NAME>" "<VALUE>"` format

Default:
### `pair()` {#pair}

empty string
| | |
| ---------------- | ---------------- |
| Type: | name-value pairs in `"<NAME>" "<VALUE>"` format |
| Default: | empty string |

*Description:* This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

Expand All @@ -159,46 +122,41 @@ empty string
)
```

*rekey()*
### `rekey()` {#rekey}

Type:

<pattern-to-select-names>, <list of transformations>

Default:

empty string
| | |
| ---------------- | ---------------- |
| Type: | `<pattern-to-select-names>`, `<list of transformations>` |
| Default: | empty string |

*Description:* This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the `rekey()` option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see [glob]({{< relref "/chapter-manipulating-messages/regular-expressions/reference-regexp-types/_index.md" >}}).

If you want to modify the names of several message fields, see also {{% xref "/chapter-manipulating-messages/modifying-messages/parser-map-value-pairs/_index.md" %}}.

- If `rekey()` is used within a `key()` option, the name-value pairs specified in the glob of the `key()` option are transformed.

- If `rekey()` is used outside the `key()` option, every name-value pair of the `scope()` is transformed.
- If `rekey()` is used within a `key()` option, the name-value pairs specified in the glob of the `key()` option are transformed.
- If `rekey()` is used outside the `key()` option, every name-value pair of the `scope()` is transformed.

The following transformations are available:

- `add-prefix("<my-prefix>")`
- `add-prefix("<my-prefix>")`

Adds the specified prefix to every name. For example, `rekey( add-prefix("my-prefix."))`

- `replace-prefix("<prefix-to-replace>", "<new-prefix>")`
- `replace-prefix("<prefix-to-replace>", "<new-prefix>")`

Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, `replace-prefix(".class", ".patterndb")` changes the beginning tag `.class` to `.patterndb`

This option was called `replace()` in {{% param "product.abbrev" %}} version 3.4.

- `shift("<number>")`
- `shift("<number>")`

Cuts the specified number of characters from the beginning of the name.

- `shift-levels("<number>")`

Similar to --shift, but instead of cutting characters, it cuts dot-delimited "levels" in the name (including the initial dot). For example, `--shift-levels 2` deletes the prefix up to the second dot in the name of the key: `.iptables.SRC` becomes `SRC`
- `shift-levels("<number>")`

Similar to --shift, but instead of cutting characters, it cuts dot-delimited "levels" in the name (including the initial dot). For example, `--shift-levels 2` deletes the prefix up to the second dot in the name of the key: `.iptables.SRC` becomes `SRC`

## Example: Using the rekey() option {#example-value-pairs-rekey}
#### Example: Using the rekey() option {#example-value-pairs-rekey}

The following sample selects every value-pair that begins with `.cee.`, deletes this prefix by cutting 4 characters from the names, and adds a new prefix (`events.`).

Expand All @@ -219,48 +177,35 @@ The `rekey()` option can be used with the `format-json` template-function as wel
$(format-json --rekey .cee.* --add-prefix events.)
```

### `scope()` {#scope}

*scope()*

Type:

space-separated list of macro groups to include in selection

Default:

empty string
| | |
| ---------------- | ---------------- |
| Type: | Space-separated list of macro groups to be included in the selection. |
| Default: | empty string |

*Description:* This option selects predefined groups of macros. The following groups are available:

- *nv-pairs*: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within {{% param "product.abbrev" %}} and are not originally part of the message, therefore are not included in this group.

- *dot-nv-pairs*: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, `.classifier.rule_id` and `.sdata.\*`. Macros starting with a dot character are generated within {{% param "product.abbrev" %}} and are not originally part of the message.
- *nv-pairs*: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within {{% param "product.abbrev" %}} and are not originally part of the message, therefore are not included in this group.
- *dot-nv-pairs*: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, `.classifier.rule_id` and `.sdata.\*`. Macros starting with a dot character are generated within {{% param "product.abbrev" %}} and are not originally part of the message.
- *all-nv-pairs*: Include every soft macro (name-value pair). Equivalent to using both `nv-pairs` and `dot-nv-pairs`.
- *rfc3164*: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: `$FACILITY`, `$PRIORITY`, `$HOST`, `$PROGRAM`, `$PID`, `$MESSAGE`, and `$DATE`.
- *rfc5424*: The macros that correspond to the RFC5424 (IETF-syslog) message format: `$FACILITY`, `$PRIORITY`, `$HOST`, `$PROGRAM`, `$PID`, `$MESSAGE`, `$MSGID`, `$R_DATE`, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with `.SDATA.`.

- *all-nv-pairs*: Include every soft macro (name-value pair). Equivalent to using both `nv-pairs` and `dot-nv-pairs`.

- *rfc3164*: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: `$FACILITY`, `$PRIORITY`, `$HOST`, `$PROGRAM`, `$PID`, `$MESSAGE`, and `$DATE`.

- *rfc5424*: The macros that correspond to the RFC5424 (IETF-syslog) message format: `$FACILITY`, `$PRIORITY`, `$HOST`, `$PROGRAM`, `$PID`, `$MESSAGE`, `$MSGID`, `$R_DATE`, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with `.SDATA.`.

The `rfc5424` group also has the following alias: `syslog-proto`. Note that the value of `$R_DATE` will be listed under the `DATE` key.

The `rfc5424` group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the `$SOURCEIP` macro), use the `selected-macros` group instead.

- *all-macros*: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

- *selected-macros*: Include the macros of the `rfc3164` groups, and the most commonly used metadata about the log message: the `$TAGS`, `$SOURCEIP`, and `$SEQNUM` macros.

- *sdata*: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with `.SDATA.`

- *everything*: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).
The `rfc5424` group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the `$SOURCEIP` macro), use the `selected-macros` group instead.

- *none*: Reset previously added scopes, for example, to delete automatically-added name-value pairs. The following example deletes every value-pair from the scope, and adds only the ones starting with iptables: `$(format-welf --scope none .iptables.\*)`
- *all-macros*: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).
- *selected-macros*: Include the macros of the `rfc3164` groups, and the most commonly used metadata about the log message: the `$TAGS`, `$SOURCEIP`, and `$SEQNUM` macros.
- *sdata*: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with `.SDATA.`
- *everything*: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).
- *none*: Reset previously added scopes, for example, to delete automatically-added name-value pairs. The following example deletes every value-pair from the scope, and adds only the ones starting with iptables: `$(format-welf --scope none .iptables.\*)`

For example:

```shell
value-pairs(
scope(rfc3164 selected-macros)
)

```
2 changes: 1 addition & 1 deletion ...destinations/configuring-destinations-amqp/reference-destination-amqp/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ Available in {{% param "product.abbrev" %}} version 3.21 and later.
*Description:* The username used to authenticate on the AMQP server.


{{< include-headless "chunk/destination-option-value-pairs.md" >}}
{{< include-headless "chunk/destination-option-value-pairs-content.md" >}}


## vhost() {#amqp-option-vhost}
Expand Down
2 changes: 1 addition & 1 deletion ...stinations/configuring-destinations-stomp/reference-destination-stomp/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,4 +96,4 @@ The `stomp()` destination has the following options:

*Description:* The username used to authenticate on the STOMP server.

{{< include-headless "chunk/destination-option-value-pairs.md" >}}
{{< include-headless "chunk/destination-option-value-pairs-content.md" >}}
3 changes: 0 additions & 3 deletions content/headless/chunk/destination-option-value-pairs-content.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,5 @@
*Description:* The `value-pairs()` option creates structured name-value pairs from the data and metadata of the log message. For details on using `value-pairs()`, see {{% xref "/chapter-concepts/concepts-value-pairs/_index.md" %}}.

{{% alert title="Note" color="info" %}}

Empty keys are not logged.

{{% /alert %}}

4 changes: 0 additions & 4 deletions content/headless/chunk/destination-option-value-pairs.md
View file

This file was deleted.

Loading

0 comments on commit de02a8e

Please sign in to comment.