Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some filterx fixes #74

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions content/filterx/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ filterx {
The same is true for complex objects, like JSON, for example:

```shell
js = json_object({
js = json({
"key": "value",
"second-key": "another-value"
});
Expand Down Expand Up @@ -145,7 +145,7 @@ Variables can have the following types. All of these types have a matching funct
- `dict`
- `double`
- `int`
- [`json, json_object`]({{< relref "/filterx/function-reference.md#json" >}}) and [`json_array`]({{< relref "/filterx/function-reference.md#json-array" >}}) for JSON or JSON-like objects. The `json` type is an alias for the `json_object` type.
- [`json`]({{< relref "/filterx/function-reference.md#json" >}}) and [`json_array`]({{< relref "/filterx/function-reference.md#json-array" >}}) for JSON or JSON-like objects.
- `list`
- `null`
- `otel_array`
Expand Down Expand Up @@ -254,7 +254,7 @@ my_list = []; # Creates an empty list (which defaults to a JSON list)
my_array = {}; # Creates an empty dictionary (which defaults to a JSON object)

my_list2 = json_array(); # Creates an empty JSON list
my_array2 = json_object(); # Creates an empty JSON object. json() is an alias for json_object()
my_array2 = json(); # Creates an empty JSON object.
```

You can add elements to lists and dictionaries like this:
Expand Down Expand Up @@ -287,7 +287,7 @@ ${MESSAGE} = list;

In all three cases, the value of `${MESSAGE}` is the same JSON array: `["first_element", "second_element", "third_element"]`.

You can define JSON objects using the `json_object` type, for example:
You can define JSON objects using the `json()` type, for example:

```shell
js1 = json();
Expand All @@ -300,14 +300,14 @@ js1 += {
}
};

js2 = json_object({"key": "value"})
js2 = json({"key": "value"})
```

Naturally, you can assign values from other variables to an object, for example:

```shell
js = json_array(["foo", "bar", "baz"]);
${MESSAGE} = json_object({
${MESSAGE} = json({
"key": "value",
"list": list
});
Expand All @@ -316,7 +316,7 @@ ${MESSAGE} = json_object({
or

```shell
js = json_object({
js = json({
"key": ${MY-NAME-VALUE-PAIR},
"key-from-expression": isset(${HOST}) ? ${HOST} : "default-hostname",
"list": list
Expand Down
41 changes: 19 additions & 22 deletions content/filterx/function-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Functions have arguments that can be either mandatory or optional.

## cache_json_file {#cache-json-file}

Load the contents of an external JSON file in an efficient manner. You can use this to lookup contextual information. (Basically, this is a FilterX-specific implementation of the [`add-contextual-data() functionality`]({{< relref "/chapter-enrich-data/data-enrichment-add-contextual-data/_index.md" >}}).)
Load the contents of an external JSON file in an efficient manner. You can use this function to lookup contextual information. (Basically, this is a FilterX-specific implementation of the [`add-contextual-data() functionality`]({{< relref "/chapter-enrich-data/data-enrichment-add-contextual-data/_index.md" >}}).)

Usage: `cache_json_file("/path/to/file.json")`

Expand Down Expand Up @@ -61,8 +61,8 @@ date = datetime("1701350398.123000+01:00");

Usually, you use the [strptime](#strptime) FilterX function to create datetime values. Alternatively, you can cast an integer, double, string, or isodate variable into datetime with the `datetime()` FilterX function. Note that:

- When casting from an integer, the integer is the number of microseconds elapsed since the UNIX epoch (January 1, 1970 12:00:00 AM).
- When casting from a double, the double is the number of seconds elapsed since the UNIX epoch (January 1, 1970 12:00:00 AM). (The part before the floating points is the seconds, the part after the floating point is the microseconds.)
- When casting from an integer, the integer is the number of microseconds elapsed since the UNIX epoch (00:00:00 UTC on 1 January 1970).
- When casting from a double, the double is the number of seconds elapsed since the UNIX epoch (00:00:00 UTC on 1 January 1970). (The part before the floating points is the seconds, the part after the floating point is the microseconds.)
- When casting from a string, the string (for example, `1701350398.123000+01:00`) is interpreted as: `<the number of seconds elapsed since the UNIX epoch>.<microseconds>+<timezone relative to UTC (GMT +00:00)>`

## flatten
Expand Down Expand Up @@ -90,7 +90,7 @@ Only the input is mandatory, other arguments are optional. Note that the delimit

By default, the delimiter is the comma (`delimiter=","`), the `columns` and `default_value` are empty.

If the `columns` option is set, {{< product >}} checks that the number of entries in the input data matches the number of columns. If there are fewer entries, it adds the `default_value` to the missing entries.
If the `columns` option is set, {{< product >}} checks that the number of fields or entries in the input data matches the number of columns. If there are fewer items, it adds the `default_value` to the missing entries.

## format_kv {#format-kv}

Expand Down Expand Up @@ -137,18 +137,17 @@ Usage: `istype(object, "type_str")`
For example:

```shell
istype({"key": "value"}, "json_object"); # True
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a slight modification, we can keep this:

    obj = json();
    istype(obj, "json_object"); # True

istype(${PID}, "string");
istype(my-local-json-object.mylist, "json_array");
```

If the object doesn't exist, `istype()` returns with an error, causing the FilterX statement to become false, and logs an error message to the `internal()` source of {{< product >}}.

## json, json_object {#json}
## json {#json}

Cast a value into a JSON object. `json_object()` is an alias for `json()`.
Cast a value into a JSON object.

Usage: `json(<string or expression to cast as json>)`
Usage: `json(<string or expression to cast to json>)`

For example:

Expand All @@ -160,7 +159,7 @@ js = json({"key": "value"});

Cast a value into a JSON array.

Usage: `json_array(<string or expression to cast as json array>)`
Usage: `json_array(<string or expression to cast to json array>)`

For example:

Expand All @@ -176,7 +175,7 @@ Usage: `len(object)`

## lower

Converts a string into lowercase characters.
Converts all characters of a string lowercase characters.

Usage: `lower(string)`

Expand All @@ -202,19 +201,19 @@ Creates an [OpenTelemetry scope object]({{< relref "/filterx/filterx-otel/_index

## parse_csv {#parse-csv}

Separate a comma-separated or similar string.
Split a comma-separated or similar string.

Usage: `parse_csv(msg_str [columns=json_array, delimiter=string, string_delimiters=json_array, dialect=string, strip_whitespace=boolean, greedy=boolean])`

For details, see {{% xref "/filterx/filterx-parsing/csv/_index.md" %}}.

## parse_kv {#parse-kv}

Separate a string consisting of whitespace or comma-separated `key=value` pairs (for example, WELF-formatted messages).
Split a string consisting of whitespace or comma-separated `key=value` pairs (for example, WELF-formatted messages).

Usage: `parse_kv(msg, value_separator="=", pair_separator=", ", stray_words_key="stray_words")`

The `value_separator` must be a single-character string. The `pair_separator` must be a string.
The `value_separator` must be a single character. The `pair_separator` can consist of multiple characters.

For details, see {{% xref "/filterx/filterx-parsing/key-value-parser/_index.md" %}}.

Expand Down Expand Up @@ -255,7 +254,7 @@ ${MY-LIST}.named = regexp_search("first-word second-part third", /(?<one>first-w

### Mixed match groups

If you use mixed (some named, some unnamed) groups, the output is a dictionary, where {{< product >}} automatically assigns a key to the unnamed groups. For example:
If you use mixed (some named, some unnamed) groups in your regular expression, the output is a dictionary, where {{< product >}} automatically assigns a key to the unnamed groups. For example:

```shell
${MY-LIST} = json(); # Creates an empty JSON object
Expand Down Expand Up @@ -286,19 +285,17 @@ regexp_subst(${MESSAGE}, "IP", "IP-Address", global=true);

{{< include-headless "chunk/filterx-regexp-notes.md" >}}

For a case sensitive search, use the `ignorecase=true` option.

### Options

You can use the following flags with the `regexp_subst` function:

- `global=true`:

Replace every occurrence of the search string.
Replace every match of the regular expression, not only the first one.

- `ignorecase=true`:

Do case insensitive search.
Do case insensitive match.

- `jit=true`:

Expand All @@ -310,7 +307,7 @@ You can use the following flags with the `regexp_subst` function:

## string

Cast a value into a string. Note currently {{< product >}} evaluates strings and executes [template functions]({{< relref "/filterx/_index.md#template-functions" >}}) and template expressions. In the future, template evaluation will be moved to a separate FilterX function.
Cast a value into a string. Note that currently {{< product >}} evaluates strings and executes [template functions]({{< relref "/filterx/_index.md#template-functions" >}}) and template expressions within the strings. In the future, template evaluation will be moved to a separate FilterX function.

Usage: `string(<string or expression to cast>)`

Expand Down Expand Up @@ -340,11 +337,11 @@ If none of the format strings match, `strptime` returns the null value and logs

{{% /alert %}}

You can use the following elements in the format string:
You can use the following format codes in the format string:

{{< include-headless "chunk/date-string-format.md" >}}

The [`isodate`](#isodate) FilterX function is a specialized version of `strptime` that accepts only a fixed format.
The [`isodate`](#isodate) FilterX function is a specialized variant of `strptime`, that accepts only a fixed format.

## unset

Expand Down Expand Up @@ -389,7 +386,7 @@ Usage: `unset_empties(object, recursive=true)`

## upper

Converts a string into uppercase characters.
Converts all characters of a string uppercase characters.

Usage: `upper(string)`

Expand Down
5 changes: 2 additions & 3 deletions content/headless/chunk/date-string-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,10 @@

{{% alert title="Warning" color="warning" %}}

When using the %z and %Z format elements, consider that while %z strictly expects a specified timezone, and triggers a warning if the timezone is not specified, %Z does not trigger a warning if the timezone is not specified.
When using the `%z` and `%Z` format codes, consider that while `%z` strictly expects a specified timezone, and triggers a warning if the timezone is missing, `%Z` does not trigger a warning if the timezone is not specified.

For further information about the %z and %Z format elements, see the 'DESCRIPTION' section on the [srtptime(3) - NetBSD Manual Pages](https://man.netbsd.org/NetBSD-7.0/i386/strptime.3).
For further information about the `%z` and `%Z` format codes, see the 'DESCRIPTION' section on the [srtptime(3) - NetBSD Manual Pages](https://man.netbsd.org/NetBSD-7.0/i386/strptime.3).

{{% /alert %}}


For example, for the date `01/Jan/2016:13:05:05 PST` use the following format string: `"%d/%b/%Y:%H:%M:%S %Z"`
Loading