Merge remote-tracking branch 'origin' into revamp_taxo_doc

.github/workflows/build_doc.yaml

@@ -0,0 +1,22 @@
+name: Build documentation
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Use Node.js 18.x
+        uses: actions/setup-node@v3
+        with:
+          node-version: 18.x
+      - name: Install dependencies
+        run: npm ci
+        working-directory: ./crowdsec-docs/
+      - run: npm run build --if-present
+        working-directory: ./crowdsec-docs/

crowdsec-docs/docs/bouncers/cloudflare.mdx

@@ -113,8 +113,6 @@ sudo systemctl start crowdsec-cloudflare-bouncer # the bouncer now syncs the cro
 
 ### From source
 
-:warning: requires go >= 1.16
-
 ```bash
 make release
 cd crowdsec-cloudflare-bouncer-vX.X.X

crowdsec-docs/docs/bouncers/firewall.mdx

@@ -59,13 +59,6 @@ sudo apt install crowdsec-firewall-bouncer-iptables
 sudo yum install crowdsec-firewall-bouncer-iptables
 ```
 
-</TabItem>
-<TabItem value="iptables_freebsd">
-
-```bash
-sudo pkg install crowdsec-firewall-bouncer
-```
-
 </TabItem>
 </Tabs>
 
@@ -92,13 +85,6 @@ sudo apt install crowdsec-firewall-bouncer-nftables
 sudo yum install crowdsec-firewall-bouncer-nftables
 ```
 
-</TabItem>
-<TabItem value="nftables_freebsd">
-
-```bash
-sudo pkg install crowdsec-firewall-bouncer
-```
-
 </TabItem>
 </Tabs>
 
@@ -323,8 +309,8 @@ table <crowdsec6-blacklists> persist
 
  ### ipset
 
- ipset lists have to exist before crowdsec-firewall-bouncer starts
- you can create them and add them to your iptables like this:
+ ipset lists have to exist before crowdsec-firewall-bouncer starts.
+ You can create them and add them to your iptables like this:
 
 ```console
 ipset create crowdsec-blacklists hash:ip timeout 0 maxelem 150000

crowdsec-docs/docs/configuration/crowdsec_configuration.md

@@ -180,7 +180,7 @@ always replaced.
 - `bouncers/crowdsec-blocklist-mirror.yaml`
 
 In the case of `profiles.yaml`, the files are read as a whole (as if they were
-attached) instead of merged. See [profiles - introduction](/profiles/intro).
+attached) instead of merged. See [profiles - introduction](/profiles/intro.md).
 
 
 ## Configuration directives
@@ -614,7 +614,6 @@ db_config:
 ```
 The port to connect to (only if the type of database is `mysql` or `postgresql`). Must be omitted if using socket file.
 
-#### `sslmode`
 
 ```yaml
 db_config:

crowdsec-docs/docs/contributing/code.md

@@ -39,7 +39,7 @@ Run with `make bats-build bats-fixture` once, then `make bats-test-hub`.
 
 We receive contributions on the _master_ branch (or _main_, in recent repositories). To contribute, fork the repository, commit the code in a dedicated branch and ask for a Pull Request. By default it will target the master branch on the upstream repository, so in most cases you don't have to change anything. It will be reviewed by the core team and merged when ready, possibly after some changes. It is recommended to open [an Issue linked to the PR](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) in order to discuss it and track its progression.
 
-You may also receive feedback from the CI scripts (directory [.github/workflows](.github/workflows)) that run a series of linters and tests. You are encouraged to run these on your environment as well, before committing (see the "Testing" section above, and "Style guide" below).
+You may also receive feedback from the CI scripts (directory [.github/workflows](https://github.com/crowdsecurity/hub/tree/master/.github/workflows)) that run a series of linters and tests. You are encouraged to run these on your environment as well, before committing (see the "Testing" section above, and "Style guide" below).
 
 ## Release branches
 

crowdsec-docs/docs/cti_api/taxonomy/taxonomy.md

@@ -48,7 +48,7 @@ sidebar_position: 1
    - `false_positive`: A list of false positives tags associated with the IP. Any IP with `known_false_positive` tags shouldn't be considered as malicious
    - `classifications`: A list of categories associated with the IP. Those data can be sourced from 3rd parties (i.e. tor exit nodes list). [An exhaustive list](#list-of-common-classifications) is available bellow,
  - `attack details`: A more exhaustive list of the scenarios for which a given IP was reported. Each entry contains the following information :
-   - `name`: name of the scenario (see [hub.crowdsec.net](hub.crowdsec.net))
+   - `name`: name of the scenario (see [hub.crowdsec.net](https://hub.crowdsec.net/))
    - `label`, `description`: Human-friendly descriptions of said scenarios
  - `target_countries`: The top 10 reports repartition by country about the IP, as a percentage
  - `background_noise_score`: Evaluate the noisiness of an IP address, from a scale of 0 (not noisy) to 10 (extremely noisy)

crowdsec-docs/docs/data_sources/introduction.md

@@ -15,7 +15,7 @@ Name | Type | Stream | One-shot
 [file](/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes
 [journald](/data_sources/journald.md) | journald via filter | yes | yes
 [AWS cloudwatch](/data_sources/cloudwatch.md) | single stream or log group | yes | yes
-[syslog service](/data_sources/syslog.md) | read logs received via syslog protocol | yes | no
+[syslog service](/data_sources/syslog_service.md) | read logs received via syslog protocol | yes | no
 [docker](/data_sources/docker.md) | read logs from docker containers | yes | yes
 [AWS kinesis](/data_sources/kinesis.md)| read logs from a kinesis strean | yes | no
 [Kafka](/data_sources/kafka.md)| read logs from kafka topic | yes | no

crowdsec-docs/docs/getting_started/getting_started.md

@@ -27,7 +27,7 @@ The Security Engine by default uses the following ports:
   - 8080/tcp for the API
   - 6060/tcp for the Prometheus metrics / Debugging
 
-If these ports are not available on your system, you can change them in the configuration file post installation. See [Configuration](/docs/configuration.md) for more information.
+If these ports are not available on your system, you can change them in the configuration file post installation. See [Configuration](/configuration/crowdsec_configuration.md) for more information.
 
 Please note that the API is mandatory for your security engine, do not remove it from your configuration.
 

crowdsec-docs/docs/getting_started/install_freebsd.md

@@ -125,7 +125,9 @@ will compile and install the packages and all their dependencies. Then configure
 
 ## Troubleshooting
 
-In some case, Crowdsec is unable to generate the machine id.
+In some cases, CrowdSec is unable to generate the machine id and is unable to initialize properly.
+
+We saw it happen with an APU board, likely due to the open source coreboot firmware.
 
 Start `hostid` and `hostid_save`:
 
@@ -134,4 +136,4 @@ Start `hostid` and `hostid_save`:
 /etc/rc.d/hostid_save start
 ```
 
-Then start again the Crowdsec' service  `service crowdsec start`.
+Then start again the CrowdSec' service  `service crowdsec start`.

crowdsec-docs/docs/parsers/format.md

@@ -171,7 +171,7 @@ DEBU[31-07-2020 16:36:28]        evt.Parsed.program = 'nginx'           id=withe
 filter: expression
 ```
 
-`filter` must be a valid [expr](/expr/helpers.md) expression that will be evaluated against the [event](/expr/event.md).
+`filter` must be a valid [expr](/expr/intro.md) expression that will be evaluated against the [event](/expr/event.md).
 
 If `filter` evaluation returns true or is absent, node will be processed.
 
@@ -194,7 +194,7 @@ A valid grok pattern
 
 #### `expression`
 
-A valid [expr](/expr/helpers.md) expression that return a string to apply the pattern on.
+A valid [expr](/expr/intro.md) expression that return a string to apply the pattern on.
 
 #### `apply_on`
 
@@ -463,11 +463,11 @@ The name of the stash. Distinct parsers can manipulate the same cache.
 
 #### `key`
 
-The [expression](/expr/helpers.md) that defines the string that will be used as a key.
+The [expression](/expr/intro.md) that defines the string that will be used as a key.
 
 #### `value`
 
-The [expression](/expr/helpers.md) that defines the string that will be used as a value.
+The [expression](/expr/intro.md) that defines the string that will be used as a value.
 
 #### `ttl`
 

crowdsec-docs/docs/parsers/introduction.mdx

@@ -12,7 +12,7 @@ import useBaseUrl from '@docusaurus/useBaseUrl';
 
 A parser is a YAML configuration file that describes how a string must be parsed. Said string can be a log line, or a field extracted from a previous parser. 
 
-While a lot of parsers rely on the **GROK** approach (a.k.a regular expression named capture groups), parsers can also use [expressions](/expr/helpers.md) to perform parsing on specific data (ie. json), [refer to external methods for enrichment](https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich) or even [perform whitelisting](https://hub.crowdsec.net/author/crowdsecurity/configurations/whitelists.md).
+While a lot of parsers rely on the **GROK** approach (a.k.a regular expression named capture groups), parsers can also use [expressions](/expr/intro.md) to perform parsing on specific data (ie. json), [refer to external methods for enrichment](https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich) or even [perform whitelisting](https://hub.crowdsec.net/author/crowdsecurity/configurations/whitelists.md).
 
 The [event](/expr/event.md) enters the parser, and might exit successfully or not:
 

crowdsec-docs/docs/profiles/format.md

@@ -51,7 +51,7 @@ filters:
  - Alert.Remediation == true && Alert.GetScope() == "Ip"
 ```
 
-If any `filter` of the list returns `true`, the profile is eligible and the `decisions` will be applied (note: `filter` can use [expr helpers](/expr/helpers.md)).
+If any `filter` of the list returns `true`, the profile is eligible and the `decisions` will be applied (note: `filter` can use [expr helpers](/expr/intro.md)).
 
 The filter allows you to then create custom decisions for some specific scenarios for example:
 
@@ -113,7 +113,7 @@ duration_expr: "Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)"
 If the profile applies, and the `duration_expr` generates a valid [golang's duration](https://pkg.go.dev/time#ParseDuration), it will replace the decision duration.
 
 It can be used to have custom duration. For example, you can have an increased duration every time an attacker comes back.
-It relies on [expr helpers](/expr/helpers.md).
+It relies on [expr helpers](/expr/intro.md).
 
 ### `on_success`
 

crowdsec-docs/docs/profiles/intro.md

@@ -15,7 +15,7 @@ The profiles configuration allows users to configure which kind of remediation s
 The profiles configuration is located in `/etc/crowdsec/profiles.yaml`.
 
 You can also write your profiles in a `profiles.yaml.local` file (as explained
-in [Crowdsec configuration](/configuration/crowdsec_configuration)), and they
+in [Crowdsec configuration](/configuration/crowdsec_configuration.md)), and they
 will be read _before_ `profiles.yaml`. In this case, you may want to provide
 `on_success: break` because the YAML files are not merged together, but read as
 a single multi-document configuration.

crowdsec-docs/docs/scenarios/format.md

@@ -214,7 +214,7 @@ A reference to third party documents. This is a list of string.
 filter: expression
 ```
 
-`filter` must be a valid [expr](/expr/helpers.md) expression that will be evaluated against the event.
+`filter` must be a valid [expr](/expr/intro.md) expression that will be evaluated against the event.
 
 If `filter` evaluation returns true or is absent, event will be pour in the bucket.
 
@@ -272,7 +272,7 @@ groupby: evt.Meta.source_ip
 ```
 
 
-An [expression](/expr/helpers.md) that must return a string. This string will be used as a partition for the buckets.
+An [expression](/expr/intro.md) that must return a string. This string will be used as a partition for the buckets.
 
 
 #### Examples
@@ -303,7 +303,7 @@ distinct: evt.Meta.http_path
 ```
 
 
-An [expression](/expr/helpers.md) that must return a string. The event will be poured **only** if the string is not already present in the bucket.
+An [expression](/expr/intro.md) that must return a string. The event will be poured **only** if the string is not already present in the bucket.
 
 #### Examples
 
@@ -585,7 +585,7 @@ Cache size will affect the number of events you receive within an alert. If you
 overflow_filter: any(queue.Queue, { .Enriched.IsInEU  == "true" })
 ```
 
-`overflow_filter` is an [expression](/expr/helpers.md) that is run when the bucket overflows.
+`overflow_filter` is an [expression](/expr/intro.md) that is run when the bucket overflows.
 If this expression is present and returns false, the overflow will be discarded.
 
 ---
@@ -595,7 +595,7 @@ If this expression is present and returns false, the overflow will be discarded.
 cancel_on: evt.Parsed.something == 'somevalue'
 ```
 
-`cancel_on` is an [expression](/expr/helpers.md) that runs on each event poured to the bucket.
+`cancel_on` is an [expression](/expr/intro.md) that runs on each event poured to the bucket.
 If the `cancel_on` expression returns true, the bucket is immediately destroyed (and doesn't overflow).
 
 

crowdsec-docs/docs/troubleshooting.md

@@ -59,6 +59,84 @@ Each Remediation Component has its own configuration file, which is located in t
 
 They all have an `api_url` option to set the local API URL.
 
+## CrowdSec Log Location
+
+By default CrowdSec will log to the following locations depending on platform:
+
+- **Linux** `/var/log/crowdsec.log`
+- **Freebsd** `/var/log/crowdsec.log`
+  - **Opnsense** `/var/log/crowdsec/crowdsec.log`
+  - **Pfsense** `/var/log/crowdsec/crowdsec.log`
+- **Windows** `C:\Program Files\CrowdSec\logs\crowdsec.log`
+
+This is the default log location if you are using a third party integration they may change the log location.
+
+## CrowdSec doesn't start (common causes)
+
+:::info
+If it's a configuration file issue, the error message may log in a different location. For example, if a parser/scenario file is invalid, the error message will be logged in `/var/log/crowdsec.log` but if `config.yaml` is invalid, the error message will be logged in syslog instead.
+:::
+
+Here are some common causes and solutions:
+
+1. CrowdSec cannot bind to the configured port or IP
+  - **error** message might look like:
+```
+level=fatal msg="while serving local API: listen tcp 127.0.0.1:8080: bind: address already in use"
+```
+  - **solution** verify another service is not already using the port. If it is, you can edit the `listen_uri` port in the configuration file `/etc/crowdsec/config.yaml` and update `/etc/crowdsec/local_api_credentials.yaml` to the same port. Then you can restart CrowdSec with `sudo systemctl restart crowdsec`.
+
+2. CrowdSec cannot authenticate to the local API
+  - **error** message might look like:
+```
+level=fatal msg="starting outputs error : authenticate watcher (fcb7303c4df44c03ada289dd7ec3dbe7cU3GaseSWdqUaVg6): API error: ent: machine not found"
+```
+  - **solution** regenerate the credentials via [cscli machines](/docs/cscli/cscli_machines_add) command. If the local API is on the same machine you can run `sudo cscli machines add -a` (`-a` will automatically generate a random machine name and password). Then you can restart CrowdSec with `sudo systemctl restart crowdsec`.
+
+3. CrowdSec cannot connect to the local API
+  - **error** message might look like:
+```
+level=error msg="error while performing request: dial tcp 127.0.0.1:8080: connect: connection refused; 4 retries left"
+## There may be other variations of this error message just note if the message contains `connection refused` or `connection reset by peer` or `no such host` its likely a connection issue or configuration.
+```
+  - **solution** verify that the local API runs on the logged IP and port. If the logged IP and port is incorrect, you can update `/etc/crowdsec/local_api_credentials.yaml` to the correct IP and port (If local API is running on the same machine you can run `grep listen_uri /etc/crowdsec/config.yaml` to find it). Then you can restart CrowdSec with `sudo systemctl restart crowdsec`. If the logged IP and port is correct, verify that the local API is running via `sudo systemctl status crowdsec`.
+
+4. CrowdSec cannot start because of an invalid configuration file
+
+  - **error** message might look like:
+```
+level=fatal msg="/etc/crowdsec/config.yaml: yaml: unmarshal errors:\n  line 1: field test not found in type csconfig.Config"
+```
+  - **solution** CrowdSec will inform you which field or line is invalid. You can edit the configuration file and fix the error. Then you can restart CrowdSec with `sudo systemctl restart crowdsec`. If you are unsure what the configuration file should look like you can find the default configuration files [here](https://github.com/crowdsecurity/crowdsec/tree/master/config) or examples via the [documentation](/docs/configuration/crowdsec_configuration).
+
+## How can I check to see if a service is already using a port?
+
+You can use the `netstat` command to check if a service is already using a port.
+
+For example, to check if a service is using port 8080: (Our default port for the local API)
+
+```bash
+sudo netstat -tulpn | grep :8080
+```
+
+## How can I check to see if the local API is running?
+
+You can use the `cscli lapi` command to check if the local API is running and is contactable.
+
+```bash
+sudo cscli lapi status
+```
+
+## Where does CrowdSec store its data?
+
+CrowdSec stores its data in `/var/lib/crowdsec/data/` folder for UNIX and `C:\Program Files\CrowdSec\data\` for Windows.
+
+## Where does CrowdSec store its configuration files?
+
+CrowdSec stores configuration files in diffent folder dependins on platform:
+- **Linux** `/etc/crowdsec/`
+- **Freebsd** `/usr/local/etc/crowdsec/`
+- **Windows** `C:\Program Files\CrowdSec\`
 
 ## My Remediaton Component doesn't start/work (common causes)
 

crowdsec-docs/docs/whitelist/format.md

@@ -107,7 +107,7 @@ whitelist:
     - "127.0.0.1"
 ```
 
-A valid [expr](/expr/helpers.md) expression that return a string to apply the pattern on.
+A valid [expr](/expr/intro.md) expression that return a string to apply the pattern on.
 
 
 #### `cidr`
@@ -120,7 +120,7 @@ whitelist:
     - "172.16.0.0/12"
 ```
 
-A valid [expr](/expr/helpers.md) expression that return a string to apply the pattern on.
+A valid [expr](/expr/intro.md) expression that return a string to apply the pattern on.
 
 
 #### `expression`
@@ -134,7 +134,7 @@ whitelist:
     - evt.Enriched.IsoCode == 'FR'
 ```
 
-A valid [expr](/expr/helpers.md) expression that return a string to apply the pattern on.
+A valid [expr](/expr/intro.md) expression that return a string to apply the pattern on.
 
 
 ### `data`

crowdsec-docs/docs/whitelist/introduction.md

@@ -4,6 +4,7 @@ title: Introduction
 sidebar_position: 1
 ---
 
+
 Whitelists are special parsers that allow you to "discard" events, and can exist at two different steps :
 
  - *Parser whitelists* : Allows you to discard an event at parse time, so that it never hits the buckets.

crowdsec-docs/versioned_docs/version-v1.0/configuration/crowdsec_configuration.md

@@ -403,7 +403,6 @@ db_config:
 ```
 The port to connect to (only if the type of database is `mysql` or `postgresql`)
 
-#### `sslmode`
 
 ```yaml
 db_config:

crowdsec-docs/versioned_docs/version-v1.0/cscli/cscli_alerts.md

@@ -26,7 +26,7 @@ Manage alerts
 
 ### SEE ALSO
 
-* [cscli](/docs/v1.0/cscli/cscli)	 - cscli allows you to manage crowdsec
+* [cscli](/docs/v1.0/cscli/)	 - cscli allows you to manage crowdsec
 * [cscli alerts delete](/docs/v1.0/cscli/cscli_alerts_delete)	 - Delete alerts
 /!\ This command can be use only on the same machine than the local API.
 * [cscli alerts inspect](/docs/v1.0/cscli/cscli_alerts_inspect)	 - Show info about an alert