Generic SQL module

[jsoriano]

Add a generic SQL module so metrics and other data can be collected using custom SQL queries, the module will support the different drivers included in Metricbeat binary.

This is a follow-up of the initial implementation in #13257.

Other related issues:

• Allow custom queries for MySQL and PostgreSQL Mysql and Postgres modules should allow addition of custom queries #2987
• Add a Metricbeat helper for SQL Metricbeat SQL Helper #9416
• Add support for IBM DB2 [Metricbeat] IBM DB2 Module #13106
• Add support for "key/value" responses Add "key/value" mode for SQL module #15770

Modules

For a metricset to go GA, the following criterias should be met:

• Supported versions are documented
• Supported operating systems are documented (if applicable)
• Integration tests exist, for several drivers in this case
• System tests exist, also for several drivers
• Automated checks that all fields are documented
• Documentation [Metricbeat] Missing documentation in SQL module #18743
• Fields follow ECS and naming conventions
• Dashboards exists (if applicable)
• Kibana Home Tutorial (if applicable)
  • Open PR against Kibana repo with tutorial. Examples can be found here.

Metricbeat module

• Example data.json exists and an automated way to generate it exists (go test -data)
• Test environment in Docker exist for integration tests

SQL module

• Review vendorization of sql drivers, move them to common vendor directories and use released tags when possible (done as part of migration to go modules).
• Consider importing drivers from the sql module so dependency is more easily tracked.
• Add support for username and password options as best effort (it is going to depend on the DSNs supported by the driver, see Remove datasource option from SQL module and add tests #15686)
• Add support for sanitization of more DSN formats (see Remove datasource option from SQL module and add tests #15686)
• Include DSN examples in the documentation for each one of the supported databases
• Reuse connections between fetches. Reuse connections on sql module #16001
• Reconsider the use of sqlx, it doesn't seem to be needed with current implementation

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!