Apache IoTDB: Integrate IoTDB Data Quality Library and enable visualisation in Grafana

[slawr]

Summary

Changes:

1. Integrate the Apache IoTDB Data Quality Library into the CDSP IoTDB image to allow for more advanced data processing.
2. Enable the REST API in the CDSP IoTDB image to allow visualisation in Grafana.
3. To enable the above introduce a CDSP IoTDB specific Docker image.
4. Extend IoTDB section of CDSP online documentation to outline functionality added by the Library.
5. Add example illustrating use of the library.

This is a WIP as the documentation for this needs to be added. This should not be merged without discussion with the author.

User note

To use the Data Quality Library the functions must be first registered in a running IoTDB instance. This need only be done once.
A script is provided to do this for you. With the Playground running run the following command from the host:
sudo docker exec -ti iotdb-service /iotdb/sbin/register-UDF.sh

Example use of the Data Quality Library to down-sample the timeseries root.test2.dev1.`Vehicle.Speed` to 100 points using the triangle method:
select sample(`Vehicle.Speed`,'method'='triangle','k'='100') from root.test2.dev1

Instructions for connecting Grafana to IoTDB using the IoTDB Grafana-Plugin can be found upstream here:
https://iotdb.apache.org/UserGuide/latest/Ecosystem-Integration/Grafana-Plugin.html

Technical Detail

Introduce a Dockerfile to build a CDSP specific Apache IoTDB image.

To keep the build as simple and quick as possible the starting point is
the pre-built IoTDB image from the upstream IoTDB project. This avoids
the burden of building IoTDB from scratch. This can be done because
currently our needs are limited to configuration and adding some
additional files.

Beyond some specific runtime environment values that can be hot loaded
IoTDB stores configuration in files located in the 'conf' directory.
For some specific use cases it is useful to automate the configuration
so end users of the Playground do not need to have to go through a lot
of manual configuration. For example, in this commit we enable the REST
API which is used by the IoTDB Grafana Connector to make DB queries.

In addition this commit integrates the optional IoTDB UDF (User Defined
Function) Data Quality Library into the image, which greatly extends
the data processing functions of IoTDB.

[slawr]

PR update: Added documentation for the IoTDB Data Quality Library to the doc site

[slawr]

PR update 26/7: added WIP example for the using the library. Example down-samples high frequency vehicle.speed data

[slawr]

Substantially updated the README.md that documents the timeseries transformation example. Squashed a series of commits during the last week for related changes.

[chrizmc]

Is it still WIP or is it ready for review?

[slawr]

Is it still WIP or is it ready for review?

Given your time constraints before your leave I think you could review, but not yet merge, as its 95% done.
I don't intend to change the code and documentation related to the Playground core changes for IoTDB, which integrates the Data Quality Library and configures the DB for Grafana.

With the example the coding is done and I just need to complete the documentation in the readme. I planned to do that yesterday but didn't achieve it. What's left to do on the readme:

1. There are a couple of sentences I have spotted that could be improved.
2. I need to spell and grammer check.
3. Decide whether I add screen shots to the Grafana section about creating a dashboard panel that shows the down-sampling. It's standard Grafana stuff so it could be considered as not needed, or added as a second step.

It would be great of course for this to be your first merge, but depending on what time you have left a review is also useful.

[slawr]

Extended the example documentation to include notes on using Grafana.
Example documentation is also now checked.

[slawr]

Removed WIP from the title as the PR is now ready.

@chrizmc Please review and consider for possible merging. As discussed if merging please use "Rebase and merge" as the default Github merge strategy.

[slawr]

Pushed final minor documentation changes prior to merge.

[slawr]

[12/8/2024] I met with my fellow CDSP Maintainer @chrizmc just before Christian has gone on extended leave. Unfortunately we were unable to get this PR reviewed and merged by him before his leave. He however is familiar with the work and he is happy for me to merge when ready. The Docker changes have been reviewed by his colleague @sschleemilch.

After a final check I am now merging.