Skip to content

Commit

Permalink
Added required external components (#461)
Browse files Browse the repository at this point in the history
* Added required external components

* added to nav and index

* some changes to the getting started guide
  • Loading branch information
fhennig authored Aug 10, 2023
1 parent 5aed035 commit e546259
Show file tree
Hide file tree
Showing 5 changed files with 23 additions and 5 deletions.
6 changes: 4 additions & 2 deletions docs/modules/druid/pages/getting_started/first_steps.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ include::example$getting_started/getting_started.sh[tag=install-druid]

This will create the actual druid instance.

WARNING: This Druid instance uses Derby (`dbType: derby`) as a metadata store, which is an interal SQL database. It is not persisted and not suitable for production use! Consult the https://druid.apache.org/docs/latest/dependencies/metadata-storage.html#available-metadata-stores[Druid documentation] for a list of supported databases and setup instructions for production instances.

== Verify that it works

Next you will submit an ingestion job and then query the ingested data - either through the web interface or the API.
Expand Down Expand Up @@ -160,8 +162,8 @@ include::example$getting_started/expected_query_result.json[]

image::getting_started/query.png[]

Great! You've set up your first Druid cluster, ingested some data and queried it in the web interface!
Great! You've set up your first Druid cluster, ingested some data and queried it in the web interface.

== What's next

Have a look at the xref:usage-guide/index.adoc[] page to find out more about the features of the Operator, such as S3-backed deep storage or OPA-based authorization.
Have a look at the xref:usage-guide/index.adoc[] page to find out more about the features of the Operator, such as S3-backed deep storage (as opposed to the HDFS backend used in this guide) or OPA-based authorization.
6 changes: 5 additions & 1 deletion docs/modules/druid/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,11 @@ The Druid Operator has the following dependencies:
* The xref:commons-operator:index.adoc[] provides common CRDs such as xref:concepts:s3.adoc[] CRDs.
* The xref:secret-operator:index.adoc[] is required for things like S3 access credentials or LDAP integration.

Have a look at the xref:getting_started/index.adoc[getting started guide] for an example of a minimal working setup. Druid works well with other Stackable supported products, such as xref:kafka:index.adoc[Apache Kafka] for data ingestion xref:trino:index.adoc[Trino] for data processing or xref:superset:index.adoc[Superset] for data visualization. xref:opa:index.adoc[OPA] can be connected to create authorization policies. Have a look at the xref:usage-guide/index.adoc[] for more configuration options and have a look at the <<demos, demos>> for complete data pipelines you can install with a single command.
Have a look at the xref:getting_started/index.adoc[getting started guide] for an example of a minimal working setup.

The getting started guide sets up a fully working Druid cluster, but the S3 deep storage backend as well as the metadata SQL database are xref:required-external-components.adoc[required external components] and need to be set up by you as prerequisites for a production setup.

Druid works well with other Stackable supported products, such as xref:kafka:index.adoc[Apache Kafka] for data ingestion xref:trino:index.adoc[Trino] for data processing or xref:superset:index.adoc[Superset] for data visualization. xref:opa:index.adoc[OPA] can be connected to create authorization policies. Have a look at the xref:usage-guide/index.adoc[] for more configuration options and have a look at the <<demos, demos>> for complete data pipelines you can install with a single command.

== [[demos]]Demos

Expand Down
9 changes: 9 additions & 0 deletions docs/modules/druid/pages/required-external-components.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Required external components

Druid uses an SQL database to store metadata. Consult the https://druid.apache.org/docs/latest/dependencies/metadata-storage.html#available-metadata-stores[Druid documentation] for a list of supported databases and setup instructions.

## Feature specific: S3 and cloud deep storage

https://druid.apache.org/docs/latest/dependencies/deep-storage.html[Deep storage] is where segments are stored. Druid offers multiple storage backends. For the local storage there are no prerequisites. HDFS deep storage can be set up with the xref:hdfs:index.adoc[Stackable Operator for Apache HDFS]. For S3 deep storage or the Google Cloud and Azure storage backends, you need to set up the storage.

Read the xref:usage-guide/deep-storage.adoc[deep storage usage guide] to learn more about configuring Druid deep storage.
6 changes: 4 additions & 2 deletions docs/modules/druid/pages/usage-guide/deep-storage.adoc
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
= Deep storage configuration

https://druid.apache.org/docs/latest/dependencies/deep-storage.html[Deep Storage] is where Druid stores data segments. For a Kubernetes environment, either the HDFS or S3 backend is recommended.

== [[hdfs]]HDFS

Druid can use HDFS as a backend for deep storage:
Druid can use HDFS as a backend for deep storage, which requires having an HDFS instance running. You can use the xref:hdfs:index.adoc[Stackable Operator for Apache HDFS] to run HDFS. Configure the HDFS deep storage backend in your Druid cluster this way:

[source,yaml]
----
Expand Down Expand Up @@ -80,4 +82,4 @@ include::partial$s3-note.adoc[]

=== S3 Credentials

include::partial$s3-credentials.adoc[]
include::partial$s3-credentials.adoc[]
1 change: 1 addition & 0 deletions docs/modules/druid/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,6 @@
** xref:druid:usage-guide/monitoring.adoc[]
** xref:druid:usage-guide/configuration-and-environment-overrides.adoc[]
** xref:druid:usage-guide/cluster_operations.adoc[]
* xref:druid:required-external-components.adoc[]
* xref:druid:configuration.adoc[]

0 comments on commit e546259

Please sign in to comment.