diff --git a/Dockerfile b/Dockerfile index c3e1fc2a..a73daac2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -24,7 +24,6 @@ COPY ./ ./ RUN --mount=type=cache,target=/root/.m2 \ MAVEN_OPTS=-Dorg.slf4j.simpleLogger.defaultLogLevel=warn mvn -B package -DskipTests RUN mv xtable-utilities_2.12-$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)-bundled.jar target/app.jar -#RUN mv xtable-utilities/target/$(mvn help:evaluate -Dexpression=project.build.finalName -q -DforceStdout)-bundled.jar target/app.jar FROM eclipse-temurin:17-jre-jammy AS final diff --git a/website/docs/biglake-metastore.md b/website/docs/biglake-metastore.md index 4ee4c2c2..b292ea69 100644 --- a/website/docs/biglake-metastore.md +++ b/website/docs/biglake-metastore.md @@ -25,7 +25,7 @@ This document walks through the steps to register an Apache XTable™ (Incubatin export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_key.json ``` 5. Clone the Apache XTable™ (Incubating) [repository](https://github.com/apache/incubator-xtable) and create the - `xtable-utilities-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) + `xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) 6. Download the [BigLake Iceberg JAR](gs://spark-lib/biglake/biglake-catalog-iceberg1.2.0-0.1.0-with-dependencies.jar) locally. Apache XTable™ (Incubating) requires the JAR to be present in the classpath. @@ -117,7 +117,7 @@ catalogOptions: From your terminal under the cloned Apache XTable™ (Incubating) directory, run the sync process using the below command. ```shell md title="shell" -java -cp xtable-utilities/target/xtable-utilities-0.2.0-SNAPSHOT-bundled.jar:/path/to/downloaded/biglake-catalog-iceberg1.2.0-0.1.0-with-dependencies.jar org.apache.xtable.utilities.RunSync --datasetConfig my_config.yaml --icebergCatalogConfig catalog.yaml +java -cp xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:/path/to/downloaded/biglake-catalog-iceberg1.2.0-0.1.0-with-dependencies.jar org.apache.xtable.utilities.RunSync --datasetConfig my_config.yaml --icebergCatalogConfig catalog.yaml ``` :::tip Note: diff --git a/website/docs/fabric.md b/website/docs/fabric.md index 9bae2d9b..10572cbb 100644 --- a/website/docs/fabric.md +++ b/website/docs/fabric.md @@ -98,7 +98,7 @@ An example hadoop configuration for authenticating to ADLS storage account is as ``` ```shell md title="shell" -java -jar xtable-utilities/target/xtable-utilities-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml --hadoopConfig hadoop.xml +java -jar xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml --hadoopConfig hadoop.xml ``` Running the above command will translate the table `people` in Iceberg or Hudi format to Delta Lake format. To validate diff --git a/website/docs/glue-catalog.md b/website/docs/glue-catalog.md index 6d1388c9..11317820 100644 --- a/website/docs/glue-catalog.md +++ b/website/docs/glue-catalog.md @@ -19,7 +19,7 @@ This document walks through the steps to register an Apache XTable™ (Incubatin also set up access credentials by following the steps [here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html) 3. Clone the Apache XTable™ (Incubating) [repository](https://github.com/apache/incubator-xtable) and create the - `xtable-utilities-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) + `xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) ## Steps ### Running sync @@ -84,7 +84,7 @@ Replace with appropriate values for `sourceFormat`, `tableBasePath` and `tableNa From your terminal under the cloned xtable directory, run the sync process using the below command. ```shell md title="shell" - java -jar xtable-utilities/target/xtable-utilities-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml + java -jar xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml ``` :::tip Note: diff --git a/website/docs/hms.md b/website/docs/hms.md index 7a4696e8..98682df5 100644 --- a/website/docs/hms.md +++ b/website/docs/hms.md @@ -17,7 +17,7 @@ This document walks through the steps to register an Apache XTable™ (Incubatin or a distributed system like Amazon EMR, Google Cloud's Dataproc, Azure HDInsight etc. This is a required step to register the table in HMS using a Spark client. 3. Clone the XTable™ (Incubating) [repository](https://github.com/apache/incubator-xtable) and create the - `xtable-utilities-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) + `xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) 4. This guide also assumes that you have configured the Hive Metastore locally or on EMR/Dataproc/HDInsight and is already running. @@ -88,7 +88,7 @@ datasets: From your terminal under the cloned Apache XTable™ (Incubating) directory, run the sync process using the below command. ```shell md title="shell" -java -jar xtable-utilities/target/xtable-utilities-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml +java -jar xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml ``` :::tip Note: diff --git a/website/docs/how-to.md b/website/docs/how-to.md index ea18a663..a60f223f 100644 --- a/website/docs/how-to.md +++ b/website/docs/how-to.md @@ -24,7 +24,7 @@ history to enable proper point in time queries. 1. A compute instance where you can run Apache Spark. This can be your local machine, docker, or a distributed service like Amazon EMR, Google Cloud's Dataproc, Azure HDInsight etc 2. Clone the Apache XTable™ (Incubating) [repository](https://github.com/apache/incubator-xtable) and create the - `xtable-utilities-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) + `xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) 3. Optional: Setup access to write to and/or read from distributed storage services like: * Amazon S3 by following the steps [here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to install AWSCLIv2 @@ -351,7 +351,7 @@ Authentication for GCP requires service account credentials to be exported. i.e. In your terminal under the cloned Apache XTable™ (Incubating) directory, run the below command. ```shell md title="shell" -java -jar xtable-utilities/target/xtable-utilities-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml +java -jar xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml ``` **Optional:** diff --git a/website/docs/unity-catalog.md b/website/docs/unity-catalog.md index b2fb83fe..cc5ccb0d 100644 --- a/website/docs/unity-catalog.md +++ b/website/docs/unity-catalog.md @@ -17,7 +17,7 @@ This document walks through the steps to register an Apache XTable™ (Incubatin 3. Create a Unity Catalog metastore in Databricks as outlined [here](https://docs.gcp.databricks.com/data-governance/unity-catalog/create-metastore.html#create-a-unity-catalog-metastore). 4. Create an external location in Databricks as outlined [here](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-location.html). 5. Clone the Apache XTable™ (Incubating) [repository](https://github.com/apache/incubator-xtable) and create the - `xtable-utilities-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) + `xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar` by following the steps on the [Installation page](/docs/setup) ## Pre-requisites (for open-source Unity Catalog) 1. Source table(s) (Hudi/Iceberg) already written to external storage locations like S3/GCS/ADLS or local. @@ -48,7 +48,7 @@ datasets: From your terminal under the cloned Apache XTable™ (Incubating) directory, run the sync process using the below command. ```shell md title="shell" -java -jar xtable-utilities/target/xtable-utilities-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml +java -jar xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml ``` :::tip Note: