Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add generic Iceberg catalog adapter creation to Java / Python #5754

Merged
merged 18 commits into from
Sep 6, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
18 changes: 8 additions & 10 deletions .github/workflows/build-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ jobs:
distribution: 'temurin'
java-version: '17'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand Down Expand Up @@ -88,11 +91,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}

- name: Create Dockerfile and context
uses: burrunan/gradle-cache-action@v1
with:
job-id: build-server
arguments: --scan outputVersion docker-server-slim:prepareDocker docker-server:prepareDockerAll
gradle-version: wrapper
run: ./gradlew --scan outputVersion docker-server-slim:prepareDocker docker-server:prepareDockerAll

- name: Get Deephaven Version
id: deephaven_version
Expand Down Expand Up @@ -158,6 +157,9 @@ jobs:
distribution: 'temurin'
java-version: '11'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -182,11 +184,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}

- name: Create Dockerfile and context
uses: burrunan/gradle-cache-action@v1
with:
job-id: build-web
arguments: --scan outputVersion docker-web-plugin-packager:prepareDocker
gradle-version: wrapper
run: ./gradlew --scan outputVersion docker-web-plugin-packager:prepareDocker

- name: Get Deephaven Version
id: deephaven_version
Expand Down
9 changes: 4 additions & 5 deletions .github/workflows/check-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ jobs:
distribution: 'temurin'
java-version: '17'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -42,11 +45,7 @@ jobs:
cat gradle.properties

- name: Check
uses: burrunan/gradle-cache-action@v1
with:
job-id: checks
arguments: --scan --continue check
gradle-version: wrapper
run: ./gradlew --scan --continue check

- name: Upload Test Results
uses: actions/upload-artifact@v4
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/cla.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
steps:
- name: "CLA Assistant"
if: (github.event.comment.body == 'recheck' || github.event.comment.body == 'I have read the CLA Document and I hereby sign the CLA') || github.event_name == 'pull_request_target'
uses: cla-assistant/github-action@v2.4.0
uses: cla-assistant/github-action@v2.5.1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PERSONAL_ACCESS_TOKEN : ${{ secrets.CLA_PERSONAL_ACCESS_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dependency-submission.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
cat gradle.properties

- name: Generate and submit dependency graph
uses: gradle/actions/dependency-submission@v3
uses: gradle/actions/dependency-submission@v4
lbooker42 marked this conversation as resolved.
Show resolved Hide resolved
env:
# Dependencies derived from :server-jetty-app runtimeClasspath get a "runtime" scope and everything else gets
# a "development" scope. Ideally, gradle would be able to pass along the finer dependency details (for
Expand Down
55 changes: 25 additions & 30 deletions .github/workflows/docs-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ jobs:
distribution: 'temurin'
java-version: '17'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -85,12 +88,8 @@ jobs:
cat gradle.properties

- name: All Javadoc
uses: burrunan/gradle-cache-action@v1
with:
job-id: allJavadoc
arguments: --scan outputVersion combined-javadoc:allJavadoc
gradle-version: wrapper

run: ./gradlew --scan outputVersion combined-javadoc:allJavadoc

- name: Get Deephaven Version
id: dhc-version
run: echo "version=$(cat build/version)" >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -139,6 +138,9 @@ jobs:
distribution: 'temurin'
java-version: '17'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -148,12 +150,8 @@ jobs:
cat gradle.properties

- name: Run typedoc on JS API
uses: burrunan/gradle-cache-action@v1
with:
job-id: typedoc
arguments: --scan outputVersion :web-client-api:types:typedoc
gradle-version: wrapper

run: ./gradlew --scan outputVersion :web-client-api:types:typedoc

- name: Get Deephaven Version
id: dhc-version
run: echo "version=$(cat build/version)" >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -202,6 +200,9 @@ jobs:
distribution: 'temurin'
java-version: '17'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -211,12 +212,8 @@ jobs:
cat gradle.properties

- name: Generate Python Docs
uses: burrunan/gradle-cache-action@v1
with:
job-id: pythonDocs
arguments: --scan outputVersion sphinx:pythonDocs sphinx:pydeephavenDocs
gradle-version: wrapper

run: ./gradlew --scan outputVersion sphinx:pythonDocs sphinx:pydeephavenDocs

- name: Get Deephaven Version
id: dhc-version
run: echo "version=$(cat build/version)" >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -285,6 +282,9 @@ jobs:
distribution: 'temurin'
java-version: '11'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -294,12 +294,8 @@ jobs:
cat gradle.properties

- name: Generate C++ Docs
uses: burrunan/gradle-cache-action@v1
with:
job-id: cppDocs
arguments: --scan outputVersion sphinx:cppClientDocs sphinx:cppExamplesDocs
gradle-version: wrapper

run: ./gradlew --scan outputVersion sphinx:cppClientDocs sphinx:cppExamplesDocs

- name: Get Deephaven Version
id: dhc-version
run: echo "version=$(cat build/version)" >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -360,6 +356,9 @@ jobs:
distribution: 'temurin'
java-version: '11'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -369,12 +368,8 @@ jobs:
cat gradle.properties

- name: Generate R Docs
uses: burrunan/gradle-cache-action@v1
with:
job-id: rDocs
arguments: --scan outputVersion R:rClientSite
gradle-version: wrapper

run: ./gradlew --scan outputVersion R:rClientSite

- name: Get Deephaven Version
id: dhc-version
run: echo "version=$(cat build/version)" >> $GITHUB_OUTPUT
Expand Down
9 changes: 4 additions & 5 deletions .github/workflows/nightly-check-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ jobs:
distribution: 'temurin'
java-version: '22'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -64,11 +67,7 @@ jobs:
cat gradle.properties

- name: Run gradle ${{ matrix.gradle-task }} on java ${{ matrix.test-jvm-version }}
uses: burrunan/gradle-cache-action@v1
with:
job-id: gradle-run
arguments: --scan --continue --rerun-tasks ${{ matrix.gradle-task }} -PtestRuntimeVersion=${{ matrix.test-jvm-version }}
gradle-version: wrapper
run: ./gradlew --scan --continue --rerun-tasks ${{ matrix.gradle-task }} -PtestRuntimeVersion=${{ matrix.test-jvm-version }}

- name: Upload Test Results
uses: actions/upload-artifact@v4
Expand Down
9 changes: 4 additions & 5 deletions .github/workflows/nightly-image-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ jobs:
distribution: 'temurin'
java-version: '11'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -29,11 +32,7 @@ jobs:
cat gradle.properties

- name: Run gradle
uses: burrunan/gradle-cache-action@v1
with:
job-id: image-compare
arguments: --continue pullImage compareImage
gradle-version: wrapper
run: ./gradlew --continue pullImage compareImage

- name: Notify Slack
uses: slackapi/[email protected]
Expand Down
67 changes: 67 additions & 0 deletions .github/workflows/nightly-publish-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Nightly Publish CI

on:
schedule:
# 1AM EST == 5AM UTC
- cron: '0 5 * * *'

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false

jobs:
nightly-publish:
runs-on: ubuntu-24.04
if: ${{ github.repository_owner == 'deephaven' }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup JDK 11
id: setup-java-11
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '11'

- name: Setup JDK 17
id: setup-java-17
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '17'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

- name: Setup gradle properties
run: |
.github/scripts/gradle-properties.sh >> gradle.properties
cat gradle.properties

- name: Publish to OSSRH snapshots repository
# We are not worried here about using --no-parallel (as we are with release publishing), since publishing
# snapshots does not even create staging repositories.
run: ./gradlew publish
env:
ORG_GRADLE_PROJECT_ossrhUsername: ${{ secrets.SONATYPE_USERNAME }}
ORG_GRADLE_PROJECT_ossrhPassword: ${{ secrets.SONATYPE_PASSWORD }}
ORG_GRADLE_PROJECT_signingKey: ${{ secrets.CI_AT_DEEPHAVEN_KEY }}
ORG_GRADLE_PROJECT_signingPassword: ${{ secrets.CI_AT_DEEPHAVEN_PASSWORD }}
ORG_GRADLE_PROJECT_signingRequired: true

- name: Slack Nightly Failure
uses: slackapi/[email protected]
id: slack-nightly-failure
if: failure()
with:
payload: |
{
"slack_message": "Nightly publish failure @ ${{ github.head_ref }} ${{ github.sha }}"
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_NIGHTLY_FAILURE }}
17 changes: 6 additions & 11 deletions .github/workflows/publish-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ jobs:
distribution: 'temurin'
java-version: '17'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -51,20 +54,12 @@ jobs:

- name: Build all artifacts, publish to Maven Local
if: ${{ !startsWith(github.ref, 'refs/heads/release/v') }}
uses: burrunan/gradle-cache-action@v1
with:
job-id: publish-local
arguments: server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build py-client-ticking:build web-client-api:types:build publishToMavenLocal
gradle-version: wrapper
run: ./gradlew server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build py-client-ticking:build web-client-api:types:build publishToMavenLocal

- name: Build all artifacts, publish to Sonatype for staging to Maven Central
if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
uses: burrunan/gradle-cache-action@v1
with:
job-id: publish
# We need to be explicit here about no parallelism to ensure we don't create disjointed staging repositories.
arguments: --no-parallel server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build py-client-ticking:build web-client-api:types:build publish
gradle-version: wrapper
# We need to be explicit here about no parallelism to ensure we don't create disjointed staging repositories.
run: ./gradlew --no-parallel server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build py-client-ticking:build web-client-api:types:build publish
env:
ORG_GRADLE_PROJECT_ossrhUsername: ${{ secrets.SONATYPE_USERNAME }}
ORG_GRADLE_PROJECT_ossrhPassword: ${{ secrets.SONATYPE_PASSWORD }}
Expand Down
13 changes: 6 additions & 7 deletions .github/workflows/quick-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ jobs:
distribution: 'temurin'
java-version: '11'

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Set JAVA_HOME
run: echo "JAVA_HOME=${{ steps.setup-java-11.outputs.path }}" >> $GITHUB_ENV

Expand All @@ -36,12 +39,8 @@ jobs:
cat gradle.properties

- name: Quick Task
uses: burrunan/gradle-cache-action@v1
with:
job-id: quick-task
# Even though quick includes spotlessCheck, we want to make sure it runs first and fails ASAP for quick feedback
arguments: --scan spotlessCheck quick
gradle-version: wrapper
# Even though quick includes spotlessCheck, we want to make sure it runs first and fails ASAP for quick feedback
run: ./gradlew --scan spotlessCheck quick

- name: Upload JVM Error Logs
uses: actions/upload-artifact@v4
Expand All @@ -64,4 +63,4 @@ jobs:
run: pip install vermin==1.6.0

- name: Verify minimum version support
run: vermin -t=3.8 --no-tips --eval-annotations --violations py/server/deephaven py/client py/client-ticking py/embedded-server
run: vermin -t=3.8 --no-tips --eval-annotations --violations --feature fstring-self-doc --feature union-types py/server/deephaven py/client py/client-ticking py/embedded-server
Loading