Update spark 4.0.0-preview2 #17622

yaooqinn · 2024-09-26T02:06:24Z

https://github.com/apache/spark-docker/tree/master/4.0.0-preview2

github-actions · 2024-09-26T02:07:56Z

Diff for 909fa2f:

diff --git a/_bashbrew-cat b/_bashbrew-cat
index 75fa442..b3a22e8 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -61,22 +61,42 @@ Architectures: amd64, arm64v8
 GitCommit: b9f1f8e8ebed1959c2be3864a114b52f67519092
 Directory: 3.5.2/scala2.12-java17-ubuntu
 
-Tags: 4.0.0-preview1-scala2.13-java17-python3-r-ubuntu
+Tags: 4.0.0-preview2-scala2.13-java17-python3-r-ubuntu
 Architectures: amd64, arm64v8
-GitCommit: b9f1f8e8ebed1959c2be3864a114b52f67519092
-Directory: 4.0.0-preview1/scala2.13-java17-python3-r-ubuntu
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java17-python3-r-ubuntu
 
-Tags: 4.0.0-preview1-scala2.13-java17-python3-ubuntu, 4.0.0-preview1-python3, 4.0.0-preview1
+Tags: 4.0.0-preview2-scala2.13-java17-python3-ubuntu, 4.0.0-preview2-python3, 4.0.0-preview2
 Architectures: amd64, arm64v8
-GitCommit: b9f1f8e8ebed1959c2be3864a114b52f67519092
-Directory: 4.0.0-preview1/scala2.13-java17-python3-ubuntu
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java17-python3-ubuntu
 
-Tags: 4.0.0-preview1-scala2.13-java17-r-ubuntu, 4.0.0-preview1-r
+Tags: 4.0.0-preview2-scala2.13-java17-r-ubuntu, 4.0.0-preview2-r
 Architectures: amd64, arm64v8
-GitCommit: b9f1f8e8ebed1959c2be3864a114b52f67519092
-Directory: 4.0.0-preview1/scala2.13-java17-r-ubuntu
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java17-r-ubuntu
 
-Tags: 4.0.0-preview1-scala2.13-java17-ubuntu, 4.0.0-preview1-scala
+Tags: 4.0.0-preview2-scala2.13-java17-ubuntu, 4.0.0-preview2-scala
 Architectures: amd64, arm64v8
-GitCommit: b9f1f8e8ebed1959c2be3864a114b52f67519092
-Directory: 4.0.0-preview1/scala2.13-java17-ubuntu
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java17-ubuntu
+
+Tags: 4.0.0-preview2-scala2.13-java21-python3-r-ubuntu
+Architectures: amd64, arm64v8
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java21-python3-r-ubuntu
+
+Tags: 4.0.0-preview2-scala2.13-java21-python3-ubuntu, 4.0.0-preview2-java21-python3, 4.0.0-preview2-java21
+Architectures: amd64, arm64v8
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java21-python3-ubuntu
+
+Tags: 4.0.0-preview2-scala2.13-java21-r-ubuntu, 4.0.0-preview2-java21-r
+Architectures: amd64, arm64v8
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java21-r-ubuntu
+
+Tags: 4.0.0-preview2-scala2.13-java21-ubuntu, 4.0.0-preview2-java21-scala
+Architectures: amd64, arm64v8
+GitCommit: 63c0ce3c0abfc94b591da7bc8cd5f90e02f16464
+Directory: 4.0.0-preview2/scala2.13-java21-ubuntu
diff --git a/_bashbrew-list b/_bashbrew-list
index 13fb4f8..5a0fa41 100644
--- a/_bashbrew-list
+++ b/_bashbrew-list
@@ -22,14 +22,22 @@ spark:3.5.2-scala2.12-java17-python3-r-ubuntu
 spark:3.5.2-scala2.12-java17-python3-ubuntu
 spark:3.5.2-scala2.12-java17-r-ubuntu
 spark:3.5.2-scala2.12-java17-ubuntu
-spark:4.0.0-preview1
-spark:4.0.0-preview1-python3
-spark:4.0.0-preview1-r
-spark:4.0.0-preview1-scala
-spark:4.0.0-preview1-scala2.13-java17-python3-r-ubuntu
-spark:4.0.0-preview1-scala2.13-java17-python3-ubuntu
-spark:4.0.0-preview1-scala2.13-java17-r-ubuntu
-spark:4.0.0-preview1-scala2.13-java17-ubuntu
+spark:4.0.0-preview2
+spark:4.0.0-preview2-java21
+spark:4.0.0-preview2-java21-python3
+spark:4.0.0-preview2-java21-r
+spark:4.0.0-preview2-java21-scala
+spark:4.0.0-preview2-python3
+spark:4.0.0-preview2-r
+spark:4.0.0-preview2-scala
+spark:4.0.0-preview2-scala2.13-java17-python3-r-ubuntu
+spark:4.0.0-preview2-scala2.13-java17-python3-ubuntu
+spark:4.0.0-preview2-scala2.13-java17-r-ubuntu
+spark:4.0.0-preview2-scala2.13-java17-ubuntu
+spark:4.0.0-preview2-scala2.13-java21-python3-r-ubuntu
+spark:4.0.0-preview2-scala2.13-java21-python3-ubuntu
+spark:4.0.0-preview2-scala2.13-java21-r-ubuntu
+spark:4.0.0-preview2-scala2.13-java21-ubuntu
 spark:latest
 spark:python3
 spark:python3-java17
diff --git a/_bashbrew-list-build-order b/_bashbrew-list-build-order
index 5d2d659..3536513 100644
--- a/_bashbrew-list-build-order
+++ b/_bashbrew-list-build-order
@@ -2,15 +2,19 @@ spark:3.4.3-scala
 spark:3.4.3-scala2.12-java11-python3-r-ubuntu
 spark:3.5.2-java17-scala
 spark:3.5.2-scala2.12-java17-python3-r-ubuntu
-spark:4.0.0-preview1-scala
-spark:4.0.0-preview1-scala2.13-java17-python3-r-ubuntu
+spark:4.0.0-preview2-java21-scala
+spark:4.0.0-preview2-scala
+spark:4.0.0-preview2-scala2.13-java17-python3-r-ubuntu
+spark:4.0.0-preview2-scala2.13-java21-python3-r-ubuntu
 spark:python3-java17
 spark:scala
 spark:3.4.3
 spark:3.4.3-r
 spark:3.5.2-java17-r
 spark:3.5.2-scala2.12-java11-python3-r-ubuntu
-spark:4.0.0-preview1
-spark:4.0.0-preview1-r
+spark:4.0.0-preview2
+spark:4.0.0-preview2-java21
+spark:4.0.0-preview2-java21-r
+spark:4.0.0-preview2-r
 spark:latest
 spark:r
diff --git a/spark_4.0.0-preview1-r/Dockerfile b/spark_4.0.0-preview2-java21-r/Dockerfile
similarity index 94%
rename from spark_4.0.0-preview1-r/Dockerfile
rename to spark_4.0.0-preview2-java21-r/Dockerfile
index c1729ec..1d77e12 100644
--- a/spark_4.0.0-preview1-r/Dockerfile
+++ b/spark_4.0.0-preview2-java21-r/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM spark:4.0.0-preview1-scala2.13-java17-ubuntu
+FROM spark:4.0.0-preview2-scala2.13-java21-ubuntu
 
 USER root
 
diff --git a/spark_4.0.0-preview1-scala/Dockerfile b/spark_4.0.0-preview2-java21-scala/Dockerfile
similarity index 88%
copy from spark_4.0.0-preview1-scala/Dockerfile
copy to spark_4.0.0-preview2-java21-scala/Dockerfile
index 1102caf..42fc9ec 100644
--- a/spark_4.0.0-preview1-scala/Dockerfile
+++ b/spark_4.0.0-preview2-java21-scala/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM eclipse-temurin:17-jre-jammy
+FROM eclipse-temurin:21-jammy
 
 ARG spark_uid=185
 
@@ -36,9 +36,9 @@ RUN set -ex; \
 
 # Install Apache Spark
 # https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview1/spark-4.0.0-preview1-bin-hadoop3.tgz \
-    SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview1/spark-4.0.0-preview1-bin-hadoop3.tgz.asc \
-    GPG_KEY=4DC9676CEF9A83E98FCA02784D6620843CD87F5A
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview2/spark-4.0.0-preview2-bin-hadoop3.tgz \
+    SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview2/spark-4.0.0-preview2-bin-hadoop3.tgz.asc \
+    GPG_KEY=F28C9C925C188C35E345614DEDA00CE834F0FC5C
 
 RUN set -ex; \
     export SPARK_TMP="$(mktemp -d)"; \
@@ -55,10 +55,12 @@ RUN set -ex; \
     tar -xf spark.tgz --strip-components=1; \
     chown -R spark:spark .; \
     mv jars /opt/spark/; \
+    mv RELEASE /opt/spark/; \
     mv bin /opt/spark/; \
     mv sbin /opt/spark/; \
     mv kubernetes/dockerfiles/spark/decom.sh /opt/; \
     mv examples /opt/spark/; \
+    ln -s "$(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar))" /opt/spark/examples/jars/spark-examples.jar; \
     mv kubernetes/tests /opt/spark/; \
     mv data /opt/spark/; \
     mv python/pyspark /opt/spark/python/pyspark/; \
diff --git a/spark_4.0.0-preview1-scala/entrypoint.sh b/spark_4.0.0-preview2-java21-scala/entrypoint.sh
similarity index 100%
rename from spark_4.0.0-preview1-scala/entrypoint.sh
rename to spark_4.0.0-preview2-java21-scala/entrypoint.sh
diff --git a/spark_4.0.0-preview1/Dockerfile b/spark_4.0.0-preview2-java21/Dockerfile
similarity index 94%
copy from spark_4.0.0-preview1/Dockerfile
copy to spark_4.0.0-preview2-java21/Dockerfile
index 66fb618..c7155b0 100644
--- a/spark_4.0.0-preview1/Dockerfile
+++ b/spark_4.0.0-preview2-java21/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM spark:4.0.0-preview1-scala2.13-java17-ubuntu
+FROM spark:4.0.0-preview2-scala2.13-java21-ubuntu
 
 USER root
 
diff --git a/spark_3.4.3-r/Dockerfile b/spark_4.0.0-preview2-r/Dockerfile
similarity index 94%
copy from spark_3.4.3-r/Dockerfile
copy to spark_4.0.0-preview2-r/Dockerfile
index 58e228e..e3185f3 100644
--- a/spark_3.4.3-r/Dockerfile
+++ b/spark_4.0.0-preview2-r/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM spark:3.4.3-scala2.12-java11-ubuntu
+FROM spark:4.0.0-preview2-scala2.13-java17-ubuntu
 
 USER root
 
diff --git a/spark_4.0.0-preview1-scala/Dockerfile b/spark_4.0.0-preview2-scala/Dockerfile
similarity index 88%
rename from spark_4.0.0-preview1-scala/Dockerfile
rename to spark_4.0.0-preview2-scala/Dockerfile
index 1102caf..baaa68a 100644
--- a/spark_4.0.0-preview1-scala/Dockerfile
+++ b/spark_4.0.0-preview2-scala/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM eclipse-temurin:17-jre-jammy
+FROM eclipse-temurin:17-jammy
 
 ARG spark_uid=185
 
@@ -36,9 +36,9 @@ RUN set -ex; \
 
 # Install Apache Spark
 # https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview1/spark-4.0.0-preview1-bin-hadoop3.tgz \
-    SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview1/spark-4.0.0-preview1-bin-hadoop3.tgz.asc \
-    GPG_KEY=4DC9676CEF9A83E98FCA02784D6620843CD87F5A
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview2/spark-4.0.0-preview2-bin-hadoop3.tgz \
+    SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-4.0.0-preview2/spark-4.0.0-preview2-bin-hadoop3.tgz.asc \
+    GPG_KEY=F28C9C925C188C35E345614DEDA00CE834F0FC5C
 
 RUN set -ex; \
     export SPARK_TMP="$(mktemp -d)"; \
@@ -55,10 +55,12 @@ RUN set -ex; \
     tar -xf spark.tgz --strip-components=1; \
     chown -R spark:spark .; \
     mv jars /opt/spark/; \
+    mv RELEASE /opt/spark/; \
     mv bin /opt/spark/; \
     mv sbin /opt/spark/; \
     mv kubernetes/dockerfiles/spark/decom.sh /opt/; \
     mv examples /opt/spark/; \
+    ln -s "$(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar))" /opt/spark/examples/jars/spark-examples.jar; \
     mv kubernetes/tests /opt/spark/; \
     mv data /opt/spark/; \
     mv python/pyspark /opt/spark/python/pyspark/; \
diff --git a/spark_3.5.2-java17-scala/entrypoint.sh b/spark_4.0.0-preview2-scala/entrypoint.sh
similarity index 100%
copy from spark_3.5.2-java17-scala/entrypoint.sh
copy to spark_4.0.0-preview2-scala/entrypoint.sh
diff --git a/spark_4.0.0-preview1-scala2.13-java17-python3-r-ubuntu/Dockerfile b/spark_4.0.0-preview2-scala2.13-java17-python3-r-ubuntu/Dockerfile
similarity index 95%
rename from spark_4.0.0-preview1-scala2.13-java17-python3-r-ubuntu/Dockerfile
rename to spark_4.0.0-preview2-scala2.13-java17-python3-r-ubuntu/Dockerfile
index 3636a21..7c575a8 100644
--- a/spark_4.0.0-preview1-scala2.13-java17-python3-r-ubuntu/Dockerfile
+++ b/spark_4.0.0-preview2-scala2.13-java17-python3-r-ubuntu/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM spark:4.0.0-preview1-scala2.13-java17-ubuntu
+FROM spark:4.0.0-preview2-scala2.13-java17-ubuntu
 
 USER root
 
diff --git a/spark_3.5.2-scala2.12-java17-python3-r-ubuntu/Dockerfile b/spark_4.0.0-preview2-scala2.13-java21-python3-r-ubuntu/Dockerfile
similarity index 95%
copy from spark_3.5.2-scala2.12-java17-python3-r-ubuntu/Dockerfile
copy to spark_4.0.0-preview2-scala2.13-java21-python3-r-ubuntu/Dockerfile
index 80dda3b..99268f9 100644
--- a/spark_3.5.2-scala2.12-java17-python3-r-ubuntu/Dockerfile
+++ b/spark_4.0.0-preview2-scala2.13-java21-python3-r-ubuntu/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM spark:3.5.2-scala2.12-java17-ubuntu
+FROM spark:4.0.0-preview2-scala2.13-java21-ubuntu
 
 USER root
 
diff --git a/spark_4.0.0-preview1/Dockerfile b/spark_4.0.0-preview2/Dockerfile
similarity index 94%
rename from spark_4.0.0-preview1/Dockerfile
rename to spark_4.0.0-preview2/Dockerfile
index 66fb618..576fdaa 100644
--- a/spark_4.0.0-preview1/Dockerfile
+++ b/spark_4.0.0-preview2/Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-FROM spark:4.0.0-preview1-scala2.13-java17-ubuntu
+FROM spark:4.0.0-preview2-scala2.13-java17-ubuntu
 
 USER root

Relevant Maintainers:

spark: @apachespark

tianon · 2024-09-26T22:36:21Z

+    ln -s "$(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar))" /opt/spark/examples/jars/spark-examples.jar; \

This chain is definitely eyebrow-raising 😅

Can you explain a bit what it's trying to do? Does /opt/spark/examples/jars/spark-examples_*.jar match multiple things, or just one? Perhaps the --relative flag to ln is what you're looking for here so you can avoid both subshells?

yaooqinn · 2024-09-27T02:10:11Z

This is a user experience improvement for referencing it for quickstart w/ spark examples.

FYI, apache/spark-docker#67

tianon · 2024-09-30T23:22:55Z

Sorry, let me be more clear -- ls in any scripting context is almost always a "code smell" because it is primarily a user interface, not intended for scripting use.

In this case, I think what you meant by this:

+    ln -s "$(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar))" /opt/spark/examples/jars/spark-examples.jar; \

was really much more simply stated as:

+    ln -sr /opt/spark/examples/jars/spark-examples_*.jar /opt/spark/examples/jars/spark-examples.jar; \

(with what should be the exact same end result, but with less fiddly subshell layers in between where things could unintentionally go wrong)

tianon · 2024-09-30T23:24:33Z

Another alternative that's still valid, but only a single subshell (still not ideal, but more correct than the current that's shelling out to ls as well):

+    ln -s "$(basename /opt/spark/examples/jars/spark-examples_*.jar)" /opt/spark/examples/jars/spark-examples.jar; \

(The ideal solution would be to ship this symlink as part of the original upstream dist, but that's perhaps a separate topic.)

Update spark 4.0.0-preview2

909fa2f

yaooqinn requested a review from a team as a code owner September 26, 2024 02:06

github-actions bot added the library/spark label Sep 26, 2024

yaooqinn mentioned this pull request Sep 27, 2024

Add 4.0.0-preview2 news apache/spark-website#561

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update spark 4.0.0-preview2 #17622

Update spark 4.0.0-preview2 #17622

yaooqinn commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024

tianon commented Sep 26, 2024

yaooqinn commented Sep 27, 2024

tianon commented Sep 30, 2024

tianon commented Sep 30, 2024

Update spark 4.0.0-preview2 #17622

Are you sure you want to change the base?

Update spark 4.0.0-preview2 #17622

Conversation

yaooqinn commented Sep 26, 2024 • edited Loading

github-actions bot commented Sep 26, 2024

tianon commented Sep 26, 2024

yaooqinn commented Sep 27, 2024

tianon commented Sep 30, 2024

tianon commented Sep 30, 2024

yaooqinn commented Sep 26, 2024 •

edited

Loading