[BUG] test_yyyyMMdd_format_for_legacy_mode failed in Dataproc Serverless integration tests

[yinqingh]

Describe the bug
Seeing test failures in rapids-it-dataproc-serverless-2.2#32. The full test logs can be found in Dataproc Serverless job with name "rapids-it-dataproc-serverless-22-32-3-20240925124028"

FAILED rapids-it-dataproc-serverless-32/integration_tests/src/main/python/date_time_test.py::test_yyyyMMdd_format_for_legacy_mode[DATAGEN_SEED=0, TZ=UTC]

a = ('xro1511250', <py4j.clientserver.JavaClient object at 0x7f3ebd714160>, 'o1511249', 'collectToPython')
kw = {}, converted = IllegalArgumentException()

    def deco(*a: Any, **kw: Any) -> Any:
        try:
            return f(*a, **kw)
        except Py4JJavaError as e:
            converted = convert_exception(e.java_exception)
            if not isinstance(converted, UnknownException):
                # Hide where the exception came from that shows a non-Pythonic
                # JVM exception message.
>               raise converted from None
E               pyspark.errors.exceptions.captured.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
E               Project [unix_timestamp(a#175292, yyyyMMdd, Some(Etc/UTC), false) AS unix_timestamp(a, yyyyMMdd)#175294L, from_unixtime(unix_timestamp(a#175292, yyyyMMdd, Some(Etc/UTC), false), yyyyMMdd, Some(Etc/UTC)) AS from_unixtime(unix_timestamp(a, yyyyMMdd), yyyyMMdd)#175295, date_format(gettimestamp(a#175292, yyyyMMdd, TimestampType, Some(Etc/UTC), false), yyyyMMdd, Some(Etc/UTC)) AS date_format(to_timestamp(a, yyyyMMdd), yyyyMMdd)#175296]
E               +- Scan ExistingRDD[a#175292]

Steps/Code to reproduce bug
Please provide a list of steps or a code sample to reproduce the issue.
Avoid posting private or sensitive data.

Expected behavior
A clear and concise description of what you expected to happen.

Environment details (please complete the following information)

• Dataproc Serverless version 2.2.20
• Scala213

Additional context
Add any other context about the problem here.

[pxLi]

the case was added in #11449

cc @res-life to help

[res-life]

I tried to reproduce on Spark 3.5.1 Scala213, and the case passed.
Will test on Dataproc.

[res-life]

Verified on Dataproc, also did not reproduce.

[pxLi]

lets wait for another round of this job, it may miss some cudf updates, also cc @yinqingh to trigger some runs after tonight's latest snapshot is out thanks

[yinqingh]

Will monitor the test builds with the latest snapshot jar

[res-life]

#33 shows this case passed：

[2024-09-26T15:44:30.917Z] 
[2024-09-26T15:44:30.917Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/date_time_test.py::test_yyyyMMdd_format_for_legacy_mode[DATAGEN_SEED=0, TZ=UTC] PASSED [ 41%]
[2024-09-26T15:44:30.917Z] 24/09/26 12:28:09 WARN GpuOverrides: 
[2024-09-26T15:44:30.917Z]   ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
[2024-09-26T15:44:30.917Z]     @Expression <AttributeReference> a#175305 could run on GPU
[2024-09-26T15:44:30.917Z] 

@yinqingh Please help check if CI used an old Plugin jar?

[yinqingh]

Confirmed that CI job used inconsistent plugin jar (00cd422) and IT package (a34f33e) due to prerelease version shifting. Closing this issue.