Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Adding Spark35 support #2172

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

BrendanWalsh
Copy link
Collaborator

Adding support for Spark 3.5

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Briefly describe the changes included in this Pull Request.

How is this patch tested?

  • I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change any dependencies?

  • No. You can skip this section.
  • Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

  • No. You can skip this section.
  • Yes. Make sure you have added samples following below steps.
  1. Find the corresponding markdown file for your new feature in website/docs/documentation folder.
    Make sure you choose the correct class estimators/transformers and namespace.
  2. Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
  3. Make sure the DocTable points to correct API link.
  4. Navigate to website folder, and run yarn run start to make sure the website renders correctly.
  5. Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
  6. Make sure the WebsiteSamplesTests job pass in the pipeline.

@@ -3,7 +3,7 @@ FROM mcr.microsoft.com/oss/mirror/docker.io/library/ubuntu:20.04
ARG SYNAPSEML_VERSION=1.0.2
ARG DEBIAN_FRONTEND=noninteractive

ENV SPARK_VERSION=3.4.1
ENV SPARK_VERSION=3.5.0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we should use 3.5.1 instead of 3.5.0

@@ -7,7 +7,7 @@ import scala.xml.transform.{RewriteRule, RuleTransformer}
import scala.xml.{Node => XmlNode, NodeSeq => XmlNodeSeq, _}

val condaEnvName = "synapseml"
val sparkVersion = "3.4.1"
val sparkVersion = "3.5.0"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use 3.5.1

@BrendanWalsh BrendanWalsh force-pushed the brwals/spark35 branch 2 times, most recently from 049806a to d5098e3 Compare March 20, 2024 23:15
@acrolinxatmsft1
Copy link

Acrolinx Scorecards

A minimum Acrolinx score of 80 is required.

Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology.

Article Acrolinx
score
Word and
Phrases Score
Correctness
Score
Scorecard Processed
docs/Get Started/Install SynapseML.md 67 100 55 link

More information about Acrolinx

@BrendanWalsh
Copy link
Collaborator Author

/azp run

@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@microsoft microsoft deleted a comment from codecov-commenter Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@microsoft microsoft deleted a comment from acrolinxatmsft1 Mar 22, 2024
@microsoft microsoft deleted a comment from azure-pipelines bot Mar 22, 2024
@codecov-commenter
Copy link

codecov-commenter commented Mar 22, 2024

Codecov Report

Attention: Patch coverage is 87.50000% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 85.90%. Comparing base (9aea411) to head (12a8a39).

Files Patch % Lines
...rk/sql/types/injections/OptimizedCKNNFitting.scala 71.42% 2 Missing ⚠️
...crosoft/azure/synapse/ml/explainers/LIMEBase.scala 0.00% 1 Missing ⚠️
...crosoft/azure/synapse/ml/io/image/ImageUtils.scala 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2172      +/-   ##
==========================================
- Coverage   85.95%   85.90%   -0.06%     
==========================================
  Files         329      329              
  Lines       17086    17086              
  Branches     1515     1515              
==========================================
- Hits        14687    14678       -9     
- Misses       2399     2408       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants