draft: Spark33rebase #2075
draft: Spark33rebase #2075
Conversation
|
Hey @JessicaXYWang 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
JessicaXYWang
force-pushed
the
spark33rebase
branch
from
71a2acf
to
27ea1ae
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #2075 +/- ##
===========================================
- Coverage 87.06% 43.18% -43.89%
===========================================
Files 306 306
Lines 16162 16162
Branches 824 824
===========================================
- Hits 14072 6980 -7092
- Misses 2090 9182 +7092
|
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary by GPT-4
This commit updates the versions of several dependencies in the SynapseML project. The Spark version is updated from 3.2.3 to 3.3.2, and the Horovod version is updated from 0.25.0 to 0.27.0.
In addition, the versions of Torch and Torchvision are updated, and the Transformers library is updated from version 4.15.0 to 4.32.1.
The commit also updates some test files to reflect these changes, and modifies some environment setup scripts and Dockerfiles accordingly.
Overall, this commit aims to keep the project up-to-date with its dependencies, which can bring performance improvements, bug fixes, and new features from these libraries into SynapseML.
Suggestions
The changes in this PR are generally good, but there are a few areas that could be improved:
-
In the
build.sbtfile, the spark version is updated from "3.2.3" to "3.3.2". It would be helpful to include a comment explaining why this change was made. -
In the
core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegen.scalafile, several package versions are updated. Again, it would be helpful to include comments explaining why these changes were made. -
In the
core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scalafile, several package versions are updated in theGPULibrariesstring. It would be helpful to include comments explaining why these changes were made. -
In the
deep-learning/src/main/python/horovod_installation.shfile, several commands are changed and some are removed entirely. It would be helpful to include comments explaining why these changes were made. -
In the
docs/Explore Algorithms/Hyperparameter Tuning/HyperOpt.ipynbfile, the mlflow version is downgraded from 2.6.0 to 2.5.0 without explanation. -
In multiple files (e.g.,
DeepTextClassifier.py,DeepVisionClassifier.py, etc.), checks for specific versions of packages (e.g., transformers and horovod) are updated without explanation. -
The Dockerfiles in
tools/docker/demo/Dockerfileandtools/docker/minimal/Dockerfilehave their Spark and Hadoop versions updated without explanation. -
The script in
tools/dotnet/dotnetSetup.shhas its Spark version updated without explanation.
In general, it's important to explain why version updates are being made so that other developers can understand the rationale behind these changes.
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
JessicaXYWang
force-pushed
the
spark33rebase
branch
from
94ec40a
to
3f6c04b
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
JessicaXYWang
force-pushed
the
spark33rebase
branch
from
57c8674
to
ce9b01f
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
JessicaXYWang
force-pushed
the
spark33rebase
branch
from
8b76583
to
afb0054
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Related Issues/PRs
#xxx
What changes are proposed in this pull request?
Briefly describe the changes included in this Pull Request.
How is this patch tested?
Does this PR change any dependencies?
Does this PR add a new feature? If so, have you added samples on website?
website/docs/documentationfolder.Make sure you choose the correct class
estimators/transformersand namespace.DocTablepoints to correct API link.yarn run startto make sure the website renders correctly.<!--pytest-codeblocks:cont-->before each python code blocks to enable auto-tests for python samples.WebsiteSamplesTestsjob pass in the pipeline.