Add Support for Spark3.4

[KeerthiYandaOS]

Add Spark3.4

[github-actions]

Hey @KeerthiYandaOS 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[github-actions]

Summary by GPT-4

The changes in this commit include updating the Spark version from 3.2.3 to 3.4.1 and updating the Hadoop version from 2.7 to 3.3 in various files such as build.sbt, environment.yml, pipeline.yaml, and Dockerfiles. Additionally, some display functions in a notebook were updated to remove the use of .toPandas().

Suggestions

No suggestions are needed as the changes in this PR seem appropriate.

[codecov-commenter]

Codecov Report

Merging #2050 (6b1bb0d) into spark3.4 (c6d5882) will decrease coverage by 10.48%.
The diff coverage is 50.00%.

@@              Coverage Diff              @@
##           spark3.4    #2050       +/-   ##
=============================================
- Coverage     87.07%   76.60%   -10.48%     
=============================================
  Files           306      306               
  Lines         16063    16221      +158     
  Branches        852     1456      +604     
=============================================
- Hits          13987    12426     -1561     
- Misses         2076     3795     +1719     

Files Changed	Coverage Δ
...a/com/microsoft/azure/synapse/ml/nn/BallTree.scala	83.17% <ø> (+0.32%)	⬆️
...spark/ml/recommendation/RecommendationHelper.scala	36.70% <0.00%> (-50.14%)	⬇️
...osoft/azure/synapse/ml/core/env/PackageUtils.scala	100.00% <100.00%> (ø)
...se/ml/exploratory/DistributionBalanceMeasure.scala	96.42% <100.00%> (+0.06%)	⬆️
.../azure/synapse/ml/io/binary/BinaryFileFormat.scala	98.86% <100.00%> (ø)
...spark/ml/source/image/PatchedImageFileFormat.scala	89.28% <100.00%> (-1.46%)	⬇️

... and 104 files with indirect coverage changes

[mhamilton723]

lave helm charts alone for this pr please