Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Amazon linux 2023 fix #358

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

pragnesh
Copy link
Contributor

@pragnesh pragnesh commented Mar 31, 2023

This PR makes the following changes:

  • added ebs-volume-type config which allow ebs volume type from configuration since amazon linux 2023 by default comes with gp3 volume image
  • updated setup-ephemeral-storage.py since new version of lsblk doesn't produce same output for setup.
  • moved python package installation before running setup-ephemeral-storage.py
  • instance data IMDSv2 fix for spark public hostname

I tested this PR by...
launching spark cluster with latest amazon linux 2023 and amazon linux 2 image. both working fine

@nchammas
Copy link
Owner

Thanks for this PR @pragnesh and sorry to create a conflict.

Could you explain the motivation for the following changes? They seem unrelated to the subject of this PR.

  • moved python package installation before running setup-ephemeral-storage.py
  • instance data IMDSv2 fix for spark public hostname

@nchammas
Copy link
Owner

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

@pragnesh
Copy link
Contributor Author

pragnesh commented Sep 19, 2023

Hi @nchammas ,

setup-ephemeral-storage.py is not running without python package,

amazon linux 2 has default python installed so it was working earlier but amazon linux 2023 need python to be installed so moved it before setting running setup-ephemeral-storage.py

instance data IMDSv2 is enabed by default for amazon linux 2023, so SPARK_PUBLIC_DNS need to be adjusted for functional spark web UI

link for Instance data difference between amazon linux 2 and amazon linux 2023

@pragnesh
Copy link
Contributor Author

pragnesh commented Sep 19, 2023

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

no i have not. we do you VPC but , launch it with public ip enabled.

I have face similar issue with IMDSV2 so hence this fix for SPARK_PUBLIC_DNS.

I have rebased my PR.

@pragnesh pragnesh force-pushed the amazon_linux_2023_fix branch 2 times, most recently from 95e2586 to e3c394d Compare September 20, 2023 06:12
@pragnesh
Copy link
Contributor Author

pragnesh commented Sep 20, 2023

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

I have updated PR to correctly set SPARK_PUBLIC_DNS in private VPC.
I have tested change on amazon linux 2 ami.
I have tested amazon linux 2023 ami with java preinstalled since Adoptium does not work with amazon linux 2023. Hardcoded releasever to 2 since Adoptium does not have amazon linux 2023 jdk
I have tried testing private vpc but due to some vpc network configuration i am unable to launch but SPARK_PUBLIC_DNS is set correctly within private vpc.

@nchammas
Copy link
Owner

Sorry to leave this PR hanging. It is still on my mind and I will hopefully pick it up after I get Flintrock 2.1 out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants