Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add use_emulator setting to better align with object_store crate #436

Open
aersam opened this issue Nov 3, 2023 · 2 comments
Open

Add use_emulator setting to better align with object_store crate #436

aersam opened this issue Nov 3, 2023 · 2 comments

Comments

@aersam
Copy link
Contributor

aersam commented Nov 3, 2023

Hi there

In many datascience project, fsspec and object_store crate settings are used here and there. I found myself writing a conversion method to convert the settings keys from one to the other and mostly it's pretty easy to do.

  • pandas uses fsspec/adlfs
  • deltalake uses object_store
  • polars uses object_store
  • Most python libs use fsspec/adlfs

See https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants

Something I miss from adlfs is the use_emulator shortcut for Azure Blob Storage which should result in setting the connection_string to DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1; ( the default connection string of the azure storage emulator)

I think this one should be easy to add and make our lives a little easier :) would also do a PR

@aersam aersam changed the title Add use_emulator setting to better align with object_store create Add use_emulator setting to better align with object_store crate Nov 3, 2023
@TomAugspurger
Copy link
Contributor

Looks like Azure/azure-sdk-for-python#10040 is a similar issue in the upstream azure.storage.blob library. I think we should follow that.

@aersam
Copy link
Contributor Author

aersam commented Jul 3, 2024

MSFT does not seem to be willing to implement this, but I still think it would be useful

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants