feat: support storage options for spark read and write

[SaintBacchus]

#2854
Set storage options configuration in spark-default

spark.sql.catalog.lance com.lancedb.lance.spark.LanceCatalog
spark.sql.catalog.lance.access_key_id AKAKAKAKAKAKAKAKAKAKAKAKAKAKAKAKAKAKAKAK
spark.sql.catalog.lance.secret_access_key SKSKSKSKSKSKSKSKSKSKSKSKSKSKSKSKSKSK
spark.sql.catalog.lance.aws_region region
spark.sql.catalog.lance.aws_endpoint https://Endpoint
spark.sql.catalog.lance.virtual_hosted_style_request true

Then you can read and write lance file for object store.

val df = Seq(
  ("Alice", 1),
  ("Bob", 2)
).toDF("name", "id")

df.write.format("lance").option("path", "s3://test/lance/demo.lance").save()


val data = spark.read.format("lance").option("path", "s3://test/lance/demo.lance").load();

data.show(10)

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[LuQQiu]

@SaintBacchus thanks for contributing to the Java API! Awesome! Could you help fix the "cargo clippy --all-targets -- -D warnings" errors shown in the test https://github.com/lancedb/lance/actions/runs/11254749789/job/31292838444?pr=2990

[LuQQiu]

Can we put SparkOptions inside LanceConfig?
LanceConfig is expected to be the centralized place of putting Spark related configuration.
It can have Optional of ReadOptions and Optional of WriteParams?

[SaintBacchus]

OK, I will fix the ut later.

[LuQQiu]

LGTM