feat: add generic Iceberg catalog adapter creation to Java / Python

[lbooker42]

Java, connecting to a RESTCatalog using MinIO

import io.deephaven.iceberg.util.*;

properties = new HashMap<>();
properties.put("type", "rest");
properties.put("uri", "http://rest:8181");

properties.put("client.region", "us-east-1");

properties.put("s3.access-key-id", "admin");
properties.put("s3.secret-access-key", "password");
properties.put("s3.endpoint", "http://minio:9000");

adapter = IcebergTools.createAdapter("generic-adapter", properties);

Python, connecting to a RESTCatalog using MinIO

from deephaven.experimental import iceberg

adapter = iceberg.adapter(name="generic-adapter", properties={
    "type" : "rest",
    "uri" : "http://rest:8181",
    "client.region" : "us-east-1",
    "s3.access-key-id" : "admin",
    "s3.secret-access-key" : "password",
    "s3.endpoint" : "http://minio:9000"
});

Java, connecting to AWS Glue

NOTE: credentials set in local environment

import io.deephaven.iceberg.util.*;

properties = new HashMap<>();
properties.put("type", "glue");
properties.put("uri", "s3://lab-warehouse/sales");

adapter = IcebergTools.createAdapter("generic-adapter", properties);

Python, connecting to AWS Glue

NOTE: credentials set in local environment

from deephaven.experimental import iceberg

adapter = iceberg.adapter(name="generic-adapter", properties={
    "type" : "glue",
    "uri" : "s3://lab-warehouse/sales",
    "warehouse" : "s3://lab-warehouse/sales",
});

[lbooker42]

We include the support libraries for REST and Glue catalogs but others will need the user to have the libraries in the class path. These are the catalogs that will need additional files:

• org.apache.iceberg.hive.HiveCatalog
• org.apache.iceberg.hadoop.HadoopCatalog
• org.apache.iceberg.nessie.NessieCatalog
• org.apache.iceberg.jdbc.JdbcCatalog

[devinrsmith]

I'm curious; how does pyiceberg deduce the catalog in the following example:

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "default",
    **{
        "uri": "http://rest:8181/",
        "s3.endpoint": "http://minio:9000/",
        "s3.access-key-id": "minioadmin",
        "s3.secret-access-key": "minioadmin",
    }
)

taxi_dataset = catalog.load_table("default.taxi_dataset").to_arrow()

Maybe we could use the same strategy? (I'm looking into it and will report back.)

[devinrsmith]

Looks like they infer the type if the uri property is provided. https://github.com/apache/iceberg-python/blob/pyiceberg-0.6.1/pyiceberg/catalog/__init__.py#L155C5-L182

            if uri.startswith("http"):
                return CatalogType.REST
            elif uri.startswith("thrift"):
                return CatalogType.HIVE
            elif uri.startswith(("sqlite", "postgresql")):
                return CatalogType.SQL

[rcaudy]

Reviewed. Seems like a nice PR, if it works.

[chipkent]

Reviwed python

[devinrsmith]

I'm going to try and use it and report back.

[devinrsmith]

I think we should compare this directly to https://py.iceberg.apache.org/reference/pyiceberg/catalog/#pyiceberg.catalog.load_catalog, https://github.com/apache/iceberg-python/blob/pyiceberg-0.6.1/pyiceberg/catalog/__init__.py#L185-L200

def load_catalog(name: Optional[str] = None, **properties: Optional[str]) -> Catalog:

Should we call this load_catalog_adapter, load_catalog, or load_adapter instead? Should we match the same sort of signature, ie using **properties?

In addition, I think from python standpoint, we also need to compare to the wider pyiceberg API to compare / contrast, https://py.iceberg.apache.org/api/

I don't think we necessarily need to dive deep into the details today, but we could potentially add a layer of yaml config in the future like pyiceberg does with ~/.pyiceberg.yaml (in which case, the config is looked up by name).

[devinrsmith]

Based on the order of args, we are not able to do adapter("name", { "properties": ... }), which seems like the more natural order?

[lbooker42]

The new call is IcebergCatalogAdapter.adapter(), I think that is pretty clear but I'm open to new name.

[devinrsmith]

We might be able to infer the catalog type based on URI like pyiceberg does:

• https://py.iceberg.apache.org/reference/pyiceberg/catalog/#pyiceberg.catalog.infer_catalog_type
• https://github.com/apache/iceberg-python/blob/pyiceberg-0.6.1/pyiceberg/catalog/__init__.py#L155-L182

[lbooker42]

We can do this and it will bring us closer to PyIceberg configuration. My approach was not to repeat all the hand-holding that pyiceberg does, but to mirror the Java Iceberg API which is more barebones.

Not sure which approach is better, but I lean toward less code and more user responsibility (esp. where it matches Iceberg Java API).

[rcaudy]

I'm OK with introducing these kinds of helpers, as long as we can be sure that they are:

1. Unambiguous
2. Unlikely to break later because of changes to the expected properties

That said, we can merge property-driven catalog support independently and add these kinds of "nice to have" features later; they will not change the interface, as far as I can tell.

Accidental approve.

[chipkent]

Can this be unit tested yet?

[chipkent]

Sounds like it is not testable yet.

[deephaven-internal]

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#305