configuration issue

[JunseoChoJJ]

try to deploy nessie + spark + iceberg + s3a on k8s envirnoment

working on nessie with postgresql but getting this error

pod/nessie-6f86d47f77-dxv4j 0/1 CrashLoopBackOff 10 (54s ago) 28m

WARN [io.qua.config] (main) Unrecognized configuration key "quarkus.rest.gzip.enabled" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
[org.pro.ser.con.ConfigChecks] (main) Both authentication (AuthN) and authorization (AuthZ) are disabled, all requests to Nessie will be permitted. This means: everybody with access to Nessie can read, write and change everything. Recommended action: Enable AuthN & AuthZ, see https://projectnessie.org/try/configuration/
[org.pro.qua.pro.sto.JdbcBackendBuilder] (main) Selected datasource: postgresql
ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus
Caused by: java.lang.RuntimeException: java.sql.SQLException: Acquisition timeout while waiting for new connection

nessie-postgresql.yaml file

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-pvc
  namespace: nessie
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: nfs-client
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-statefulset
  namespace: nessie
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  serviceName: postgres-service
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: nessie_catalog
            - name: POSTGRES_USER
              value: postgres_user
            - name: POSTGRES_PASSWORD
              value: postgres_password
          volumeMounts:
            - name: postgres-data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-data
          persistentVolumeClaim:
            claimName: postgres-data-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
  namespace: nessie
spec:
  type: NodePort
  selector:
    app: postgres
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432
      nodePort: 30001  # Choose a suitable nodePort
---
apiVersion: v1
kind: Secret
metadata:
  name: postgresql-creds
  namespace: nessie
type: Opaque
data:
  username: 'base64'
  password: 'base64'

postgres-creds

POSTGRESQL_USERNAME=postgres_user
POSTGRESQL_PASSWORD=postgres_password

values.yaml for helm chart

versionStoreType: JDBC
jdbc:  
  jdbcUrl: jdbc:postgresql://postgres-service:30001/nessie
  secret:
    name: postgres-creds
    username: POSTGRESQL_USERNAME
    password: POSTGRESQL_PASSWORD

[adutra]

@JunseoChoJJ I see 2 potential errors in your config:

1. it seems you are trying to make Nessie contact Postgres through the service's node port: 30001. I don't think that's going to work. Could you please try the service port instead, that is, 5432? Your jdbcUrl should start with: jdbc:postgresql://postgres-service:5432....
2. It seems that the catalog name is nessie_catalog and nessie must be the schema name. If this is correct, the jdbcUrl should be: jdbc:postgresql://postgres-service:5432/nessie_catalog?currentSchema=nessie.

[JunseoChoJJ]

@adutra thanks for helping me
once I change to
jdbcUrl: jdbc:postgresql://postgres-service:5432/nessie_catalog?currentSchema=nessie

I got this error
2024-06-03 09:21:08,867 WARN [io.qua.config] (main) Unrecognized configuration key "quarkus.rest.gzip.enabled" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2024-06-03 09:21:11,909 WARN [org.pro.ser.con.ConfigChecks] (main) Both authentication (AuthN) and authorization (AuthZ) are disabled, all requests to Nessie will be permitted. This means: everybody with access to Nessie can read, write and change everything. Recommended action: Enable AuthN & AuthZ, see https://projectnessie.org/try/configuration/
2024-06-03 09:21:11,914 INFO [org.pro.qua.pro.sto.JdbcBackendBuilder] (main) Selected datasource: postgresql
2024-06-03 09:21:11,968 WARN [io.agr.pool] (agroal-11) Datasource 'postgresql': Connection to postgres-service:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
2024-06-03 09:21:12,046 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus

[JunseoChoJJ]

this is my shell script when I delete namespace and reinstall whole process

kubectl create namespace nessie
kubectl apply -f nessie-postgresql.yaml
kubectl create secret generic postgres-creds --from-env-file="$PWD/postgres-creds" -n nessie
helm install -n nessie nessie nessie-helm/nessie -f values.yaml

[adutra]

I think you need to change your postgres-service service type to ClusterIP as well.

[JunseoChoJJ]

@adutra it worked thanks a lot, but still figuring it out with spark+iceberg+nessie+S3a

do you know why you need to change Nodeport to Cluster IP? I think this is more kubernetes thing

also, for jdbc url part, is there any instructions what to write more?
https://projectnessie.org/guides/kubernetes/#configuring-jdbc-version-stores
from this docs, it is hard for me what to write more

[adutra]

do you know why you need to change Nodeport to Cluster IP? I think this is more kubernetes thing

NodePort service should be used when contacting the service from outside the K8s cluster. From inside you need a ClusterIP service. If you need to access the service from both inside and outside, you can create 2 services.

also, for jdbc url part, is there any instructions what to write more?

That's more a problem of understanding how Postgres interprets the jdbc URL:

https://jdbc.postgresql.org/documentation/use/

In general, you need to provide at least: jdbc:postgresql://host:5432/database. If you also need to specify the schema, use jdbc:postgresql://host:5432/database?currentSchema=schema.

Note that the database and the schema need to be created upfront.

CREATE DATABASE mydb;
\connect mydb
CREATE SCHEMA myschema;

[JunseoChoJJ]

@adutra thanks a lot

for second question,
for the database name it already made from yaml file so I just need to go into database and make schema

 env:
            - name: POSTGRES_DB
              value: nessie_catalog
 

kubectl exec -it postgres-statefulset-0 -n nessie -- /bin/bash
psql -U postgres_user -d nessie_catalog
CREATE schema nessie;
#list the schema
\dn

#quit
\q

[adutra]

Yes, I think that's all you need to do. Nessie will create the tables for you, inside the catalog nessie_catalog and schema nessie.

Your URL must be like this: jdbc:postgresql://host:5432/nessie_catalog?currentSchema=nessie.

[JunseoChoJJ]

@adutra Can I ask you about namespace? Is it schema name?

I got this error when I am deploying sparkapplication on spark-operator

 org.apache.iceberg.exceptions.NoSuchNamespaceException: Namespace does not exist: nessie
	at org.apache.iceberg.nessie.NessieUtil.maybeUseSpecializedException(NessieUtil.java:302)
	at org.apache.iceberg.nessie.NessieUtil.handleExceptionsForCommits(NessieUtil.java:224)
	at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:125)
	at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
	at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
	at org.apache.iceberg.CachingCatalog$CachingTableBuilder.lambda$create$0(CachingCatalog.java:261)

https://github.com/projectnessie/nessie-demos/blob/main/notebooks/nessie-iceberg-demo-nba.ipynb
From above docs, it seems like creating a namespace is similar to creating a schema. Is that correct? Is there a specific reason for using this terminology

and for spark with nessice.uri
.set("spark.sql.catalog.nessie_catalog.uri", "http://...:32327/api/v2"

why do you need to put api/v2 in the end?

when I didn't put api/v2, i got an error for "please configure client-api-version in the catalog properties explicity.

[JunseoChoJJ]

spark.sql("CREATE NAMESPACE IF NOT EXISTS nessie_catalog.nessie")
use this command to solve problem
and finally worked I'm deeply thankful to @adutra

but still need to look at branch thing :)

[adutra]

Is there a specific reason for using this terminology

That's the common terminology in the datalake space 🤷‍♂️

use this command to solve problem

Glad to hear! And yes, you need to create a namespace first before creating tables inside the namespace. Namespaces are not implicitly created.

[JunseoChoJJ]

@adutra
does nessie ui support tag, branch ui ? or does it not supported yet?

spark.sql("Alter Table nessie_catalog.nessie.sample CREATE TAG JUNE_2024 RETAIN 10 DAYS")

spark.sql("INSERT INTO nessie_catalog.nessie.sample values (4, 'd'), (5, 'e')")
df = spark.sql("SELECT * FROM nessie_catalog.nessie.sample")
df.show()


df = spark.sql("SELECT * FROM nessie_catalog.sample VERSION AS OF 'JUNE_2024'")
df.show()

tag and branch are creating well but I cannot get into nessie ui tag above images .
Did I do something wrong?
might be wrong bc I see this article https://www.dremio.com/blog/exploring-branch-tags-in-apache-iceberg-using-spark/
Is it vendor specialize sql command so that I cannot see the tag in the nessie web ui?

[dimas-b]

@JunseoChoJJ : That web UI is very old and badly needs maintenance 😅

Please consider using the new CLI: https://projectnessie.org/nessie-latest/cli/

[dimas-b]

Alter Table nessie_catalog.nessie.sample CREATE TAG JUNE_2024 RETAIN 10 DAYS

This looks to me like it's create an Iceberg tag in that table.

Nessie tags are at catalog level (apply to all tables). https://projectnessie.org/guides/sql/