How can i use terminal to read data from minio?

[NThanhDuong]

Hi, i have already minio running, now i wanna read data from minio storage. im newbie. Thanks!

[zhoujinsong]

Hi,

Amoro is a Lakehouse management system based on open data lake formats like Apache Iceberg.
So you may want to choose a data lake table format before using Amoro, I suggest that you try using the Iceberg Format first, as it is already a popular open-source data lake table format. You can find more information about Apache Iceberg at:https://iceberg.apache.org/
Apache Iceberg has integrated with S3 storage. I understand that MinIO also supports the S3 protocol, so you can use Iceberg on MinIO just like Iceberg on S3.
You can follow the steps below to try using Iceberg on MinIO：

• Clone project sources into your local envrionment
• Switch to 0.5.x branch
• Build project with aws profile: mvn clean package -DskipTests -Paws -pl '!trino'
• Restart the AMS through the new newly packaged binary file, you can find it under: {PROJECT_ROOT}/dist/target/amoro-0.5.0-bin.zip
• Create a new catalog on AMS Dashboard and configure it as:
  1. Choose Internal Catalog
  2. Choose Iceberg table format
  3. Ignore the storage configuration
  4. Fill the Authentication configuration with SIMPLE and amoro
  5. Add the warehouse property in properties configuration, set the value to the path you want to store your Iceberg tables.
  6. Add some extra properties about the MinIO like AWS_REGION/AWS_SECRET_KEY/AWS_ACESS_KEY/s3.endpoint if needed

Then you can create Iceberg tables through Terminal now, you can start your quickstart journey followed by:https://amoro.netease.com/quick-demo/

I paste a screenshot to help you to configue the catalog, wish it can help.

[NThanhDuong]

i built project with aws but in folder lib dont have file jar to resolve error "org.apache.hadoop.fs.s3a.S3AFileSystem". And if i need download file hadoop-aws-.jar and aws-java-sdk-.jar so what is the version i need choose

[Kyofin]

Maybe you can try hadoop-aws 3.2.2 and Ams 0.5.0 with hadoop3

[NThanhDuong]

i met this error after i add file jar

this's my config catalog:

how can i fix this error

[baiyangtx]

you can use s3:// instead of s3a://

[baiyangtx]

you can task a look at this https://zhuanlan.zhihu.com/p/644789092

[baiyangtx]

The profile aws only add aws-sdk to final dist. It can access s3 via s3:// but can't use s3a:// .

The s3a:// required a hadoop-aws bound and visit s3 throuth hadoop api.

[zhoujinsong]

Do you copy the 3 jars into the lib folder manually?

If you rebuild the project with command mvn clean package -DskipTests -Paws -pl '!trino', you will find these 6 jars in the lib folder. And they are the necessary dependencies for running iceberg on S3.

[NThanhDuong]

after i rebuild project with command mvn clean package -DskipTests -Paws -pl '!trino' finished.
this's my result:

but it show new error:

2023/09/11 15:59:37 new sql script submit, current thread pool state. [Active: 1, PoolSize: 1]
2023/09/11 15:59:37 terminal session dose not exists. create session first
2023/09/11 15:59:41 create a new terminal session.
2023/09/11 15:59:41 fetch terminal session: node0657qsohdosa6xh7ybazun2tb0.node0-SIMPLE-amoro-aws
setup session, session factory: com.netease.arctic.server.terminal.local.LocalSessionFactory
spark.sql.catalog.aws org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.aws.AWS_REGION us-east-1
spark.sql.catalog.aws.table.self-optimizing.group local
spark.sql.catalog.aws.s3.endpoint http://10.159.19.101:9003
spark.sql.catalog.aws.catalog-impl org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.aws.table-formats ICEBERG
spark.sql.catalog.aws.AWS_SECRET_KEY minioadmin
spark.sql.catalog.aws.warehouse aws
spark.sql.catalog.aws.AWS_ACCESS_KEY minioadmin
spark.sql.catalog.aws.uri http://127.0.0.1:1630/api/iceberg/rest
spark.sql.arctic.refresh-catalog-before-usage true
2023/09/11 15:59:41 session configuration: catalog-url-base => thrift://127.0.0.1:1260
2023/09/11 15:59:41 session configuration: catalog.aws.table.self-optimizing.group => local
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.table.self-optimizing.group => local
2023/09/11 15:59:41 session configuration: spark.sql.arctic.refresh-catalog-before-usage => true
2023/09/11 15:59:41 session configuration: catalog.aws.catalog-impl => org.apache.iceberg.rest.RESTCatalog
2023/09/11 15:59:41 session configuration: catalog.aws.warehouse => aws
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.AWS_REGION => us-east-1
2023/09/11 15:59:41 session configuration: session.catalogs => aws
2023/09/11 15:59:41 session configuration: catalog.aws.AWS_SECRET_KEY => minioadmin
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.uri => http://127.0.0.1:1630/api/iceberg/rest
2023/09/11 15:59:41 session configuration: session.catalog.aws.connector => iceberg
2023/09/11 15:59:41 session configuration: catalog.aws.table-formats => ICEBERG
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.catalog-impl => org.apache.iceberg.rest.RESTCatalog
2023/09/11 15:59:41 session configuration: catalog.aws.AWS_ACCESS_KEY => minioadmin
2023/09/11 15:59:41 session configuration: catalog.aws.AWS_REGION => us-east-1
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.AWS_SECRET_KEY => minioadmin
2023/09/11 15:59:41 session configuration: session.fetch-size => 1000
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws => org.apache.iceberg.spark.SparkCatalog
2023/09/11 15:59:41 session configuration: catalog.aws.uri => http://127.0.0.1:1630/api/iceberg/rest
2023/09/11 15:59:41 session configuration: catalog.aws.s3.endpoint => http://10.159.19.101:9003
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.table-formats => ICEBERG
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.s3.endpoint => http://10.159.19.101:9003
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.AWS_ACCESS_KEY => minioadmin
2023/09/11 15:59:41 session configuration: spark.sql.catalog.aws.warehouse => aws
2023/09/11 15:59:41
2023/09/11 15:59:41 prepare execute statement, line:1
2023/09/11 15:59:41 USE db
switch to new catalog via: use aws
2023/09/11 15:59:45 statement execute down, result is empty, execution cost: 4202ms
2023/09/11 15:59:45
2023/09/11 15:59:45 prepare execute statement, line:2
2023/09/11 15:59:45 CREATE TABLE IF NOT EXISTS user ( id INT, name string, ts TIMESTAMP ) USING iceberg PARTITIONED BY (days(ts))
2023/09/11 15:59:59 meet exception during execution.
2023/09/11 15:59:59 org.apache.iceberg.exceptions.ServiceFailureException: Server error: SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@6b4446e7: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@7c16a90d: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@69748135: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@5daf632a: Unable to contact EC2 metadata service.]
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:162)
at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:109)
at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:93)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:176)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:285)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:219)
at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:330)
at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:112)
at org.apache.iceberg.rest.RESTSessionCatalog$Builder.create(RESTSessionCatalog.java:603)
at org.apache.iceberg.CachingCatalog$CachingTableBuilder.lambda$create$0(CachingCatalog.java:261)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2344)
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2342)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2325)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at org.apache.iceberg.CachingCatalog$CachingTableBuilder.create(CachingCatalog.java:257)
at org.apache.iceberg.spark.SparkCatalog.createTable(SparkCatalog.java:237)
at org.apache.spark.sql.execution.datasources.v2.CreateTableExec.run(CreateTableExec.scala:45)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at com.netease.arctic.server.terminal.local.LocalTerminalSession.executeStatement(LocalTerminalSession.java:73)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.executeStatement(TerminalSessionContext.java:277)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.execute(TerminalSessionContext.java:240)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.lambda$get$0(TerminalSessionContext.java:201)
at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:363)
at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:343)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:343)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.get(TerminalSessionContext.java:193)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.get(TerminalSessionContext.java:164)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

[baiyangtx]

Please add AWS_REGION to the environment variables launched by AMS. This may be due to compatibility issues with the AWS SDK itself.

[NThanhDuong]

did i add right ?:

[baiyangtx]

No, please add to ${AMORO_HOME}/bin/config.sh

[NThanhDuong]

i had re-config catalog below:

and already read table into MinIO:

but when i run query in terminal to read table then met this error:

2023/09/12 15:12:31 new sql script submit, current thread pool state. [Active: 0, PoolSize: 1]
2023/09/12 15:12:31 fetch terminal session: node0wsxet2pojlj7hrregis210o20.node0-SIMPLE-amoro-test
2023/09/12 15:12:31 session configuration: catalog-url-base => thrift://127.0.0.1:1260
2023/09/12 15:12:31 session configuration: session.catalog.test.connector => iceberg
2023/09/12 15:12:31 session configuration: catalog.test.type => hive
2023/09/12 15:12:31 session configuration: spark.sql.catalog.test.table.self-optimizing.group => local
2023/09/12 15:12:31 session configuration: spark.sql.arctic.refresh-catalog-before-usage => true
2023/09/12 15:12:31 session configuration: spark.sql.catalog.test => org.apache.iceberg.spark.SparkCatalog
2023/09/12 15:12:31 session configuration: spark.sql.catalog.test.table-formats => ICEBERG
2023/09/12 15:12:31 session configuration: session.catalogs => test
2023/09/12 15:12:31 session configuration: catalog.test.table-formats => ICEBERG
2023/09/12 15:12:31 session configuration: spark.sql.catalog.test.type => hive
2023/09/12 15:12:31 session configuration: spark.sql.catalog.test.warehouse => s3://datalake/lakehouse
2023/09/12 15:12:31 session configuration: catalog.test.table.self-optimizing.group => local
2023/09/12 15:12:31 session configuration: session.fetch-size => 1000
2023/09/12 15:12:31 session configuration: catalog.test.warehouse => s3://datalake/lakehouse
2023/09/12 15:12:31
2023/09/12 15:12:31 prepare execute statement, line:1
2023/09/12 15:12:31 use default
2023/09/12 15:12:31 statement execute down, result is empty, execution cost: 32ms
2023/09/12 15:12:31
2023/09/12 15:12:31 prepare execute statement, line:2
2023/09/12 15:12:31 select * from airbyte_raw_test_user1
2023/09/12 15:12:32 meet exception during execution.
2023/09/12 15:12:32 java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3341)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3373)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:125)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3424)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3392)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:485)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.iceberg.hadoop.Util.getFs(Util.java:56)
at org.apache.iceberg.hadoop.HadoopInputFile.fromLocation(HadoopInputFile.java:56)
at org.apache.iceberg.hadoop.HadoopFileIO.newInputFile(HadoopFileIO.java:90)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:266)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:189)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:208)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:208)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:185)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:180)
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:178)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)
at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:80)
at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:47)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2344)
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2342)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2325)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at org.apache.iceberg.CachingCatalog.loadTable(CachingCatalog.java:166)
at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:642)
at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:160)
at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:311)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$lookupRelation$3(Analyzer.scala:1202)
at scala.Option.orElse(Option.scala:447)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$lookupRelation$1(Analyzer.scala:1201)
at scala.Option.orElse(Option.scala:447)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupRelation(Analyzer.scala:1193)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$13.applyOrElse(Analyzer.scala:1064)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$13.applyOrElse(Analyzer.scala:1028)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$2(AnalysisHelper.scala:135)
at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1228)
at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1227)
at org.apache.spark.sql.catalyst.plans.logical.OrderPreservingUnaryNode.mapChildren(LogicalPlan.scala:208)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:135)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1028)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:987)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:211)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:208)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:200)
at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:231)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:227)
at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:173)
at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:227)
at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:188)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:179)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:179)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:212)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:211)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:76)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:185)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:184)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:76)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:74)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:66)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at com.netease.arctic.server.terminal.local.LocalTerminalSession.executeStatement(LocalTerminalSession.java:73)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.executeStatement(TerminalSessionContext.java:277)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.execute(TerminalSessionContext.java:240)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.lambda$get$0(TerminalSessionContext.java:201)
at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:363)
at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:343)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:343)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.get(TerminalSessionContext.java:193)
at com.netease.arctic.server.terminal.TerminalSessionContext$ExecutionTask.get(TerminalSessionContext.java:164)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
... 110 more
when select a table :

[zhoujinsong]

Since you have chosen Hive MetaStore as the metadata service, you need to configure the implementation of Iceberg FileIO in catalog properties. Specifically, you can try adding the following configuration in catalog properties：

io-impl=org.apache.iceberg.aws.s3.S3FileIO

[NThanhDuong]

i have many small file in MinIO while streaming data by kafka. even update data will have a new file created. i wanna compact and merge all file into one file  i use local optimize :   tables committing but it error:  this's error logs: com.netease.arctic.server.exception.OptimizingCommitException: unexpected commit error at com.netease.arctic.server.optimizing.UnKeyedTableCommit.replaceFiles(UnKeyedTableCommit.java:214) at com.netease.arctic.server.optimizing.UnKeyedTableCommit.commit(UnKeyedTableCommit.java:174) at com.netease.arctic.server.optimizing.OptimizingQueue$TableOptimizingProcess.commit(OptimizingQueue.java:529) at com.netease.arctic.server.table.executor.OptimizingCommitExecutor.execute(OptimizingCommitExecutor.java:32) at com.netease.arctic.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:53) at com.netease.arctic.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$4(BaseTableExecutor.java:65) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.iceberg.exceptions.RuntimeIOException: Failed to create file: s3a://datalake/lakehouse/postgres_ducdn__public_test_user/metadata/9dddf437-89d1-4c99-b95e-5a940f228609-m9.avro at org.apache.iceberg.hadoop.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:87) at org.apache.iceberg.avro.AvroFileAppender.<init>(AvroFileAppender.java:56) at org.apache.iceberg.avro.Avro$WriteBuilder.build(Avro.java:191) at org.apache.iceberg.ManifestWriter$V2Writer.newAppender(ManifestWriter.java:243) at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:58) at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:34) at org.apache.iceberg.ManifestWriter$V2Writer.<init>(ManifestWriter.java:220) at org.apache.iceberg.ManifestFiles.write(ManifestFiles.java:163) at org.apache.iceberg.SnapshotProducer.newManifestWriter(SnapshotProducer.java:488) at org.apache.iceberg.MergingSnapshotProducer$DataFileFilterManager.newManifestWriter(MergingSnapshotProducer.java:1057) at org.apache.iceberg.ManifestFilterManager.filterManifestWithDeletedFiles(ManifestFilterManager.java:414) at org.apache.iceberg.ManifestFilterManager.filterManifest(ManifestFilterManager.java:319) at org.apache.iceberg.ManifestFilterManager.lambda$filterManifests$0(ManifestFilterManager.java:195) at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:69) at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Caused by: java.io.InterruptedIOException: getFileStatus on s3a://datalake/lakehouse/postgres_ducdn__public_test_user/metadata/9dddf437-89d1-4c99-b95e-5a940f228609-m9.avro: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at org.apache.hadoop.fs.s3a.S3AUtils.translateInterruptedException(S3AUtils.java:352) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:177) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:151) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2278) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160) at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:804) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1105) at org.apache.hadoop.fs.FileSystem.create( Vào Th 3, 12 thg 9, 2023 vào lúc 18:23 ZhouJinsong < ***@***.***> đã viết:

…

Since you have chosen Hive MetaStore as the metadata service, you need to configure the implementation of Iceberg FileIO in catalog properties. Specifically, you can try adding the following configuration in catalog properties： io-impl=org.apache.iceberg.aws.s3.S3FileIO — Reply to this email directly, view it on GitHub <#1933 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AU22MFFX7OOIM4DRU5ETTG3X2BA2HANCNFSM6AAAAAA4OTDCGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[baiyangtx]

I can't see the full logs.Can you paste the full log?

[NThanhDuong]

i can't see full log too

[baiyangtx]

you can try this.

• catalog: internal catalog
• format: iceberg
• properties:
  • s3.access-key-id: $YOUR_AK
  • s3.secret-access-key: $YOUR_SK
  • s3.endpoint: https://$YOUR_S3_ENDPOINT
  • warehouse: s3://${warehouse_path}
  • client.region: s3-region

This is a solution that other users in the community have implemented.

[baiyangtx]

Look at this: https://iceberg.apache.org/docs/latest/configuration/

history.expire.max-snapshot-age-ms

[NThanhDuong]

can i use snapshot.base.keep.minutes in Data-cleaning configurations to set time delete small files ?

[zhoujinsong]

Yes, Amoro will expire snapshot automatically with comfiguration snapshot.base.keep.minutes while table-expire.enabled is true.

[NThanhDuong]

when t use hive metastore, amoro auto show tables like into my minio:

but i cant remove this catalog. it errors "Some tables in catalog!"

[zhoujinsong]

It seems to be a bug about removing a catalog.
Can you help to create a issue?
We will fix it soon and released it in the next version.

BTW, you can delete it from catalog_metadata table in database and restart the AMS before we fixing it.