Transfer a file to an AWS S3 bucket

[darxsky]

Hi everyone,
I'm trying to transfer a file to an AWS S3 bucket. When I launch the transfer request /v2/transferprocesses, I get the following error:

SEVERE 2024-05-31T14:11:56.901014111 StateMachineManager [TransferProcessManagerImpl] error caught
 Caused by: org.eclipse.edc.spi.persistence.EdcPersistenceException: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.eclipse.edc.connector.transfer.spi.types.ResourceDefinition$Builder` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
connectorb             |  at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 57] (through reference chain: org.eclipse.edc.connector.transfer.spi.types.ResourceManifest$Builder["definitions"]->java.util.ArrayList[0])
connectorb             |        at org.eclipse.edc.sql.store.AbstractSqlStore.fromJson(AbstractSqlStore.java:94)
connectorb             |        at org.eclipse.edc.connector.store.sql.transferprocess.store.SqlTransferProcessStore.mapTransferProcess(SqlTransferProcessStore.java:309)
connectorb             |        at org.eclipse.edc.sql.SqlQueryExecutor$1.tryAdvance(SqlQueryExecutor.java:128)
connectorb             |        ... 27 more
connectorb             | Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.eclipse.edc.connector.transfer.spi.types.ResourceDefinition$Builder` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
connectorb             |  at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 57] (through reference chain: org.eclipse.edc.connector.transfer.spi.types.ResourceManifest$Builder["definitions"]->java.util.ArrayList[0])
connectorb             |        at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
connectorb             |        at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1887)
connectorb             |        at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
connectorb             |        at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1375)
connectorb             |        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1508)
connectorb             |        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserializeFromObject(BuilderBasedDeserializer.java:319)
connectorb             |        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:240)
connectorb             |        at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170)
connectorb             |        at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
connectorb             |        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:1380)
connectorb             |        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361)
connectorb             |        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
connectorb             |        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
connectorb             |        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:158)
connectorb             |        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:293)
connectorb             |        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:217)
connectorb             |        at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
connectorb             |        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4899)
connectorb             |        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3846)
connectorb             |        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3814)
connectorb             |        at org.eclipse.edc.sql.store.AbstractSqlStore.fromJson(AbstractSqlStore.java:92)
connectorb             |        ... 29 more

I have added the "provision-aws-s3" dependency in the build.gradle.kts file and configured the environment variables (edc.aws.access.key and edc.aws.secret.access.key).

This is the dataDestination property in the transfer request:

"dataDestination": {
    "type": "AmazonS3",
    "region": "<my_region>",
    "bucketName": "<my_bucket>"
  }

Despite the fact that I registered the dataplane via the api /v2/dataplanes , I always get this error.

{
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@id": "http-pull-provider-dataplane",
  "url": "http://connectorb:22004/control/transfer",
  "allowedSourceTypes": [
    "HttpData"
  ],
  "allowedDestTypes": [
    "HttpProxy",
    "HttpData",
    "AmazonS3"
  ],
  "properties": {
    "https://w3id.org/edc/v0.0.1/ns/publicApiUrl": "http://connectorb:29291/public/"
  }
}

Should I setup anything else to transfer a file to an S3 bucket?

Thank you in advance.

P.S. I am using EDC Connector version 0.6.0

[darxsky]

Does anyone know how to solve the issue?

[ndr-brt]

ProviderPushTransferDataFlowController is the problem, it has been deprecated, you should use DataPlaneSignalingFlowController that is registered by transfer-data-plane-signaling (use it in the place of transfer-data-plane)

[darxsky]

Thanks again for the reply. Now the data flow starts correctly but stops due to a json deserialization problem. Is it my fault or the S3 extension? I used Json validator, passing the json of the request for the transfer which is valid. I think it refers to the transfer request json.
[EDIT] There was a problem with the AWS credentials vault entry. I removed the entry and deleted the "keyName" field from the transfer request json. Now the data flow starts correctly and the asset transfer from S3 to S3 works.
Is there a possibility to insert the asset into an existing bucket? Because as it is set up now, the bucket is created at the time of the transfer request.

Thanks again.

[ndr-brt]

Looking at the S3 provisioner it always creates a new bucket, so the possibility to use an already existing bucket would be a new feature.

[darxsky]

Ok, thanks a lot for the support.

[darxsky]

Looking at the S3 provisioner it always creates a new bucket, so the possibility to use an already existing bucket would be a new feature.

HIi @ndr-brt,
sorry to bring up the discussion again. Do you think it is possible to do this by adding an extension to the Eclipse connector?
Thanks.