feat: BigQuery common and provisioner classes #138

man8pr · 2024-03-08T05:58:11Z

What this PR changes/adds

Added:

extensions/common/gcp/gcp-core/.../bigquery : common classes including BQ DataAddress, service client, and theBigQuery DataAddress Validator Extension to register DataAddress validators
extensions/control-plane/provision/provision-bigquery : BQ provisioner for generating access token for data plane sink, including the BigQuery Provision Extension to register the provisioner

Why it does that

The PR adds first basic version of the BigQuery extension to exchange data to and from BigQuery tables.

Further notes

To be added in following PR:
- BigQuery data plane
Limitations for this version:
- provisioner requires destination table to exist, doesn't create new table if it doesn't
- data is committed to the destination table only if no error occurred during the data transmission
- no transmission retry in case of error
- results are fetched via pages (size is fixed, 4 rows), max. size for serialized page data is 16 KiB
- credentials for BQ services have default 1hr lifetime

Linked Issue(s)

Closes #50

Added: - extensions/common/gcp/gcp-core/.../bigquery : common classes including BQ DataAddress, service client, and theBigQuery DataAddress Validator Extension to register DataAddress validators - extensions/control-plane/provision/provision-bigquery : BQ provisioner for generating access token for data plane sink, including the BigQuery Provision Extension to register the provisioner Limitations: - provisioner requires destination table to exist, doesn't create new table if it doesn't - data is committed to the destination table only if no error occurred during the data transmission - no transmission retry in case of error - results are fetched via pages (size is fixed, 4 rows), max. size for serialized page data is 16 KiB

paullatzelsperger · 2024-03-08T15:08:57Z

@man8pr I added the "Closes" keyword to the PR description, to automatically link the PR with the issue

paullatzelsperger

cool stuff overall, comments inline

...ions/common/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/BigQueryDataAddress.java

extensions/common/gcp/gcp-core/src/test/java/org/eclipse/edc/gcp/bigquery/Asserts.java

paullatzelsperger · 2024-03-11T07:02:28Z

.../common/gcp/gcp-core/src/test/java/org/eclipse/edc/gcp/bigquery/BigQueryServiceImplTest.java

+        testRunSinkQuery(false);
+    }
+
+    private void testRunSourceQuery(String sinkType) throws InterruptedException, IOException {


nit: method ordering should be such, that test methods go first, then protected methods then private methods.

Updated ordering

paullatzelsperger · 2024-03-11T07:03:33Z

...cp-core/src/test/java/org/eclipse/edc/gcp/bigquery/BigQuerySinkDataAddressValidatorTest.java

+    }
+
+    @Test
+    void testSinkValidatorShouldNotValidateIncompleteAddresses() {


nit: i'd prefer having one Test method per scenario, i.e. every violation should have its own method. That greatly improves maintenance and readability

Split the method into single violation methods

paullatzelsperger · 2024-03-11T07:03:46Z

extensions/common/gcp/gcp-core/src/test/java/org/eclipse/edc/gcp/bigquery/PartAssert.java

+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class PartAssert extends AbstractAssert<PartAssert, Part> {


doc. and does this have to be public? seems like package-private would be fine too.

paullatzelsperger · 2024-03-11T07:05:02Z

...ava/org/eclipse/edc/connector/provision/gcp/BigQueryConsumerResourceDefinitionGenerator.java

+
+    @Override
+    public @Nullable
+    ResourceDefinition generate(DataRequest dataRequest, Policy policy) {


heads up: the signature for this interface changed from DataRequest -> TransferProcess. you should rebase onto latest main

Thank you, updated interface

ndr-brt

I focused mainly on the service implementation, the big issue here is the thread safety and how this class is used.

...ions/common/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/BigQueryServiceImpl.java

.../common/gcp/gcp-core/src/test/java/org/eclipse/edc/gcp/bigquery/BigQueryServiceImplTest.java

...ions/common/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/BigQueryServiceImpl.java

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java

...ions/common/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/BigQueryServiceImpl.java

...p/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQuerySourceServiceImpl.java

…n error fix, ExecutorService for running the query - Split BigQueryService and BigQueryServiceImpl in 3 parts, provision, source and sink, to address the specific needs - BigQueryDataAddress not serializable - BigQuerySourceServiceImpl using ExecutorService to track threading and thread pools - BigQuerySourceServiceImplTest.java less mocks to test the runSourceQuery function - BigQueryConsumerResourceDefinitionGenerator and test updated to reflect the new API

ndr-brt

the thread-safety issues are still there.

As a general rule, I suggest to remove from this PR all the code that's only used by tests (because it's dead-code) and to put it into a separate pr that will show their usage within an integration/e2e test, that will make the review easier.

...cp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryProvisionServiceImpl.java

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java

…arameter in BQ ProvisionService tableExists method

…h BigQueryFactory - replaced BigQueryProviderService* classes with BigQueryFactory / BigQuery for simpler implementation - removed TypeManager member class from Provisioner (not used) - removed unused classes in the test implementation (will be bundled with the data plane classes)

.../common/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryFactory.java

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java

- remove last empty line

ndr-brt · 2024-03-20T14:49:17Z

...cp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryProvisionServiceImpl.java

+        var credentialProject = project;
+        if (credentialProject == null) {
+            credentialProject = gcpConfiguration.projectId();
+        }


this credentialProject could be a field of the service and initiated in the build() method, so there'll be no reason to initialize it twice (line 95 and line 131)

The class has been removed

ndr-brt · 2024-03-20T14:50:17Z

...cp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryProvisionServiceImpl.java

+        this.monitor = monitor;
+    }
+
+    private void initCredentials() throws IOException {


this method could return a GoogleCredentials without setting fields (like it's done at line 111 and 116). Doing this these side effects will disappear

The class has been removed

ndr-brt · 2024-03-20T14:56:53Z

...ore/src/test/java/org/eclipse/edc/gcp/bigquery/service/BigQueryProvisionServiceImplTest.java

+        reset(gcpConfiguration);
+        reset(monitor);
+        reset(bigQuery);
+        reset(table);
+        reset(typeManager);
+


reset is not needed here, please take a look at other tests around the EDC repositories to see how we mock

The class has been removed

ndr-brt · 2024-03-20T14:57:41Z

...ore/src/test/java/org/eclipse/edc/gcp/bigquery/service/BigQueryProvisionServiceImplTest.java

+            .build();
+
+        // New seed for each test.
+        random = new Random();


this is not used

The class has been removed

.../common/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryFactory.java

ndr-brt · 2024-03-26T13:48:24Z

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java

+            }
+
+            if (bqFactory == null) {
+                bqFactory = new BigQueryFactoryImpl();


why the factory is instantiated here? I'd say that it should be done in the extension and passed as collaborator of the provisioner

ndr-brt · 2024-03-26T13:49:43Z

...mon/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryFactoryImpl.java

+    }
+
+    @Override
+    public BigQuery createBigQuery(GcpConfiguration gcpConfiguration, Monitor monitor) throws IOException {


both GcpConfiguration and Monitor should be passed in construction, there's no point in passing them here.
Plus, if the method is not depending on "flow data" like "projectId" (as it was in the previous implementation), there's no need at all to have this factory, because the BigQuery instance build by the factory will always be the same, so it could be instantiated by the extension and injected directly

Done, BigQuery instance is different if the transfer request includes the name of a service account to be impersonated for the access

ndr-brt · 2024-03-26T13:51:05Z

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java

+        // TODO update target with the generated table name.
+
+        try {
+            String serviceAccountEmail = null;


this variable is not used, please remove it

ndr-brt · 2024-03-26T13:59:10Z

...-bigquery/src/test/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisionerTest.java

+        when(table.exists()).thenReturn(true);
+        when(bigQuery.getTable(TEST_TARGET.getTableId())).thenReturn(table);
+
+        var response = bigQueryProvisioner.provision(resourceDefinition, policy).join();


please avoid using join and use assertJ async capabilities (assertThat(future).succeedsWithin(...))

Done (thanks for the hint, the succeedsWithin is really good)

ndr-brt · 2024-03-26T14:00:22Z

...-bigquery/src/test/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisionerTest.java

+        assertThat(response.succeeded()).isTrue();
+        assertThat(response.getContent().getResource()).isInstanceOfSatisfying(BigQueryProvisionedResource.class, resource -> {
+            assertThat(resource.getId()).isEqualTo(RESOURCE_ID);
+            assertThat(resource.getTransferProcessId()).isEqualTo(TRANSFER_ID);
+            assertThat(resource.getProject()).isEqualTo(TEST_PROJECT);
+            assertThat(resource.getDataset()).isEqualTo(TEST_DATASET);
+            assertThat(resource.getTable()).isEqualTo(TEST_TABLE);
+            assertThat(resource.hasToken()).isTrue();
+            assertThat(resource.getCustomerName()).isEqualTo(CUSTOMER_NAME);
+        });


all the token creation logic is not tested

Added tests for testing the creation of the token

…e with token creation test

ndr-brt · 2024-03-27T11:00:57Z

...mon/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryFactoryImpl.java

+                .build().getService();
+    }
+
+    private BigQuery createBigQuery() throws IOException {


unused, please delete

ndr-brt · 2024-03-27T11:02:01Z

...-bigquery/src/test/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisionerTest.java

+                .isEqualTo(true);
+    }
+
+    private class BigQueryFactoryTest implements BigQueryFactory {


please use a mockito mock instead

ndr-brt · 2024-03-27T11:03:12Z

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java

+        return new BigQueryTarget(project, dataset, table);
+    }
+
+    public static class Builder {


this builder is not really needed as it's relying on a single newInstance method + build, just use the constructor instead

ndr-brt · 2024-03-27T11:06:06Z

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java

+            GcpAccessToken token = null;
+
+            if (serviceAccount != null) {
+                token = iamService.createAccessToken(serviceAccount);
+            } else {
+                serviceAccount = ADC_SERVICE_ACCOUNT;
+                token = iamService.createDefaultAccessToken();
+            }


this piece of code is duplicated in the GcsProvisioner, could be the iamService.createAccessToken that detects if the service account passed is a default one and in that case it will return the default access token?
This will simplify things here and give the knowledge about accounts to the iam service (that's the one that should have them)

Definitely, delegating the responsibility to IAM is good. Since I would change then both BQ and GCS provisioner, along with IAM, does it make sense to have this in a dedicated PR?

ndr-brt · 2024-03-27T11:08:25Z

...-bigquery/src/test/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisionerTest.java

+        }
+    }
+
+    private void provisionSucceeds(String serviceAccountName) {


this is not a safe way to test, because you pretty much replicated the production code logic, so it's more error prone, please structure the tests in the way that they are understandable from the test method itself.

Really good point, as exchanged, I make a first try, by clearly showing the setup using default credentials, and the setup for testing the use of specific service account

ndr-brt · 2024-03-27T11:09:03Z

...gquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryResourceDefinition.java

+        @Override
+        protected void verify() {
+            super.verify();
+            // TODO verify required fields.


please solve this TODO as it seems not too hard to do

ndr-brt · 2024-03-27T11:11:10Z

.../common/gcp/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryService.java

+/**
+ * Schema for BigQuery service.
+ */
+public interface BigQueryService {


Suggested change

public interface BigQueryService {

public interface BigQueryServiceSchema {

… BigQueryProvisioner with public ctor

… method - better grouping of test set up when using default credentials and when using specific service account

ndr-brt

looking forward for the "service account refactoring", please solve DEPENDENCIES and I will merge this

man8pr · 2024-04-02T07:01:01Z

looking forward for the "service account refactoring", please solve DEPENDENCIES and I will merge this

Sure, on the refactoring now, it shouldn't be a large change

man8pr added the enhancement New feature or request label Mar 8, 2024

man8pr requested a review from paullatzelsperger March 8, 2024 06:02

ndr-brt self-requested a review March 8, 2024 07:39

paullatzelsperger reviewed Mar 11, 2024

View reviewed changes

ndr-brt requested changes Mar 11, 2024

View reviewed changes

github-advanced-security bot found potential problems Mar 14, 2024

View reviewed changes

...p/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQuerySourceServiceImpl.java Fixed Show fixed Hide fixed

...p/gcp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQuerySourceServiceImpl.java Fixed Show fixed Hide fixed

man8pr force-pushed the feature/50_bigquery branch from 56175a1 to d9f4c9d Compare March 14, 2024 10:48

man8pr added 2 commits March 14, 2024 12:01

fix: test compilation error, update extension path

9ac3f2f

refactor: fix CodeQl findings

9b97a77

man8pr requested review from paullatzelsperger and ndr-brt March 14, 2024 11:16

ndr-brt requested changes Mar 19, 2024

View reviewed changes

...cp-core/src/main/java/org/eclipse/edc/gcp/bigquery/service/BigQueryProvisionServiceImpl.java Outdated Show resolved Hide resolved

...sion-bigquery/src/main/java/org/eclipse/edc/connector/provision/gcp/BigQueryProvisioner.java Outdated Show resolved Hide resolved

man8pr added 2 commits March 20, 2024 08:58

refactor: removed source and sink service implementations, restored p…

9f4a11e

…arameter in BQ ProvisionService tableExists method

refactor: removed member variable for service from BigQueryProvisioner

c7ea317

man8pr requested a review from ndr-brt March 20, 2024 10:29

man8pr and others added 5 commits March 20, 2024 11:58

chore: DEPENDENCIES alignment

1064bb6

Merge branch 'eclipse-edc:main' into feature/50_bigquery

1b15b39

refactor: sync with new GcpConfiguration API

40f88b6

chore: DEPENDENCIES

9706c14

github-advanced-security bot found potential problems Mar 26, 2024

View reviewed changes

man8pr and others added 5 commits March 26, 2024 13:00

fix: provisioner to use provided BigQuery factory

04f5f89

core: DEPENDENCIES

d835af2

chore: DEPENDENCIES

280353b

Merge branch 'eclipse-edc:main' into feature/50_bigquery

1c83b88

chore: DEPENDENCIES

56a1c0e

- remove last empty line

ndr-brt requested changes Mar 26, 2024

View reviewed changes

man8pr added 3 commits March 26, 2024 19:28

refactor: simplified BigQueryFactory, improved BQ Provisioner coverag…

2444088

…e with token creation test

chore: DEPENDENCIES

95c4b3b

test: removed mock resets in the Provisioner test

84c9b49

man8pr requested a review from ndr-brt March 26, 2024 18:51

ndr-brt requested changes Mar 27, 2024

View reviewed changes

man8pr added 2 commits March 28, 2024 07:51

refactor: rename BigQueryService to BigQueryServiceSchema, simplified…

31c7d38

… BigQueryProvisioner with public ctor

refactor: simplify test arrangement for BigQueryProvision 'provision'…

8c34097

… method - better grouping of test set up when using default credentials and when using specific service account

man8pr requested a review from ndr-brt March 28, 2024 13:54

ndr-brt approved these changes Mar 29, 2024

View reviewed changes

man8pr and others added 2 commits April 2, 2024 08:43

chore: DEPENDENCIES

8cf57fc

Merge branch 'main' into feature/50_bigquery

2d260a1

man8pr added 2 commits April 2, 2024 09:07

chore: DEPENDENCIES

12c8f0c

fix: compilation errors

16721c8

ndr-brt merged commit aa67ca9 into eclipse-edc:main Apr 2, 2024
14 checks passed

man8pr mentioned this pull request Apr 10, 2024

BigQuery dataplane extension #160

Closed


		import static org.assertj.core.api.Assertions.assertThat;

		public class PartAssert extends AbstractAssert<PartAssert, Part> {

	public interface BigQueryService {
	public interface BigQueryServiceSchema {

feat: BigQuery common and provisioner classes #138

feat: BigQuery common and provisioner classes #138

Conversation

man8pr commented Mar 8, 2024 • edited Loading

What this PR changes/adds

Why it does that

Further notes

Linked Issue(s)

paullatzelsperger commented Mar 8, 2024

paullatzelsperger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndr-brt left a comment

Choose a reason for hiding this comment

ndr-brt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndr-brt left a comment • edited Loading

Choose a reason for hiding this comment

man8pr commented Apr 2, 2024

man8pr commented Mar 8, 2024 •

edited

Loading

ndr-brt left a comment •

edited

Loading