Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] ppl #1770

Open
wants to merge 89 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
658c952
Pipeline operations AST
wu-hui Feb 14, 2024
4ab3382
Getting ready for fluent api
wu-hui Feb 15, 2024
a3d7234
Basic fluent
wu-hui Feb 15, 2024
a837f62
Joins too
wu-hui Feb 17, 2024
c8e7854
Some fixups, and group bys
wu-hui Feb 20, 2024
c5a01a6
joins but nicer
wu-hui Feb 20, 2024
191eb44
address feedback
wu-hui Feb 22, 2024
27f2748
pagination and others
wu-hui Feb 23, 2024
55ace39
minor adds
wu-hui Feb 26, 2024
56c3a41
adding more sugar to improve brevity
wu-hui Feb 27, 2024
a5a90ab
address some feedback
wu-hui Feb 28, 2024
424ac43
support fromData
wu-hui Feb 28, 2024
4d8a205
pipeline result basic
wu-hui Mar 7, 2024
9119b95
execute
wu-hui Mar 7, 2024
3aa73de
fixing compilation error and add some high level comments
wu-hui Mar 11, 2024
b583791
setup nix for idx
wu-hui Mar 11, 2024
670ae65
Merge pull request #3 from wu-hui/wuandy/IdxSetup
wu-hui Mar 11, 2024
7216745
tweaks
wu-hui Mar 11, 2024
c46601f
pipeline result basic
wu-hui Mar 7, 2024
99e9067
execute
wu-hui Mar 7, 2024
f20e409
fixing compilation error and add some high level comments
wu-hui Mar 11, 2024
742d3c4
tweaks
wu-hui Mar 11, 2024
637d494
Merge remote-tracking branch 'pipelines/wuandy/PipelineResult' into w…
wu-hui Mar 11, 2024
7c67e99
fix errors
wu-hui Mar 11, 2024
ba521a0
fix errors
wu-hui Mar 12, 2024
6ae7b13
Add function composition example.
wu-hui Mar 12, 2024
c1825e0
build into a single jar
wu-hui Mar 13, 2024
0fe5faa
better aliasing for joins
wu-hui Mar 15, 2024
0d52712
Merge pull request #2 from wu-hui/wuandy/PipelineResult
wu-hui Mar 15, 2024
7ab3099
add stuff to support challenges
wu-hui Mar 18, 2024
57f0b74
edits
wu-hui Apr 8, 2024
0629026
Initial PP pipeline api
wu-hui Apr 9, 2024
a557d4d
pull in proto change and regenerate
wu-hui Apr 12, 2024
3369473
pull in proto change and regenerate
wu-hui Apr 12, 2024
9486027
Basic serialization
wu-hui Apr 17, 2024
ed2d276
query to pipeline
wu-hui Apr 26, 2024
f551073
e2e tests fine tuning
wu-hui Apr 29, 2024
1ceab8a
visibility changes
wu-hui May 6, 2024
df21cf2
refactoring
wu-hui May 14, 2024
cfea0fd
disable failing tests temparorily
wu-hui May 16, 2024
c2b0771
Revert "disable failing tests temparorily"
wu-hui May 16, 2024
bb16ca1
Fix missing asAlias
wu-hui May 21, 2024
424cf8b
Rename project to select
wu-hui May 21, 2024
8bc968f
fix countAll
wu-hui May 22, 2024
8c864f7
order normalization and field unify
wu-hui May 27, 2024
a79e8a7
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 27, 2024
a4ea45e
Merge branch 'main' into wuandy/PplPP
wu-hui May 28, 2024
ca9a22b
Merge remote-tracking branch 'origin/wuandy/PplPP' into wuandy/PplPP
wu-hui May 28, 2024
93135a8
Add jvmfile name and string as field heuristics
wu-hui May 29, 2024
7c15d5d
convert to java
wu-hui May 30, 2024
42cc0e5
Tweaks
wu-hui Jun 11, 2024
b27c5b8
Tweaks
wu-hui Jun 14, 2024
392dd84
minor fixes
wu-hui Jun 26, 2024
2568d6a
add missing functions
wu-hui Jul 3, 2024
478c46c
Function and stage renames
wu-hui Jul 16, 2024
12d4f29
Add annotations.
wu-hui Jul 17, 2024
593d39e
executions, group bys, etc.
wu-hui Jul 18, 2024
a1a6076
Fixes
wu-hui Jul 19, 2024
64622b1
startsWith and endsWith
wu-hui Jul 19, 2024
8165ce4
no Fields.ofAll
wu-hui Jul 22, 2024
29fcb1d
Add Distinct
wu-hui Jul 22, 2024
0c69e9f
Renames
wu-hui Jul 23, 2024
508eb7a
Add exists to toPipeline()
wu-hui Jul 25, 2024
f3934d8
Minor fixes
wu-hui Jul 30, 2024
a5f852b
Delete AccumulatorTarget
wu-hui Jul 30, 2024
ab2aaab
add public docs
wu-hui Aug 2, 2024
dc1b06b
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 2, 2024
7c697a5
Add public ref doc
wu-hui Aug 3, 2024
edc1cc9
Merge remote-tracking branch 'origin/wuandy/JavaPplPP' into wuandy/Ja…
wu-hui Aug 3, 2024
77decb5
Simplify Pipeline
tom-andersen Aug 12, 2024
f8b9386
Mirror Query class pattern
tom-andersen Aug 13, 2024
7355e49
Cleanup and sync with server function updates
wu-hui Aug 21, 2024
c51a9e5
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 21, 2024
07226db
Merge branch 'main' into wuandy/JavaPplPP
wu-hui Aug 26, 2024
e021fdf
Add proper vector value support
wu-hui Aug 26, 2024
8f4f27a
Merge remote-tracking branch 'origin/wuandy/JavaPplPP' into wuandy/Ja…
wu-hui Aug 26, 2024
eb5a815
Fix tests and license
wu-hui Aug 29, 2024
a62cdde
dot product and arrayConcat fix
wu-hui Sep 3, 2024
762b005
findnearest update and length rename
wu-hui Sep 11, 2024
5825cb6
remove __path__ hack
wu-hui Sep 12, 2024
3a965e9
Address feedback
wu-hui Sep 17, 2024
8d63fb6
Fix immutablelist parameters
wu-hui Sep 17, 2024
2f0e505
Merge branch 'tomandersen/ppSuggestion' into wuandy/JavaPplPP
wu-hui Sep 19, 2024
de7a147
Merge Tom's suggestions.
wu-hui Sep 19, 2024
f775b29
Sync with private preview catelog
wu-hui Sep 24, 2024
13fc2ec
more sync
wu-hui Sep 25, 2024
11a2ef2
cleanup countAll
wu-hui Sep 26, 2024
dda3c42
add new overloads
wu-hui Oct 2, 2024
17dc248
add transaction support
wu-hui Oct 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .idx/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@

gc/
17 changes: 17 additions & 0 deletions .idx/dev.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{ pkgs, ... }: {

# Which nixpkgs channel to use.
channel = "stable-23.11"; # or "unstable"

# Use https://search.nixos.org/packages to find packages
packages = [
pkgs.jdk11 # Or jdk8, jdk17, etc. - match your project's requirements
pkgs.maven
pkgs.kotlin
];

# Sets environment variables in the workspace
env = {
SOME_ENV_VAR = "hello";
};
}
3 changes: 3 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"java.compile.nullAnalysis.mode": "automatic"
}
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,20 @@ If you are using Maven without the BOM, add this to your dependencies:
If you are using Gradle 5.x or later, add this to your dependencies:

```Groovy
implementation platform('com.google.cloud:libraries-bom:26.39.0')
implementation platform('com.google.cloud:libraries-bom:26.43.0')

implementation 'com.google.cloud:google-cloud-firestore'
```
If you are using Gradle without BOM, add this to your dependencies:

```Groovy
implementation 'com.google.cloud:google-cloud-firestore:3.21.1'
implementation 'com.google.cloud:google-cloud-firestore:3.24.2'
```

If you are using SBT, add this to your dependencies:

```Scala
libraryDependencies += "com.google.cloud" % "google-cloud-firestore" % "3.21.1"
libraryDependencies += "com.google.cloud" % "google-cloud-firestore" % "3.24.2"
```
<!-- {x-version-update-end} -->

Expand Down Expand Up @@ -222,7 +222,7 @@ Java is a registered trademark of Oracle and/or its affiliates.
[kokoro-badge-link-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-firestore/java11.html
[stability-image]: https://img.shields.io/badge/stability-stable-green
[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-firestore.svg
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-firestore/3.21.1
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-firestore/3.24.2
[authentication]: https://github.com/googleapis/google-cloud-java#authentication
[auth-scopes]: https://developers.google.com/identity/protocols/oauth2/scopes
[predefined-iam-roles]: https://cloud.google.com/iam/docs/understanding-roles#predefined_roles
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 +486,15 @@ public FirestoreAdminStub createStub() throws IOException {
"Transport not supported: %s", getTransportChannelProvider().getTransportName()));
}

/** Returns the endpoint set by the user or the the service's default endpoint. */
@Override
public String getEndpoint() {
if (super.getEndpoint() != null) {
return super.getEndpoint();
}
return getDefaultEndpoint();
}

/** Returns the default service name. */
@Override
public String getServiceName() {
Expand Down Expand Up @@ -1377,6 +1386,15 @@ public UnaryCallSettings.Builder<RestoreDatabaseRequest, Operation> restoreDatab
return deleteBackupScheduleSettings;
}

/** Returns the endpoint set by the user or the the service's default endpoint. */
@Override
public String getEndpoint() {
if (super.getEndpoint() != null) {
return super.getEndpoint();
}
return getDefaultEndpoint();
}

@Override
public FirestoreAdminStubSettings build() throws IOException {
return new FirestoreAdminStubSettings(this);
Expand Down
36 changes: 36 additions & 0 deletions google-cloud-firestore/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,42 @@
<groupId>org.codehaus.mojo</groupId>
<artifactId>flatten-maven-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<executions>
<execution>
<id>default-compile</id>
<phase>none</phase>
</execution>
<execution>
<id>default-testCompile</id>
<phase>none</phase>
</execution>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>testCompile</id>
<phase>test-compile</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,15 @@
package com.google.cloud.firestore;

import com.google.api.core.ApiFuture;
import com.google.api.core.BetaApi;
import com.google.api.core.InternalExtensionOnly;
import com.google.api.core.SettableApiFuture;
import com.google.api.gax.rpc.ResponseObserver;
import com.google.api.gax.rpc.ServerStreamingCallable;
import com.google.api.gax.rpc.StatusCode;
import com.google.api.gax.rpc.StreamController;
import com.google.cloud.Timestamp;
import com.google.cloud.firestore.pipeline.expressions.ExprWithAlias;
import com.google.cloud.firestore.v1.FirestoreSettings;
import com.google.common.collect.ImmutableMap;
import com.google.firestore.v1.RunAggregationQueryRequest;
Expand Down Expand Up @@ -65,6 +67,17 @@ public Query getQuery() {
return query;
}

@Nonnull
@BetaApi
public Pipeline pipeline() {
return getQuery()
.pipeline()
.aggregate(
this.aggregateFieldList.stream()
.map(PipelineUtils::toPipelineAggregatorTarget)
.toArray(ExprWithAlias[]::new));
}

/**
* Executes this query.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

package com.google.cloud.firestore;

import com.google.api.core.InternalApi;
import com.google.api.core.InternalExtensionOnly;
import com.google.cloud.Timestamp;
import com.google.firestore.v1.Value;
Expand Down Expand Up @@ -189,4 +190,9 @@ public boolean equals(Object object) {
public int hashCode() {
return Objects.hash(query, data);
}

@InternalApi
Map<String, Value> getData() {
return data;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

package com.google.cloud.firestore;

import com.google.api.core.InternalApi;
import com.google.auto.value.AutoValue;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
Expand Down Expand Up @@ -79,7 +80,8 @@ static boolean isDocumentId(String path) {
}

/** Returns a field path from a dot separated string. Does not support escaping. */
static FieldPath fromDotSeparatedString(String field) {
@InternalApi
public static FieldPath fromDotSeparatedString(String field) {
if (PROHIBITED_CHARACTERS.matcher(field).matches()) {
throw new IllegalArgumentException("Use FieldPath.of() for field names containing '˜*/[]'.");
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ public interface Firestore extends Service<FirestoreOptions>, AutoCloseable {
*/
CollectionGroup collectionGroup(@Nonnull String collectionId);

PipelineSource pipeline();

/**
* Executes the given updateFunction and then attempts to commit the changes applied within the
* transaction. If any document read within the transaction has changed, the updateFunction will
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

import com.google.api.core.ApiClock;
import com.google.api.core.ApiFuture;
import com.google.api.core.BetaApi;
import com.google.api.core.NanoClock;
import com.google.api.core.SettableApiFuture;
import com.google.api.gax.rpc.ApiStreamObserver;
Expand All @@ -29,6 +30,8 @@
import com.google.api.gax.rpc.StreamController;
import com.google.api.gax.rpc.UnaryCallable;
import com.google.cloud.Timestamp;
import com.google.cloud.firestore.Transaction.AsyncFunction;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I am inclined not to import nested classes. In this case, the qualifier Transaction helps understand that this is not a lambda, such as java.util.function.Function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

import com.google.cloud.firestore.Transaction.Function;
import com.google.cloud.firestore.spi.v1.FirestoreRpc;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
Expand Down Expand Up @@ -385,30 +388,35 @@ public CollectionGroup collectionGroup(@Nonnull final String collectionId) {

@Nonnull
@Override
public <T> ApiFuture<T> runTransaction(@Nonnull final Transaction.Function<T> updateFunction) {
@BetaApi
public PipelineSource pipeline() {
return new PipelineSource(this);
}

@Nonnull
@Override
public <T> ApiFuture<T> runTransaction(@Nonnull final Function<T> updateFunction) {
return runAsyncTransaction(
new TransactionAsyncAdapter<>(updateFunction), TransactionOptions.create());
}

@Nonnull
@Override
public <T> ApiFuture<T> runTransaction(
@Nonnull final Transaction.Function<T> updateFunction,
@Nonnull TransactionOptions transactionOptions) {
@Nonnull final Function<T> updateFunction, @Nonnull TransactionOptions transactionOptions) {
return runAsyncTransaction(new TransactionAsyncAdapter<>(updateFunction), transactionOptions);
}

@Nonnull
@Override
public <T> ApiFuture<T> runAsyncTransaction(
@Nonnull final Transaction.AsyncFunction<T> updateFunction) {
public <T> ApiFuture<T> runAsyncTransaction(@Nonnull final AsyncFunction<T> updateFunction) {
return runAsyncTransaction(updateFunction, TransactionOptions.create());
}

@Nonnull
@Override
public <T> ApiFuture<T> runAsyncTransaction(
@Nonnull final Transaction.AsyncFunction<T> updateFunction,
@Nonnull final AsyncFunction<T> updateFunction,
@Nonnull TransactionOptions transactionOptions) {

if (transactionOptions.getReadTime() != null) {
Expand Down Expand Up @@ -519,10 +527,10 @@ public void shutdownNow() {
closed = true;
}

private static class TransactionAsyncAdapter<T> implements Transaction.AsyncFunction<T> {
private final Transaction.Function<T> syncFunction;
private static class TransactionAsyncAdapter<T> implements AsyncFunction<T> {
private final Function<T> syncFunction;

public TransactionAsyncAdapter(Transaction.Function<T> syncFunction) {
public TransactionAsyncAdapter(Function<T> syncFunction) {
this.syncFunction = syncFunction;
}

Expand Down
Loading