Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid memcpy in GRPC write path #810

Open
wants to merge 1,016 commits into
base: branch-2.2.x
Choose a base branch
from
Open

Avoid memcpy in GRPC write path #810

wants to merge 1,016 commits into from

Conversation

mayanks
Copy link
Contributor

@mayanks mayanks commented Jun 23, 2022

No description provided.

dependabot bot and others added 30 commits October 12, 2020 16:45
@dependabot
Bump junit from 4.13 to 4.13.1 (GoogleCloudDataproc#469)
7db83cb 
Bumps [junit](https://github.com/junit-team/junit4) from 4.13 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.1.md)
- [Commits](junit-team/junit4@r4.13...r4.13.1)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@medb
Fetch parent info in delete method asynchronously (GoogleCloudDatapro…
b7a96b2 
…c#465)
@medb
Always infer implicit directories (GoogleCloudDataproc#467)
ef91e6e
@hongyegong
Revert "Add interceptor to inject remote address to non-ok status dis… (
e0ae512 
GoogleCloudDataproc#471)

* Revert "Add interceptor to inject remote address to non-ok status discription (GoogleCloudDataproc#441)"

This reverts commit 815e773.
@medb
Migrate glob algorithm configuration to enum (GoogleCloudDataproc#470)
0f9f5e4
@medb
Cleanup and update dependencies (GoogleCloudDataproc#472)
99225ab
@medb
Improve test coverage of GoogleCloudStorage.list* methods (GoogleClou…
cd031b0 
…dDataproc#466)
@medb
Do not create parent directories for new objects (GoogleCloudDataproc…
90e0b50 
…#468)
@medb
Remove Slf4j Flogger backend. (GoogleCloudDataproc#473)
0d5d333
@medb
Use shared executor instance for patallel file info requests (GoogleC…
bd6bec7 
…loudDataproc#474)
@medb
Do not create parent directory objects and bucket for new directories (…
c6e7bd8 
…GoogleCloudDataproc#475)
@medb
Refactor to use utility method getFromfuture (GoogleCloudDataproc#476)
9f87977
@medb
Revert fs.gs.overwrite.generation.mismatch.ignore feature (GoogleClou…
e605420 
…dDataproc#477)
@medb
Update CONFIGURATION.md
18f4ee6
@hongyegong
refactor methods handing out gRPC channels syncly (GoogleCloudDatapro…
84f30b4 
…c#479)
@hongyegong
Made input stream chunk size fixed to reduce copies (GoogleCloudDatap…
cc6fafc 
…roc#481)
@hongyegong
Bumped the readFrom chunk size to be maximum size of gRPC message (Go…
3bafeaa 
…ogleCloudDataproc#484)
@hongyegong
Fixed constructor of GCSImpl to respect gRPC flag (GoogleCloudDatapro…
0538297 
…c#486)
@hongyegong
Cancel the request context when no more data available (GoogleCloudDa…
89b6a38 
…taproc#487)
@medb
Add HCFS benchmark tool (GoogleCloudDataproc#489)
c2f8ad6
@hongyegong
Update README.md
e44af2c
@hongyegong
Added basic integration tests for gRPC (GoogleCloudDataproc#490)
7870126
@hongyegong
retry getObjectRequest for grpc read (GoogleCloudDataproc#491)
1f57bb5
@medb
Remove redundant constructors (GoogleCloudDataproc#492)
159cf3f
@medb
Remove unused listFileNames methods (GoogleCloudDataproc#493)
098d40d
@medb
Use factory methods to create GoogleCloudStorageItemInfo instances (G…
2572af7 
…oogleCloudDataproc#494)
@medb
Use projection to request only object fields that are used in get and…
2528f5e 
… list requests (GoogleCloudDataproc#495)
@medb
Cleanup and delete unused code (GoogleCloudDataproc#496)
0694533
@medb
List +1 object only when prefix object is not included in the list re…
37a275e 
…sult (GoogleCloudDataproc#497)
@medb
Fix flaky tests (GoogleCloudDataproc#499)
15ef184
Deependra-Patel and others added 21 commits August 3, 2022 09:47
@Deependra-Patel @mayanks
Preparing for 2.2.7 release (GoogleCloudDataproc#794)
8717ffd
@Deependra-Patel @mayanks
Prepare for next release (GoogleCloudDataproc#798)
93dca13
@mayanks
FsBenchmark test to support write benchmark (GoogleCloudDataproc#795)
77633fa 
* FsBenchmark test to support write tests

* updated the implementation to separate write from read benchmark
@JerryLeiDing @mayanks
Upgrade Google auth dependency to 1.7.0 (GoogleCloudDataproc#803)
fb96e13
@Deependra-Patel @mayanks
Upgrade google-oauth-client version (GoogleCloudDataproc#806)
e796e57
@arunkumarchacko @mprashanthsagar @mayanks
Add support for tracing GCS API calls (GoogleCloudDataproc#796)
7383a9b 
Add support for tracing GCS Storage API calls

This is disabled by default behind <fs.gs.tracelog.enable> flag

Co-authored-by: Prashanth Sagar <[email protected]>
@mayanks
Optimize write performance by avoiding a memcopy
2c03849 
This feature removes the Pipe based communication between upstream
thread and the upload thread which involved an additional memcopy.
@mayanks
fetch the committed write offset in an async manner before the buffer…
d73cc08 
…s are full
@mayanks
added comments and logs
934b463
@mayanks
fixed spotless:clean issue
6117a33
@mayanks
fixed review comments
3f8ee59
@abmodi @mayanks
Fixing seek back to same position with grpc channel (GoogleCloudDatap…
5971675 
…roc#808) (GoogleCloudDataproc#812)

(cherry picked from commit 2a6b063)
@abmodi @mayanks
Grpc read optimization to not prematurely close existing requests (Go…
08040ac 
…ogleCloudDataproc#809) (GoogleCloudDataproc#813)

* Changes to avoid current request cancellation on seek

* fixing checkstyle

* Fixing checkstyle issues

(cherry picked from commit bc44d0c)
@mayanks
handling interrupted exceptions and added comments
aa8fa93
@mayanks
added explanation on how lastChunk is used
923d8dd
@mayanks
remove dependency of GoogleCloudStorageGrpcWriteChannel on BaseAbstra…
5433d35 
…ctGoogleAsyncWriteChannel
@mayanks
removed optimization to fetch committed offset asynchronously
376aac6
@mayanks
added a test case when exception is raised on second write after gcs …
b36962f 
…write of first chunk fails
@mayanks
fix write stall on ready by timing out
25ee9f2
@mayanks
fixed the comment and write channel initialization logic
971b845
@arunkumarchacko @mayanks
Add bucket information as header (GoogleCloudDataproc#815) (GoogleClo…
126ba74 
…udDataproc#818)

Add bucket information as header
@mayanks mayanks force-pushed the ms-write-save-memcpy-branch-2xx branch from 8b72c82 to 40a0e36 Compare August 3, 2022 04:18
@mayanks
Copy link
Contributor Author

mayanks commented Aug 3, 2022

/gcbrun

1 similar comment
@mayanks
Copy link
Contributor Author

mayanks commented Aug 3, 2022

/gcbrun

@mayanks mayanks force-pushed the ms-write-save-memcpy-branch-2xx branch from aa396c0 to 4aaab2b Compare August 3, 2022 17:33
@abmodi abmodi requested review from abmodi and removed request for mprashanthsagar August 5, 2022 08:51
@abmodi abmodi dismissed their stale review August 5, 2022 08:52

Branch is corrupted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mprashanthsagar mprashanthsagar mprashanthsagar left review comments

@medb medb Awaiting requested review from medb

@arunkumarchacko arunkumarchacko Awaiting requested review from arunkumarchacko

@Deependra-Patel Deependra-Patel Awaiting requested review from Deependra-Patel

@abmodi abmodi Awaiting requested review from abmodi

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.