Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Move retries createreadstream get resumption logic working (#1492)
* getRanges * Slight refactor of createReadStream * Add header to table utils * Refactor range and keys getting * Pull request opts into a separate function * Revert "Pull request opts into a separate function" This reverts commit c2f4dfe. * logical separation of ranges and keys * Revert "Revert "Pull request opts into a separate function"" This reverts commit 4817863. * set up the test frame * Test is set up to evaluate streaming behavior * Modify test with the mock server to pass first tst * Fix tests that are appending startKey and endKey * Getting all the tests working - Transform the rowsLimit parameter into an integer - Change the hook into a before hook so that we don’t attempt to create multiple mock servers - Create a guard so that the stream only writes if there are row keys to write * eliminate old createreadstream test define new interfaces too * Remove only. rowsLimit should be optional * Make rowKeysRead type more specific * Add after hook to shut down server * Add the less than or equal to fn to utils * remove server start * Add splice ranges back to diagnose problem * Revert "Add splice ranges back to diagnose problem" This reverts commit 5edaf82. * get all tests passing readRowsReqOpts should have an ECMAscript prefix to completely hide it from the user. Also remove a useless Filter.parse. * Use tableUtils lessthanorequalto * refactor: Move retries finish making createreadstream smaller * More specific type for options * Import type and replace with any * Turn on gax streaming retries * Try out the shouldRetryFn * Service error or Google error for this function Also uncommented retry logic to see how the test would behave. * First PR correction - remove any * Get rid of another any * Get rid of the any suppressed by typescript It was necessary to add guards to make this work. Also changed the error type to Google error. * Should use CreateReadStreamRequest as the type * Remove the comment disabling linting on any * Remove the import. It is not used anymore. * Remove the custom retry logic * Wrote a test that mocks gapic that will compile * Refactor the test so that it is easy to Move the check out into a separate function. * Add expected values for request * Do not append retryRequestOptions to gax options Appending retryRequestOptions will not work for any call that is also passing in the retry gax parameter. * Factor out the retry options logic Move functions out of table.ts that other methods should not be concerned with. Export the function that creates retry logic from this new module * Rename the file to retry-options This is going to be used by the mutate row retry options so the filename should not be so specific. * Initial attempt at adding resumption logic * skip * Comment out check in test * Keep the resumption strategy in table.ts for now There is currently a proxyquire issue preventing us from moving code out of table.ts so make a TODO and move it later. * Get proxyquire logic right Move the resumption strategy back to its own file and get the proxyquire configs right so that tests pass. * Remove unused code * Wrap with arrow functions. We can’t pass class methods in * Use the resumption request * Set up a unit test framework This ensures the right parameters are passed along to the gapic layer. * Make sure to pass along maxRetries to the gapic * Do an empty write to ensure server called only one * Gets all 10 tests passing Some issue with the mock server is making it so that responses don’t get sent back the way they should. This change allows all the tests to pass. * Pull testing code out into a dedicated object * Mock out project id properly. * Delete code * maxRetries test rename * Pass in the filter * Remove unused parameter * Create tableStrategyInfo case class * Fix appProfileId issue * Basic resumption test added * More resumption tests * Get resumption logic tests into its own module * Modify test name with reumption strategy * Skip a test * Remove a function that isn’t used anymore * inline building request options * Remove unused imports * Remove imports that are not used * Inline some functions to eliminate confusion * Move comment to where it was before * Better test parameterization * Replace grpc error codes with programmatic result * Correct interface * Uncomment some code removed from before * Remove this TODO * Move tests to unit test folder. * Better comments * Update comment * better spacing * correct grammar * Change to retry options construction * Add a test for passing custom retry options * Fix the complex example passing options along * Pull functions out of TableUtils These functions do not need to exist as a static method. Put them in the readrows resumption module. * Import bug fix * Moved some tests over Moved some unit tests over to the system test json file. * Check to see if maxRetries is undefined * Change retry options - wrap in gaxOpts * Move over should remove ranges which were already read * move more tests over should not retry if limit is reached should remove the keys which were already read 'should remove ranges which were already read * Add limit test and retry if all keys are read test * Move should not retry if all the ranges are read * Expected error code is 4 * Move location of updates * Correct tests with new info * Correct the json test file * Modify test to match old test * Moved rststream test over * Add framework to mock server so that error message * Remove the RST stream test * Delete retries test block * Make sure to call canResume in all cases * chore: update copyright year for auto-generated protos (#1417) * chore: update copyright year for auto-generated protos PiperOrigin-RevId: 631538781 Source-Link: googleapis/googleapis@3597f7d Source-Link: googleapis/googleapis-gen@8021fdf Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiODAyMWZkZjhkNjZmNjAwNTUxOWMwNDRkNTgzNDEyNGI2NzdkYzkxOSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * Remove retryOpts from mutate function * indent so diff is easier to read * Expand on comment * Fix side effects from turning on streaming retries * Simplify the diff * Add comments for a better description * Remove any and replace with more specific type * Correct test types to CreateReadStreamRequest * Coerce types and reintroduce parseInt to match expectations * Remove unused import * feat: Add feature for copying backups (#1153) * src test and protos copied over * lint fix * implemented copy backup * part of test done * workaround * corrections to get code working * Add expire time * This allows the copy backup endpoint to work * Working system test * delete try block * tests pass with new refactor * Create backup and copy on another cluster * modifications to second test * copy backup unit test * Revert "lint fix" This reverts commit 576550d. * Revert "src test and protos copied over" This reverts commit 46ce5b8. * First correction * Check the list of backups * fetched backup * get name and id * Remove only * refactor request mock * Add unit test for copying backup to another projec * feat: add experimental reverse scan for public preview PiperOrigin-RevId: 543539118 Source-Link: googleapis/googleapis@ae18706 Source-Link: googleapis/googleapis-gen@5d05516 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNWQwNTUxNmY4NGU1M2FhYmE2M2E0Yjg3NjdmZjk1NWFjNWJiNGE4NyJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: Increase the maximum retention period for a Cloud Bigtable backup from 30 days to 90 days PiperOrigin-RevId: 544356969 Source-Link: googleapis/googleapis@c35889a Source-Link: googleapis/googleapis-gen@c00326e Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzAwMzI2ZWM3ODU2NWI1ZDE2ZjkyYzg0NWZmMGJiMThmMTFjYTA1ZCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * docs: fix formatting for reversed order field example PiperOrigin-RevId: 547553954 Source-Link: googleapis/googleapis@c4e6427 Source-Link: googleapis/googleapis-gen@f552269 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjU1MjI2OTYwOWQ0MTgzNTQ2NTQzYmZlM2EwMjJmNTQ0ZDRmNWJkYiJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add last_scanned_row_key feature PiperOrigin-RevId: 551191182 Source-Link: googleapis/googleapis@51e04ba Source-Link: googleapis/googleapis-gen@4b90e8e Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNGI5MGU4ZWFkNDQ3N2VmZjk2YzMxYjliMGZkZWYzNmVkOTc1YjE1ZiJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: fix typings for IAM methods docs: fixed links in the generated Markdown documentation PiperOrigin-RevId: 551610576 Source-Link: googleapis/googleapis@73b1313 Source-Link: googleapis/googleapis-gen@8bec066 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiOGJlYzA2NjQ5MmE2ZGEyODU1YjFiOGNlNTYyNjY0YzBhNmIzMGIwMSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * test: disable retry-request for streaming tests PiperOrigin-RevId: 554648220 Source-Link: googleapis/googleapis@53cd9ad Source-Link: googleapis/googleapis-gen@7e8867e Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiN2U4ODY3ZWZiZWQ3ZGJmZTVlZjZlYzNjMmM5MmE0YmNlNDI4MGY3YSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: publish CopyBackup protos to external customers PiperOrigin-RevId: 557192020 Source-Link: googleapis/googleapis@b4c238f Source-Link: googleapis/googleapis-gen@feccb30 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZmVjY2IzMGUzMTc3ZGE4YjdiN2U2ODE0OWNhNGJiOTE0ZjhmYWYyYSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: simplify logic for HTTP/1.1 REST fallback option For the `fallback` parameter, all values considered as `true` in Boolean context will enable HTTP/1.1 REST fallback, since the other fallback transport, proto over HTTP, is removed from `google-gax` v4. PiperOrigin-RevId: 559812260 Source-Link: googleapis/googleapis@6a6fd29 Source-Link: googleapis/googleapis-gen@56c1665 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNTZjMTY2NTdlN2E1OTEyMmIxZGE5NDc3MWE5ZWY0MDk4OWMyODJjMCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add feature flag for improved mutate rows throttling PiperOrigin-RevId: 565090488 Source-Link: googleapis/googleapis@e8a136f Source-Link: googleapis/googleapis-gen@9a8dcca Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiOWE4ZGNjYTBmYjIxMTc2MjhhMWE2YTZjMzYyNWE2YWEzMmZjMmY3NSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * run lint * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * build: update typescript generator version to publish in dual format (ESM) PiperOrigin-RevId: 568643156 Source-Link: googleapis/googleapis@f95afc0 Source-Link: googleapis/googleapis-gen@bbd2c49 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYmJkMmM0OWQyZTQyM2E4Y2U1Y2M4NTYyNzQwMmQ1MTJhZWVmYzU4YiJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: Add support for Cloud Bigtable Request Priorities in App Profiles PiperOrigin-RevId: 571158646 Source-Link: googleapis/googleapis@bc3c83b Source-Link: googleapis/googleapis-gen@93366e8 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiOTMzNjZlODRlNGU2ODYxZTJlNTgwZWIwMDA3MjFkOTliZjU0YTBhNCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * build: update Node.js generator to compile protos PiperOrigin-RevId: 582493526 Source-Link: googleapis/googleapis@7c4e4b5 Source-Link: googleapis/googleapis-gen@368cfb6 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzY4Y2ZiNjUxMDE2ZDZhOTNjYTZlNDg4Y2JjMzRlMmQxZDlkMjEyYyJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Use the destination backup expiry time Expiry time of the destination should be used for the call. Not for the source expiry time. * Use the config instead for copy backup Use the config and not the backup object for copying backups. The expiry time is needed so a backup object can’t be used because the metadata cannot be introspected. * Rename CopyBackupConfig Rename it to DestinationBackupConfig. * Fix signature of copyBackup Fix the signature of copyBackup to include the fact that the config now has the gax options. * Change the copy signature to require callback If the callback is not required then all calls match first signature overload and async calls return void which isn’t correct. * Some debugging on the system tests The system tests were giving an error about the source expiry time and now this is fixed. * DestinationBackupConfig Don’t use the name DestinationBackupConfig anymore. * lint fix * Move the config and create a generic callback Create a generic callback data structure in order to avoid callback and promise data structures not matching up. Also move the configs to a different file. * Eliminate the generic callback data structure Eliminate the generic data structure as it can’t be used to capture the pattern we need exactly. Also modify the way that errors are passed into the callback function to be more consistent. * Get the first copy backup test working Copy the backup into the same cluster test is working. Still need to work on copy backup for test that looks at a different cluster to get that test working. * Fix different cluster, different instance test Fix the issue preventing the second test from working by allowing the test function to specify the instance where the new backups should be created. * Add another test for copy backup The test should address a backup copied to a different cluster, but the same instance. * Add a test case for copying to another project Test case for copying to another project uses environment variable to point to new project for CI pipeline. * Restructure tests for multiple expire time inputs Take one of the tests and structure it into a describe block instead. Each test in this block will now run the old test with a different expiry time format. * Various changes to test interaction with copy back Rename backupId to id. id is a simpler name and it is the name that is used everywhere. Rename a function to testWithExpiryTimes. Modify source code for copy function to return a backup with the new id. Add test stub for restore backup. * Finish the restore tests and modify copy backup The restore test should test to see if a table can be restored. It should also return a backup object that matches the new backup created. * Add comment describing second argument Second argument is a backup that corresponds to a new backup. This needs to be explained in a comment. * Remove TODO The TODO is done * Modify copy backup unit test Modify the copy backup unit test to compare against copying the a destination with customized cluster, backup, instance and project. * Modify the name of the test The test is not necessarily about copying to a different project. It is more a test about copying to a specific project. * Test for gax options in the test Make sure that the gax options get passed down to the request layer. * TODO is done The promise and callbacks follow the same pattern that is applied everywhere. * Change comment slightly It is actually more of a check than an action that ensures something. * Eliminate TODO The id is required. It can’t be generated automatically by the server. It needs to be provided. * Fix documentation for source function Add description for the parameters and explain what the function does. * Move copy backup code to proper place The tests were originally placed in a block at the very beginning so that they were easier to work through. This change moves them to the proper place. * Use cluster instead of parent cluster should be used instead of parent to specify the destination cluster. * Change parent to cluster cluster name should be used instead of parent to match the design doc. * A couple cleanup changes Id is required so no question mark needed. Also inline the function that replaces the project name. * Use parent instead of cluster This test was accidentally changed by a refactor and should not change further. * Eliminate unused time references Variables were created so that time can be used in a block of tests. That block of tests were moved so these. variables are no longer needed. * This test should not have changed Use parent instead of cluster because that is what it was before. * Eliminate unused import CreateBackupConfig is not used anywhere. * Remove assert statement that is not needed * Move two lines of code to location used General cleanup for readability * Improve readability Reduce number of lines required in testWithExpiryTimes function. This makes the function easier to read. * Inline variables instead of using them explicitly Don’t use the variables explicitly. Inline them instead. * Generate the backup id inline An extra variable is not needed for this. * readability - indent code so it is separate Also inline the id generation as it is not used elsewhere. * Indent block to separate check Separate expiry time check from the rest of the code. * indent cluster creation, expiry time check Indenting the code makes it easier to see variable relationships * Another test cleanup Inline Bigtable instance creation options. Indent code for checking expiry time. * Add comment Comment is for test that copies a table onto another project. * Indent code for table data insertion Indenting this code shows it is not referenced later. * Rename variables to distinguish operations Differentiate copy operation from create operation. * Eliminate unused imports * should be lower case * Change comment to respect alignment * Inline callback, config and cluster Inline variables for better readability * Remove unused operation * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Delete an instance Delete the instance after it is used just like the other test. * Inline generate id A generateId function I need to inline. * Add second project to config Import the second project to use in the integration tests * Rename the function to setBackupExpiryTime Rename this function to be more specific. * Replace the project for backup path Currently this test is too specific. It does not work if a specific project is specified in the client so we must only check the rest of the backup path. * Run the linter Linter removes unnecessary comma. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update comments for the copy function This function needs to be more clear and explain exactly what it does without ambiguity. * Rename function that sets correct value of expiry expiryTime should be in the right format so rename function to more clearly indicate what it is doing. * Add an environment variable for second project The variable for the second project is needed for the the integration tests to run properly for the copy backup test that copies to another project. * Add log for second project See what kokoro prints out. * Remove the console log This was just for testing. Remove this now. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Rename bigtable to bigtableSecondaryProject * PR follow-ups - inline code and rename variables * Remove unused import * Add comments for expire time * Add better garbage cleanups of the backups * These comments are not needed anymore. * Remove only * Update system-test/bigtable.ts Co-authored-by: Leah E. Cole <[email protected]> * Update system-test/bigtable.ts Co-authored-by: Leah E. Cole <[email protected]> --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: sofisl <[email protected]> Co-authored-by: Sofia Leon <[email protected]> Co-authored-by: meredithslota <[email protected]> Co-authored-by: Leah E. Cole <[email protected]> * Rename the comment as a TODO * Complete comment * Add latency measurements to mock server tests This commit contains a tool useful for printing latency results. * Revert "Add latency measurements to mock server tests" This reverts commit d3d05a2. * Revert "Complete comment" This reverts commit 32fe3e5. * Revert "Rename the comment as a TODO" This reverts commit 80cf68c. * Resolve the spacing issues * Solve remaining spacing issues * Add spaces back * Add the new test back in Add the test for the max retries 0 case. * feat: Add String type with Utf8Raw encoding to Bigtable API (#1419) * feat: update Nodejs generator to send API versions in headers for GAPICs PiperOrigin-RevId: 634109303 Source-Link: googleapis/googleapis@998ade8 Source-Link: googleapis/googleapis-gen@000ca6f Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDAwY2E2ZjAwODAxZjY1Yjg0N2U2MDI5Y2IwNTExMTQwNGRmMjFlYyJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: Add String type with Utf8Raw encoding to Bigtable API Bigtable will allow users to configure the type of a column family with string type PiperOrigin-RevId: 636631633 Source-Link: googleapis/googleapis@89a8364 Source-Link: googleapis/googleapis-gen@d776700 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZDc3NjcwMDdlYWUwZmU4Nzc1NWIyMWNmZTU2OWI4Nzc5ZjAyMTUxYyJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: danieljbruce <[email protected]> * Change the Gapic layer back to what it was * Fix max retries test * Remove only * Add header * Add comments to the tester * Add comments for the test functions. * Remove the TODOs * Fix lint warning * Address another lint warning * Remove Filter * Add stronger type checking to userStream.end fn * Create a better overloaded version of userStreamen * Remove reference to proto * Added a comment about streaming retries set to tru * Add comments to determine what table.ts should do * chore(main): release 5.1.0 (#1321) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> * Update test/util/gapic-layer-tester.ts Co-authored-by: Leah E. Cole <[email protected]> * Update test/util/gapic-layer-tester.ts Co-authored-by: Leah E. Cole <[email protected]> * Update test/util/gapic-layer-tester.ts Co-authored-by: Leah E. Cole <[email protected]> * Update system-test/read-rows.ts Co-authored-by: Leah E. Cole <[email protected]> * For max retries have a default retry count * Define a retry count constant * Update the comment * Move check for rst error. Rename MockGapicLayer. * linter * Make test case description more specific * Rename the file * RST stream error message inclusion * Don’t retry on resource exhausted * Remove service error from imports * RetryInfo in the status details should retry * Remove unnecessary code for compilation * Update system-test/read-rows.ts Co-authored-by: Leah E. Cole <[email protected]> * Remove the arrow function * Remove comments and replace with constants * Update the comment for getRowKeys * Move variables inside the if statement * Add a test case for row ranges and row keys * Add a test case that doesn’t do any filtering * Remove TODO * remove type any * Allow user to override codes Also update the keys and ranges every time. * Add another test for overriding the retry codes * Make the test a little bit more robust * Update the test framework to measure shouldRetry * Add another test ensuring a retry doesn’t happen * rename the test * Make canResume always run unless shouldRetryFn pr * Gapic layer should expect empty retry codes * Remove copied file * Change the resumption strategy * Replace 13 with enum * Removed example * npm run fix * Revert "npm run fix" This reverts commit dd42118. * Remove overrides * Remove irrelevant test * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Daniel Bruce <[email protected]> Co-authored-by: gcf-owl-bot[bot] <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: danieljbruce <[email protected]> Co-authored-by: sofisl <[email protected]> Co-authored-by: Sofia Leon <[email protected]> Co-authored-by: meredithslota <[email protected]> Co-authored-by: Leah E. Cole <[email protected]> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
- Loading branch information
10 people
authored
Sep 27, 2024
1 parent
3f5db96
commit b43468e