Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Console] Add support for JSON with long numerals #4562

Merged
merged 1 commit into from
Sep 6, 2023

Conversation

AMoo-Miki
Copy link
Collaborator

Add support for JSON with long numerals in Dev Tools.

Also:

  • Add support for parsing and stringifying JSON with long numerals into @osd/std
  • Upgrade @opensearch/[email protected] which supports long numerals
  • Add support for long numerals to http/fetch

Description

  • When long numerals were used in the editor, they lost precision when
    • requests were run
    • text was auto-indented
  • When a result contained long numerals, it lost precision when displayed in console.

Issues Resolved

Fixes #4536

Screenshot

Before

before-long-numerals-fix.mp4

After

after-long-numerals-fix.mp4

Check List

  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
    • yarn test:ftr
  • New functionality includes testing.
  • New functionality has been documented.
  • Update CHANGELOG.md
  • Commits are signed per the DCO using --signoff

@codecov
Copy link

codecov bot commented Jul 13, 2023

Codecov Report

Merging #4562 (40b4783) into main (9d7f576) will increase coverage by 0.04%.
Report is 1 commits behind head on main.
The diff coverage is 92.03%.

@@            Coverage Diff             @@
##             main    #4562      +/-   ##
==========================================
+ Coverage   66.39%   66.43%   +0.04%     
==========================================
  Files        3397     3398       +1     
  Lines       64804    64904     +100     
  Branches    10360    10385      +25     
==========================================
+ Hits        43026    43120      +94     
- Misses      19218    19226       +8     
+ Partials     2560     2558       -2     
Flag Coverage Δ
Linux_1 34.80% <15.46%> (-0.05%) ⬇️
Linux_2 55.30% <92.00%> (+0.15%) ⬆️
Linux_3 44.46% <66.37%> (+0.23%) ⬆️
Linux_4 34.89% <26.47%> (-0.03%) ⬇️
Windows_1 34.82% <15.46%> (-0.05%) ⬇️
Windows_2 ?
Windows_3 44.46% <66.37%> (+0.22%) ⬆️
Windows_4 34.89% <26.47%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
src/core/server/opensearch/client/mocks.ts 98.21% <ø> (ø)
...lugins/console/public/lib/opensearch/opensearch.ts 75.00% <ø> (ø)
src/plugins/console/public/services/storage.ts 0.00% <0.00%> (ø)
.../server/routes/api/console/proxy/create_handler.ts 67.39% <40.00%> (-2.38%) ⬇️
packages/osd-std/src/json.ts 95.74% <95.74%> (ø)
packages/osd-std/src/index.ts 100.00% <100.00%> (ø)
src/core/public/http/fetch.ts 96.05% <100.00%> (ø)
...rc/core/server/opensearch/client/cluster_client.ts 85.71% <100.00%> (+1.50%) ⬆️
...equest_to_opensearch/send_request_to_opensearch.ts 70.49% <100.00%> (ø)
src/plugins/console/public/lib/utils/index.ts 86.04% <100.00%> (+27.90%) ⬆️
... and 1 more

... and 8 files with indirect coverage changes

@ashwin-pc
Copy link
Member

Nice addition, can you add documentation for the change in the package @AMoo-Miki ?

@zhongnansu
Copy link
Member

re-running failed CIs

@zhongnansu
Copy link
Member

@AMoo-Miki Hi Miki, Would you check the CI failures? Is it because we haven't release the js client with the fix?

@AMoo-Miki
Copy link
Collaborator Author

Thanks. This needs to be bumped to use the 2.3.1 client.

@zhongnansu
Copy link
Member

Thanks. This needs to be bumped to use the 2.3.1 client.

js client 2.3.1 is out. Would you update the PR? Thx! @AMoo-Miki

@joshuarrrr joshuarrrr added the needs more info Requires more information from poster label Aug 4, 2023
@AMoo-Miki
Copy link
Collaborator Author

I am stuck with making a decision for how this should be implemented as we don't want to add it all over OSD. Will continue thinking.

@ashwin-pc
Copy link
Member

@AMoo-Miki should i remove the 2.10 tag for this then?

@ashwin-pc ashwin-pc marked this pull request as draft August 29, 2023 23:47
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-4562-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0709333c641f1e2bcaf116eb7d5504b33afcd71d
# Push it to GitHub
git push --set-upstream origin backport/backport-4562-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-4562-to-2.x.

Copy link
Member

@ashwin-pc ashwin-pc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one concern about how we are exposing the custom client. The rest are just minor nits or opinions, feel free to use them if you think it makes sense :)

Comment on lines +69 to +71
public readonly asCurrentUser: OpenSearchClient,
public readonly asInternalUserWithLongNumeralsSupport: OpenSearchClient,
public readonly asCurrentUserWithLongNumeralsSupport: OpenSearchClient
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this is verbose, cant we make it more generic? This is the first customization that we are introducing but it doubt it will be the last. And since this client will become a part of the API it will be annoying to change this in the future.

interface ExtendedClient extends OpenSearchClient {
    _custom : {
        [key: string]: OpenSearchClient,
    }
}
Suggested change
public readonly asCurrentUser: OpenSearchClient,
public readonly asInternalUserWithLongNumeralsSupport: OpenSearchClient,
public readonly asCurrentUserWithLongNumeralsSupport: OpenSearchClient
public readonly asCurrentUser: ExtendedClient,

And add the custom long numeral client like:

internalClient.customClient = {
    longNumeral: client,
}

const maxIntAsString = String(Number.MAX_SAFE_INTEGER);
const maxIntLength = maxIntAsString.length;
// Sub-patterns for each digit
const longNumeralMatcherTokens = [`\\d{${maxIntAsString.length + 1},}`];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Cant we make this a lot simpler by just converting anything greater than a generic high number? e.g. 9000000000000000. What you are doing should work, but this reduces the matchers from 12 to 2

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained why not in the comments in the code. Before parsing the long string, I wouldn't know what's in it. Assuming 8192 is max int, if i take everything greater than 8000 and turn them into BigInt, that would be unexpected to everyone who don't use numbers bigger than 8192. Hence, doing this one time activity of making a marginally more complex regex.

? BigInt(val.substring(length))
: val;

/* For better performance, instead of testing for existence of `reviver` on each value, two almost
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didnt understand this comment

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code could be

loop {
  condition ? doA : doB
}

but then condition is evaluated repeatedly. I decided to do

condition ? loop { doA } : loop { doB }

this results in a little less maintainable code.

/* Experiments with different combinations of various lengths, until one is found to not be in
* the input string.
*/
const getMarker = (text: string): { marker: string; length: number } => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: cant we use UUID here? That way instead of a marker system, we can keep a dictionary of uuid's and values that can be substituted when parsing fails.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if a value uses UUIDs? also, the longer the UUID, the more expensive my regex will be. Some more details here.

kavilla pushed a commit to kavilla/OpenSearch-Dashboards-1 that referenced this pull request Sep 7, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
Signed-off-by: Kawika Avilla <[email protected]>
ashwin-pc pushed a commit that referenced this pull request Sep 7, 2023
Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
Signed-off-by: Kawika Avilla <[email protected]>
Co-authored-by: Miki <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 7, 2023
Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
Signed-off-by: Kawika Avilla <[email protected]>
Co-authored-by: Miki <[email protected]>
(cherry picked from commit e69d38d)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
joshuarrrr pushed a commit that referenced this pull request Sep 8, 2023
)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`




(cherry picked from commit e69d38d)

Signed-off-by: Miki <[email protected]>
Signed-off-by: Kawika Avilla <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Miki <[email protected]>
kavilla added a commit to kavilla/OpenSearch-Dashboards-1 that referenced this pull request Sep 8, 2023
@kavilla kavilla added v2.11.0 and removed v2.10.0 labels Sep 8, 2023
@kavilla
Copy link
Member

kavilla commented Sep 8, 2023

@AMoo-Miki I have reverted this on 2.10 and tagged with 2.11. The reason being I'm not positive if the JS client cherry pick commit opensearch-project/opensearch-js#550 is intended to make the client 2.x version to be non-breaking for major version bump. So that we can utilize the declaration of opensearch instead of using opensearch-next

kavilla added a commit that referenced this pull request Sep 8, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 8, 2023
) (#4973)" (#4977)

This reverts commit e3cebe7.

Signed-off-by: Kawika Avilla <[email protected]>
(cherry picked from commit b75edfa)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
manasvinibs pushed a commit that referenced this pull request Sep 11, 2023
) (#4973)" (#4977) (#4978)

This reverts commit e3cebe7.


(cherry picked from commit b75edfa)

Signed-off-by: Kawika Avilla <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@kavilla
Copy link
Member

kavilla commented Sep 12, 2023

@AMoo-Miki will address the backport

AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 22, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 22, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 22, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 22, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 22, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 25, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 26, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 26, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 26, 2023
…#4562)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`

Signed-off-by: Miki <[email protected]>
AMoo-Miki added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Sep 27, 2023
…#4562)(opensearch-project#5130)

Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`
* Improve testing of clients with long numerals support

Signed-off-by: Miki <[email protected]>
joshuarrrr pushed a commit that referenced this pull request Oct 2, 2023
Also:
* Add support for parsing and stringifying JSON with long numerals into `@osd/std`
* Upgrade to `@opensearch/[email protected]` which supports long numerals
* Add support for long numerals to `http/fetch`
* Improve testing of clients with long numerals support

Signed-off-by: Miki <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Decouple] [BUG] Dev tool console is losing big number precision due to client
5 participants