Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(sanity): migrate sanity package to vitest #7578

Open
wants to merge 6 commits into
base: next
Choose a base branch
from

Conversation

binoy14
Copy link
Contributor

@binoy14 binoy14 commented Oct 3, 2024

Description

Migrates sanity tests to vitest

What to review

Changes makes sense and all tests pass

Testing

All tests should pass

Notes for release

N/A

ricokahler and others added 5 commits October 1, 2024 21:34
* test: revamp eFPS suite

* chore: add efps github actions

* chore: use different token for efps tests

* test: add comment with perf report result

* test: keep playwright install script

* feat: add workflow dispatch inputs

* fix: use `requestAnimationFrame` instead

---------

Co-authored-by: Binoy Patel <[email protected]>
* feat(sanity): allow `extractSchema` worker to emit schemas for all workspaces

* feat(sanity): include workspace and dataset names when extracting schema

* feat(cli): add `manifest` commands

* feat(manifest): add `@sanity/manifest` package

* refactor(sanity): use manifest schemas from `@sanity/manifest`

* chore: format files

* feat(schema): include `title`, `description`, and `deprecated` attributes when extracting schema

* feat(sanity): add `direct` schema format to schema extractor

* Revert "feat(schema): include `title`, `description`, and `deprecated` attributes when extracting schema"

This reverts commit 60cb576.

* feat(sanity): export `ConcreteRuleClass` class

* feat(sanity): include validation rules in manifests

* refactor(sanity): move manifest extraction code

* feat(sanity): extract manifest during build

* feat(sanity): adopt `.studioschema.json` filename suffix for manifest schemas

* refactor(sanity): rename manifest extraction functions (remove plural)

* fix(sanity): remove redundant success message

* fix(sanity): stop build spinner before starting manifest extraction

* feat(sanity): add `unstable_extractManifestOnBuild` CLI config option

* feat(test-studio): enable `unstable_extractManifestOnBuild`

* fix(sanity): switch to node crypto for node 18 compatibility

* feat(cli): add `unstable_staticAssetsPath` CLI configuration option

* chore(cli): refine `unstable_extractManifestOnBuild` CLI configuration option description

* feat(sanity): remove extraneous `types` wrapper from manifests

* debug(test-studio): remove Mux plugin to unblock typegen

* feat(embedded-studio): enable manifest extraction

* feat(starter-next-studio): enable manifest extraction

* wip

* feat(sanity): normalize type constraints in manifest validation

* wip

* chore: merge fix

* feat: serialize userland properties and validation rules in manifest

* fix: remove @sanity/manifest package

* chore: cleanup

* fix: serialize fieldsets

* fix: omit default titles on fields and array-members

* fix: ensure manifest schema is restoreble and supports cross dataset references

* chore: mergefix

* fix: serialization of type aliases no longer inlines fields and of props

* fix: removes double dot in filename

* feat: manifest command

* chore: tweaks

* chore: revert redundant changes

* fix: adds manifest group to CLI

* chore: wording change

* fix: adds a 2-minute timeout to manifest extract

* fix: ensures error code when mainfest extract fails and changes failed spinner message to info

* chore: use *ENABLED instead of *DISABLED for constant

* chore: defensive optional chaining for option extraction

* chore: reworded EXTRACT_FAILURE_MESSAGE

---------

Co-authored-by: Ash <[email protected]>
### Description

<!--
What changes are introduced?
Why are these changes introduced?
What issue(s) does this solve? (with link, if possible)
-->

Updates the validate command to use studioHost configured in the CLI
config instead of relying on the projects endpoint studioHost for this
information.

### What to review

<!--
What steps should the reviewer take in order to review?
What parts/flows of the application/packages/tooling is affected?
-->

Changes makes sense

### Testing

<!--
Did you add sufficient testing for this change?
If not, please explain how you tested this change and why it was not
possible/practical for writing an automated test
-->

Tests Updates

### Notes for release

<!--
Engineers do not need to worry about the final copy,
but they must provide the docs team with enough context on:

* What changed
* How does one use it (code snippets, etc)
* Are there limitations we should be aware of

If this is PR is a partial implementation of a feature and is not
enabled by default or if
this PR does not contain changes that needs mention in the release notes
(tooling chores etc),
please call this out explicitly by writing "Part of feature X" or "Not
required" in this section.
-->

N/A (does not seem it is important to mention in release notes)
Copy link

vercel bot commented Oct 3, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
page-building-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 3, 2024 5:49pm
performance-studio ✅ Ready (Inspect) Visit Preview Oct 3, 2024 5:49pm
test-compiled-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 3, 2024 5:49pm
test-next-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 3, 2024 5:49pm
test-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 3, 2024 5:49pm
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
studio-workshop ⬜️ Ignored (Inspect) Visit Preview Oct 3, 2024 5:49pm

Copy link
Contributor Author

binoy14 commented Oct 3, 2024

Copy link
Contributor

github-actions bot commented Oct 3, 2024

No changes to documentation

@binoy14 binoy14 changed the title chore(test): move jest config to @repo/test-config/jest test(sanity): migrate sanity package to vitest Oct 3, 2024
Copy link
Contributor

github-actions bot commented Oct 3, 2024

Component Testing Report Updated Oct 3, 2024 6:01 PM (UTC)

✅ All Tests Passed -- expand for details
File Status Duration Passed Skipped Failed
comments/CommentInput.spec.tsx ✅ Passed (Inspect) 44s 15 0 0
formBuilder/ArrayInput.spec.tsx ✅ Passed (Inspect) 9s 3 0 0
formBuilder/inputs/PortableText/Annotations.spec.tsx ✅ Passed (Inspect) 30s 6 0 0
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx ✅ Passed (Inspect) 37s 11 7 0
formBuilder/inputs/PortableText/copyPaste/CopyPasteFields.spec.tsx ✅ Passed (Inspect) 0s 0 12 0
formBuilder/inputs/PortableText/Decorators.spec.tsx ✅ Passed (Inspect) 18s 6 0 0
formBuilder/inputs/PortableText/DisableFocusAndUnset.spec.tsx ✅ Passed (Inspect) 10s 3 0 0
formBuilder/inputs/PortableText/DragAndDrop.spec.tsx ✅ Passed (Inspect) 3m 0s 0 0 0
formBuilder/inputs/PortableText/FocusTracking.spec.tsx ✅ Passed (Inspect) 45s 15 0 0
formBuilder/inputs/PortableText/Input.spec.tsx ✅ Passed (Inspect) 1m 36s 21 0 0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx ✅ Passed (Inspect) 1m 16s 18 0 0
formBuilder/inputs/PortableText/PresenceCursors.spec.tsx ✅ Passed (Inspect) 9s 3 9 0
formBuilder/inputs/PortableText/RangeDecoration.spec.tsx ✅ Passed (Inspect) 25s 9 0 0
formBuilder/inputs/PortableText/Styles.spec.tsx ✅ Passed (Inspect) 17s 6 0 0
formBuilder/inputs/PortableText/Toolbar.spec.tsx ✅ Passed (Inspect) 35s 12 0 0
formBuilder/tree-editing/TreeEditing.spec.tsx ✅ Passed (Inspect) 0s 0 3 0
formBuilder/tree-editing/TreeEditingNestedObjects.spec.tsx ✅ Passed (Inspect) 0s 0 3 0

Copy link
Contributor

github-actions bot commented Oct 3, 2024

⚡️ Editor Performance Report

Updated Thu, 03 Oct 2024 18:05:20 GMT

Benchmark reference
latency of sanity@latest
experiment
latency of this branch
Δ (%)
latency difference
article (title) 18.5 efps (54ms) 17.5 efps (57ms) +3ms (+5.6%)
article (body) 59.5 efps (17ms) 63.3 efps (16ms) -1ms (-6.0%)
article (string inside object) 18.9 efps (53ms) 19.6 efps (51ms) -2ms (-3.8%)
article (string inside array) 14.8 efps (68ms) 15.0 efps (67ms) -1ms (-1.5%)
recipe (name) 30.3 efps (33ms) 30.8 efps (33ms) -1ms (-1.5%)
recipe (description) 33.3 efps (30ms) 33.3 efps (30ms) +0ms (-/-%)
recipe (instructions) 99.9+ efps (6ms) 99.9+ efps (6ms) +0ms (-/-%)
synthetic (title) 15.4 efps (65ms) 15.2 efps (66ms) +1ms (+1.5%)
synthetic (string inside object) 15.9 efps (63ms) 16.1 efps (62ms) -1ms (-1.6%)

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark latency p75 p90 p99 blocking time test duration
article (title) 54ms 61ms 67ms 153ms 1041ms 13.2s
article (body) 17ms 19ms 26ms 136ms 255ms 5.8s
article (string inside object) 53ms 57ms 63ms 143ms 721ms 8.6s
article (string inside array) 68ms 72ms 78ms 246ms 1592ms 9.9s
recipe (name) 33ms 36ms 40ms 66ms 48ms 9.0s
recipe (description) 30ms 33ms 54ms 118ms 54ms 6.5s
recipe (instructions) 6ms 8ms 9ms 10ms 0ms 3.3s
synthetic (title) 65ms 67ms 74ms 246ms 1905ms 16.2s
synthetic (string inside object) 63ms 66ms 75ms 158ms 1707ms 9.8s

🧪 Experiment result

The performance result of this branch

Benchmark latency p75 p90 p99 blocking time test duration
article (title) 57ms 61ms 113ms 195ms 1084ms 13.8s
article (body) 16ms 18ms 21ms 97ms 157ms 5.1s
article (string inside object) 51ms 54ms 58ms 136ms 816ms 8.5s
article (string inside array) 67ms 70ms 79ms 123ms 1617ms 9.7s
recipe (name) 33ms 34ms 38ms 72ms 27ms 9.0s
recipe (description) 30ms 32ms 54ms 97ms 62ms 6.4s
recipe (instructions) 6ms 9ms 9ms 10ms 0ms 3.2s
synthetic (title) 66ms 70ms 80ms 317ms 2130ms 16.1s
synthetic (string inside object) 62ms 69ms 74ms 241ms 1798ms 9.8s

📚 Glossary

column definitions

  • benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".
  • latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.
  • p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.
  • p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.
  • p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.
  • blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.
  • test duration — how long the test run took to complete.

@ryanbonial
Copy link
Contributor

@binoy14 The test pass, but I am seeing the addition of about 35 warnings about Warning: An update to ... inside a test was not wrapped in act(...). warnings.

CleanShot 2024-10-03 at 13 58 22@2x

@binoy14
Copy link
Contributor Author

binoy14 commented Oct 4, 2024

@binoy14 The test pass, but I am seeing the addition of about 35 warnings about Warning: An update to ... inside a test was not wrapped in act(...). warnings.

CleanShot 2024-10-03 at 13 58 22@2x

Pretty sure they existed during jest as well but were getting swallowed by jest. Do you see any instances where it could be causing this?

My suggestion is if other changes looks good we merge this and do a follow up PR that fixes the warnings

Copy link
Contributor

@ryanbonial ryanbonial left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Sounds like a good plan for a follow-up PR that fixes the warnings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants