Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OuterLoop, and SkipOnPlatform attributes conflict, and don't work correctly when used together #908

Open
radical opened this issue Jun 14, 2022 · 4 comments
Assignees
Labels
test-runners TestRunner area

Comments

@radical
Copy link
Member

radical commented Jun 14, 2022

Consider
https://github.com/dotnet/runtime/blob/978df67ced885aeca5f7e75379451bc1a57cc219/src/libraries/System.Net.WebSockets/tests/WebSocketCreateTest.cs#L40-L45

        [OuterLoop("Uses external servers")]
        [Theory]
        [MemberData(nameof(EchoServers))]
        [SkipOnPlatform(TestPlatforms.Browser, "System.Net.Sockets is not supported on this platform.")]
        [ActiveIssue("https://github.com/dotnet/runtime/issues/34690", TestPlatforms.Windows, TargetFrameworkMonikers.Netcoreapp, TestRuntimes.Mono)]
        public async Task WebSocketProtocol_CreateFromConnectedStream_CanSendReceiveData(Uri echoUri)

This has both OuterLoop, and SkipOnPlatform attributes. This should run when using category=outerloop, but get skipped if it is running on the browser. Instead, the tests run anyway, and fail because the operations are not supported on browser.

I can reproduce this with ./dotnet.sh build /bl src/libraries/System.Net.WebSockets/tests/System.Net.WebSockets.Tests.csproj /p:Configuration=Release /p:TargetOS=Browser /p:TargetArchitecture=wasm /t:Test -p:Scenario=WasmTestOnBrowser -p:TestScope=outerloop -p:XUnitClassName=System.Net.WebSockets.Tests.WebSocketTests

this is the xharness command line:
wasm test-browser --app=. --output-directory=/workspaces/runtime/artifacts/bin/System.Net.WebSockets.Tests/Release/net7.0/browser-wasm/AppBundle/xharness-output --browser-arg=--no-sandbox -s dotnet.js.symbols --symbol-patterns wasm-symbol-patterns.txt --browser-arg=--no-sandbox -- --run WasmTestRunner.dll System.Net.WebSockets.Tests.dll -class System.Net.WebSockets.Tests.WebSocketTests -trait category=OuterLoop -notrait category=failing.

I don't think this is wasm, or SkipOnPlatform specific.

cc @akoeplinger @premun

@premun
Copy link
Member

premun commented Jun 15, 2022

I think this is something that is actually implemented in runtime?
https://github.com/dotnet/runtime/blob/2f38fb55e965d685fff9ee07e0f9c50d3e6bafaf/docs/workflow/testing/libraries/filtering-tests.md

My understanding was that you provide the filters and also this file with ignores/excludes and XHarness only triggers the regular Xunit runner over that? But @akoeplinger might know more

@akoeplinger
Copy link
Member

That doc is mostly relevant for the regular xunit console runner, but we implement most of that logic ourselves, e.g.

runner.SkipCategories(ExcludedTraits);
runner.SkipCategories(IncludedTraits, isExcluded: false);

It's most likely some issue there.

@premun premun added the test-runners TestRunner area label Jun 20, 2022
@premun
Copy link
Member

premun commented Mar 28, 2023

Closing this as this is not implemented in xharness but in runtime

@premun premun closed this as completed Mar 28, 2023
@akoeplinger
Copy link
Member

This is still an issue, reopening.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test-runners TestRunner area
Projects
None yet
Development

No branches or pull requests

3 participants