Test class logs are attached to all test cases in given test class

[MarcinKozak005]

Describe the Bug

Test class TestClass has multiple test methods: TestMethod1, TestMethod2, TestMethod3.

Suppose that TestMethod1 and TestMethod2 fail and TestMethod3 succeeds.

In the Allure report for TestClass, all three test methods have linked error logs of TestMethod1 and TestMethod2.

It's very unintuitive, that a test method which passes (Passed status) has logs with words like: "Failure", "Error" etc.

Steps to Reproduce

1. Clone repo https://github.com/allure-examples/junit5-java-maven
2. Remove test methods in com.example.junit5.StepTest.java
3. Add failing and passing tests, for ex:

    @Test
    public void failingTest1(){
        System.out.println("failing11");
        assertFalse("MessageAssertFalse", true);
        System.out.println("failing12");
    }

    @Test
    public void failingTest2(){
        System.out.println("failing21");
        assertEquals("allure", "all");
        System.out.println("failing22");
    }

    @Test
    public void passingTest(){
        System.out.println("passing1");
        assertTrue(true);
        System.out.println("passing2");
    }

4. mvn clean
5. mvn test
6. Generate allure report from surefire-reports folder: allure generate target/surefire-reports
7. Open allure report in the web browser
8. Go to Suites -> com.example.junit5.StepTest -> passingTest -> Overview -> Test body -> System out
9. System out contains error messages of failingTest1 and failingTest2

Expected Behaviour

1. Test cases with Failed status shoud have in their System out only error messages which applies to them.
2. Test cases with Passed status should not have any error messages from other test cases in their System out

Screenshots or Additional Context

What Language are you using?

Java

What Framework/Allure Integration you are using?

What version of Allure Integration you are using?

What version of Allure Report you are using?

2.25.0

Code of Conduct

• I agree to follow this project's Code of Conduct

[baev]

The issue is that JUnit XML stores system output files per class. So, there is nothing Allure can do about it.

Why do you use surefire-reports instead of allure-results?

[baev]

The recommended way is to use Allure#step methods to log additional data to the report. If you really want to capture system output you can implement your own PrintStream and set it to System.

[MarcinKozak005]

If you really want to capture system output you can implement your own PrintStream and set it to System.

Sorry for probable miscommunication, by saying:

When I use allure-reports no System out is generated:

I meant that when using allure-reports as suggested by You, in browser there is nothing in the Execution part (no test logs). See image in this message.

[MarcinKozak005]

JUnit XML is widely used across different languages and frameworks. The log format depends on the test framework. I wouldn't think we'll be able to support that.

Correct me if I'm wrong. The XML test report format is standarized (JUnit XML format with <testcase>,<testsuite> ...), but logs in *.txt are not, right?

[baev]

Correct me if I'm wrong. The XML test report format is standarized (JUnit XML format with <testcase>,<testsuite> ...), but logs in *.txt are not, right?

There is no published schema (at least officially), and in fact, we are already supporting different dialects. Some frameworks may store log files in similar named files (by class name) but with a .txt extension. But I'm not aware of someone doing that except for Maven Surefire. At least in JUnit 5, the legacy reporter only produces XML files.

Allure itself supports JUnit.XML, XUnit2.Xml, XCTest and TRX results. However, all of these formats are usually quite limited in terms of the amount of stored data. That's why Allure provides integrations with other frameworks (reporters that store test data in Allure format). The integrations store test data in the allure-results folder, and you have plenty of options to add additional logs & metadata to it. You can find the ways to do so in the docs https://allurereport.org/docs/junit5/#specify-description-links-and-other-metadata

[MarcinKozak005]

Ok, thanks for the explanation and clarification

[aptester]

I'm still a little confused. Does Allure still support JUnit's node? Because that is the default output in the JUnit xml when System.out.println() is called in a test.