Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Describe minicore test auxiliary and directive #2097

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
+89 −34
Draft

Describe minicore test auxiliary and directive #2097

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4108747
Drop undefined footnotes
jieyouxu Oct 11, 2024
0f568c4
Use directives consistently
jieyouxu Oct 11, 2024
0e54ad1
Describe `minicore` test auxiliary and directive
jieyouxu Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@
- [Best practices](./tests/best-practices.md)
- [Compiletest](./tests/compiletest.md)
- [UI tests](./tests/ui.md)
- [Test headers](./tests/directives.md)
- [Test directives](./tests/directives.md)
- [Minicore](./tests/minicore.md)
- [Ecosystem testing](./tests/ecosystem.md)
- [Crater](./tests/crater.md)
- [Fuchsia](./tests/fuchsia.md)
Expand Down
6 changes: 6 additions & 0 deletions src/tests/compiletest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,9 @@ more information.

See also the [assembly tests](#assembly-tests) for a similar set of tests.

If you need to work with `#![no_std]` cross-compiling tests, consult the
[`minicore` test auxiliary](./minicore.md) chapter.

[`tests/codegen`]: https://github.com/rust-lang/rust/tree/master/tests/codegen
[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html

Expand All @@ -303,6 +306,9 @@ information.

See also the [codegen tests](#codegen-tests) for a similar set of tests.

If you need to work with `#![no_std]` cross-compiling tests, consult the
[`minicore` test auxiliary](./minicore.md) chapter.

[`tests/assembly`]: https://github.com/rust-lang/rust/tree/master/tests/assembly


Expand Down
66 changes: 33 additions & 33 deletions src/tests/directives.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,46 +65,46 @@ not be exhaustive. Directives can generally be found by browsing the
See [Controlling pass/fail
expectations](ui.md#controlling-passfail-expectations).

| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------|---------------------------------------------|--------------------------------------------------|-----------------|
| `check-pass` | Building (no codegen) should pass | `ui`, `crashes`, `incremental`[^inc1] | N/A |
| `check-fail` | Building (no codegen) should fail | `ui`, `crashes` | N/A |
| `build-pass` | Building should pass | `ui`, `crashes`, `codegen`, `incremental`[^inc1] | N/A |
| `build-fail` | Building should fail | `ui`, `crashes` | N/A |
| `run-pass` | Running the test binary should pass | `ui`, `crashes`, `incremental`[^inc1] | N/A |
| `run-fail` | Running the test binary should fail | `ui`, `crashes` | N/A |
| `ignore-pass` | Ignore `--pass` flag | `ui`, `crashes`, `codegen`, `incremental`[^inc1] | N/A |
| `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental` | N/A |
| `failure-status` | Check | `ui`, `crashes` | Any `u16` |
| `should-ice` | Check failure status is `101` | `coverage`, `incremental` | N/A |
| `should-fail` | Compiletest self-test | All | N/A |
| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------|---------------------------------------------|-------------------------------------------|-----------------|
| `check-pass` | Building (no codegen) should pass | `ui`, `crashes`, `incremental` | N/A |
| `check-fail` | Building (no codegen) should fail | `ui`, `crashes` | N/A |
| `build-pass` | Building should pass | `ui`, `crashes`, `codegen`, `incremental` | N/A |
| `build-fail` | Building should fail | `ui`, `crashes` | N/A |
| `run-pass` | Running the test binary should pass | `ui`, `crashes`, `incremental` | N/A |
| `run-fail` | Running the test binary should fail | `ui`, `crashes` | N/A |
| `ignore-pass` | Ignore `--pass` flag | `ui`, `crashes`, `codegen`, `incremental` | N/A |
| `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental` | N/A |
| `failure-status` | Check | `ui`, `crashes` | Any `u16` |
| `should-ice` | Check failure status is `101` | `coverage`, `incremental` | N/A |
| `should-fail` | Compiletest self-test | All | N/A |

### Controlling output snapshots and normalizations

See [Normalization](ui.md#normalization), [Output
comparison](ui.md#output-comparison) and [Rustfix tests](ui.md#rustfix-tests)
for more details.

| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------------------------------------------|
| `check-run-results` | Check run test binary `run-{pass,fail}` output snapshot | `ui`, `crashes`, `incremental`[^inc1] if `run-pass` | N/A |
| `error-pattern` | Check that output contains a regex pattern | `ui`, `crashes`, `incremental`[^inc1] if `run-pass` | Regex |
| `check-stdout` | Check `stdout` against `error-pattern`s from running test binary[^check_stdout] | `ui`, `crashes`, `incremental`[^inc1] | N/A |
| `compare-output-lines-by-subset` | Check output contains the contents of the snapshot by lines opposed to checking for strict equality | `ui`, `coverage` | N/A |
| `normalize-stderr-32bit` | Normalize actual stderr (for 32-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-64bit` | Normalize actual stderr (for 64-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-test` | Normalize actual stderr with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stdout-test` | Normalize actual stdout with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `dont-check-compiler-stderr` | Don't check actual compiler stderr vs stderr snapshot | `ui` | N/A |
| `dont-check-compiler-stdout` | Don't check actual compiler stdout vs stdout snapshot | `ui` | N/A |
| `run-rustfix` | Apply all suggestions via `rustfix`, snapshot fixed output, and check fixed output builds | `ui` | N/A |
| `rustfix-only-machine-applicable` | `run-rustfix` but only machine-applicable suggestions | `ui` | N/A |
| `exec-env` | Env var to set when executing a test | `ui`, `crashes` | `<KEY>=<VALUE>` |
| `unset-exec-env` | Env var to unset when executing a test | `ui`, `crashes` | Any env var name |
| `stderr-per-bitwidth` | Generate a stderr snapshot for each bitwidth | `ui` | N/A |
| `forbid-output` | A pattern which must not appear in `cfail` output | `incremental` | Regex pattern |
| `run-flags` | Flags passed to the test executable | `ui` | Arbitrary flags |
| `known-bug` | No error annotation needed due to known bug | `ui`, `crashes`, `incremental` | Issue number `#123456` |
| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------|----------------------------------------------|-----------------------------------------------------------------------------------------|
| `check-run-results` | Check run test binary `run-{pass,fail}` output snapshot | `ui`, `crashes`, `incremental` if `run-pass` | N/A |
| `error-pattern` | Check that output contains a regex pattern | `ui`, `crashes`, `incremental` if `run-pass` | Regex |
| `check-stdout` | Check `stdout` against `error-pattern`s from running test binary[^check_stdout] | `ui`, `crashes`, `incremental` | N/A |
| `compare-output-lines-by-subset` | Check output contains the contents of the snapshot by lines opposed to checking for strict equality | `ui`, `coverage` | N/A |
| `normalize-stderr-32bit` | Normalize actual stderr (for 32-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-64bit` | Normalize actual stderr (for 64-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-test` | Normalize actual stderr with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stdout-test` | Normalize actual stdout with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `dont-check-compiler-stderr` | Don't check actual compiler stderr vs stderr snapshot | `ui` | N/A |
| `dont-check-compiler-stdout` | Don't check actual compiler stdout vs stdout snapshot | `ui` | N/A |
| `run-rustfix` | Apply all suggestions via `rustfix`, snapshot fixed output, and check fixed output builds | `ui` | N/A |
| `rustfix-only-machine-applicable` | `run-rustfix` but only machine-applicable suggestions | `ui` | N/A |
| `exec-env` | Env var to set when executing a test | `ui`, `crashes` | `<KEY>=<VALUE>` |
| `unset-exec-env` | Env var to unset when executing a test | `ui`, `crashes` | Any env var name |
| `stderr-per-bitwidth` | Generate a stderr snapshot for each bitwidth | `ui` | N/A |
| `forbid-output` | A pattern which must not appear in `cfail` output | `incremental` | Regex pattern |
| `run-flags` | Flags passed to the test executable | `ui` | Arbitrary flags |
| `known-bug` | No error annotation needed due to known bug | `ui`, `crashes`, `incremental` | Issue number `#123456` |

[^check_stdout]: presently <!-- date-check: Oct 2024 --> this has a weird quirk
where the test binary's stdout and stderr gets concatenated and then
Expand Down
45 changes: 45 additions & 0 deletions src/tests/minicore.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# `minicore` test auxiliary

[`tests/auxiliary/minicore.rs`][`minicore`] is a test auxiliary for
ui/codegen/assembly test suites. It provides `core` stubs for tests that need to
build for cross-compiled targets but do not need/want to run.

A test can use [`minicore`] by specifying the `//@ use-minicore` directive. Then,
mark the test with `#![feature(no_core)]` + `#![no_std]` + `#![no_core]`. Due to
Edition 2015 extern prelude rules, you will probably need to declare `minicore`
as an extern crate.

Due to the `no_std` + `no_core` nature of these tests, `//@ use-minicore`
implies and requires that the test will be built with `-C panic=abort`.
Unwinding panics are not supported.

If you find a `core` item to be missing from the [`minicore`] stub, consider
adding it to the test auxiliary. However, please note that [`minicore`] is only
intended for `core` items, and explicitly not `std` or `alloc` items because
`core` items are applicable to a wider range of tests.

## Example codegen test that uses `minicore`

```rust,no_run
//@ use-minicore
//@ revisions: meow
//@[meow] compile-flags: --target=x86_64-unknown-linux-gnu
//@[meow] needs-llvm-components: x86

#![crate_type = "lib"]
#![feature(no_core)]
#![no_std]
#![no_core]

extern crate minicore;
use minicore::*;

struct Meow;
impl Copy for Meow {} // `Copy` here is provided by `minicore`

// CHECK-LABEL: meow
#[no_mangle]
fn meow() {}
```

[minicore]: https://github.com/rust-lang/rust/tree/master/tests/auxiliary/minicore.rs
3 changes: 3 additions & 0 deletions src/tests/ui.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ used for many other purposes. For example, tests can also be configured to [run
the resulting program](#controlling-passfail-expectations) to verify its
behavior.

If you need to work with `#![no_std]` cross-compiling tests, consult the
[`minicore` test auxiliary](./minicore.md) chapter.

[`tests/ui`]: https://github.com/rust-lang/rust/blob/master/tests/ui

## General structure of a test
Expand Down