Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add first draft of a spin-test SIP #2462

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 190 additions & 0 deletions docs/content/sips/000-spin-test.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
title = "SIP 000 - `spin-test`"
template = "main"
date = "2024-04-18"
---

Summary: create a component based testing framework for testing Spin applications within a WebAssembly sandbox.

Owner(s): [email protected]

Created: April 18, 2024

This SIP describes the `spin-test` tool that allows running unmodified Spin applications in WebAssembly component based testing environment where all potential imports to the Spin app are virtualized *and* can be configured by the user in a language independent way.

Implementation of this SIP has already begun [here](https://github.com/fermyon/spin-test).

## Requirements

The requirements for such a tool are the following:

- Does not require the user to modify their application in any way - what runs in the test is what runs in production.
- The app and all the imports to the app run in WebAssembly and not on the host.
- The test itself also runs in WebAssembly allowing tests to be written in any language with WebAssembly component guest support.
- Allows the user to modify the testing environment in a programmatic way in any language they want to use.
- Have the test environment match functionality as much as is possible with other Spin runtimes such as found in Spin CLI, Fermyon Cloud, and SpinKube.

## Proposal

To achieve the goals above, we propose a new tool tentatively called `spin-test`.

The `spin-test` binary would be callable from any Spin project in much the same way that `spin up` is project aware (i.e., it doesn’t need to be told where the `spin.toml` manifest is).

Invoking `spin-test` will automatically resolve where the Spin app component lives. The tool will automatically find tests that should be run and will run those tests against the Spin app component reporting whether the tests passed or failed. `spin-test` will come with basic ways of filtering tests so that a subset of tests can be run.

**Note:** See the *Open Questions* section for discussion on how `spin-test` discovers tests to run.

### Implementation

At the core of `spin-test` is component composition. The user’s Spin app is composed together with other components to ultimately produce a component (referred to as “the composition” below) that has none of the original Spin or WASI imports. The composition has the following components internal to it:

- A single component from the Spin app (see the *Open Questions* section for why only one component from a Spin app can be supported)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This refers to "a single component" from the perspective of the Spin application manifest I assume — if my Spin component build process generates a statically linked component, I would expect testing to continue to work as expected.

Is that accurate?

- A Spin virtualization component that virtualizes all `fermyon:spin/platform` interfaces.
- A WASI virtualization component that virtualizes all the `wasi:cli/imports` interfaces.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I was going through the test examples from the implementation repository, I was thinking about the wasi:cli/imports imports — will this contain a programmatic API to set the behavior of all WASI functionality?
Will there be a subset available to begin with?

I'm curious what the API through which we expose that to users in tests will look like.

- A router component that decides from a `spin.toml` manifest file whether an `incoming-request` should be routed to the Spin component or not.
- A test driver component that can configure the Spin and WASI virtualization components, make one or more requests to the router, and then make assertions on the response returned and the state of the Spin and WASI virtualizations.

The composition only has the following imports:

- A way to receive a `spin.toml` manifest file. This is needed by the router, Spin virtualization, and WASI virtualization components to know how they should be configured.
- A few imports for working around limitations of guest support for `wasi:http`:
* An `http-helper` interface which allows creation of `wasi:http/[email protected].{incoming-request, response-outparam, incoming-response}` resources since these can only be created by a host.
* Another function that can turn `wasi:http@[email protected].{outgoing-response}` into a `wasi:http@[email protected].{future-incoming-response}`

*Note*:

> **A few notes:**
* We may still wish to have the composition import `wasi:cli/stdout` and `wasi:cli/stderr` so that it can easily log things.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example test I ran from the implementation repo did log to stdout -- is this already supported?

* It might be possible to statically configure the virtualized Spin runtime component given a manifest file by synthesizing the component on the file (in much the way that WASI-virt works). This would make it possible to no longer require that the manifest is passed to the composition component at runtime since it has already been statically configured to run correctly with that Spin manifest. This has its downsides though as composition components for the same app but different manifests cannot be shared between tests.
* The `wasi:http` work around imports may hopefully eventually overcome with changes to the `wasi:http` package.
* If this and the manifest import are eliminated, tests could potentially be run by an component runtime with no imports whatsoever.
>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: leftover >


## Wit definitions

**NOTE**: These wit definitions are still very much under development! Bike-shedding welcome!

### The `fermyon:spin-test/test` world

This is the world that all `spin-test` compliant test components target.

```
package fermyon:spin-test;

world test {
/// The following allow the test to both configure and observe
/// the environment the Spin app is running in.

/// Gives the test the ability to read and write to the kv store
/// independently of the Spin app.
import fermyon:spin/[email protected];
/// A handle into the configuration of the `wasi:http/outbound-handler`
/// implementation.
import fermyon:spin-test-virt/http-handler;
/// A handle into an interface that tracks calls to the key-value store
import fermyon:spin-test-virt/key-value-calls;
/// TODO: more handles to configure and observe the other Spin
/// and WASI interfaces.

/// The following allow the test to make requests against the Spin app
/// and view its response.
import http-helper;
/// The ability to call the Spin app
import wasi:http/[email protected];

/// Actually call the test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: formatting for WIT source and comments seems off in parts of this file.

export run: func();
}

interface http-helper {
use wasi:http/[email protected].{incoming-request, response-outparam, incoming-response, outgoing-request};
resource response-receiver {
get: func() -> option<incoming-response>;
}
new-request: func(request: outgoing-request) -> incoming-request;
new-response: func() -> tuple<response-outparam, response-receiver>;
}
```

### The `fermyoin:spin-test-virt/plug` world

This is the world of the component that virtualizes a Spin runtime.

```
package fermyon:spin-test-virt;

/// The exports that can be composed with a Spin app creating
/// a virtualized component.
world plug {
/// All of the `fermyon:spin/platform` world's interfaces are
/// exported here.
export fermyon:spin/[email protected];
export fermyon:spin/[email protected];
export fermyon:spin/[email protected];
export fermyon:spin/[email protected];
export fermyon:spin/[email protected];
export fermyon:spin/[email protected];
export fermyon:spin/[email protected];
export fermyon:spin/[email protected];
export wasi:http/[email protected];

/// The virtualization needs the component's id and
/// the `spin.toml` manifest because many of the spin
/// interfaces are configured through that combination.
export set-component-id: func(component-id: string);
import get-manifest: func() -> string;

/// A way to say that a certain URL is associated with a response
export http-handler;
/// How the calls to the kv store are tracked
export key-value-calls;

}

interface http-handler {
use wasi:http/[email protected].{future-incoming-response};
set-response: func(url: string, response: future-incoming-response);
}

interface key-value-calls {
reset: func();
get: func() -> list<tuple<string, list<get-call>>>;
set: func() -> list<tuple<string, list<set-call>>>;

record get-call {
key: string
}

record set-call {
key: string,
value: list<u8>
}
}
```

### The `fermyoin:spin-test/runner` world

```
world runner {
/// The host must supply a manifest
import get-manifest: func() -> string;
/// The host must give the runner the ability to create HTTP resources
import http-helper;
/// The host can then run the test
export run: func();
}
```

## Open Questions

- Should `spin-test` be a stand alone CLI experience, a plugin for `spin` CLI, or both?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say plugin is probably best - at least as a starting point - as it would benefit from the plugin update framework.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on the plugin route for delivering the functionality to users.

- How do we handle Spin applications with more than component?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignoring service chaining (?), presumably you could create a test composition per Spin component? That might even be ideal if you could compose multiple components as you wouldn't have to build anything you aren't currently testing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But which test composition do we end up actually calling? The spin-test host calls the run function to kick off the test. This run function is exported by the user-written test runner component. This test runner component is ultimately what calls the router (one or more times) as part of the test. If we create N test compositions, N - 1 of those compositions are not meant to receive that request (and calling their run function would just fail). At the core of the issue is that we don't really known which component needs to called before hand - the router is what makes that decision, but the router cannot compose with an arbitrary number of components to route to.

The only way I can see see to work around this are the two proposed solutions: moving routing into the host or synthesizing a router component.

Copy link
Collaborator

@lann lann Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohhh I was thinking that the developer would explicitly map tests to components, but you are looking at doing that via regular URL path routing. Yeah I would think in the short term that would require the host to do the routing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a scenario that would become (at least for me) much clearer with an example of the (proposed/desired/currently possible/default) developer experience for writing tests. I think there are a few decisions that might impact the decisions here:

  • does each Spin component come with its own sets of tests, or is there a single test "project" for the entire Spin application?
  • depending on ^^, there could be a scenario where the test is linked to a single component, which means the question is no longer difficult to answer.

- There’s no way to compose a statically defined component with N number of components. This means the router component must know how many components it will be composed with ahead of time, and so the reasonable number of 1 is picked.
- We could work around this by moving the router out of a component and into the host, but this makes the host smarter which would like to avoid. We could also potentially synthesize a router component based on the `spin.toml` manifest instead of relying on a statically defined one, but this is certainly not trivial.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could also (statically/macro-ly) generate a family of routers (router1, router2, ...). Annoying, but probably fine as a medium-term hack.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wouldn't work either (without changing the host to be smarter) as something needs to split one request into a call to one of N possible components. Composing N components with 1 component (where N is not known ahead of time) is not possible.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about 1 router with many identical component instance imports? It would be a little fiddly but you should be able to derive one consistent ordering of spin components from a manifest which would let the composition step fill the imports in the same order that the router then expects them. You would also need some way to fill all the unused import slots, either a static dummy component that always traps or some Linker jiggery-pokery.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you're suggesting a single router component with some arbitrary amount of imports that components can plug into and we either plug those with actual Spin app components or with a dummy trapping component once we run out of Spin app components?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep

- How should test discovery work?
- The most straightforward way would be for `spin-test` to require the user to specify the path to a `spin-test` compliant test component binary.
- Another way would be for there to be a convention around a `tests` directory. This `tests` directory could include both built `spin-test` compliant test component binaries as well as directories with source code and a `spin-test.toml` configuration file that specifies how the source code is built into a `spin-test` compliant test component binary. `spin-test` would then build all the tests that need to be built and run each test binary.
lann marked this conversation as resolved.
Show resolved Hide resolved
- We could also carve away space in the `spin.toml` manifest (e.g., by using the `[component.<id>.tool]` section). Such a solution might be more appropriate if we determine that users will normally want very few test component binaries. If, however, it proves to be useful to potentially have many test binaries, splitting the configuration out to each test binary in the form of a bespoke `spin-test.toml` per test binary might be the better solution.
Copy link
Collaborator

@lann lann Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose we could also do both (eventually), with a compatible schema reused between spin.toml [component.<id>.tool] for one-test-per-component and tests/spin-test.toml for more complex scenarios.

Copy link
Collaborator

@lann lann Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option to consider might be the application.tool section in spin.toml, e.g. [application.tool.spin-test.test-a]. I'm not sure I would actually like that but it would avoid having another config file.

- How exactly should the various worlds look like?
- The proposal above is enough to get things working, but there is plenty of room for bike-shedding the exact shape of the worlds.
- Which triggers should we support?
- Can we support an arbitrary number of triggers through some sort of plugin system? How would that work?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like something that may become dramatically easier with CM async (to the extent that magically fixes all our nested composition woes).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I'm not sure I follow - none of the triggers (with the exception of wasi:http/incoming-handler which we already support) are necessarily built to try to accommodate async. I don't see how CM async improves the situation here. Could you explain more about what you mean?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah sorry the train of consciousness left the station early there...

I think each trigger type could have a test driver adapter to wrap the component-under-test and test driver to implement the runner world; this is impossible today because the host has to provide resource factories to glue everything together.