Test Suites

If you've gone through all of the previous articles, you are now ready to begin testing your builds with some experiences and generating metrics! We do this by creating and running a Test Suite.

Creating A Test Suite

To create a test suite, you need to have a system in mind, a metrics build and a set of experiences you would like to use. You can create a test suite using the ReSim CLI with this command, assuming you already have a project selected

resim suites create \
    --name "Perception Nightly Regression"
    --description "A test suite for perception system nightly regression tests"
    --system "perception"
    --metrics-build-id <METRICS BUILD UUID> \
    --experiences "<experience name or UUID>, <experience name or UUID>, ..."

You can get or list the test suites for that project. For example, the following command displays the latest revision:

resim suites get \
    --test-suite "Perception Nightly Regression"

To get a specific revision append the --revision 1 key. To get a list of all revisions, pass --all-revisions.

In order to revise the test suite, you simply need to pass the elements that you would like to change. For example, to change the experience list, you would use the following command:

resim suites revise \
    --test-suite "Perception Nightly Regression"
    --experiences "<new experience name or UUID>, <new experience name or UUID>, ..."

Warning

You need to specify the exact list of experiences. The CLI will be extended in the near future to support specifying a query to describe the list of experiences. In the short term, the ReSim app supports query-based test suite creation and building.

Running A Test Suite

If you have a test suite, you can then run it to create a Test Batch:

resim suites run \
    --test-suite "Perception Nightly Regression"
    --build-id "<UUID of the build>"

If you would like to specify a particular revision, you can pass --revision 0.

You can list the test batches that have been created from this test suite via:

resim suites batches \
    --test-suite "Perception Nightly Regression"

This will list all the test batches from all revisions of the test suite. Simply pass the --revision flag to see test batches from a specific test suite revision.

Tracking A Test Batch

Once you have created a test batch by running a test suite, the CLI lets you track the status of that batch. You can list all of the batch details -- as JSON -- either by passing the friendly name of the batch (e.g. rejoicing-aquamarine-starfish) or its id:

resim batches get \
    --batch-id <Batch ID>

or

resim batches get \
    --batch-name <Batch friendly name>

If you additionally pass the exit status key:

resim batches get \
    --batch-id <Batch ID> \
    --exit-status

The application will return only the exit status, which are: - 0: Succeeded: the batch has completed. This does not indicate whether the metrics have passed or failed. - 1: Internal Error: the CLI command errored (which is nothing to do with the test batch). - 2: Error: the batch has finished with an error.
- 3: Submitted: the batch is in the submitted state. - 4: Running: the batch is still running. - 5: Cancelled: the batch has been cancelled.

This is commonly used in CI to trigger a failing test.

You can also list the individual tests in that batch with:

resim batches tests \
    --batch-name <Batch friendly name>

This will return the JSON associated with the test to track the individual status and return their IDs. The IDs are helpful to, for example, list the log files associated with that test:

resim logs list \
    --batch-id <Batch ID> \
    --test-id <Test ID>

The output of this includes pre-signed URLs that would enable you to download logs directly from the command line. In particular, every test contains the experience-container.log and metrics-container.log files which contain the console logs from your container (experience-container.log) and the metrics stage (metrics-container.log).

Ad Hoc Test Suites

It is not always desirable to create a test suite: sometimes, you may simply want to try a few experiences on an experimental branch to make sure that you haven't broken everything. For this, the ReSim App supports the concept of an ad hoc test suite. That is, an unnamed test suite.

You can create this in the CLI as follows:

resim batches create \
    --project "my-project" \
    --build-id <BUILD UUID> \
    --metrics-build-id <METRICS BUILD UUID> \
    --experiences "<experience name or UUID>, <experience name or UUID>, ..." \
    --experience-tags "<tag name>, <tag name>"

Use --experiences and/or --experience-tags to control which experiences are run in the batch.