Re-Running Partial Batches

You might have a need to re-run some subset of tests inside a batch. Maybe one experience encountered a transient error. Maybe it has non-deterministic behaviour. But you don't want to re-run all of the tests in a batch, just one or a few problematic ones.

Limitations

The tests will be re-run as if they were being freshly run now. They do not take a snapshot of what the configuration was when they were first run and re-use that data. This means that if you have updated any of the dependencies, including system, image, experience data, etc - then the test will run with the latest definitions of that data.

When re-running tests, both the experience and metrics stages for that test will be re-run. The metrics stage will run against only the results from the new run; results from previous runs will be ignored.

It is not currently possible to only re-run the metrics stage for a test re-using previously generated experience output.

The Batch Metrics stage will always be re-run. It will reuse the output from any test that did not need to be re-run, combining it with the fresh output from any re-run tests, ignoring old output for anything that was superseded.

CLI Instructions

To re-run one or more tests:

resim batches rerun --batch-id={batch id} --test-ids={test id 1, test id 2, ...}

Specify the test IDs (aka Job IDs) from the batch in question. Do not use (eg) experience IDs.

To re-run just the Batch Metrics stage, simply specify no test IDs:

resim batches rerun --batch-id={batch id}

App Instructions

Coming Soon