Skip to content

Batches and Tests

In this document, we will cover two fundamental concepts in the ReSim infrastructure that enable customers to run a large number of tests and aggregate the results: tests and test batches. This guide will help you understand what is happening when you are running thousands of tests.

Understanding Batches

You can trigger a batch in many ways in the ReSim platform, such as:

  • the ReSim CLI tool
  • ReSim App
  • GitHub Action

If you need to run a thousand tests, you will initiate one batch that will encompass all those tasks. The Batch is the unit of work that processes an entire set of experiences.

Understanding Tests

Within each batch, individual work items are broken down into tests. When running experiences, Each test corresponds to a specific experience that runs.

Detailed Workflow

The following sections will illustrate each of the phases in the overall batch execution process defined here:

Full Workflow

Batch Submission

When you submit a batch, you define:

  • Experiences: the set of experiences you wish to run against your version of code.
  • Build: Contains your simulator and autonomy code.
  • Metrics Build: Used for analyzing test outputs.

For each experience a separate, parallel, test will be created that runs that experience through the build and then computes metrics on the outputs from the experience using a metrics build.

Test Execution and Metrics

The execution of a test is split into two separate stages: an experience execution stage and a metrics stage:

  1. Experience Execution: Each test runs a specific experience against that build, generating some outputs. The output of this stage is processed in the following metrics stage.

  2. Metrics Stage: After execution, the test's output is analyzed using the metrics build. This is where your pass fail signal is configured and computed.

Batch Metrics and Aggregation

Once all tests within a batch are completed, ReSim moves to the batch metrics stage. During this stage, you can define aggregation methods such as weighted averages in whatever way is best suited to your system.

Conclusion

Running thousands of tests efficiently in the cloud is a challenge. Our use of batches and tests helps us organize the work and make it clear what state your test is in while it is running.