Run Build Parameter Sweeps

Introduction

It is usually the case that robotics, AI, and autonomy applications have a number of configurable parameters that impact their performance. For example, the acceptable minimum distances to maintain to other agents, weightings for a tracker, number of particles in a particle filter. Another common use of parameters is to toggle a prototype feature.

At ReSim, we call these Build Parameters, because they impact how a particular version of an autonomy application (encapsulated as a build) will perform across the experiences that it encounters.

The ReSim platform supports the ability to run what we call a Build Parameter Sweep (or sweep hereafter) just like you would create a normal test batch. The benefit of this is that one can quickly experiment with different parameterizations of an autonomy application in order to quickly determine candidate optimal values for parameters without the slow (and more qualitative) feedback loop of field testing.

Build Parameter Sweep Components

A sweep requires:

• A build parameter configuration, which is the specification of the parameters to sweep. See below for more details.
• A build that can override default parameters with the values provided by ReSim at run time. See below for the interface.
• A set of experiences to execute, as either a list of experiences or experience tags (or both).
• A metrics build that will evaluate the performance of the parameterized build on those experiences. See the metrics overview for more details.

For each combination of values from the parameter configuration, the ReSim platform will create a test batch containing all the experiences. The ReSim App provides an experimental page to allow one to compare the performance of each experience and the batch as a whole across the parameters.

Supported Sweep Types

The ReSim platform currently only supports one type of sweep configuration: a grid search across parameters. With a grid search, one specifies the parameters and their desired values individually and ReSim will generate tests for the cross-product.

For example, a single parameter my_first_parameter with n values (value1, value2 etc.) will simply generate n test batches. However, two parameters, one with n and one with m values will generate n x m test batches: for each combination of the parameters.

Creating a Build Parameter Sweep

One can currently only create a parameter sweep in the ReSim CLI (as of version 0.1.27). The syntax is similar to creating a test batch, with the exception of two alternative ways to specify the parameters.

For a one-dimensional grid search, the CLI offers a shortcut:

resim sweeps create \
    --project <PROJECT NAME> \
    --build-id <BUILD UUID> \
    --metrics-build-id <METRICS BUILD UUID> \
    --experience-tags "<TAG 1>, <TAG 2>"
    --parameter-name "my_first_parameter"
    --parameter-values "value1","value2","value3" ...

This creates a sweep with three batches, testing out the build with three parameterizations.

For a multi-dimensional grid search, or to allow one to persist the parameters on a local machine, the CLI offers:

resim sweeps create \
  --build-id <BUILD UUID> \
  --metrics-build-id <METRICS BUILD UUID> \
  --experience-tags "<TAG 1>, <TAG 2>"
  --grid-search-config <path to json config file>

The syntax for a config file is a JSON list of sweep parameters and their values:

[
  {
    "name": "parameter-1",
    "values": ["parameter-1-value-1", "parameter-1-value-2"]
  },
  {
    "name": "parameter-2",
    "values": ["parameter-2-value-1", "parameter-2-value-2"]
  }
]

For more parameters extend the list of parameters, for more values extend each individual values list. To modify a default parameter with only one option, simply have a singleton list, which will not contribute to the growth in the number of tests. Caution: grid searches with many parameters can result in a combinatorial explosion i.e. many more tests than you expected!

Visualizing Batch Parameter Sweeps

The ReSim web app has an experimental page to help with visualizing sweeps. In particular the page enables:

• For each experience in the sweep: a direct comparison of any scalar metrics (seen as a bar chart) or line chart metrics (e.g. time series) superimposed.
• At the test batch level, a direct comparison of any scalar or line chart batch metrics. For more information about batch metrics, see the metrics page
• A list of the test batches and tests created in the sweep to navigate quickly to the full metrics.

In the near future, we plan to support direct comparisons of all metrics types as well as the ability to create sweeps in the web app.

Build Parameter Sweep Mechanics

In order to support sweeps, some small modifications are required to the builds that are registered with ReSim.

Now, in addition to the experience data, the /tmp/resim/inputs directory will, for all batches, now contain a file called parameters.json which contains the parameters and their values as strings that are to be used in place of defaults for this particular test.

The format of parameters.json is very simple: a set of fields of parameter:value. For example:

{
  "parameter-1": "parameter-1-value",
  "parameter-2": "parameter-2-value"
}

If the test batch is not part of a sweep (and parameter overrides are not specified), then this will be an empty JSON file.

Your startup script/entrypoint should, therefore, read this file, parse the parameter values and set arguments or environment variables as required. Since ReSim do not require any further details of how a customer's docker entrypoint operates, we can provide ad-hoc support on parsing this data in a particular application.

Test Suites and Parameters

It is not yet directly possible to create a parameter suite from a test suite, but is on the development roadmap.

It is, however, possible and often extremely valuable to run a test suite with specific parameter overrides passed to the build. This means, you can run your test suite with my-beta-feature : true selected, for example.

To do this in the ReSim CLI, simply adapt the following command:

resim suites run \
    --test-suite "Perception Nightly Regression"
    --build-id "<UUID of the build>"
    --parameter "my-param : my-value", "my-param-2 : my-value-2"

This will create a test batch and pass the parameters in the parameters.json file described above.