Core Concepts
In order to easily manage the execution of your code on the cloud, ReSim introduces a few basic concepts that it is useful to understand.
Project
A Project is a flexible container for all related concepts within the ReSim Ecosystem. All data within the ReSim platform is isolated across projects, enabling different teams or embodied AI applications to be developed within a dedicated space. Projects can be used to represent a full embodied AI system, for an individual subsystem or as a personal sandbox.
Projects typically map to repositories, in which case it is helpful to use the same name for the project as the repository so that CI services associated with repositories (e.g. GitHub, GitLab) have easy ways to reference repository names inside CI workflows and commands can be parameterised. It is not mandatory to maintain this distinction.
We recommend that a Project be used to reflect an autonomy stack as a whole, and that systems be used to partition individual components of that autonomy stack e.g. perception, localization, etc. In this way, the reporting functionalities of the ReSim app can be used to offer an overview of the state of your stack.
Projects have a name and an ID.
Experience
An experience is a set of interchangeable input files for a test you want to execute in the ReSim app. For a simulation test, an experience will likely include a scenario description (e.g. where the ego is in the world and what its goals are), autonomy stack parameters, and any other assets the simulation test needs to run.
Experiences can be stored in cloud blob storage, such as AWS S3, or a local path in the build container. You will need to make any S3 storage available to ReSim in order for us to consume it.
Experiences can be tagged to help filter and search them.
System
A System is an abstraction of a software component that can be that can be run on a specific set of data with its own resource needs, such as an embodied AI solution. This solution may be composed of several subsystems that can be tested individually or collectively at scale. Examples might include perception system, motion planning, full stack.
A system definition includes the hardware resources requirements for a particular component, such as CPUs, memory, and GPUs. It can then be run against a suite of experiences using a build definition and a metrics build.
You may wish to use experiences and metrics builds with multiple systems. ReSim allows you to tag experiences and metrics builds with one or more systems to show that a particular system is compatible with them. This flexible representation enables users to filter individual test results based on which system is being tested, or get a unified view of the status of the systems that make up a given project.
Note
We have a feature on the roadmap to introduce interactive prompts and compatibility validation to the UI and CLI, but for now, you'll need to visually check
Build
A Build is an instantiation of a system in time. It is a snapshot of the code, represented by a container image, that can actually run the system under test with inputs provided by an experience.
Builds are typically versioned, as the executable test is modified over time. For instance, in continuous integration workflows, you might generate a new build for every new commit to your autonomy code repository so that you can run tests on the newest software.
The executable and dependencies for the build is packaged up in a container image which is pushed to a container registry and registered with the ReSim app. Again, you will need to grant us access to any non-public container registry you use.
Metrics Build
A Metrics Build Image is a Docker image that contains the code for evaluating the performance of your system in the test.
As one tests an autonomy application at increasing scale in a virtual environment, it becomes ever more critical to have good metrics to quantitatively evaluate the performance of your system.
The ReSim platform provides an open-source, general purpose metrics SDK and framework to enable the calculation of metrics on the data output from the execution of your autonomy application.
ReSim enables metrics to be specified and computed at two different levels of granularity: at the test level, where metrics are evaluating the performance across a single experience, and, at the test batch level, where one can compute aggregate metrics across many experiences.
For example, you may compute mean localization error magnitude for your application for each experience in a test metric. However, if you run your application on hundreds of experiences, rather than checking that value for each experience, a test batch metric could provide batch mean localization error magnitude, which aggreagates the individual localization error metrics.
The batch metrics framework is flexible enough to enable one to create a weighted mean, which takes into account that some experiences may have more frames than others.
Branch
A branch is a collection of builds. Branches in the ReSim app are similar to the idea of a branch in git, although there is no automatic association between branches in the two systems. Branches are useful for associating builds that represent different iterations of the same work in the same way that git branches associate a graph of commits. In a continuous integration workflow, each pull request could therefore have its own branch in the ReSim app to contain associated builds.
Test
A test is the fundamental unit of cloud-based work at ReSim. It is a single execution of a given build with a given experience.
It is canonical terminology in software engineering to refer to a Test as a single function that checks some functionality of your system and a Test Suite as a... well, a suite of tests.
A test returns either a pass or a fail, and a test suite passes if and only if all of its tests pass.
At ReSim, we adopt this terminology with a small modification:
- A Test is the execution of a single experience against a build, where performance is evaluated by a metrics build
- A Test Suite is a specification of a number of tests to run and evaluate with the same metrics build. The aggregate performance of the test suite is also determined by the metrics build.
A test suite can then be run with a build to generate tests. Running a test suite will generate a test for each experience in the test suite and then generate aggregate metrics.
To distinguish between the test suite qua specification and group of tests it creates, we use the term Test Batch A test batch is a group of tests that have been run in the ReSim platform. As has been discussed in the Metrics Build page, the test suite's metrics build can also be instrumented to compute so-called batch metrics, which aggregate the metrics from each individual test to display, for example, the average performances of the build or a list of the worst performing experiences.
Test Suite
A test suite is a named template for a set of tests. It is an organizational component to allow you to specify a commonly used set of experiences to run. A common example is to define the set of regression tests for a given system.
A test suite has five main pieces of data:
- The name of the test suite e.g. Nightly Regression Test.
- An optional description to provide some more detail about the intent and purpose of this test suite.
- The system that this test suite is intended to be run with. The ReSim app will warn if you attempt to run the test suite with a build that is not from that system, since it is likely that there will be compatibility issues and it will fail.
- The metrics build that will be used to compute the metrics for this test suite.
- A list of experiences that will be run against the build you choose.
In order to provide traceability of previous results, test suites are versioned. Any time a change
is made to a test suite (updating the metrics build, adding or removing experiences), a new revision
is created. Test suites start at revision 0 and increment.
One of the main benefits for using test suites -- other than the time saving from having to choose what experiences and metrics you want repeatedly -- is that it makes it very easy to:
- Filter and Compare your tests. If there are many test batches created in your project, it can be helpful to filter a list of results by test suite to find what you are looking for and to then compare like for like e.g. compare the last two times a test suite has been run against main
- Control your tests. If a systems and test engineering team exists separately to engineering and have responsibility for defining acceptance criteria, test suites can be used to define what tests the engineering team should be passing for a given feature.
- Construct longitudinal reports. If a test suite is run regularly against a particular branch, it can be helpful to generate a longitudinal report that shows how performance is improving over time i.e. is your system better. Using test suites makes this easy in the ReSim app, as Reports can be used to generate this longitudinal analysis.
Test Batch
A test batch, often referred to as simply a batch, is a collection of tests that have been triggered simultaneously using the same build. In a continuous integration workflow, a batch of simulation tests could be kicked off on each change to the code under development. These would all use the same build (based on the latest iteration of the software), but each could use a different experience.
Test Suite Report
A test suite report is a longitudinal report, displaying evolving performance of a system over time. We have already seen how the ReSim platform supports analysis of an atomic test via metrics, then the aggregate analysis of a group of those tests in a test batch via batch metrics. It can also be useful to construct a longitudinal report, displaying evolving performance of a system over time.
In ReSim, a Test Suite Report offers such a capability, by enabling users to generate completely custom reports that display how a System has performed for a given test suite over time, for a particular branch.
This means, for example, that it is possible to generate a report displaying how the main branch has performed in
Nightly Progression Tests over the past month (ideally ever better). One could also use it to generate a report on the
performance of a PR against a defined set of tests My Acceptance Tests that need to be passed to merge the PR.
Since the philosophy at ReSim is to support maximum customizability for customers, reports can be calculated using a metrics build of your own construction. ReSim does, however, offer an open-source generic report metrics build that creates sensible default reports.
The default ReSim report metrics build generates aggregate statistics such as:
- The total number of tests and batches run over the report period.
- The list of experiences in the test suite, ordered by the number of failures.
- The list of all metrics computed for the batches, ordered by number of failures.
- A line chart showing how the number of tests passing per batch has evolved over the time period
- For any scalar batch metrics, a line chart displaying that scalar value's evolution over time e.g. mean average precision over time.
View
A view is a set of visualizable 3D content generated by ReSim View. This content can be visualized using Foxglove Studio by clicking the Open in Foxglove link provided on the view's page in the ReSim app. ReSim View is a C++ library that users can link to their code to make it easy to visualize 3D transforms in their code.