In order to easily manage the execution of your code on the cloud, ReSim introduces a few basic concepts that are useful to understand. ## Project A **Project** is a flexible container for all related concepts within the ReSim Ecosystem. All data within the ReSim platform is isolated across projects, enabling different teams or embodied AI applications to be developed within a dedicated space. Projects can be used to represent a full embodied AI system, for an individual subsystem or as a personal sandbox. Projects typically map to repositories, in which case it is helpful to use the same name for the project as the repository so that CI services associated with repositories (e.g. GitHub, GitLab) have easy ways to reference repository names inside CI workflows and commands can be parameterised. It is not mandatory to maintain this distinction. We recommend that a **Project** be used to reflect an autonomy stack as a whole, and that [systems](#system) be used to partition individual components of that autonomy stack e.g. perception, localization, etc. In this way, the reporting functionalities of the ReSim app can be used to offer an overview of the state of your stack. Projects have a name and an ID. - [Create a project](../setup/projects/) ## Experience An **experience** is a set of interchangeable input files for a test you want to execute in the ReSim app. For a simulation test, an experience will likely include a scenario description (e.g. where the ego is in the world and what its goals are), autonomy stack parameters, and any other assets the simulation test needs to run. Experiences can be stored in cloud blob storage (such as AWS S3 or Google Cloud Storage), in the Foxglove Data Platform, or at a local path in the [build container](#build). You will need to [configure ReSim to access your experience data](../setup/experience-data-sources/) in order for us to consume it. Experiences can be stored in multiple locations, all of which is fetched by ReSim. Experiences can be tagged to help filter and search them. - [Add experiences](../setup/adding-experiences/) Experiences also support important metadata for when they are executed as a test. In particular, you can set: - A set of environment variables that can be injected into your [build](#build) - A timeout (in seconds) that can be used as an upper bound on your expected experience runtime, to guard against infinite loops - A [docker compose profile](https://docs.docker.com/compose/how-tos/profiles/) that should be used to run this experience if you are using a multiple container build. Note: we do not support multiple profiles right now, please contact us if this is important. ## Asset An **asset** is a set of shared data files that a [build](#build) needs at runtime but that are not specific to any single [experience](#experience). Common examples include HD maps, 3D environment meshes, ML model weights, physics parameters, and calibration data -- anything that remains constant across many experiences. While experiences represent per-test inputs (mounted at `/tmp/resim/inputs/`), assets represent shared data that is associated with a **build** and mounted read-only at `/tmp/resim/assets//`. This separation means you don't need to duplicate large shared files across experience locations or bake them into your Docker image. Assets support **revisions**: each time the source data changes, a new revision is created with a version string, preserving an immutable history. Assets are [cached](../guides/asset-caching/) by default, so only changed files are synchronized between runs. Assets are explicitly linked to builds, and we recommend managing these associations in your CI/CD pipeline to ensure reproducibility. - [Set up assets](../setup/assets/) ## System A **System** is an abstraction of a software component that can be run on a specific set of data with its own resource needs, such as an embodied AI solution. This solution may be composed of several subsystems that can be tested individually or collectively at scale. Examples might include perception system, motion planning, full stack. A **system** definition includes the hardware resources requirements for a particular component, such as CPUs, memory, and GPUs. It can then be run against a suite of [experiences](#experience) using a [build](#build) definition and a [metrics build](#metrics-build). You may wish to use experiences and metrics builds with multiple systems. ReSim allows you to tag experiences and metrics builds with one or more systems to show that a particular system is compatible with them. This flexible representation enables users to filter individual test results based on which system is being tested, or get a unified view of the status of the systems that make up a given [project](#project). Note We have a feature on the roadmap to introduce interactive prompts and compatibility validation to the UI and CLI, but for now, you'll need to visually check - [Define a system](../setup/systems/) ## Build A **Build** is an instantiation of a system in time. It is a snapshot of the code, represented by a container image, that can actually run the [system](#system) under test with inputs provided by an [experience](#experience). Builds are typically versioned, as the executable test is modified over time. For instance, in continuous integration workflows, you might generate a new build for every new commit to your autonomy code repository so that you can run tests on the newest software. The name of a build usually references the relevant system (e.g. "Perception Build") whereas its description contains a descriptive commit message or SHA. The executable and dependencies for the build is packaged up in a [container image](https://opencontainers.org/) which is pushed to a container registry and [registered with the ReSim app](../setup/build-images/). ReSim supports images from AWS ECR, Docker Hub, GitHub Container Registry, and Google Artifact Registry. You will need to [grant us access](../setup/container-registry-access/) to any non-public container registry you use. - [Build a system image](../setup/build-images/) - [Push a system image from CI](../setup/ci/) ### Multi-container builds In place of a single container image, a **Build** may also be a collection of container images tied together by a [Docker Compose](https://docs.docker.com/reference/compose-file/) file. Defining a multi-container build like this allows you to have multiple containers executed together during the **Experience** phase. These containers can communicate with each other, share data, etc, and may allow you to better simulate a more complex system. ## Metrics build A **Metrics Build Image** is a Docker image that contains the code for evaluating the performance of your system in the test. As one tests an autonomy application at increasing scale in a virtual environment, it becomes ever more critical to have good *metrics* to quantitatively evaluate the performance of your system. The ReSim platform provides an open-source, [general purpose metrics SDK](https://github.com/resim-ai/open-core/tree/main/resim/metrics) and framework to enable the calculation of metrics on the data output from the execution of your autonomy application. ReSim enables metrics to be specified and computed at two different levels of granularity: at the [*test*](#test) level, where metrics are evaluating the performance across a single experience, and, at the [*test batch*](#test-batch) level, where one can compute aggregate metrics across many experiences. For example, you may compute *mean localization error magnitude* for your application for each experience in a *test* metric. However, if you run your application on hundreds of experiences, rather than checking that value for each experience, a *test batch* metric could provide *batch mean localization error magnitude*, which aggregates the individual localization error metrics. The batch metrics framework is flexible enough to enable one to create a *weighted mean*, which takes into account that some experiences may have more frames than others. - [Build a metrics image](/open-core/metrics/metrics_builds/) ## Branch A **branch** is a collection of builds. Branches in the ReSim app are similar to the idea of a branch in [git](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell), although there is no automatic association between branches in the two systems. Branches are useful for associating builds that represent different iterations of the same work in the same way that git branches associate a graph of commits. In a continuous integration workflow, each pull request could therefore have its own branch in the ReSim app to contain associated builds. - [Create a branch](../setup/build-images/#creating-a-branch) ## Test A **test** is the fundamental unit of cloud-based work at ReSim. It is a single execution of a given build with a given experience. It is canonical terminology in software engineering to refer to a **Test** as a single function that checks some functionality of your system and a **Test Suite** as a... well, a suite of tests. A test returns either a pass or a fail, and a test suite passes if and only if all of its tests pass. At ReSim, we adopt this terminology with a small modification: - A **Test** is the execution of a single [*experience*](#experience) against a [*build*](#build), where performance is evaluated by a [*metrics build*](#metrics-build) - A **Test Suite** is a *specification* of a number of tests to run and evaluate with the same metrics build. The aggregate performance of the test suite is also determined by the *metrics build*. A test suite can then be run with a build to generate tests. Running a test suite will generate a test for each experience in the test suite and then generate aggregate metrics. To distinguish between the test suite *qua* specification and group of tests it creates, we use the term [**Test Batch**](#test-batch). A test batch is a group of tests that have been run in the ReSim platform. As has been discussed in the [Metrics Build](../setup/metrics-builds/) page, the test suite's metrics build can also be instrumented to compute so-called batch metrics, which aggregate the metrics from each individual test to display, for example, the average performances of the build or a list of the worst performing experiences. ## Test suite A **test suite** is a named template for a set of [tests](#test). It is an *organizational* component to allow you to specify a commonly used set of [experiences](#experience) to run. A common example is to define the set of regression tests for a given [system](#system). A test suite has five main pieces of data: - The **name** of the test suite e.g. Nightly Regression Test. - An optional **description** to provide some more detail about the intent and purpose of this test suite. - The **system** that this test suite is intended to be run with. The ReSim app will warn if you attempt to run the test suite with a build that is not from that system, since it is likely that there will be compatibility issues and it will fail. - The **metrics build** that will be used to compute the metrics for this test suite. - A list of **experiences** that will be run against the build you choose. In order to provide traceability of previous results, test suites are *versioned*. Any time a change is made to a test suite (updating the metrics build, adding or removing experiences), a new *revision* is created. Test suites start at `revision 0` and increment. One of the main benefits for using test suites -- other than the time saving from having to choose what experiences and metrics you want repeatedly -- is that it makes it very easy to: - Filter and Compare your tests. If there are many test batches created in your project, it can be helpful to filter a list of results by test suite to find what you are looking for and to then compare like for like e.g. compare the last two times a test suite has been run against main - Control your tests. If a systems and test engineering team exists separately to engineering and have responsibility for defining acceptance criteria, test suites can be used to define what tests the engineering team should be passing for a given feature. - Construct longitudinal reports. If a test suite is run regularly against a particular branch, it can be helpful to generate a longitudinal report that shows how performance is improving over time i.e. is your system better. Using test suites makes this easy in the ReSim app, as [Reports](../setup/reports/) can be used to generate this longitudinal analysis. - [Create test suites](../setup/test-suites/) ## Test batch A **test batch**, often referred to as simply a **batch**, is a collection of tests that have been triggered simultaneously using the same build. In a continuous integration workflow, a batch of simulation tests could be kicked off on each change to the code under development. These would all use the same build (based on the latest iteration of the software), but each could use a different experience. - [Run a batch](../setup/test-suites/#running-a-test-suite) ## Test suite report A **test suite report** is a longitudinal report, displaying *evolving performance of a [system](#system) over time*. We have already seen how the ReSim platform supports analysis of an atomic test via [metrics](#metrics-build), then the aggregate analysis of a group of those tests in a test batch via batch metrics. It can also be useful to construct a longitudinal report, displaying evolving performance of a system over time. In ReSim, a **Test Suite Report** offers such a capability, by enabling users to generate completely custom reports that display how a System has performed for a given test suite over time, for a particular [**branch**](#branch). This means, for example, that it is possible to generate a report displaying how the `main` branch has performed in *Nightly Progression Tests* over the past month (ideally ever better). One could also use it to generate a report on the performance of a *PR* against a defined set of tests *My Acceptance Tests* that need to be passed to merge the PR. Since the philosophy at ReSim is to support maximum customizability for customers, reports can be calculated using a **metrics build** of your own construction. ReSim does, however, offer an open-source generic report metrics build that creates sensible default reports. The default ReSim report metrics build generates aggregate statistics such as: - The total number of tests and batches run over the report period. - The list of experiences in the test suite, ordered by the number of failures. - The list of all metrics computed for the batches, ordered by number of failures. - A line chart showing how the number of tests passing per batch has evolved over the time period - For any scalar batch metrics, a line chart displaying that scalar value's evolution over time e.g. mean average precision over time. - [Test suite reports](../setup/reports/) ## View A **view** is a set of visualizable 3D content generated by ReSim View. This content can be visualized using Foxglove Studio by clicking the **Open in Foxglove** link provided on the view's page in the ReSim app. ReSim View is a C++ library that users can link to their code to make it easy to visualize 3D transforms in their code. - [Visualization](../open-core/visualization/)