# ReSim Docs

> Documentation for ReSim, the simulation-based evaluation platform for autonomous systems.

ReSim is a simulation-based evaluation platform for autonomous systems.
These docs cover setting up the platform, running test batches and
parameter sweeps, integrating with CI, building metrics, and consuming
results via the CLI, SDK, MCP server, and web app. They also include
reference material for the open-core libraries and conceptual
explanations of simulation-based evaluation.


# Tutorials

# Tutorials

Hands-on walkthroughs that take you from zero to working results.

~10 min Step 1

Get Your First Metrics in ReSim

Use the Python SDK to push data from your own machine and see metrics in the dashboard. No Docker or cloud infrastructure needed.

[Start tutorial →](./first-external-batch/)

~20 min Step 2

Run Your First Test Batch

Set up a project, register builds and experiences using the demo repo, and run three tests in parallel. No Docker or AWS needed.

[Start tutorial →](./first-batch/)

______________________________________________________________________

Once you've completed the tutorials, the [How-to Guides](../setup/) cover bringing your own system, experiences, and metrics into ReSim.

# Get Your First Metrics in ReSim

In this tutorial you'll get data into ReSim and see real metrics in the dashboard — without Docker, without a container registry, and without running anything in ReSim's infrastructure. You'll use the ReSim Python SDK to create a batch, emit structured data from two tests, and view the results.

**Time:** about 10 minutes

**Before you start:**

- You have a ReSim account. Sign up at [app.resim.ai](https://app.resim.ai) if you don't have one.
- You have Python 3.10+ installed.
- You can authenticate in a browser (the script will prompt you with a URL the first time).

______________________________________________________________________

## What you're building

The [resim-sdk-example](https://github.com/resim-ai/resim-sdk-example) repo contains a minimal Python script that:

1. Creates a batch in your ReSim project
1. Runs two tests, each emitting a series of `(x, y)` position data points
1. Uploads the data to ReSim and triggers metrics processing

You'll end up with a batch in the ReSim app showing a line chart of `y over time` per test, and a batch-level table of min/avg/max values across both tests.

______________________________________________________________________

## Step 1: Create a project

If you don't have a ReSim project yet, create one:

Go to [app.resim.ai/projects/create](https://app.resim.ai/projects/create) and create a project.

Shell

```
resim projects create --name "my-project" --description "My project"
```

Note the project name — you'll need it in Step 4.

______________________________________________________________________

## Step 2: Clone the example repo

Shell

```
git clone https://github.com/resim-ai/resim-sdk-example.git
cd resim-sdk-example
```

______________________________________________________________________

## Step 3: Install dependencies

Shell

```
pip install -r requirements.txt
```

Or, if you're using `uv`:

Shell

```
uv sync
```

This installs `resim-sdk`.

______________________________________________________________________

## Step 4: Set your project name

Open `main.py` and update `PROJECT_NAME` to match the project you created in Step 1:

main.py

```
PROJECT_NAME = "my-project"
```

______________________________________________________________________

## Step 5: Look at what you're running

Before running, take a look at the two files that matter:

**`main.py`** — creates the batch and emits test data:

main.py

```
client = DeviceCodeClient()

with Batch(
    client=client,
    project_name=project_name,
    branch="metrics-test-branch",
    metrics_set_name="my metrics",
    metrics_config_path="resim/config.resim.yml",
) as batch:

    with Test(client, batch, "test 1") as test:
        for i in range(10):
            test.emit("position", {"x": float(i), "y": float(i)}, i)

    with Test(client, batch, "test 2") as test:
        for i in range(10):
            test.emit("position", {"x": float(i), "y": float(i * 2)}, i)
```

**`resim/config.resim.yml`** — defines the data schema and metrics:

resim/config.resim.yml

```
version: 1

topics:
  position:
    schema:
      x: float
      y: float

metrics:
  y over time:
    type: test
    query_string: select 'y', timestamp, y from position order by timestamp
    template_type: system
    template: line

  min/avg/max:
    type: batch
    query_string: select min(y) as min, avg(y) as avg, max(y) as max from position
    template_type: system
    template: table

metrics sets:
  my metrics:
    metrics:
      - y over time
      - min/avg/max
```

The config defines one topic (`position` with `x` and `y` floats), two metrics (a per-test line chart and a batch-level table), and a metrics set that groups them.

______________________________________________________________________

## Step 6: Run it

Shell

```
python main.py
# or
uv run main.py
```

The first time you run it, you'll be prompted to authenticate in a browser:

Output

```
Authenticating by Device Code

Please navigate to: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX

Created batch rejoicing-aquamarine-starfish with id dc71ebee-...
test 1 done
test 2 done
Batch done. After a few minutes, you can view your metrics here: https://app.resim.ai/projects/.../batches/...
```

Visit the URL, authenticate with your ReSim account, and the script will continue automatically. Subsequent runs reuse the cached token. The script exits as soon as the data is uploaded; metrics processing runs in ReSim and takes a couple of minutes.

Running in CI

For non-interactive environments (CI pipelines), replace `DeviceCodeClient` with `UsernamePasswordClient` and pass ReSim API credentials as secrets. Contact [info@resim.ai](mailto:info@resim.ai) if you need API credentials.

Python

```
from resim.sdk.auth.username_password_client import UsernamePasswordClient
client = UsernamePasswordClient(username="...", password="...")
```

______________________________________________________________________

## Step 7: View results in the dashboard

Open the URL printed by the script, or navigate to your project in [app.resim.ai](https://app.resim.ai) and find the batch.

For each test you'll see:

- **y over time** — a line chart of `y` values across the 10 data points

At the batch level:

- **min/avg/max** — a table comparing the minimum, average, and maximum `y` values across both tests

`test 2` has `y = i * 2` (steeper slope) so its values will be higher than `test 1` (`y = i`), which you'll see clearly in the batch table.

______________________________________________________________________

## What's next

You've now pushed data into ReSim and seen it as metrics — without any Docker or cloud infrastructure setup.

The natural next steps are:

- **[Adapt this to your own data](../../guides/resim-sdk/)** — swap out the `position` topic for your own data schema and write SQL metrics that match your system
- **[Run your first test batch](../first-batch/)** — when you're ready to run your system inside ReSim's infrastructure for full execution management

# Run your first test batch

In this tutorial you'll set up a complete ReSim project from scratch and run your first batch of tests. By the end you'll have three tests running in parallel against a drone flight simulator and be looking at real metrics in the ReSim dashboard.

The demo uses pre-built public Docker images, so you don't need AWS ECR, Docker, or any build tooling to follow along.

About metrics in this tutorial

This tutorial uses a pre-built metrics build image from the demo repo — a Docker container that runs after each test and computes metrics. In your own projects you likely won't need a metrics build at all: ReSim's [Metrics](../../guides/metrics/) framework lets you emit data directly from your experience build and define visualizations in a config file, with no separate image required.

**Time:** about 20 minutes

**Before you start:**

- You have a ReSim account. Sign up at [app.resim.ai](https://app.resim.ai) if you don't have one.
- You have the ReSim CLI installed and authenticated. See [Install the CLI](../../setup/cli/) if not.

______________________________________________________________________

## What you're building

The [getting-started-demo](https://github.com/resim-ai/getting-started-demo) repo contains a simple drone flight simulator. The "system under test" reads a flight log (a JSON file with position, speed, and state data sampled once per second) and produces a processed output. The metrics build then reads that output and computes things like maximum speed, altitude over time, and whether any warning conditions were triggered.

Three flight logs are bundled into the demo image as experiences:

| Experience                | What it represents                     |
| ------------------------- | -------------------------------------- |
| Maiden Flight Voyage      | A normal, nominal flight               |
| Drone Flight Fast         | The same route flown at higher speed   |
| Drone Flight with Warning | A flight that trips warning thresholds |

You'll register these experiences with ReSim, point ReSim at the pre-built images, create a test suite, and run it. ReSim will launch all three tests in parallel and aggregate the results.

______________________________________________________________________

## Step 1: Create a project

A project is the top-level container for your work. Create one for this tutorial:

Shell

```
resim projects create \
  --name "drone-demo" \
  --description "Getting started tutorial"
```

Then select it so you don't have to pass `--project` on every subsequent command:

Shell

```
resim projects select "drone-demo"
```

______________________________________________________________________

## Step 2: Create a system

A system defines the software stack you're testing and the compute resources it needs. The demo is lightweight, so defaults are fine:

Shell

```
resim systems create \
  --name "drone-flight" \
  --description "Demo drone flight system"
```

______________________________________________________________________

## Step 3: Register the experiences

Experiences in this demo are bundled inside the Docker image at `/app/experiences/`. When ReSim runs a test, it writes the experience location into `/tmp/resim/test_config.json`, and the sim reads from there. Since the files live inside the container (not in S3), the `--locations` is just the path inside the image.

Register all three:

Shell

```
resim experiences create \
  --name "Maiden Flight Voyage" \
  --description "A nominal drone flight" \
  --locations "experiences/maiden_drone_flight/" \
  --systems "drone-flight"

resim experiences create \
  --name "Drone Flight Fast" \
  --description "A faster version of the nominal flight" \
  --locations "experiences/fast_drone_flight/" \
  --systems "drone-flight"

resim experiences create \
  --name "Drone Flight with Warning" \
  --description "A flight that triggers warning conditions" \
  --locations "experiences/warning_drone_flight/" \
  --systems "drone-flight"
```

Each command returns the UUID of the created experience and you can verify them in [app.resim.ai](https://app.resim.ai) under **Experiences**.

______________________________________________________________________

## Step 4: Register the build

A build is the Docker image that contains your system under test. The demo image is hosted on a public AWS ECR registry — no auth needed. Register it with ReSim:

Shell

```
resim builds create \
  --name "Demo Build v25" \
  --description "Getting started demo build v25" \
  --version "v25" \
  --image "public.ecr.aws/resim/open-builds/getting-started-demo:experience-build-v25" \
  --branch "demo" \
  --system "drone-flight" \
  --auto-create-branch
```

Note the build UUID in the output. You'll need it in Step 6. You can also look it up later:

Shell

```
resim builds list --system "drone-flight"
```

______________________________________________________________________

## Step 5: Register the metrics build

The metrics build is a second Docker image that runs after each test. It reads the sim's output and produces structured metrics that ReSim can store, display, and aggregate across batches.

Register the demo metrics image:

Shell

```
resim metrics-builds create \
  --name "Demo Metrics v25" \
  --version "v25" \
  --image "public.ecr.aws/resim/open-builds/getting-started-demo:metrics-build-v25" \
  --systems "drone-flight"
```

Note the metrics-build UUID from the output, or retrieve it:

Shell

```
resim metrics-builds list
```

______________________________________________________________________

## Step 6: Create a test suite

A test suite pairs a system with a fixed set of experiences and a metrics build. Every time you run the suite, ReSim launches one test per experience and then runs the metrics build against each result.

Shell

```
resim suites create \
  --name "Demo Regression Suite" \
  --description "All three demo experiences" \
  --system "drone-flight" \
  --metrics-build "<UUID from Step 5>" \
  --experiences "Maiden Flight Voyage,Drone Flight Fast,Drone Flight with Warning"
```

______________________________________________________________________

## Step 7: Run the suite

Run the suite against the experience build you registered in Step 4 (not the metrics build from Step 5):

Shell

```
resim suites run \
  --test-suite "Demo Regression Suite" \
  --build-id "<experience build UUID from Step 4>"
```

ReSim returns a batch name (something like `rejoicing-aquamarine-starfish`) and a batch ID. Keep these handy.

______________________________________________________________________

## Step 8: Watch it run

Check the status of the batch:

Shell

```
resim batches get --batch-name <your-batch-name>
```

The batch goes through these states: `Submitted` → `Running` → `Succeeded`. With three lightweight tests, it typically completes in a few minutes.

To watch individual tests:

Shell

```
resim batches tests --batch-name <your-batch-name>
```

If anything fails, the per-test container logs are useful for debugging:

Shell

```
resim logs list \
  --batch-id <batch-id> \
  --test-id <test-id>
```

The output includes pre-signed URLs so you can download `experience-container.log` (your sim's stdout) and `metrics-container.log` (the metrics stage output) directly.

______________________________________________________________________

## Step 9: View results in the dashboard

Once the batch completes, open [app.resim.ai](https://app.resim.ai), navigate to your **drone-demo** project, and click into the batch.

The metrics build computes the following for each test:

- **Maximum Speed** — scalar pass/fail metric
- **Speed Over Time** — line chart
- **Altitude Over Time** — line chart with a warning threshold drawn in
- **Flight States Over Time** — state transitions plotted over time
- **X Position Over Time** — position trace
- **Speed Distribution** — histogram
- **3D Flight Path** — interactive Plotly chart
- **Flight Summary** — text summary of the run

At the batch level you'll also see:

- **Highest Recorded Speed** across all three flights
- **Average Max Speed**
- **Overall Success Rate**
- **Altitude Comparison** — all three flights overlaid

The "Drone Flight with Warning" experience is designed to trip warning thresholds, so you should see it come up differently in the batch-level pass/fail summary compared to the other two.

______________________________________________________________________

## What's next

You've now completed a full ReSim cycle: register experiences and images, create a test suite, run a batch, and inspect metrics.

The next step is to swap out the demo images for your own system:

- [Build your own system image](../../setup/build-images/) — packaging your code into a Docker image with the right inputs/outputs contract
- [Add your own experiences](../../setup/adding-experiences/) — pointing ReSim at your own data in S3 or bundled in your image
- [Write a metrics build](../../setup/metrics-builds/) — if you need to do post processing on your emitted data from the experience build you can register a metrics build
- [Set up CI](../../setup/ci/) — automatically triggering batches on pull requests
# How-to Guides

# Getting Started with Your Own System

If you haven't used ReSim before, start with the [tutorial](../tutorials/first-batch/) — it walks you through a complete project from scratch using pre-built demo images, with no Docker or AWS setup required.

This section covers the same steps with your own system: your Docker images, your experiences, and your metrics.

- **Connect your experiences and builds**

  ______________________________________________________________________

  Grant ReSim limited access to your data in S3 and your images in ECR.

  [Guide](resim-data-access/)

- **Install the CLI**

  ______________________________________________________________________

  Install the ReSim CLI to create resources and trigger batches from your terminal.

  [Guide](cli/)

- **Create a project**

  ______________________________________________________________________

  Create a [project](../core-concepts/#project) to house your work and collaborate with your team.

  [Guide](projects/)

- **Create a system**

  ______________________________________________________________________

  Define a [system](../core-concepts/#system) to run tests against.

  [Guide](systems/)

- **Add experiences**

  ______________________________________________________________________

  Register your own [experiences](../core-concepts/#experience) as inputs for your tests.

  [Guide](adding-experiences/)

- **Build your system image**

  ______________________________________________________________________

  Package your system into a Docker image with the right inputs/outputs contract.

  [Guide](build-images/)

- **Create test suites**

  ______________________________________________________________________

  Create [test suites](../core-concepts/#test-suite) that pair a system with a fixed set of experiences.

  [Guide](test-suites/)

- **(Optional) Set up CI**

  ______________________________________________________________________

  Automatically trigger batches on pull requests using GitHub Actions or GitLab CI.

  [Guide](ci/)

- **Create a report**

  ______________________________________________________________________

  See how your system performs over time across test suite revisions.

  [Guide](reports/)

- **Enable the Overview page**

  ______________________________________________________________________

  Track how your test suites and metrics are trending over time.

  [Guide](overview/)

For ReSim to run your tests, it needs to be able to access four types of data: [experiences](../../core-concepts/#experience), [assets](../../core-concepts/#asset), [system build images](../../core-concepts/#build), and [metrics build images](../../core-concepts/#metrics-build).

- [**Experience Data Sources**](../experience-data-sources/)

  ______________________________________________________________________

  Configure access to experience data stored in AWS S3, Google Cloud Storage, the Foxglove Data Platform, or locally in your build container.

- [**Asset Data Sources**](../assets/#prerequisites)

  ______________________________________________________________________

  Assets use the same cloud storage sources as experiences. If you have already configured experience data access, no additional setup is needed.

- [**Container Registry Access**](../container-registry-access/)

  ______________________________________________________________________

  Configure access to container images stored in AWS ECR, Docker Hub, GitHub Container Registry, or Google Artifact Registry.

## IAM roles

By default, ReSim uses a generic role when accessing S3 and ECR data.

This role is only used programmatically by ReSim's internal components.

We also support customer-specific IAM roles. If you would like to use a customer-specific role, speak to your ReSim contact.

The generic role ARN is:

Text

```
arn:aws:iam::083585393365:role/resim-customer-prod
```

This role is referenced in the setup instructions for [AWS S3](../experience-data-sources/#aws-s3) and [AWS ECR](../container-registry-access/#aws-ecr).

ReSim needs to pull your [build images](../../core-concepts/#build) and [metrics build images](../../core-concepts/#metrics-build) from a container registry. This page describes how to configure access for each supported registry.

## AWS ECR

To use images hosted in AWS Elastic Container Registry (ECR) with ReSim, you can set up access to an existing ECR repository, or create a new ECR repository.

If you're creating a new repository, we recommend creating it in `us-east-1` as that's where our app runs. You can follow [the AWS documentation for creating ECR repositories](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html).

The repository needs a policy applied so that ReSim components can pull images from it. Check whether you are using a generic role or a customer-specific role as described in [IAM Roles](../resim-data-access/#iam-roles).

### New repository, or existing repository without policy

JSON

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReSimToPull",
      "Effect": "Allow",
      "Principal": {
        "AWS": ["arn:aws:iam::083585393365:role/resim-customer-prod"]
      },
      "Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer"]
    }
  ]
}
```

### Existing repository with policy

If you have an existing ECR repository with images you would like to test, and that repository already has a policy set, add the below statement to the repository's policy. Check whether you are using a generic role or a customer-specific role as described in [IAM Roles](../resim-data-access/#iam-roles).

JSON

```
"Statement": [{
    "Sid": "AllowReSimToPull",
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::083585393365:role/resim-customer-prod"
        ]
    },
    "Action": [
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
    ]
}]
```

## Docker Hub

You can use an existing repository or [create a new one](https://docs.docker.com/docker-hub/repos/create/). Once it's ready, you will need to [set up a new service account](https://docs.docker.com/docker-hub/service-accounts/#creating-a-new-service-account) and share the Docker ID and personal access token (PAT) securely with us.

Docker Image Pull Rate

We use a mechanism to pull and mirror your container images securely inside our platform. This means image retrieval is quicker, and also means that we only make requests to Docker Hub when you publish images we haven't yet mirrored. Thus the limit of 5000 pulls per day Docker imposes should be more than enough.

## GitHub Container Registry

To use ReSim with GitHub Container Registry (`ghcr.io`), [create a Personal Access Token (classic)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) with access to the relevant repository and the `read:packages` scope.

Share the token and your GitHub username with your contact at ReSim and we will configure secure storage and automated use of your credentials.

To restrict the access this token grants, create a new GitHub account and add it as an [outside collaborator](https://docs.github.com/en/organizations/managing-user-access-to-your-organizations-repositories/managing-outside-collaborators/adding-outside-collaborators-to-repositories-in-your-organization) with access only to the relevant repository/repositories.

## Google Artifact Registry

You can use an existing repository or [create a new one](https://cloud.google.com/artifact-registry/docs/docker). Once it's ready, you will need to [set up a new service account](https://cloud.google.com/iam/docs/service-accounts-create) and [generate a key](https://cloud.google.com/iam/docs/keys-create-delete) for it. ReSim needs the service account to have `Artifact Reader` access to the repository in question.

To grant ReSim access, please share a base64 encoded version of the `key.json`, [which can be used to login to the artifact registry](https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key).

## AWS GovCloud or private cloud solutions

Please get in touch with ReSim if you would like to discuss other, more bespoke, access patterns or if you are a user of AWS GovCloud, which we also support.

ReSim supports multiple sources for [experience](../../core-concepts/#experience) data. This page describes how to configure access for each supported source.

## AWS S3

You can use an existing bucket where you already have input data, or create a new one for ReSim.

If you create a new bucket, we recommend creating it in the `us-east-1` region as that's where our app runs. You can follow the [AWS documentation for creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). We don't require any specific settings except that the data isn't stored in [Glacier archive](https://aws.amazon.com/s3/storage-classes/glacier/).

Whether you have an existing bucket, or have created one specifically for use with ReSim, you need to give a ReSim IAM role access to read from the bucket. As described in [IAM Roles](../resim-data-access/#iam-roles), the IAM role to which you need to grant access will either be a generic role or a customer-specific role.

The IAM role is used by components of the ReSim platform if they need to fetch Experience data. The generic role is:

Text

```
arn:aws:iam::083585393365:role/resim-customer-prod
```

### New bucket, or bucket without existing policy

The following policy needs to be applied to your bucket. Make sure you replace the `bucket-name` placeholder value with the name of your bucket. Also check whether you are using a generic role or a customer-specific role as described above.

JSON

```
{
  "Version": "2012-10-17",
  "Id": "ReSimAccess",
  "Statement": [
    {
      "Sid": "ReSimAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::083585393365:role/resim-customer-prod"
      },
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::bucket-name", "arn:aws:s3:::bucket-name/*"]
    }
  ]
}
```

### Existing bucket with policy

If your bucket already has a bucket policy set, update the policy to add the following statement. Make sure you replace the `bucket-name` placeholder value with the name of your bucket. Also check whether you are using a generic role or a customer-specific role as described above.

JSON

```
"Statement": [
  {
      "Sid": "ReSimAccess",
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::083585393365:role/resim-customer-prod"
      },
      "Action": [
          "s3:GetObject",
          "s3:ListBucket"
      ],
      "Resource": [
          "arn:aws:s3:::bucket-name",
          "arn:aws:s3:::bucket-name/*"
      ]
  }
]
```

### Using S3 experiences

When creating experiences, use the `s3://` prefix for your location:

Bash

```
resim experiences create \
    --project "my-project" \
    --name "My S3 Experience" \
    --locations "s3://my-bucket/path/to/experience/"
```

Important

Any location that points to an S3 bucket must be prefixed with `s3://` rather than the `https://...` form.

## Google Cloud Storage

You can use an existing GCS bucket or create a new one for ReSim. If creating a new bucket, we recommend using the `us-east1` region for optimal performance.

### Setting up access

To grant ReSim access to your GCS bucket, you need to create a service account and share its credentials with us.

1. **Create a service account** following the [Google Cloud documentation](https://cloud.google.com/iam/docs/service-accounts-create).
1. **Grant the service account read access** to your bucket. The service account needs the `Storage Object Viewer` role on the bucket containing your experiences.
1. **Generate a key** for the service account following the [Google Cloud documentation](https://cloud.google.com/iam/docs/keys-create-delete).
1. **Share the key with ReSim**. Provide the contents of the `key.json` file to your ReSim contact. We will configure secure storage and automated use of your credentials.

### Using GCS experiences

When creating experiences, use the `gs://` prefix for your location:

Bash

```
resim experiences create \
    --project "my-project" \
    --name "My GCS Experience" \
    --locations "gs://my-gcs-bucket/path/to/experience/"
```

## Foxglove Data Platform

ReSim integrates with the [Foxglove Data Platform](https://foxglove.dev/data-platform), allowing you to use recordings stored in Foxglove as experience data for your tests.

### Setting up access

Foxglove access is configured per ReSim project using an API key.

1. **Generate an API key** in your Foxglove organization settings. Navigate to your [Foxglove settings](https://app.foxglove.dev/~/settings) and create a new API key with read access to the recordings you want to use.
1. **Share the API key with ReSim**. Provide the API key to your ReSim contact along with the ReSim project you want to associate it with. We will configure secure storage and automated use of your credentials.

One API key per project

Each ReSim project can have one Foxglove API key associated with it. If you need to access recordings from multiple Foxglove organizations, please contact us to discuss your requirements.

### Using Foxglove experiences

When creating experiences, use the `foxglove://` prefix with the recording ID:

Bash

```
resim experiences create \
    --project "my-project" \
    --name "My Foxglove Experience" \
    --locations "foxglove://rec_abc123def456"
```

You can find the recording ID in the Foxglove web app when viewing a recording.

## Local experiences

Experience data can also be stored locally within your [build container](../../core-concepts/#build). This is useful when experience data is tightly coupled to changes in your system and lives in the same repository.

### Providing access to local experiences in your container

In your [experience build](../build-images/) Dockerfile, copy the files from your repository to your container:

Dockerfile

```
COPY <path>/<to>/<experience>/<in>/<repo> /experiences/
```

### Creating local experiences

When creating a local experience, provide the path where the data will be located inside the container:

Bash

```
resim experiences create \
    --project "my-project" \
    --name "My Local Experience" \
    --locations "/experiences/my-scenario/"
```

### Programmatically accessing local experiences

For local experiences, ReSim does not perform any automatic file copying. Instead, the `/tmp/resim/test_config.json` file will contain an `experienceLocation` field that you can use to locate the experience data:

Python

```
import json

# Read the test_config.json file
try:
    with open("/tmp/resim/test_config.json", "r") as f:
        test_config = json.load(f)
except FileNotFoundError:
    raise FileNotFoundError("test_config.json not found in /tmp/resim")

# Get the experience location
experience_location = test_config["experienceLocation"]
```

Cloud vs Local experiences

For cloud-based experiences (S3, GCS, Foxglove), ReSim automatically copies the experience files into `/tmp/resim/inputs/` before your build runs. For local experiences, files remain at their original location and you must access them directly using the path from `test_config.json`.

# Install the CLI

ReSim has released an [open source repository](https://github.com/resim-ai/api-client) for the ReSim Command Line Interface, which can be used to easily access our API endpoints. Follow the [installation](https://github.com/resim-ai/api-client#installation) and [authentication](https://github.com/resim-ai/api-client#installation) instructions in that repo before continuing to the subsequent articles.

The first thing for us to do is to create a [project](../../core-concepts/#project) to work in. As discussed in the Core Concepts, we advise you use a project to reflect an entire stack, but otherwise, the project is arbitrary.

## Creating a project

You can create a project using the ReSim CLI.

Bash

```
resim projects create --name "autonomy-stack" --description "Our autonomy stack"
```

## Using the CLI with projects

All further commands will expect you to declare a project, for example:

Bash

```
resim experiences create --project "autonomy-stack" --name "my experience" --description "My experience description" --locations "s3://..."
```

In order to make this a better user experience, you can set a default project:

Bash

```
resim projects select "autonomy-stack"
```

Once selected, the previous command can be written:

Bash

```
resim experiences create --name "my experience" --description "My experience description" --locations "s3://..."
```

To find out what project you have selected and to select another, you can use:

Bash

```
resim projects list
```

## Navigating between projects

In the [ReSim app](https://app.resim.ai), the project name is displayed in the top right hand corner. You can select your project via a dropdown.

Warning

This dropdown is currently not persisted through user sessions: the app will, by default, auto-select the most recently created project. If you have feedback on this feature or requests for more functionality around projects, feel free to reach out to info@resim.ai

Next, we will create a [system](../../core-concepts/#system) in order to define a particular software stack to test.

## Creating a system

You can create a system using the ReSim CLI.

Bash

```
resim systems create --project "autonomy-stack" --name "perception" \
    --description "The perception subsystem of our autonomy stack" \
    --architecture "amd64" --build-vcpus 4 --build-memory-mib 16384 --build-gpus 0
```

The above example details some common resource requirements. Sensible defaults are provided for those unspecified. For a full list of available flags and their defaults, run `resim systems create --help`.

## Architectures

ReSim supports both the `amd64` and `arm64` architecture. This can be specified at the system level via the `--architecture` flag.

## Listing systems via CLI

To list all the systems in your project:

Bash

```
resim systems list --project "autonomy-stack"
```

To test our [system](../../core-concepts/#system) we need some [experiences](../../core-concepts/#experience).

## Prerequisites

Before creating experiences, you need to configure ReSim to access your experience data. See the [Experience Data Sources](../experience-data-sources/) guide for instructions on setting up access to:

- [AWS S3](../experience-data-sources/#aws-s3)
- [Google Cloud Storage](../experience-data-sources/#google-cloud-storage)
- [Foxglove Data Platform](../experience-data-sources/#foxglove-data-platform)
- [Local paths](../experience-data-sources/#local-experiences) in your build container

## Experience locations

Each experience can have multiple locations. In many cases, a single location is sufficient, but it can be helpful to store MCAPs recorded from different sources in different locations. [Assets](../assets/) that are common across all (or many) experiences are also supported in ReSim via a separate path.

ReSim supports the following location prefixes:

| Source               | Prefix        | Example                               |
| -------------------- | ------------- | ------------------------------------- |
| AWS S3               | `s3://`       | `s3://my-bucket/experiences/exp_001/` |
| Google Cloud Storage | `gs://`       | `gs://my-bucket/experiences/exp_001/` |
| Foxglove             | `foxglove://` | `foxglove://rec_abc123def456`         |
| Local                | None          | `/experiences/my-scenario/`           |

For cloud storage locations (S3, GCS), each location should have a unique prefix containing the input files. Anything contained in that prefix will be treated as the same experience and downloaded into your tasks. We recommend using a prefix structure that represents your ReSim projects:

Text

```
📂 .
├── 📂 project_a
│  └── 📂 experiences
│     └── 📂 experience_56ccce99
│        └── ⚙️ test_file
│
└── 📂 project_b
   └── 📂 experiences
      └── 📂 experience_679bef80
         └── ⚙️ test_file
```

### How experience files are accessed

For cloud-based experiences (S3, GCS, Foxglove), files are copied into `/tmp/resim/inputs/` when the experience build runs. If you use *multiple locations*, all files are copied directly into the `inputs` directory. Take care to ensure there are no file or directory name clashes at the root of the locations.

Avoiding file clashes with multiple locations

If your experience has two locations with the same directory structure:

Text

```
s3://your-bucket/experience1      s3://your-bucket/experience2
└── mcaps                         └── mcaps
    └── log.mcap                      └── log.mcap
```

The `mcaps/log.mcap` file will clash - one will overwrite the other in `/tmp/resim/inputs/`.

**Solution:** Change the path or filename to ensure uniqueness. For example, use distinct directory names:

Text

```
s3://your-bucket/experience1      s3://your-bucket/config
└── mcaps                         └── params.yaml
    └── log.mcap
```

This results in:

Text

```
/tmp/resim/inputs/
├── mcaps/
│   └── log.mcap
└── params.yaml
```

[Experience caching](../../guides/experience-caching/) is enabled by default, so we only copy files that have been added or changed since the last run.

For local experiences, files remain at their original location and you access them using the path from `/tmp/resim/test_config.json`. See [Local Experiences](../experience-data-sources/#local-experiences) for details.

## Experience timeouts

Each experience has a timeout. This is the maximum amount of time that a build can run for a given experience. If the build does not finish running before the timeout is reached, it will be sent a SIGTERM and then a SIGKILL after a grace period.

The default timeout is 1 hour, but you can specify a different timeout when creating an experience.

## Creating an experience

You can create new experiences like so:

Bash

```
resim experiences create \
    --project "my-project" \
    --name "Experience_56ccce99" \
    --description "In this experience, we test that our robot can navigate to the goal without colliding with any other agents." \
    --locations "s3://my-experiences-bucket/project_hal9000/experiences/experience_56ccce99/" \
    --systems "perception" "localization" \
    --tags "navigation" "collision-avoidance"
```

This command should return a message containing the UUID of the created experience, and it should now be browsable through the [ReSim app](https://app.resim.ai).

The `--systems` flag enables you to specify any [systems](../../core-concepts/#system) that this experience is compatible with. You can specify multiple systems by providing multiple system names or IDs.

The `--tags` flag allows you to apply tags to the experience for better organization and filtering. You can specify multiple tags by providing multiple tag names or IDs.

Optional flags are:

Bash

```
    --timeout "1h10m" # the maximum expected duration of the container \
    --environment-variable "SYSTEM_CONFIGURATION=Foo" # many allowed \
    --environment-variable "AN=other" \
    --profile "left-arm" # a docker compose profile to select the subset of compose services to run \
```

### Examples with different sources

**AWS S3:**

Bash

```
resim experiences create \
    --project "my-project" \
    --name "S3 Experience" \
    --description "An experience stored in S3" \
    --locations "s3://my-bucket/experiences/exp_001/"
```

**Google Cloud Storage:**

Bash

```
resim experiences create \
    --project "my-project" \
    --name "GCS Experience" \
    --description "An experience stored in GCS" \
    --locations "gs://my-gcs-bucket/experiences/exp_001/"
```

**Foxglove:**

Bash

```
resim experiences create \
    --project "my-project" \
    --name "Foxglove Experience" \
    --description "An experience stored in Foxglove" \
    --locations "foxglove://rec_abc123def456"
```

**Local:**

Bash

```
resim experiences create \
    --project "my-project" \
    --name "Local Experience" \
    --description "An experience stored locally in the build container" \
    --locations "/experiences/my-scenario/"
```

## Updating an experience

You can update various aspects of an experience, including its systems and tags:

Bash

```
resim experiences update \
    --project "my-project" \
    --experience "Experience_56ccce99" \
    --name "Updated Experience Name" \
    --description "Updated description" \
    --locations "s3://my-experiences-bucket/project_hal9000/experiences/updated_experience/" \
    --timeout "2h" \
    --systems "perception" "planning" \
    --tags "updated-tag" "new-functionality"
```

When updating systems or tags, the provided lists will replace the existing systems and tags associated with the experience.

You can also add or remove individual systems from an experience:

Bash

```
resim experiences add-system \
    --project "my-project" \
    --system "<system name>" \
    --experience "Experience_56ccce99"

resim experiences remove-system \
    --project "my-project" \
    --system "<system name>" \
    --experience "Experience_56ccce99"
```

Similarly you can add and remove individual tags:

Bash

```
resim experiences tag \
    --project "my-project" \
    --tag "<tag string>" \
    --experience "Experience_56ccce99"

resim experiences untag \
    --project "my-project" \
    --tag "<tag string>" \
    --experience "Experience_56ccce99"
```

To share common data across [builds](../../core-concepts/#build) and [experiences](../../core-concepts/#experience), ReSim supports [assets](../../core-concepts/#asset).

## Why use assets?

Consider a simulation stack that requires a large HD map, 3D environment meshes, and ML model weights. Every experience in your test suite needs this data, but none of it changes between experiences, it only changes when your simulation environment or models are updated.

Without assets, you have two options, neither ideal:

- **Bake the data into your Docker image** - This can bloat the image, slow down CI builds, and force a full image rebuild every time a map or model is updated even if the code hasn't changed
- **Duplicate the data across experience locations** - This can waste storage, make updates error-prone, and increase data transfer costs

*Assets* solve this by letting you define shared data once, revision it independently of your code, and mount it into any build at runtime. Your Docker image stays lean, your experiences stay focused on per-test inputs, and your shared data is versioned and cached.

```
flowchart LR
    Asset["Asset\n(e.g. HD Map)"] -->|"linked to"| Build[Build]
    Experience["Experience\n(e.g. Scenario)"] -->|"run with"| Test[Test]
    Build -->|"run in"| Test
    Test -->|"part of"| Batch[Test Batch]

    subgraph container ["Container at Runtime"]
        AssetMount["/tmp/resim/assets/world/"]
        InputMount["/tmp/resim/inputs/"]
    end

    Asset -->|"mounted at"| AssetMount
    Experience -->|"mounted at"| InputMount
```

## Prerequisites

Assets are stored in cloud blob storage, using the same data sources as experiences. If you have already configured access for your experience data, no additional setup is needed.

If not, see the [Experience Data Sources](../experience-data-sources/) guide for instructions on setting up access to:

- [AWS S3](../experience-data-sources/#aws-s3)
- [Google Cloud Storage](../experience-data-sources/#google-cloud-storage)

## Asset locations

Each asset can have multiple locations. ReSim supports the same location prefixes as experiences:

| Source               | Prefix  | Example                            |
| -------------------- | ------- | ---------------------------------- |
| AWS S3               | `s3://` | `s3://my-bucket/assets/sim-world/` |
| Google Cloud Storage | `gs://` | `gs://my-bucket/assets/sim-world/` |
| Local                | None    | `/assets/sim-world/`               |

For cloud storage locations, each location should have a unique prefix containing the asset files. Anything within that prefix will be synced. We recommend a prefix structure that mirrors your ReSim projects:

Text

```
📂 .
└── 📂 my-project
   └── 📂 assets
      ├── 📂 sim-world
      │  ├── 🗺️ hd_map.bin
      │  └── 📂 meshes
      │     └── 🏙️ urban_env.glb
      └── 📂 ml-models
         └── 🤖 perception_v3.onnx
```

## How asset files are accessed

For cloud-based assets, files are synced from cloud storage and mounted **read-only** at `/tmp/resim/assets/<mountFolder>/` when a build runs. The `mountFolder` is specified when creating the asset and determines where inside the container the files appear.

For example, an asset with `mountFolder` set to `world` and location `s3://my-bucket/assets/sim-world/` results in:

Text

```
/tmp/resim/assets/world/
├── hd_map.bin
└── meshes/
    └── urban_env.glb
```

Your entrypoint script can then read from `/tmp/resim/assets/world/` just as it would from any local directory.

If an asset has multiple locations, all files are synced into the same mount directory. Take care to avoid file or directory name clashes at the root of the locations, just as with [experience locations](../adding-experiences/#experience-locations).

[Asset caching](../../guides/asset-caching/) is enabled by default, so only files that have been added or changed since the last run are downloaded.

Note

Because assets are shared across potentially concurrent jobs, the `/tmp/resim/assets/<mountFolder>/` directory is mounted **read-only**. If your workload needs to modify asset data, copy the files to a writable location first.

## Creating an asset

You can create a new asset like so:

Bash

```
resim assets create \
    --project "my-project" \
    --name "Simulation World" \
    --description "HD map and 3D meshes for urban simulation environment" \
    --locations "s3://my-assets-bucket/my-project/assets/sim-world/" \
    --mount-folder "world" \
    --version "1.0.0"
```

This command returns the UUID and revision number of the created asset. The asset starts at *revision* `0`. Revisions are automatically incremented.

| Flag             | Required | Description                                         |
| ---------------- | -------- | --------------------------------------------------- |
| `--name`         | Yes      | A human-readable name for the asset                 |
| `--description`  | Yes      | A description of what the asset contains            |
| `--locations`    | Yes      | One or more cloud storage locations                 |
| `--mount-folder` | Yes      | The subdirectory name under `/tmp/resim/assets/`    |
| `--version`      | Yes      | A version string (e.g. `1.0.0`, `v2`, a commit SHA) |
| `--cache-exempt` | No       | If set, the asset will not be cached between runs   |

### Examples with different sources

**AWS S3:**

Bash

```
resim assets create \
    --project "my-project" \
    --name "ML Models" \
    --description "Perception model weights" \
    --locations "s3://my-bucket/assets/ml-models/" \
    --mount-folder "models" \
    --version "v3.1"
```

**Google Cloud Storage:**

Bash

```
resim assets create \
    --project "my-project" \
    --name "Calibration Data" \
    --description "Sensor calibration parameters" \
    --locations "gs://my-gcs-bucket/assets/calibration/" \
    --mount-folder "calibration" \
    --version "2024-Q4"
```

## Revising an asset

When the underlying data changes (e.g. a new map version or updated model weights), we create a **new revision** of the asset rather than modifying it in place. This preserves an immutable history of every version of the asset data, enabling reproducibility.

Bash

```
resim assets revise \
    --project "my-project" \
    --asset "Simulation World" \
    --locations "s3://my-assets-bucket/my-project/assets/sim-world-v2/" \
    --version "2.0.0"
```

Each revision increments the revision number (0, 1, 2, ...). The `--version` flag is required when revising -- it should reflect the new state of the data (e.g. a semantic version, date, or commit SHA). You can also optionally update the `mountFolder`:

Bash

```
resim assets revise \
    --project "my-project" \
    --asset "Simulation World" \
    --locations "s3://my-assets-bucket/my-project/assets/sim-world-v2/" \
    --mount-folder "world-v2" \
    --version "2.0.0"
```

You can list all revisions of an asset:

Bash

```
resim assets list-revisions \
    --project "my-project" \
    --asset "Simulation World"
```

## Updating asset metadata

To update an asset's name or description without changing its data:

Bash

```
resim assets update \
    --project "my-project" \
    --asset "Simulation World" \
    --name "Urban Simulation World" \
    --description "HD map and 3D meshes for the urban simulation environment (downtown)"
```

## Associating assets with builds

Assets must be explicitly **linked to a build** before they are available at runtime. This association specifies which asset revision the build should use.

### At build creation time (recommended)

The simplest approach is to attach assets when you create the build. The `resim builds create` command accepts an `--assets` flag with a comma-separated list of asset references:

Bash

```
resim builds create \
    --image "123456789.dkr.ecr.us-east-1.amazonaws.com/my-sim:abc1234" \
    --version "abc1234" \
    --name "Autonomy Build" \
    --description "Path planning v2.3" \
    --branch "my-branch" \
    --project "my-project" \
    --system "full-stack" \
    --auto-create-branch \
    --assets "Simulation World:2,ML Models:5"
```

This creates the build and links the specified asset revisions in a single step. Each asset will be mounted at its respective `mountFolder` when the build runs.

Each entry in the comma-separated list can take one of four forms where the `uuid` is the revision ID:

| Format          | Example              | Behaviour                     |
| --------------- | -------------------- | ----------------------------- |
| `name`          | `Simulation World`   | Links the **latest** revision |
| `name:revision` | `Simulation World:2` | Links a specific revision     |
| `uuid`          | `a1b2c3d4-...`       | Links the **latest** revision |
| `uuid:revision` | `a1b2c3d4-...:2`     | Links a specific revision     |

When the revision is omitted, the latest revision at the time of build creation is used as the static revision for that build. We recommend pinning explicit revision numbers in CI/CD for reproducibility.

### Adding assets to an existing build

If you need to add assets to a build after it has been created, use `resim builds add-assets`:

Bash

```
resim builds add-assets \
    --project "my-project" \
    --build "<build-name-or-id>" \
    --assets "Simulation World:2"
```

### Listing assets for a build

Bash

```
resim builds list-assets \
    --project "my-project" \
    --build "<build-name-or-id>"
```

### Removing assets from a build

Bash

```
resim builds remove-assets \
    --project "my-project" \
    --build "<build-name-or-id>" \
    --asset "Simulation World"
```

### Querying which builds use an asset

Bash

```
resim assets list-builds \
    --project "my-project" \
    --asset "Simulation World"
```

## Archiving assets

Assets that are no longer needed can be archived. Archived assets are hidden from default list views but are not deleted.

Bash

```
resim assets archive \
    --project "my-project" \
    --asset "Simulation World"
```

To restore an archived asset:

Bash

```
resim assets restore \
    --project "my-project" \
    --asset "Simulation World"
```

To include archived assets in a list:

Bash

```
resim assets list \
    --project "my-project" \
    --archived
```

## Recommended CI/CD workflow

We recommend associating assets with builds as part of your CI/CD pipeline. This ensures that every build is pinned to specific asset revisions, making test results fully reproducible.

A typical workflow:

1. **Create and revise assets independently of your code** - When a map, model, or calibration dataset is updated, create a new asset revision. This can be done manually or in a separate pipeline triggered by changes to your data repository
1. **Pin asset revisions in CI** - When your CI pipeline creates a new build, associate it with the desired asset revisions. Store the asset revision numbers as CI variables or derive them from a configuration file in your repository
1. **Update asset revisions deliberately** - When you want to test against a new map or model, bump the revision number in your CI configuration and let the pipeline pick it up

### GitHub Actions example

Extending the [GitHub Actions workflow](../ci/github/), you can pass asset revisions directly when creating the build using the `--assets` flag:

YAML

```
- name: Create build with assets and launch batch
  env:
    RESIM_CLIENT_ID: ${{ secrets.RESIM_CLIENT_ID }}
    RESIM_CLIENT_SECRET: ${{ secrets.RESIM_CLIENT_SECRET }}
  run: |
    resim builds create \
      --project "my-project" \
      --system "full-stack" \
      --image "${{ steps.docker_meta.outputs.tags }}" \
      --version "${{ github.sha }}" \
      --branch "${{ github.head_ref || github.ref_name }}" \
      --auto-create-branch \
      --assets "Simulation World:${{ vars.SIM_WORLD_REVISION }},ML Models:${{ vars.ML_MODELS_REVISION }}"
```

Store `SIM_WORLD_REVISION` and `ML_MODELS_REVISION` as [GitHub Actions variables](https://docs.github.com/en/actions/learn-github-actions/variables) so they can be updated without changing the workflow file.

Tip

Separating asset revisions from code changes means you can update your simulation environment without rebuilding your Docker image, and vice versa. This significantly speeds up iteration when only data has changed.

## End-to-end example

Here is a complete walkthrough using a simulation world asset shared across many experiences.

### 1. Create the asset

Bash

```
resim assets create \
    --project "my-project" \
    --name "Simulation World" \
    --description "Downtown urban environment: HD map, 3D meshes, traffic light configs" \
    --locations "s3://my-assets-bucket/my-project/assets/sim-world/" \
    --mount-folder "world" \
    --version "1.0.0"
```

### 2. Create a build with the asset attached

Bash

```
resim builds create \
    --image "123456789.dkr.ecr.us-east-1.amazonaws.com/my-sim:abc1234" \
    --version "abc1234" \
    --name "Autonomy Build" \
    --description "Path planning v2.3" \
    --branch "my-branch" \
    --project "my-project" \
    --system "full-stack" \
    --auto-create-branch \
    --assets "Simulation World:0"
```

### 3. Run a batch

Bash

```
resim batches create \
    --project "my-project" \
    --build "Autonomy Build" \
    --test-suite "Nightly Regression"
```

When each test in the batch runs, the simulation world data is automatically mounted at `/tmp/resim/assets/world/`.

### 4. Access asset data in your entrypoint

Python

```
import pathlib

map_path = pathlib.Path("/tmp/resim/assets/world/hd_map.bin")
mesh_dir = pathlib.Path("/tmp/resim/assets/world/meshes/")

map_data = map_path.read_bytes()
meshes = list(mesh_dir.glob("*.glb"))
```

Every experience in the test suite gets the same simulation world without any duplication.

In order to run our [experiences](../../core-concepts/#experience) against our [system](../../core-concepts/#system) we need a system [build](../../core-concepts/#build) available as a container image hosted somewhere we can access.

## Prerequisites

For your build image to be usable in ReSim, it needs to be hosted in an image registry that we are granted permission to access.

Uploading the image to a registry is documented below, but you should ensure you have followed [Resim Data Access](../resim-data-access/) to set up an AWS ECR repository to work with ReSim.

Furthermore, you must have permission to push to that registry.

If you're using an IAM user, the easiest way to get the needed permissions is to attach the AWS-managed IAM policy called `AmazonEC2ContainerRegistryPowerUser` to the user. You can do this through the [AWS browser UI](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) (see "Adding permissions by attaching policies directly to the user") or using the CLI:

Bash

```
aws iam attach-user-policy --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser" --user-name <name of user>
```

If you're using an IAM role instead, it's a [similar operation](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_permissions-policy).

If you'd prefer to be stricter with your permissions, you really only need the permissions from the policy [here](https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-push.html#image-push-iam) rather than all of those in `AmazonEC2ContainerRegistryPowerUser`.

If there is a [repository policy](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policies.html#repository-policy-vs-iam-policy) on the repository already, check it doesn't explicitly deny access by the IAM identity (user or role) you're using.

## Entrypoint

The first step in creating a build image is to identify a shell command that you would like to run in order to test your software. If your software test involves multiple different processes, this is fine, but you may want to write a shell script or another piece of software to correctly initialize and run them. For instance, if you are using ros2 with gazebo, you might write a launch file for this and then your test command might be a call to a script containing something like:

Bash

```
#!/bin/bash

# run_sim_test.sh

# Source our ros2 distribution
source /opt/ros/<distro>/setup.bash

# Navigate to, build, and source the ros2 package we're building
cd <workspace-path>
colcon build
source install/setup.bash

# Launch our test sim
ros2 launch my_package launch_test_sim.py
```

But really, any script which your docker container can execute will suffice. Once you have identified such a command, your goal is to create a [Dockerfile](https://docs.docker.com/engine/reference/builder/) which builds an image that has everything it needs to run your test command. Then, that command should be denoted as the `ENTRYPOINT` of that Dockerfile.

## Inputs and outputs

It is typically desirable to run the same *Build* with a variety of different *Experiences* (sets of input files). Therefore, the ReSim worker will mount a volume containing the inputs for each run at `/tmp/resim/inputs/`. Using the ros2 example above, the `launch_test_sim.py` launch file might reference ROS2 Parameter YAML files stored in `/tmp/resim/inputs/`. You would then put these YAML files in an Amazon S3 bucket and add them as experiences as described [here](../adding-experiences/). Again, there's nothing special about using ros2 in this case and any file stored correctly in the experiences bucket will be available there.

In addition to experience inputs, any [assets](../../core-concepts/#asset) associated with the build are mounted read-only at `/tmp/resim/assets/<mountFolder>/`. See the [assets setup guide](../assets/) for details on creating assets and linking them to builds.

The ReSim platform additionally places a file called `test_config.json` in the `/tmp/resim/` directory. This JSON contains the following fields:

JSON

```
{
  "experienceID": "the unique id of the experience",
  "experienceName": "the name of the experience",
  "experienceLocation": "the location of the experience",
  "branchName": "the name of the branch",
  "buildVersion": "the version of the build running",
  "buildID": "the unique id of the build running",
  "systemName": "the name of the system the build running belongs to",
  "systemID": "the unique id of the system"
}
```

This config file can be helpful for determining the experience location, if the experience is not stored in S3 but within the build image, by reading `/tmp/resim/test_config.json`

If custom parameters are used in the test command, they should be stored in the `/tmp/resim/parameters.json` file. Please see the [Parameter Sweeps](../../guides/run-sweeps/) guide for more information.

When the test command is run, it will produce some sort of valuable outputs that can be accessed through the ReSim app. Any such outputs (e.g. console logs or rosbags) should be stored in the `/tmp/resim/outputs/` directory. If this is done correctly, you will be able to download them using a link in the ReSim app.

If you place files in nested subdirectories beneath `/tmp/resim/outputs/`, ReSim will archive those nested directories into `.zip` files before uploading them. If you want log files to remain unarchived, place them in `/tmp/resim/outputs/logs/`. Files in that directory are uploaded as logs without being zipped and are browsable on the web app.

It can, however, be the case that you generate logs that you do not wish to upload, but wish to use for computing metrics. If you place any logs in `/tmp/resim/outputs/ignore` these will be made available in the metrics phase, but are not uploaded to the app.

If your image exits `0` in the context of a ReSim batch, we consider it a successful run and will automatically compute the next, metrics, stage. If your image exits non-zero, we will create an `Error` status on that test and will not run metrics.

It is possible to provide a custom error message that will be reported, alongside the exit code, on the ReSim web app by leaving a file called `error.txt` in the `/tmp/resim/outputs` folder.

## Developing an image

For an example of a very simple build image please refer to [this directory](https://github.com/resim-ai/open-core/tree/main/resim/examples/sim_runner) which contains directions and example scripts to create and test run a build image. In this case, we use ReSim's open source simulator to run a simple scripted simulation of two quadcopter drones.

The example uses a build script to build our test executable and then uses Docker's `COPY` command in its `Dockerfile` to move the resulting executable into the finished image. The general rule is that the Docker image must contain everything your test needs to run *except* for the the inputs (i.e. the experience).

In order to develop a working test image, a good workflow is generally to iterate on it locally. This involves the following steps:

### 1. Inputs and Outputs

Create local `inputs/` and `outputs/` folders with the `inputs/` folder containing a set of representative input files. **If you are doing this from within a Docker container**, creating volumes and copying content to them is probably better. Here's how you would do that:

Bash

```
# Create a helper container to copy content over
docker container create \
    --name tmp \
    --volume test_inputs:/inputs \
    busybox

# Copy test data to the volume
docker cp path/to/test/inputs tmp:/inputs

# Remove the helper container
docker rm tmp
```

### 2. Build

Assuming your Dockerfile is in a folder called `runner/`, build your Docker image using:

Bash

```
docker build runner/ -t test-runner-image:latest
```

Note that the name `test-runner-image` here is arbitrary. You can use whatever name you would like as long as you're consistent when developing because the image will be re-tagged with `docker tag` before it's pushed.

### 3. Run

Try out a run with your Docker image using:

Bash

```
docker run \
  --volume /absolute/path/to/inputs:/tmp/resim/inputs \
  --volume /absolute/path/to/outputs:/tmp/resim/outputs \
  test-runner-image:latest
```

The `--volume` flags here ensure that the inputs and outputs are available to the container at `/tmp/resim/inputs` and `/tmp/resim/outputs`. Note that `/absolute/path/to/inputs` and `/absolute/path/to/outputs` are paths on the *host* system. **If you are iterating within a Docker container**, you would use the Docker volumes approach mentioned in step 1. In this case, you would **instead** run:

Bash

```
docker run \
  --volume test_inputs:/tmp/resim/inputs \
  --volume test_outputs:/tmp/resim/outputs \
  test-runner-image:latest
```

And getting your outputs back would look like:

Bash

```
# Create a helper container to copy content back:
docker container create \
    --name tmp \
    --volume test_outputs:/outputs \
    busybox

# Copy outputs from the volume
docker cp tmp:/outputs .

# Remove the helper container
docker rm tmp
```

### 4. Iterate

If any step is unsuccessful, make modifications and return to step 2.

## Pushing

Once the build image is created and working, the next step is to push it to your AWS Elastic Container Registry (set up according to [ReSim Data Access](../resim-data-access/)) like so:

Bash

```
docker tag test-runner-image <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/test-runner-image:<commit-sha>
aws ecr get-login-password --region <AWS region> | docker login --username AWS --password-stdin <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com
docker push <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/<ecr-name>:<build ID>
```

Note the use of a commit SHA here. This is what we recommend as a unique ID associated with your software version if you're using `git` for version control. You can get a short form of the current commit hash by running:

Bash

```
git rev-parse --short HEAD
```

On a successful push, the command will report something like:

Text

```
The push refers to repository [<AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/<ecr-name>]
a9950558cded: Pushed
29563d2ab359: Pushed
aa5d6a7236b9: Pushed
...
bce45ce613d3: Pushed
latest: digest:
sha256:aa47e2d0c6dcc5d48d3c9b8c9c8a57433b2d241c27ef92aa53de885683f06487 size: 7480
```

If you instead get something like the following, it is an indicator either that the ECR repository URL (account number or repository name) you are trying to push to is incorrect, or that you don't have the required permissions to push to it. Check the [Prerequisites](#prerequisites) for additional info.

Text

```
The push refers to repository [<AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/<ecr-name>]
a9950558cded: Retrying in 7 seconds
29563d2ab359: Retrying in 7 seconds
aa5d6a7236b9: Retrying in 7 seconds
...
bce45ce613d3: Waiting
EOF
```

You might want to automate these build and push steps by adding them to a new or existing CI pipeline. See our articles on [Pushing Build Images from CI](../ci/) for how.

## External links

- [AWS ECR authentication documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html)

## Creating a branch

Each build is associated with a branch. The branch represents the particular change you're testing. A build can be thought of as one implementation of that change. A given branch may have many builds associated with it.

You can manually create a branch with the CLI:

Bash

```
resim branches create \
    --name "my-branch" \
    --type CHANGE_REQUEST \
    --project "my-project"
```

It's often easier to use the `--auto-create-branch` flag when creating a build which is the approach taken below.

The ReSim app supports three types of branch, for three common git workflows:

- `CHANGE_REQUEST`: a pull or merge request, a short-lived branch
- `MAIN`: a long-lived, protected branch that most change requests will land to
- `RELEASE`: a release of a piece of software

The `--auto-create-branch` flag will create a `CHANGE_REQUEST` branch, but if the branch is named `main` it will make it a `MAIN` branch.

## Creating a build

Now that the build image has been pushed to your elastic container registry, the final step is to formally create the build by informing the ReSim app as to its location. This is done using the [ReSim CLI](../cli/). We assume that we already have a [project](../projects/) called `my-project`, a [system](../systems/) called `my-system` and a [branch](../../core-concepts/#branch) called `my-branch`. These are required for build creation. After [authenticating](https://github.com/resim-ai/api-client#authentication) we can create a new build like so:

Bash

```
resim builds create \
    --image "<AWS ID>.dkr.ecr.<region>.amazonaws.com/<ecr-name>:<commit-sha>" \
    --version "<commit-sha>" \
    --name "Autonomy Build" \
    --description "Optimized path planning to avoid unexpected squirrels" \
    --branch "my-branch" \
    --project "my-project" \
    --system "my-system" \
    --auto-create-branch
```

The `--project`, `--branch`, and `--system` flags will accept either a name or an ID. When using `--auto-create-branch`, one must also pass a branch name using `--branch`. If you have selected a project via the CLI, you can omit the `--project` for convenience.

This command should return a message containing the UUID of the created build, and it should now be browsable through the [ReSim app](https://app.resim.ai).

Note that the image itself requires the full tagged URI, since we allow an arbitrary version to be stored that need not reflect the image tag.

## Multi-container builds

You can also submit a [Docker Compose](https://docs.docker.com/reference/compose-file/) file when creating a build, instead of a single build image (`--image`).

Bash

```
resim builds create \
  --build-spec=my-docker-compose.yml
  [...]
```

See [the guide](../../guides/multi-container-builds/) for more details on how to set up a Compose file to execute your code during the **Experience** phase.

When executing multiple containers like this, all containers have access to the `/tmp/resim` directory for inputs and outputs, although it is certainly common for only a subset of containers to actually read/write there as necessary.

All images specified in a Compose file must be in an accessible Container Registry as per above. The Compose file used is passed via the API at build creation time and does not need to be hosted anywhere.

You may not need this

If your emitted data from your experience build does not require post-processing you can skip this step and go straight to [Metrics](../../guides/metrics/). A metrics build is only needed if your workflow requires post-processing of raw experience build outputs before metrics can be computed. Metrics builds used to be a required step in the ReSim workflow, but with the [ReSim SDK](../../guides/resim-sdk/) they are only used for post-processing data.

This document introduces our open source metrics SDK and describes how the ReSim platform supports executing metrics.

## ReSim metrics framework

The ReSim metrics framework is an open-source set of libraries to support constructing metrics functions that generate metrics data that the ReSim platform can parse. The libraries can be found on GitHub in [resim-ai/open-core](https://github.com/resim-ai/open-core/tree/main/resim/metrics). The Python metrics API can be installed from PyPI with `pip install resim-open-core` and its documentation can be found [here](../../open-core/metrics/).

## Metrics in the ReSim platform

Similar to [builds](../../core-concepts/#build), customers can register a [**Metrics Build**](../../core-concepts/#metrics-build) with ReSim, which will be executed after each test. A metrics build is also a container image, with a similar contract to [builds](../build-images/#1-inputs-and-outputs).

For convenience, the same metrics build is expected to be able to act in three modes: `test mode`, `batch mode`, and `report mode`. Using the same metrics build for all three related tasks means we have less code repetition, fewer metrics builds, and a lower chance of compatibility errors.

In `test mode`, the ReSim platform will populate the `/tmp/resim/inputs` directory with:

- a subdirectory, `experience`, which contains any files as part of the experience
- a subdirectory, `logs`, which contains any files generated as logs from the execution of the simulation.

The ReSim platform expects to find a file called `metrics.binproto` in the `/tmp/resim/outputs` directory and will use this to register metrics with the system. It is highly recommended to follow the ReSim Metrics SDK for constructing this protobuf binary, since it provides an abstraction from the raw schema.

Any additional files placed in the outputs directory will also be uploaded to the ReSim platform as logs.

In `batch mode`, the ReSim platform will only populate the `/tmp/resim/inputs` directory with a configuration file that the Metrics SDK can use to fetch the metrics for each individual test. For more information, see the open source documentation. It is expected that a `metrics.binproto` exists in the outputs directory.

For more details about `report mode`, see [Test Suite Reports](../reports/#reports-mode-for-metrics-build).

## Creating metrics builds

It is possible to create metrics builds in CI/CD or with the CLI tool:

Bash

```
resim metrics-builds create \
    --project "my-project" \
    --name "My Metrics Build" \
    --version <commit sha or version> \
    --image <Docker image URI> \
    --systems "perception" "localization"
```

Where, again, if one has a project selected, that parameter can be omitted. Similarly to builds, ReSim recommend the set-up of CI/CD workflows to generate metrics builds when the metrics generation code is updated.

## Systems and metrics builds

Since metrics are closely tied to the test application which runs, the ReSim app enables metrics builds to be aligned to a given system, to enable compatibility checking at test creation time. One can register an existing metrics build against a given system:

Bash

```
resim metrics-builds add-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"
```

and remove them with:

Bash

```
resim metrics-builds remove-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"
```

## Debugging metrics

The ReSim app provides a page which can be used for debugging Metrics output by uploading a binproto. The page will render what the app would render given the proto. See: [Metrics Debugger](https://app.resim.ai/metrics-debugger)

If you've gone through all of the previous articles, you are now ready to begin testing your builds with some [experiences](../../core-concepts/#experience) and generating metrics! We do this by creating and running a [**Test Suite**](../../core-concepts/#test-suite).

## Creating a test suite

To create a test suite, you need to have a [system](../../core-concepts/#system) in mind, a [metrics build](../../core-concepts/#metrics-build) and a set of experiences you would like to use. You can create a test suite using the [ReSim CLI](../cli/) with this command, assuming you already have a [project selected](../projects/#using-the-cli-with-projects)

Bash

```
resim suites create \
    --name "Perception Nightly Regression"
    --description "A test suite for perception system nightly regression tests"
    --system "perception"
    --metrics-build <METRICS BUILD UUID> \
    --experiences "<experience name or UUID>, <experience name or UUID>, ..."
```

You can `get` or `list` the test suites for that project. For example, the following command displays the latest revision:

Bash

```
resim suites get \
    --test-suite "Perception Nightly Regression"
```

To get a specific revision append the `--revision 1` key. To get a list of all revisions, pass `--all-revisions`.

In order to `revise` the test suite, you simply need to pass the elements that you would like to change. For example, to change the experience list, you would use the following command:

Bash

```
resim suites revise \
    --test-suite "Perception Nightly Regression"
    --experiences "<new experience name or UUID>, <new experience name or UUID>, ..."
```

Warning

You need to specify the exact list of experiences. The CLI will be extended in the near future to support specifying a query to describe the list of experiences. In the short term, the [ReSim app](https://app.resim.ai) supports query-based test suite creation and building.

## Running a test suite

If you have a test suite, you can then run it to create a **Test Batch**:

Bash

```
resim suites run \
    --test-suite "Perception Nightly Regression"
    --build-id "<UUID of the build>"
```

If you would like to specify a particular revision, you can pass `--revision 0`.

You can list the test batches that have been created from this test suite via:

Bash

```
resim suites batches \
    --test-suite "Perception Nightly Regression"
```

This will list all the test batches from all revisions of the test suite. Simply pass the `--revision` flag to see test batches from a specific test suite revision.

## Tracking a test batch

Once you have created a test batch by running a test suite, the CLI lets you track the status of that batch. You can list all of the batch details -- as JSON -- either by passing the friendly name of the batch (e.g. rejoicing-aquamarine-starfish) or its id:

Bash

```
resim batches get \
    --batch-id <Batch ID>
```

or

Bash

```
resim batches get \
    --batch-name <Batch friendly name>
```

If you additionally pass the exit status key:

Bash

```
resim batches get \
    --batch-id <Batch ID> \
    --exit-status
```

The application will return only the exit status, which are:

- 0: `Succeeded`: the batch has completed. This does not indicate whether the metrics have passed or failed.
- 1: `Internal Error`: the CLI command errored (which is nothing to do with the test batch).
- 2: `Error`: the batch has finished with an error.
- 3: `Submitted`: the batch is in the submitted state.
- 4: `Running`: the batch is still running.
- 5: `Cancelled`: the batch has been cancelled.

This is commonly used in CI to trigger a failing test.

You can also list the individual tests in that batch with:

Bash

```
resim batches tests \
    --batch-name <Batch friendly name>
```

This will return the JSON associated with the test to track the individual status and return their IDs. The IDs are helpful to, for example, list the log files associated with that test:

Bash

```
resim logs list \
    --batch-id <Batch ID> \
    --test-id <Test ID>
```

The output of this includes pre-signed URLs that would enable you to download logs directly from the command line. In particular, every test contains the `experience-container.log` and `metrics-container.log` files which contain the console logs from your container (`experience-container.log`) and the metrics stage (`metrics-container.log`).

## Ad hoc test suites

It is not always desirable to create a test suite: sometimes, you may simply want to try a few experiences on an experimental branch to make sure that you haven't broken everything. For this, the ReSim App supports the concept of an ad hoc test suite. That is, an unnamed test suite.

You can create this in the CLI as follows:

Bash

```
resim batches create \
    --project "my-project" \
    --build-id <BUILD UUID> \
    --metrics-build-id <METRICS BUILD UUID> \
    --experiences "<experience name or UUID>, <experience name or UUID>, ..." \
    --experience-tags "<tag name>, <tag name>"
```

Use `--experiences` and/or `--experience-tags` to control which experiences are run in the batch.

## Configuration options for running test suites

There are several configuration options available when one is running a test suite or creating an ad-hoc batch:

- **A custom batch name:** Rather than use the auto-generated friendly name (e.g. `rejoicing-aquamarine-starfish`), you can provide your own. In the CLI, this comes from the `--batch-name <my-name>` flag.
- **An allowable error rate:** By default, if any tests in the test suite `ERROR` -- that is, have a failure at the execution phase -- the ReSim orchestration system will mark the whole test batch as `ERROR` and not compute any batch metrics. In many cases, this is the correct default behaviour, but it can be overridden with `--allowable-failure-percent <e.g. 100>`. By default, tests which error will not have test metrics associated with them, but if you set `--allowable-failure-percent 100`, ReSim will attempt to compute metrics even when a test has errored.
- **Parameter overrides:** One can pass key-value parameters to your build to change how it runs via a parameter override flag e.g. `--parameters "key=value"`. This is described in much more detail in [the parameter sweeps documentation](../../guides/run-sweeps/).

## Rerunning a subset of tests in a test suite

It can be frustrating if one or two tests within a test suite has a flaky error or unexpected behavior. The ReSim platform makes it easy to re-run just a subset of tests so that the whole suite doesn't need to be run again. In the CLI, this can be achieved with

Bash

```
resim batches rerun --batch-name <rejoicing-aquamarine-starfish> --test-ids "<test-id>,<test-id>"
```

Finally, we can get a view of how our [system](../../core-concepts/#system) is performing over time by creating a [**test suite report**](../../core-concepts/#test-suite-report).

## Reports in the CLI and app

A **Test Suite Report** is defined as a summary of all the test batches created from a given **Test Suite** and from builds of a given **Branch** between a **Start Timestamp** and an **End Timestamp**.

For maximum flexibility, test suite reports are executed as a cloud workflow. While this adds a small amount of friction (the reports need to be run before being viewed), we feel the benefits of flexibility for customers outweigh the potential delays.

In order to generate a **Test Suite Report** the user simply submits a report via the UI or CLI:

Bash

```
resim reports create --project "my-project" --report-name "<My Report Name>" --length "30" --test-suite "Report Test" --branch "main" --metrics-build-id <Metrics Build ID>
```

The flags are defined as:

- **Report name**: An optional name to give to your report. If you do not provide a name, a friendly name will be generated e.g. `rejoicing-aquamarine-starfish`.
- **Length**: The number of days to use to generate a report. You can alternatively explicitly state a `--start-timestamp` and (optionally) `--end-timestamp`. If you specify a start timestamp, but not an end timestamp, it will be assumed that the report will be generated up until the current time.
- **Test Suite**: This flag requires you to specify which test suite to use as the basis of the report. Reports are expected to be generated as a longitudinal analysis of all batches created from this test suite.
- **Branch**: This flag requires you to specify the branch to concentrate on for this report. We require a specific branch to be chosen because batches across multiple branches can generate a confusing report as work-in-progress branches may cause unexpected errors to be shown in the analysis.
- **Metrics Build**: As has been discussed above, we rely on a metrics build to generate the report. The details of how this metrics build is expected to operate are described in the next section.

Once a report is submitted, it goes through the following simple workflow:

```
flowchart LR
  SUBMITTED
  RUNNING
  ERROR
  SUCCEEDED

  SUBMITTED-->RUNNING
  SUBMITTED-->ERROR
  RUNNING-->ERROR
  RUNNING-->SUCCEEDED
```

The ReSim CLI allows one to `wait` for a transition via `resim reports wait --report <my-report>`.

Test Suite Reports behave similarly to a test metrics or a batch metrics stage in that they generate metrics using the ReSim Metrics SDK and can potentially output logs. As such, you can obtain the logs via `resim reports logs --report <my-report>`.

## Reports mode for metrics build

Reports can, in theory, be generated with any metrics build and it is good practice to use the same metrics build version to generate test, batch, and report metrics for a given test suite, though this is not required.

Generating reports with a metrics build follows a similar pattern to batch metrics. In `report mode`, the ReSim platform will only populate the `/tmp/resim/inputs` directory with a configuration file (called `report_config.json`) that the Metrics SDK can use to fetch the batches associated with that report for further processing and longitudinal analysis. More information is available in the [open source documentation](https://docs.resim.ai/open-core/metrics/).

It is expected that a `metrics.binproto` file exists in the outputs directory, `/tmp/resim/outputs`, which the ReSim platform will process to present the reports in the web app. Any other log files placed in the outputs directory will also be made available as logs in the app.

## Automating reports in CI/CD

It can be convenient to use CI/CD tooling to ensure that reports are generated at a regular cadence. This is possible with some simple workflow code in, for example, GitHub:

### GitHub example

In GitHub, you create a workflow to run the report nightly:

YAML

```
name: Run Report
on:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *"
jobs:
  run-report:
    name: Generate Updated ReSim Report
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Fetch ReSim CLI
        run: curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim-cli
      - name: Make ReSim CLI Executable
        run: chmod +x resim-cli
      - name: Run Report
        run: |
          REPORT_ID_OUTPUT=$(./resim-cli reports create --client-id $CLI_CLIENT_ID --client-secret $CLI_CLIENT_SECRET \
            --project $PROJECT_NAME --metrics-build-id ${METRICS_BUILD_ID} \
            --branch ${BRANCH_NAME} --test-suite ${TEST_SUITE_NAME} --length ${REPORT_LENGTH} --github)
          echo $REPORT_ID_OUTPUT
        env:
          PROJECT_NAME: "<your project name>"
          TEST_SUITE_NAME: "<your test suite name>"
          BRANCH_NAME: "<your branch name>"
          METRICS_BUILD_ID: "<your metrics build id>"
          REPORT_LENGTH: "<the number of days to generate the report for>"
          CLI_CLIENT_ID: ${{ secrets.CLI_CLIENT_ID }}
          CLI_CLIENT_SECRET: ${{ secrets.CLI_CLIENT_SECRET }}
```

### Gitlab example

In Gitlab, the process is similar, with slightly amended syntax:

Text

```
stages:
  - build

run-report:
  stage: build

  # Run this job every night at midnight
  rules:
    - when: "schedule"
      cron: "0 0 * * *"

  variables:
    PROJECT_NAME: <your project name>
    TEST_SUITE_NAME: <your test suite name>
    BRANCH_NAME: <your branch name>
    METRICS_BUILD_ID: <your metrics build id>
    REPORT_LENGTH: <the number of days to generate the report for>

  script:
    # Install the latest version of the ReSim CLI
    - curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim
    - chmod +x resim
    # Run the report
    - REPORT_ID_OUTPUT=$(./resim reports create --project $PROJECT_NAME --metrics-build-id ${METRICS_BUILD_ID} \
      --branch ${BRANCH_NAME} --test-suite ${TEST_SUITE_NAME} --length ${REPORT_LENGTH} --github)
    - echo $REPORT_ID_OUTPUT
```

Now that we have our [test suites](../test-suites/) defined, and [reports](../reports/) running against them, we can setup the Overview page.

The overview page will show you test suites of your choosing, and how your tests and metrics are performing between the latest 2 builds.

## What is the overview page?

This page will compare test suite results against the latest 2 builds on your main branch for the given test suite and system. It will also contain some details to help you measure progress, and get an overall view of the health of your systems:

- How many tests have been fixed in the latest build
- How many tests have starting failing in the latest build
- How many new tests have been added to the test suite
- How have key metric(s) changed from the previous build

## Setting up the overview page

In order for a [Test Suite](../test-suites/) to show up on this page you need to do 2 things:

1. Have a report running against the test suite (completed in the previous step)
1. Explicitly enable the test suite to show up on this page, by using the CLI:

Bash

```
./resim test-suites revise --test-suite nightly --show-on-summary true
```

That's it!

## (optional) Selecting a scalar metric

For each test suite you can choose 1 key scalar metric you want to display. In this example we have chosen to display the "batch mean ego speed" metric. It will also show the change from the previous build:

In order for the scalar metric to show up here you need to add the special tag `RESIM_SUMMARY` during your **batch** metrics build:

Python

```
metrics_writer
    .add_scalar_metric("batch mean ego speed")
    .with_value(9.93)
    .with_unit("m/s")
    .with_tag(key="RESIM_SUMMARY", value="1")
```

Notes:

- this metric needs to be calculated in your **batch** metrics, not **job** metrics
- only `SCALAR` metrics are allowed, other metric types with this tag will be ignored
- the value on the tag is currently unused

The summary page will retrieve this metric from batches run against the 2 most recent builds and display the diff automatically.

# Metrics

Still on Metrics 1.0?

Metrics 1.0 will no longer be supported starting June 1st. If you would like assistance migrating to our new Metrics offering, please reach out to ReSim in our shared Slack channel. For legacy documentation, see the [Metrics 1.0 reference](metrics-1/).

ReSim's Metrics framework recently underwent a major refactor to provide a more flexible and powerful way to collect, plot, and track data in ReSim. At its core, Metrics works by ingesting data from various sources in your system; ROS topics, log files, or custom instrumentation, into a centralized data lake. When you emit data during a test run, it's automatically captured and stored in this data lake. After each run completes, the system automatically queries this data lake using SQL queries you've defined to generate metrics, visualizations, and pass/fail status checks. The data lake persists over time, which means you can continue to explore, query, and visualize your test data long after the run has finished, enabling powerful post-hoc analysis and trend tracking across releases.

### Why use Metrics?

The data lake architecture provides several key benefits. Because your data persists, you're not limited to viewing only the metrics you defined before the test ran. You can return to the ReSim UI days, weeks, or months later to explore the raw data, write new queries, create new visualizations, and answer questions you didn't anticipate when you first ran the test. This makes Metrics ideal for debugging unexpected failures, conducting retrospective analysis, and iterating on your metrics definitions without needing to re-run expensive tests.

### Branch isolation

Metrics configurations are tied to specific branches in your project. This means your `main` branch can maintain a stable, well-tested set of metrics and data schemas, while development branches can experiment with new metrics, modify schemas, or test breaking changes without affecting production workflows. Each branch maintains its own independent metrics configuration, allowing engineers to iterate safely and merge changes only when they're ready.

### What you'll create

To use Metrics, you'll create a single configuration file (`.resim/metrics/config.resim.yaml`) that defines three key components:

1. **Topics** — The schema for data you'll emit from your system (e.g., robot velocity, localization error, goal status). Topics define the structure and types of the data flowing into the data lake.
1. **Metrics** — SQL queries that transform your emitted data into visualizations and scalar values. Metrics can be test-level (computed per individual test run) or batch-level (aggregated across multiple test runs). Metrics can have status checks - optional thresholds attached to metrics that determine pass/fail/warning states. These allow you to automatically flag tests that miss performance requirements.
1. **Metrics sets** — Collections of metrics that are run against a set of tests, helping you to validate performance and learn about robot behavior.

The rest of this guide will walk you through each of these components step by step, starting with identifying the data you want to collect.

## Identifying Key Data

The first step to adopting ReSim's Metrics framework is to identify the key data that you want to collect, plot, and track over time in the ReSim Platform. To start with, we'd recommend simple data that you're interested in plotting simply; i.e. your robot's velocity over time. Identify where this data comes from; in our example, we'll be reading it from our robot's `/odom` topic.

## Writing the schema

Firstly, we strongly recommend installing the [ReSim VSCode Extension](https://marketplace.visualstudio.com/items?itemName=resim-ai.resim) (or from [Open VSX Registry](https://open-vsx.org/extension/resim-ai/resim) for VSCode forks); it provides language support for ReSim config files and will help you avoid mistakes when emitting your data!

Look at the structure of the data you want to emit to ReSim; in our example case, we're looking at the `Odometry.twist.twist.linear` data, which is a `Vector3` representing the velocity of the robot in the x, y and z axis. We can write a ReSim config for this like so:

YAML

```
# required version header
version: 1

topics:
  odom_linear_velocity:
    schema:
      x: float
      y: float
      z: float
```

To ease your experience with writing queries, we recommend flattening nested data structures into their own topics. Additionally, we don't support slash characters `/` in topic names (sorry!). Use the content assist from the VSCode extension to explore the available data types. In-depth descriptions are provided by the extension on hover.

The config file should be saved to `.resim/metrics/config.resim.yaml` in your repository. You're also welcome to save it to another path - when using the ReSim CLI to start a batch, simply pass `--metrics-config-path=<YOUR_PATH>` as an option. The extension will only recognise files with the `.resim.yaml` suffix, so follow that pattern.

## Emitting your data

There are two approaches to take here; you can either emit the data live from your stack or post-hoc when your test is complete. We'll discuss both here.

The `emit` API is primarily used through the `Emitter` class. You can pass it a `config_path` and an `output_path`. `output_path` is where we will write your emissions file - by default it is `/tmp/resim/outputs/emissions.resim.jsonl`. When `config_path` is both specified and is visible to the application, we will validate the types & shape of the data being emitted to ensure it will be ingested by the ReSim platform correctly. See below for an example:

Python

```
from resim.metrics.python.emissions import Emitter

emitter = new Emitter(config_path=".resim/metrics/config.resim.yaml")
emitter.emit("odom_linear_velocity", {"x": 1.0, "y": 2.0, "z": 3.0}, timestamp=0)
emitter.emit("odom_linear_velocity", {"x": 1.1, "y": 1.9, "z": 0.0}, timestamp=500000000)
# rejected
emitter.emit("odom_linear_velocity", {"x": "left", "y": "right", "z": "up"}, timestamp=1000000000)
```

Under the hood, we append these messages to your `output_path`. In this example, the lines would look like this:

/tmp/resim/outputs/emissions.resim.jsonl

```
{"$metadata": {"topic": "odom_linear_velocity", "timestamp": 0}, "$data": {"x": 1.0, "y": 2.0, "z": 3.0}}
{"$metadata": {"topic": "odom_linear_velocity", "timestamp": 500000000}, "$data": {"x": 1.1, "y": 1.9, "z": 0.0}}
...
```

### Stack Emissions

Here's a more fully fleshed-out code example of a stack-based emission pattern. Continuing with our ROS example, we will create a new node for our stack, the `MetricsEmitter` which will receive messages from the `/odom` ROS topic and emit them to the `odom_linear_velocity` ReSim topic.

Python

```
from rclpy.node import Node
from resim.metrics.python.emissions import Emitter
from nav_msgs.msg import Odometry
from builtin_interfaces.msg import Time as MsgTime
from rclpy.time import Time
from typing import Optional

class MetricsEmitter(Node):
    def __init__(self):
        super().__init__('metrics_emitter')
        self.emitter = new Emitter(config_path=".resim/metrics/config.yaml")
        self.first_timestamp = None

        self.odom_subscriber = self.create_subscription(Odometry, '/odom', self.odom_callback, 10)

    def get_relative_timestamp(self, msg_time: Optional[MsgTime] = None) -> Optional[int]:
        if self.first_goal_received_time is None:
            return None

        current_time: Time
        if msg_time is not None:
            current_time = Time.from_msg(msg_time)
        else:
            current_time = self.get_clock().now()

        return current_time.nanoseconds - self.first_timestamp.nanoseconds

    def odom_callback(self, msg: Odometry):
        if self.first_timestamp == None:
            self.first_timestamp = Time.from_msg(msg.header.stamp).nanoseconds
        self.emitter.emit("odom_linear_velocity", {
            "x": msg.twist.twist.linear.x,
            "y": msg.twist.twist.linear.y,
            "z": msg.twist.twist.linear.z
        }, timestamp=self.get_relative_timestamp(msg.header.stamp))
```

### Post-hoc Emissions

If you're already recording log files from your simulations, you might want to take advantage of ReSim's metrics builds. We can follow much the same pattern, but instead read the data from log files like MCAPs or rosbags. See a developed example below using ROS:

Python

```
from resim.metrics.python.emissions import Emitter
from rosidl_runtime_py.utilities import get_message
import rosbag2_py
from pathlib import Path
import rclpy.serialization

def emit_velocity_data(emitter: Emitter, input_bag: Path):
    reader = rosbag2_py.SequentialReader()
    reader.open(
        rosbag2_py.StorageOptions(uri=str(input_bag), storage_id="mcap"),
        rosbag2_py.ConverterOptions(
            input_serialization_format="cdr", output_serialization_format="cdr"
        ),
    )

    # Create a dictionary mapping topic names to their types
    topic_type_map = {topic.name: topic.type for topic in reader.get_all_topics_and_types()}
    odom_topic = "/odom"

    if odom_topic not in topic_type_map:
        raise ValueError(f"topic {odom_topic} not in bag")

    msg_type = get_message(topic_type_map[odom_topic])

    while reader.has_next():
        topic, data, timestamp = reader.read_next()
        if topic == odom_topic:

            # Only emit if enough time has passed since last emission
            msg = rclpy.serialization.deserialize_message(data, msg_type)
            emitter.emit('odom_linear_velocity', {
                'x': msg.twist.twist.linear.x,
                'y': msg.twist.twist.linear.y,
                'z': msg.twist.twist.linear.z,
            }, timestamp=timestamp)

if __name__ == "__main__":
    input_bag = Path("/tmp/resim/inputs/logs/record.mcap")
    with Emitter(config_path=".resim/metrics/config.yaml") as emitter:
        emit_velocity_data(emitter, Path(input_bag))
```

### Series Emissions

If you're emitting a series of data from a single call, there is an additional helper function - `Emitter.emit_series`. This will spread the data from this single emit call to multiple emission lines, to aid with query simplicity. For the initial example above, this can be written as:

Python

```
emitter = new Emitter(config_path=".resim/metrics/config.resim.yaml")
emitter.emit_series("odom_linear_velocity", {
  "x": [1.0, 1.1, 1.1, 1.2],
  "y": [2.0, 1.9, 1.8, 1.7],
  "z": [3.0, 3.0, 3.0, 3.0]
}, timestamps=[0, 500000000, 1000000000, 1500000000])

# Equivalent to
emitter.emit("odom_linear_velocity", {"x": 1.0, "y": 2.0, "z": 3.0}, timestamp=0)
emitter.emit("odom_linear_velocity", {"x": 1.1, "y": 1.9, "z": 3.0}, timestamp=500000000)
emitter.emit("odom_linear_velocity", {"x": 1.1, "y": 1.8, "z": 3.0}, timestamp=1000000000)
emitter.emit("odom_linear_velocity", {"x": 1.2, "y": 1.7, "z": 3.0}, timestamp=1500000000)
```

The config file remains the same as the original example, as the resulting emissions match the existing definition of the topic. The function will validate the types of the data as it is emitted & ensure all series are of the same length.

### Event Emissions

ReSim supports special types of emissions called "events". These cause an event to be registered for your job at the given timestamp, which will appear in the Events tab in the ReSim app.

Each event refers to a specific point in time - such as reaching a goal, or when a system stop occurs. Metrics can be attached to each event by including them in the emission to help you understand the system state at that time. Events are emitted using the `emitter.emit_event` method - see the example below:

YAML

```
topics:
  # ...
  goal_reached:
    event: true
    schema:
      name: string
      description: string
      status: status
      tags: string[]
      metrics: metric[]
```

Python

```
emitter = new Emitter(config_path=".resim/metrics/config.resim.yaml")
emitter.emit_event({
  name: "Goal 1 Reached",
  description: "Robot successfully reached goal number 1",
  status: "PASSED",
  tags: ["navigation"],
  metrics: [{
   "name": "Stereo View",
   "type": "image",
   "value": ["goal_1.png"]
   "status": "PASSED"
  }]
}, event=True, timestamp=1750000000)
```

The structure of the `metric[]` object is defined as

JSON

```
"metrics": [
  {
      "name": "string",
      "description": "optional",
      "status": "optional. PASSED | FAIL_BLOCK | etc."
      "type": "scalar | image | plotly | text",
      "value": "the value of the metric. Changes based on the type.",
      // examples:
      // Scalar and text metrics are treated the same way.
      // "type": "scalar" | "text",
      // "value": "2.37",
      // "type": "image",
      // "value": ["1.jpg", "2.jpg"]
      // "type": "plotly"
      // "value": "{ a valid plotly blob}", ie. the output of fig.to_json()
   }
]
```

### Custom emissions API

If Python isn't your jam, you're more than welcome to write your own emitter in the language of your choice. The minimal implementation of the `emit` API (without validation) in Python is [here](https://github.com/resim-ai/open-core/blob/538b339550dc8c63c57dfc8a9cee6659bbb5da36/resim/sdk/metrics/emissions.py#L86). If you'd like official support for your language of choice, please reach out to our ProServe team and we'll look into it!

## Metrics development

### Adding your first metric

We recommend developing your first couple of metrics using debug dashboards. The first step is to create an empty metrics set at the end of your config file:

YAML

```
...
metrics sets:
  My Metrics:
    metrics: []
```

Next, create the debug dashboard using the [ReSim CLI](../../setup/cli/) - see the linked docs for installation instructions. Once you've generated an emissions file, it can be ingested using the `resim metrics debug` command, invoked like so:

Bash

```
resim metrics debug --project "metrics-demo" --emissions-file "emissions.resim.jsonl" --metrics-config-path ".resim/metrics/config.resim.yaml" --metrics-set "My Metrics"
```

After a short time, this will print a URL to the debug dashboard which is linked to the config and emissions file provided. From here, you can start to develop your metrics. See the video below for a demonstration.

\[

Your browser does not support the video tag. \](https://resim-public-assets.s3.us-east-1.amazonaws.com/Adding_your_first_metric.webm)

Once you're happy with your metrics, you can export the current state of the dashboard by clicking the "Export Config" button. This will download a zip file containing the config file and any custom templates configured. You can then replace your local config file with this - you may need to do some formatting & reordering to get it how you like it.

You can also develop your metrics in the config file directly, and then reusing `resim metrics debug` to test them. See the [metric templates](#metric-templates) below for more details. Once you've added a metric in the config file, make sure that it's included in your metrics set:

YAML

```
...
metrics:
  Robot Speed:
    type: test
    description: Robot speed over time
    query_string: |
      SELECT
        'Speed',
        timestamp / 1E9 AS "Time (s)",
        SQRT(POWER(x, 2) + POWER(y, 2)) AS "Speed (m/s)"
      FROM odom_linear_velocity;
    template_type: system
    template: line

metrics sets:
  My Metrics:
    metrics:
      - Robot Speed
```

And then you can view it in the debug dashboard by running:

Bash

```
resim metrics debug --project "metrics-demo" --emissions-file "emissions.resim.jsonl" --metrics-config-path ".resim/metrics/config.resim.yaml" --metrics-set "My Metrics"
```

#### Image Metrics

At this time, image metrics are not supported in the debug dashboard as they have a more complex ingestion process. Comment them out in your config in development, and add them back in when you're ready to test them with a real batch.

#### Status checks

You can make a metric affect pass/fail by adding a status block. Status checks run a separate query that is evaluated against threshold values: if the query returns any rows, the metric is marked as blocking (or warning).

Under `status` you configure:

- **`query_string`** — A SQL query with exactly one `?` parameter. The `?` is replaced with the `block` or `warn` threshold value when the check runs (e.g. `HAVING COUNT(*) < ?` or `WHERE value > ?`).
- **`block`** — Threshold value for a blocking failure. If the status query returns any rows when run with this value, the job is marked as failed (blocker).
- **`warn`** — Optional. Threshold value for a warning. If the query returns any rows when run with this value, the metric is marked as a warning instead of a pass.

Example: require at least 3 goals to be reached; otherwise the metric blocks. The status query returns a row when the count is *below* the threshold, so we use `HAVING COUNT(*) < ?` with `block: 3`:

YAML

```
Time to reach final goal:
  type: test
  description: Time between receiving first goal and reaching final goal. Blocks if less than 3 goals are reached.
  query_string: |
    SELECT
      CASE
        WHEN (SELECT COUNT(*) FROM time_to_goal) != 3 THEN 75.0
        ELSE sum(time_s)
      END
    FROM time_to_goal;
  template_type: system
  template: scalar
  status:
    query_string: SELECT '1' FROM time_to_goal HAVING COUNT(*) < ?
    block: 3
```

### Running your first Metrics batch

Once you're happy with your metrics set, you can give your stack its first run in ReSim. First step is to register your config with the ReSim platform:

Bash

```
$ resim metrics sync --project "metrics-demo" --branch "metrics-setup"
```

If you have a test suite you want to run your metrics set against, you can update the test suite as so:

Bash

```
$ resim suite revise \
   --project "metrics-demo" \
   --test-suite "Nightly Suite" \
   --metrics-set "My Metrics"

# Now run the test suite:
$ resim suite run \
   --project "metrics-demo" \
   --test-suite "Nightly Suite" \
   --build-id "dc71ebee-1601-4d98-aab4-d4bdecfafca1"
```

If running a batch directly, the parameters are similar:

Bash

```
$ resim batches create --project "metrics-demo" --metrics-set "My Metrics" ...
```

**Note**: Each branch in your project has its own copy of the config file, which might result in lots of CLI calls to re-register config files. To help with this, at the time of triggering a batch your config can automatically be synced to your branch with the `--sync-metrics-config` flag:

Bash

```
$ resim suite run --project "metrics-demo" \
   --test-suite "Nightly Suite" \
   --build-id "dc71ebee-1601-4d98-aab4-d4bdecfafca1" \
   --sync-metrics-config
```

With that, your batch should be running. Wait for it to complete, and the metrics you have configured will be shown automatically in the ReSim app. If you want to make more changes, you can simply repeat the process of exporting the config and replacing your local config file.

#### Changing your metrics

If you make changes to your metrics on a completed batch, the changes will be saved for the instance of metrics you are currently viewing. For example, if you make a change to the metrics for a single job, it will not be reflected on other jobs in the batch or going forward until you synchronise the config file manually.

### Authoring batch metrics

When defining a metric in the config file, you may have spotted the `type: test` definition. This helps our metrics system determine where that metric should be displayed - in this case, it will be shown as a test result. The other option is `type: batch`, which will be shown as a batch metric - which are used for creating aggregate metrics across an entire batch. This changes the data which is available to the metric when authoring the query. This allows really easy authoring of high level metrics - such as "What was the average speed of the robot across all jobs in the batch?" which can be defined as:

YAML

```
Average Speed:
  type: batch
  description: Average speed of the robot across all jobs in the batch.
  query_string: SELECT AVG(speed) FROM robot_speed
  template_type: system
  template: scalar
```

Along with the data you have emitted, you may also want to join against the metadata table to get more human-readable information. See the [metadata schema](#metadata-schema) below for more details.

### Dashboards

Dashboards aggregate metrics across many batches over a trailing time window, so you can track how your system, tests, or test suite are performing as a whole. See the [Dashboards guide](dashboards/) for details.

## Metric Templates

Each metric uses a **template** that defines how its query results are rendered. The following system templates are available; in all cases the query runs against your emitted topic data, and column order (and for charts, column aliases) determine layout and axis titles.

### Line Chart

The line template expects exactly three columns in order: a series identifier, then the x value, then the y value. Each row is `[series_name, x_value, y_value]`. Different `series_name` values produce multiple series on the same chart. The axis titles are taken from the column names at positions 2 and 3, so use descriptive aliases with units (e.g. `"Time (s)"`, `"Speed (m/s)"`). Emitted timestamps are in nanoseconds; divide by `1E9` to show seconds.

Example config:

YAML

```
Distance to Goal:
  type: test
  description: Distance to each goal over time, measured from odometry. Each line represents a different goal.
  query_string: |
    SELECT
      goal_name as group_name,
      timestamp / 1E9 AS "Time (s)",
      distance_m as "Distance to Goal (m)"
    FROM goal_distance;
  template_type: system
  template: line
```

### Bar Chart

The bar template uses the same shape as the line template: three columns `[series, x, y]` and rows `[series_name, x_value, y_value]`. It is well suited to aggregated or categorical comparisons (e.g. per experience, per goal).

Example config:

YAML

```
Maximum Localization Error:
  type: batch
  description: Maximum position difference between odometry and AMCL localization across all experiences.
  query_string: |
    SELECT
      'Maximum',
      m.experience_name as "Experience Name",
      MAX(position_diff_m) as "Maximum Localization Error (m)"
    FROM pose_difference pd
    JOIN metadata m on pd.job_id = m.job_id
    GROUP BY m.experience_name ORDER BY m.experience_name;
  template_type: system
  template: bar
```

#### Click-through links

Bar charts support an optional click-through feature: each bar can link to a specific page within ReSim (e.g. a batch, test, or experience). Click-through is most useful in [Dashboards](dashboards/) where you'll be aggregating data from tests over time. To enable click-through, include a `link_path` column in your query. Hovering over a bar will expose the link, and clicking it will navigate to the target page:

The `link_path` value must be a path under ReSim (third-party URLs are not supported). You are responsible for constructing the path yourself by interpolating the relevant IDs into the query. All the data you need to construct the path is available when running queries: project_id, batch_id, test_id, experience_id, and more. View the `metadata` topic in the app to see what is available.

Example config, for the metric seen above:

YAML

```
altitude over time:
  type: batch
  description: Maximum drone altitude per batch, with click-through to the source batch.
  query_string: |
    SELECT
      'altitude',
      timestamp,
      altitudes,
      '/projects/' || project_id || '/batches/' || batch_id AS link_path
    FROM drone_altitude
  template_type: system
  template: bar
```

### Table

The table template accepts any number of columns and any number of rows. Column names become the table headers and each row is rendered as a table row. Use it for summary statistics, key-value pairs, or any tabular output.

Example config:

YAML

```
AMCL Covariance Calibration:
  type: test
  description: Evaluates whether AMCL's uncertainty estimates are realistic.
  query_string: |
    SELECT "Metric", "Value" FROM (
      SELECT 1 as sort_order, 'Total Samples' as "Metric", CAST(COUNT(*) AS VARCHAR) as "Value"
      FROM covariance_accuracy
      UNION ALL
      SELECT 2 as sort_order, 'Within 1σ (target: ~68%)' as "Metric", CONCAT(CAST(ROUND(100.0 * AVG(CASE WHEN within_1_sigma = 1 THEN 1.0 ELSE 0.0 END), 1) AS VARCHAR), '%') as "Value"
      FROM covariance_accuracy
      ORDER BY sort_order
    );
  template_type: system
  template: table
```

### Scalar

The scalar template expects the query to return a single value. The renderer uses the first row and first column. Optional `status` thresholds can block or warn based on a separate query (e.g. checking that a count meets a minimum). You can also provide `units`, which will be displayed after the value.

Example config:

YAML

```
Time to reach final goal:
  type: test
  description: Time between receiving first goal and reaching final goal.
  query_string: |
    SELECT
      CASE
        WHEN (SELECT COUNT(*) FROM time_to_goal) != 3 THEN 75.0
        ELSE sum(time_s)
      END
    FROM time_to_goal;
  template_type: system
  template: scalar
  units: "s"
  status:
    query_string: SELECT '1' FROM time_to_goal HAVING COUNT(*) < ?
    block: 3
```

### Image

The image template expects one column whose values are image identifiers (e.g. filenames or paths) available in the job logs. Each row produces one image in the metric output.

Example config:

YAML

```
Stereo Camera Feed:
  type: test
  description: Camera feed from navigation, sped up 2x, starting after first goal received.
  query_string: SELECT filename FROM camera_gif
  template_type: system
  template: image
```

### Video

The video template expects one column whose values are video identifiers (e.g. filenames or paths) available in the job logs. Each row produces one video in the metric output.

Example config:

YAML

```
Stereo Camera Feed:
  type: test
  description: Camera feed from navigation, starting after first goal received.
  query_string: SELECT filename FROM camera_video
  template_type: system
  template: video
```

### State Timeline

The state timeline template shows discrete states over time for one or more systems. The query must return exactly three columns in order: system identifier, timestamp, and state name. Each row is `[system_name, timestamp, state_name]`. Use different `system_name` values to show multiple systems. Consecutive rows with the same state are merged into segments. Timestamps are in nanoseconds and are converted to elapsed seconds from the first timestamp for the x-axis; the x-axis title is "Elapsed time (s)" and the y-axis title comes from the alias you give the system column. Each unique state is assigned a color from the ReSim palette.

Example config:

YAML

```
Goal Status:
  type: test
  description: Status of the goals over time.
  query_string: |
    SELECT
      'Navigation' as "System",
      timestamp,
      state
    FROM goal_status;
  template_type: system
  template: state_timeline
```

### Custom templates

If the system templates above don't fit, you can author a Plotly chart yourself with a Liquid template. See [Custom Metric Templates](custom-templates/) for the full guide.

## Metadata Schema

When you `SELECT *` on your topics, you will notice some additional metadata such as `batch_id` and `job_id` are returned. These can be used to join against a special `metadata` table which contains information about your build, experience, and more. The `metadata` table has one row per job in the batch. Batch metrics (`type: batch`) query across all rows. This allows you to, in something like a batch metric, compute

- what was the min, average, and max speed, for each experience in my test set?
- how did the performance of this metric vary across sunny, cloudy, and rainy test scenarios?

An example batch metric:

"How does the min, avg, and max speed vary across each experience?"

SQL

```
SELECT m.experience_name, MIN(speed), AVG(speed), MAX(speed)
FROM drone_speed d
JOIN metadata m on d.job_id = m.job_id
GROUP BY m.experience_name;
```

If you want to see all metadata available you can run this query:

SQL

```
SELECT * FROM metadata
```

### Column reference

| Column                     | Type          | Description / Observed values                                                                        |
| -------------------------- | ------------- | ---------------------------------------------------------------------------------------------------- |
| `org_id`                   | string        | Organisation identifier, e.g. `"resim.ai"`                                                           |
| `batch_id`                 | string (UUID) | ID of the batch this job belongs to                                                                  |
| `job_id`                   | string (UUID) | Unique identifier for the individual job                                                             |
| `project_id`               | string (UUID) | ReSim project ID                                                                                     |
| `branch_id`                | string (UUID) | Branch ID within the project                                                                         |
| `branch_name`              | string        | Human-readable branch name, e.g. `"main"`                                                            |
| `build_id`                 | string (UUID) | ID of the build under test                                                                           |
| `build_name`               | string        | Human-readable build name                                                                            |
| `build_creation_timestamp` | timestamp     | When the build was created                                                                           |
| `experience_id`            | string (UUID) | ID of the experience (scenario) for this job                                                         |
| `experience_name`          | string        | Human-readable experience name                                                                       |
| `experience_tag_ids`       | string[]      | UUIDs of the experience tags attached to this job                                                    |
| `experience_tag_names`     | string[]      | Names of the experience tags attached to this job                                                    |
| `test_suite_id`            | string (UUID) | ID of the test suite                                                                                 |
| `test_suite_name`          | string        | Human-readable test suite name                                                                       |
| `test_suite_revision`      | int           | Revision number of the test suite                                                                    |
| `job_status`               | string        | Raw execution status — e.g.: `'SUCCEEDED'`, `'FAILED'`                                               |
| `job_metrics_status`       | string        | Metrics evaluation status — e.g. `'PASSED'`, `'FAILED'`                                              |
| `job_conflated_status`     | string        | Combined status used for pass/fail decisions — e.g.: `'PASSED'`, `'ERROR'`, `'BLOCKER'`, `'WARNING'` |
| `custom_field_keys`        | string[]      | Custom field names set on the job (may be `[]`)                                                      |
| `custom_field_values`      | string[]      | Custom field values, parallel to `custom_field_keys` (may be `[]`)                                   |
| `time`                     | timestamp     | When the relevant job started                                                                        |

### Array column usage

`experience_tag_names` and `experience_tag_ids` are Trino arrays. Common patterns:

SQL

```
-- Check membership
WHERE contains(experience_tag_names, 'my_tag')

-- Filter to tags matching a prefix, take the first result (1-based indexing)
filter(experience_tag_names, x -> starts_with(x, 'my_prefix_'))[1]

-- Conditional count across jobs
count_if(job_conflated_status = 'PASSED')

-- Group jobs by a tag prefix
WITH classified AS (
  SELECT
    filter(experience_tag_names, x -> starts_with(x, 'my_prefix_'))[1] AS group_name,
    job_conflated_status
  FROM metadata
  WHERE contains(experience_tag_names, 'my_tag')
)
SELECT group_name, count_if(job_conflated_status = 'PASSED') AS passed
FROM classified
WHERE group_name IS NOT NULL
GROUP BY group_name
```

## Introduction

This document is intended to help you build and push **build images** to an image registry from a CI service so that ReSim can use the images to run tests.

Currently ReSim supports authenticating against, and pulling images from, an AWS ECR, Dockerhub or Google Artifact Registry repository. If you use a different image registry provider and would like ReSim to integrate with it, please get in touch.

The example workflows provided are for GitLab and GitHub but almost any CI service should be able to push to ECR. If you have a suggested service to add to this list, or that you would like help configuring to push build images, please reach out to ReSim.

Note

If you already have well-established processes for pushing images build in a CI/CD environment, this guide is not required. Instead, follow the [Next Steps](./#next-steps) described below.

## Prerequisites

- We recommend (although it's not mandatory) that you have built, pushed and tested an image from a local machine before setting up a CI pipeline to do so. See [Build Images](../build-images/) for guidance.
- You need to have set up an image repository that ReSim can access, as described in [Resim Data Access](../resim-data-access/). You will need the **repository URI** or: your AWS account number, the AWS region in which the account was created, and the repository name.
- Your ReSim contact should have provided you with a **client ID** and **client secret**. These are the credentials your CI service will use to authenticate with ReSim.
- You will need a project in the ReSim app with which to associate your builds. See [Projects](../projects/).

## Creating an IAM user with AWS ECR permissions

Your CI pipeline will need to authenticate with AWS to push images. A thorough exploration of AWS configuration is outside the scope of this article, but as a starting point if you do not have existing AWS identities that your CI service uses, you can create an IAM user with the required permissions.

You will need to create a user, create security credentials for the user, and assign the `AmazonEC2ContainerRegistryPowerUser` managed IAM policy to the user.

If you have the AWS CLI installed and configured, we recommend using it rather than the browser UI for this task. See the [Command line interface](#command-line-interface) section below.

If you don't have the AWS CLI set up, you can create the user in the AWS browser interface.

### Browser interface (AWS Console)

Following along with the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) as required, you need to:

1. [Create a user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console), named `ci-user` or any other name you choose. You **do not** need to enable console access for this user.
1. [Create credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) in the form of an **access key** and **secret access key**. Store these locally in a password manager or other secure location, they are equivalent to a username and password for this CI user.
1. [Assign permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) to the user so it can push to ECR. Following the instructions to "add permissions by attaching policies directly to the user", assign the `AmazonEC2ContainerRegistryPowerUser` IAM policy. This policy is managed by AWS.

### Command line interface

Create a new user:

Bash

```
$ aws iam create-user --user-name ci-user
{
  "User": {
      "Path": "/",
      "UserName": "ci-user",
      "UserId": "AIDAWRXCWKBZSLQOSKX4V",
      "Arn": "arn:aws:iam::<AWS account number>:user/ci-user",
      "CreateDate": "2023-08-10T18:47:20+00:00"
  }
}
```

Create an access key pair:

Note

Store the access key and secret access key from the following command somewhere secure - they are static credentials equivalent to a username and password. We will add them to your CI service's secrets configuration later.

Bash

```
$ aws iam create-access-key --user-name ci-user
{
  "AccessKey": {
      "UserName": "ci-user",
      "AccessKeyId": "AKIAWRXCWKBZVEXAMPLE",
      "Status": "Active",
      "SecretAccessKey": "kKEZbvjb6UV/jEFPEED5sdcSiJUGeezDuEXAMPLE",
      "CreateDate": "2023-08-10T18:47:53+00:00"
  }
}
```

Attach a policy to the new user which will allow it to push to ECR repositories.

Bash

```
$ aws iam attach-user-policy --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser" --user-name ci-user
```

### Testing permissions

Once you have created a user, generated credentials for it, and assigned permissions to it following one of the above methods, you may wish to test the configuration locally before configuring your CI service.

To log in to your ECR repository as the CI user:

Bash

```
AWS_ACCESS_KEY_ID=<CI user access key> AWS_SECRET_ACCESS_KEY=<CI user secret key> aws ecr get-login-password --region <AWS region> | docker login --username AWS --password-stdin <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com
```

Then pull and tag an example image and push it to your repository:

Bash

```
docker pull public.ecr.aws/docker/library/hello-world
docker tag public.ecr.aws/docker/library/hello-world <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/<repository name>:hello-world
docker push <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/<repository name>:hello-world
```

Try pulling the image:

Note

This will not pull any data because the image already exists locally, but it will confirm that the image exists in the remote repository.

Bash

```
docker pull <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/<repository name>:hello-world
```

Log out from ECR in your local environment:

Bash

```
docker logout <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com
```

You have now confirmed that the CI user you've created can manage images in your ECR repository. (For more information about working with ECR images, see https://docs.aws.amazon.com/AmazonECR/latest/userguide/images.html)

## Next steps

The next step is to configure your CI system to build and push images and run tests in ReSim:

- [GitHub](github/)
- [GitLab](gitlab/)

# GitHub

In this example, you have a GitHub repository which contains the application you would like to test.

Note that you will need a ReSim project. You can [create a project](../../projects/) with the ReSim CLI.

This example uses [our GitHub action](https://github.com/resim-ai/action).

## Prerequisites

Referring to [GitHub's documentation about setting repository secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) as required, navigate to your repository's Settings and go to the "Secrets and variables" page in the Security section.

Add the following secrets:

- `AWS_ACCESS_KEY_ID` - set the value of this to the access key generated when you [set up your ECR repository for use with ReSim](../#creating-an-iam-user-with-aws-ecr-permissions)
- `AWS_SECRET_ACCESS_KEY` - set the value of this to the secret access key generated when you [set up your ECR repository for use with ReSim](../#creating-an-iam-user-with-aws-ecr-permissions)
- `SLACK_WEBHOOK` - set this to a Slack webhook URL (see the [Slack documentation](https://api.slack.com/messaging/webhooks)), if you want nightly summaries

Also add one of these sets of secrets. Your ReSim contact will provide you with either a client ID and secret, or a username and password:

- `RESIM_CLIENT_ID` and `RESIM_CLIENT_SECRET`, or
- `RESIM_USERNAME` and `RESIM_PASSWORD`

Add a variable containing your ECR registry URL:

- `ECR_REGISTRY_URL` - for example `123456789.dkr.ecr.us-east-1.amazonaws.com`

## Workflow

Now you need to create a GitHub workflow by adding a file to `.github/workflows/` in your repository.

Below is an example of a workflow that builds a Docker image, pushes it to an ECR repository and launches a batch in ReSim to test it.

If you use this example, make sure to replace the following placeholder values:

- `<your build command>` - any commands that need to run before the Docker build, e.g. `bazel build`
- `<your image name>` - replace with the name of the ECR repository, e.g. the name of the image/application
- `<your project name>` - replace with the name of your ReSim project
- `<your system name>` - replace with the name of your ReSim system
- `<your test suite>` - the test suite you want to test against (see [Test Suites](../../test-suites/))

YAML

```
name: build-and-test
on:
  pull_request:
  push:
    branches:
      - "main"
  schedule:
    - cron: '0 0 * * *'

jobs:
  build_and_test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Build
        run: <your build command>

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to ECR
        uses: docker/login-action@v2
        with:
          registry: ${{ vars.ECR_REGISTRY_URL }}
          username: ${{ secrets.AWS_ACCESS_KEY_ID }}
          password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Prepare Docker metadata
        id: docker_meta
        uses: docker/metadata-action@v5
        with:
          images: |
            ${{ vars.ECR_REGISTRY_URL }}/<your image name>
          tags: |
            type=sha

      - name: Build and push image
        id: docker_build
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ steps.docker_meta.outputs.tags }}

      - name: Launch Batch in ReSim
        uses: resim-ai/action@v1
        id: launch_batch
        with:
          client_id: ${{ secrets.RESIM_CLIENT_ID }}
          client_secret: ${{ secrets.RESIM_CLIENT_SECRET }}
          project: <your project name>
          system: <your system name>
          image: ${{ steps.docker_meta.outputs.tags }}
          test_suite: <your test suite>
          # You may use experience names or tags instead
          # experiences: <your experience names>
          # experience_tags: <your experience tags>
          metrics-build-id: <your metrics build id>
          ## If you would like the action to post a link to the results on PRs:
          comment_on_pr: true
          github_token: ${{ secrets.GITHUB_TOKEN }}
      - name: Fetch ReSim CLI
        if: github.event_name == 'schedule'
        run: |
            curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim-cli
            chmod +x resim-cli
      - name: Wait for completion and send Slack summary
        if: github.event_name == 'schedule'
        env:
          PROJECT_ID: ${{ steps.launch_batch.outputs.project_id }}
          BATCH_ID: ${{ steps.launch_batch.outputs.batch_id }}
          RESIM_CLIENT_ID: ${{ secrets.RESIM_CLIENT_ID }}
          RESIM_CLIENT_SECRET: ${{ secrets.RESIM_CLIENT_SECRET }}
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
        run: |
          # the default wait time is one hour. If you think your batch will reliably take longer, you may need to edit this
          ./resim batch wait --project "$PROJECT_ID" --batch-id "$BATCH_ID" || exit_status=$?
          ./resim batch get --project "$PROJECT_ID" --batch-id "$BATCH_ID" --slack | \
          curl -X POST -H "content-type: application/json" --data @- $SLACK_WEBHOOK
          exit "${exit_status:-0}"
```

# GitLab

In this example, you will configure a GitLab project which contains the test application that you would like to run.

Note that you will need a ReSim project. You can [create a project](../../projects/) with the ReSim CLI.

Referring to [GitLab's documentation about CI variables](https://docs.gitlab.com/ee/ci/variables/) as required, navigate to your project's CI/CD Settings and scroll to and expand the Variables section.

Add the following variables:

- `AWS_ACCESS_KEY_ID` - set the value of this to the access key generated [above](../#creating-an-iam-user-with-aws-ecr-permissions)
- `AWS_SECRET_ACCESS_KEY` - set the value of this to the secret access key generated [above](../#creating-an-iam-user-with-aws-ecr-permissions)
- `SLACK_WEBHOOK` - if you want to run nightly tests with summaries, add a Slack webhook URL (see the [Slack documentation](https://api.slack.com/messaging/webhooks))

Also add one of these sets of variables. Your ReSim contact will provide you with either a client ID and secret, or a username and password:

- `RESIM_CLIENT_ID` and `RESIM_CLIENT_SECRET`, or
- `RESIM_USERNAME` and `RESIM_PASSWORD`

If you plan to use a merge request workflow, these variables should **not** be `protected` (so that branches can use them to push images and run tests in ReSim), or the branches you raise MRs for should be covered by a branch protection rule.

If you would like CI pipelines to be able to comment on merge requests, you also need to set a variable containing a [GitLab authentication token](https://docs.gitlab.com/ee/security/token_overview.html), either a personal access token or a project token, with at least the `api` scope.

Add a [`.gitlab-ci.yml`](https://docs.gitlab.com/ee/ci/quick_start/#create-a-gitlab-ciyml-file) file to the root of the repository.

Below is an example of building and pushing a Docker image to an ECR repository. If you use this example, make sure to replace the following placeholder values:

- `<AWS account number>` - replace with your AWS account number
- `<AWS region>` - replace with the AWS region in which you have deployed your ECR repository, e.g. `us-east-1`
- `<repository name>` - replace with the name of the ECR repository, e.g. the name of the application
- `<your project name>` - replace with the name of your ReSim project
- `<your project id>` - replace with the id of your ReSim project
- `<your build commands>` - the commands you run to build a Docker image containing your application
- `<your image name>` - the name of the Docker image created by your build
- `<your experience ids>` - the experience(s) you want to test against (see [Adding Experiences](../../adding-experiences/)), comma-separated

Text

```
stages:
  - build
  - comment # Optional - configures this workflow to post a results link if it's triggered by an MR
  - notify  # Optional - configures this workflow to send the results of a completed nightly batch to a slack webhook

variables:
  PROJECT_ID: <your project id>

push-image:
  stage: build

  # Start a Docker daemon we can use to build images
  services:
   - name: docker:dind
     alias: dockerdaemon

  # Run this job if this is an MR, a tag, or the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_TAG
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

  variables:
    DOCKER_HOST: tcp://dockerdaemon:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
    SYSTEM_NAME: <your system name>

  script:
    # First, you need to build a Docker image containing your application.  Put your build commands below.
    <your build commands>
    # For example
    # - docker build . -t <your image name>:latest
    # Store our new image's URI as a variable to use later. Make sure you update these placeholder values
    - IMAGE_URI=<AWS account number>.dkr.ecr.<AWS region>.amazonaws.com/<repository name>:$CI_COMMIT_SHORT_SHA
    # We are using $CI_COMMIT_SHORT_SHA as image tag - this variable is set automatically by GitLab to the first 8 characters of the commit that triggered the run
    # Using this tag lets us map built images to code (and the inverse) easily
    - docker tag <your image name>:latest $IMAGE_URI
    # Log in to ECR so we can push the image. Make sure you update these placeholder values
    - aws ecr get-login-password --region <AWS region> | docker login --username AWS --password-stdin <AWS account number>.dkr.ecr.<AWS region>.amazonaws.com
    - docker push $IMAGE_URI
    # Install the latest version of the ReSim CLI
    - curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim
    - chmod +x resim
    # Register our new image with ReSim as a build
    - BUILD_ID_OUTPUT=$(./resim builds create --system $SYSTEM_NAME --description "A quadcopter simulation" --version "$CI_COMMIT_SHORT_SHA" --project $PROJECT_ID --branch "$CI_COMMIT_REF_NAME" --image "$IMAGE_URI" --auto-create-branch --github)
    # Run a batch with the new build
    - BATCH_ID_OUTPUT=$(./resim batches create --project $PROJECT_ID --build-id ${BUILD_ID_OUTPUT#build_id=} --experiences "<your experience IDs or names>" --github)
    - echo $BATCH_ID_OUTPUT >> build.env

  # This passes data to the next job
  artifacts:
    reports:
      dotenv: build.env

# To comment a results link for the batch above on an MR, this command uses a GitLab token from the variable $MY_GITLAB_TOKEN. You may need to change the name of this variable here or in your project's settings.
comment-on-mr:
  stage: comment
  # Configure this job to run only on MRs
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

  script:
    # this syntax (a folded block scalar) stops YAML complaining about the colons in this string
    - >
      curl --location --request POST "https://gitlab.com/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes" \
      --header "PRIVATE-TOKEN: $MY_GITLAB_TOKEN" --header "Content-Type: application/json" \
      --data-raw "{ \"body\": \"View results on [app.resim.ai](https://app.resim.ai/projects/$PROJECT_ID/batches/$batch_id)\" }"

slack-notify:
  stage: notify
  # Run only on a schedule - for nightly tests
  rules:
    - if: $CI_PIPELINE_SOURCE == 'schedule'
  script:
    # Install the latest version of the ReSim CLI
    - curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim
    - chmod +x resim
    # Wait for the job to complete and capture the exit code
    # the default wait time is one hour. If you think your batch will reliably take longer, you may need to edit this
    - ./resim batch wait --project "$PROJECT_ID" --batch-id "$BATCH_ID" || exit_status=$?
    # Build the slack payload and post it using curl
    - > 
      ./resim batch get --project "$PROJECT_ID" --batch-id "$BATCH_ID" --slack | \
      curl -X POST -H "content-type: application/json" --data @- $SLACK_WEBHOOK
    # Exit with error code
    - exit "${exit_status:-0}"
  # optional - you may want to permit job failure for batch errors
  allow_failure:
    exit_codes:
      - 2 # batch failure
      - 5 # batch cancelled
```

# Managing your artifacts

This tutorial will cover the starting point for managing your tests and other data artifacts within ReSim. This is a quick illustration of what those concepts are within the ReSim app, and how you might use them for your robot.

## Problem

When you begin to scale your offline testing, you begin to generate many artifacts in the form of [experiences](../../core-concepts/#experience), versions of code to test, and metrics you want to collect. Often, you end up with many artifacts that are not relevant or compatible, so managing those artifacts becomes a burden. During development, you want to only see the tests that matter to you and your [system](../../core-concepts/#system), and ensure your metrics are managed correctly.

Before you run any tests and waste time and resources doing so, you want to ensure that your [tests](../../core-concepts/#test), [metrics](../../core-concepts/#metrics-build), and [system build](../../core-concepts/#build) are all compatible.

## Examples

This section runs through some examples of the above concepts, applied to an example basic autonomy architecture.

### Systems

In this diagram, you see the sensors feed into a perception module, which feeds down into motion planning and ultimately controls. This is a basic representation of the pieces of an autonomy system, but sometimes these modules are combined or there are additional modules. The above example can be defined as its own System, which represents the end to end modules that make up the on-robot system. In the ReSim Platform, you could define a set of Systems as follows:

**Perception**: A perception system is likely concerned with detecting everything of interest and tracking those objects over time. A perception system is likely to require high fidelity images (either real or simulated), but the metrics you care about are likely to be independent of the motion of the robot. By defining a perception System, you can isolate the tests you run and metrics you generate to only those that apply to a perception system.

**Multi-Module Systems**: You can define a System in any way that is relevant to your robot. For example, the motion planner and controller in a robot system are closely tied together, but separating them for testing can still be useful. So you can define Systems for both individual modules, or the combination together to represent more of the realistic interaction of those modules on the robot.

### Metrics

In this example autonomy stack, we have defined two main systems: the Perception System and the Multi Module (Motion Planning) System. These two systems would have very different metrics. For example, if you are not processing image data, it doesn't make sense to attempt to compute image processing time. To support efficiently running metrics, you could define Metrics Builds for each of these Systems that capture the types of metrics that are appropriate such as:

Perception:

- Classification accuracy
- Precision & recall
- Time to detect

Multi-Module Motion Planning:

- Robot State and Derivatives such as pose, velocity, acceleration of the robot.
- Collision Checking
- Planning metrics associated with the plans produced and safety constraints

# Re-running partial batches

You might have a need to re-run some subset of tests inside a batch. Maybe one experience encountered a transient error. Maybe it has non-deterministic behaviour. But you don't want to re-run all of the tests in a batch, just one or a few problematic ones.

## Limitations

The tests will be re-run as if they were being freshly run now. They do not take a snapshot of what the configuration was when they were first run and re-use that data. This means that if you have updated any of the dependencies, including **system**, **image**, **experience data**, etc - then the test will run with the latest definitions of that data.

When re-running tests, both the experience and metrics stages for that test will be re-run. The metrics stage will run against only the results from the new run; results from previous runs will be ignored.

It is not currently possible to only re-run the metrics stage for a test re-using previously generated experience output.

The Batch Metrics stage will always be re-run. It will reuse the output from any test that did not need to be re-run, combining it with the fresh output from any re-run tests, ignoring old output for anything that was superseded. Batch Metrics may also be re-run in isolation, without needing to re-run any tests within the batch.

## CLI instructions

There are two ways to re-run batches using the CLI.

### Automated reruns using `batches supervise`

The `supervise` command under resim batches is responsible for monitoring a batch run until completion, and then re-runs the subset of failed tests automatically for a set number of times.

#### Configuration

The `supervise` command takes several required and optional parameters. To see full documentation of each, please refer to `resim batches supervise --help`.

### Required parameters

- `project`: the name or ID of the project to supervise
- `max-rerun-attempts`: maximum number of rerun attempts for failed tests (default: 1)
- `rerun-max-failure-percent`: maximum percentage of failed jobs before stopping (1-100, default: 50). This is intended to guard against re-running and wasting compute on a faulty version and only to rerun for flakiness of tests. For example, if more than 50% of tests fail, it is more likely that the code under test is failing vs failure due to non-determinism.
- `rerun-on-states`: Status of test results to trigger automatic reruns on (e.g. Warning, Error, Blocker)

#### Optional parameters

- `batch-id`: the ID of the batch to supervise
- `batch-name`: the name of the batch to supervise. One of `batch-name` or `batch-id` is necessary.
- `poll-every`: interval between checking batch status, expressed in Golang duration string (default: "30s")
- `wait-timeout`: amount of time to wait for a batch to finish, expressed in Golang duration string (default: "1h")

#### Exit codes

The supervise command returns an exit code corresponding to the final batch status:

- `0`: Success
- `1`: Internal error
- `2`: Error
- `5`: Cancelled
- `6`: Timed out

The exit codes can be used on CI pipelines to pass or fail a stage.

#### Example

Bash

```
resim batches supervise \
    --project "my-project" \
    --batch-name "my-batch-name" \
    --max-rerun-attempts 3 \
    --rerun-max-failure-percent 30.0 \
    --rerun-on-states "Error,Warning" \
    --poll-every "1m" \
    --wait-timeout "2h"
```

### Manual re-run using `batches rerun`

If there is a need to partially re-run certain tests to test for flaky failures manually, we can use the `rerun` command.

To re-run one or more tests:

Bash

```
resim batches rerun --batch-id={batch id} --test-ids={test id 1, test id 2, ...}
```

Specify the test IDs (aka Job IDs) from the batch in question. Do not use (eg) experience IDs.

To re-run just the Batch Metrics stage, simply specify no test IDs:

Bash

```
resim batches rerun --batch-id={batch id}
```

Metrics 1.0 only

Parameter sweeps are currently only supported with [Metrics 1.0](../metrics/metrics-1/). If your project uses the current metrics framework (this likely applies to you) sweeps are not yet available.

## Introduction

It is usually the case that robotics, AI, and autonomy applications have a number of configurable parameters that impact their performance. For example, the acceptable minimum distances to maintain to other agents, weightings for a tracker, number of particles in a particle filter. Another common use of parameters is to toggle a prototype feature.

At ReSim, we call these *Build Parameters*, because they impact how a particular version of an autonomy application (encapsulated as a [build](../../core-concepts/#build)) will perform across the experiences that it encounters.

The ReSim platform supports the ability to run what we call a *Build Parameter Sweep* (or *sweep* hereafter) just like you would create a normal test batch. The benefit of this is that one can quickly experiment with different parameterizations of an autonomy application in order to quickly determine candidate optimal values for parameters without the slow (and more qualitative) feedback loop of field testing.

## Build parameter sweep components

A sweep requires:

- A build parameter configuration, which is the specification of the parameters to sweep. See [below](#supported-sweep-types) for more details.
- A build that can override default parameters with the values provided by ReSim at run time. See [below](#build-parameter-sweep-mechanics) for the interface.
- A set of experiences to execute, as either a list of experiences or experience tags (or both).
- A metrics build that will evaluate the performance of the parameterized build on those experiences. See [the metrics overview](../../setup/metrics-builds/) for more details.

For each combination of values from the parameter configuration, the ReSim platform will create a test batch containing all the experiences. The ReSim App provides an experimental [page](https://app.resim.ai/sweeps) to allow one to compare the performance of each experience and the batch as a whole across the parameters.

## Supported sweep types

The ReSim platform currently only supports one type of sweep configuration: a grid search across parameters. With a grid search, one specifies the parameters and their desired values individually and ReSim will generate tests for the cross-product.

For example, a single parameter `my_first_parameter` with *n* values (`value1`, `value2` etc.) will simply generate *n* test batches. However, two parameters, one with *n* and one with *m* values will generate *n x m* test batches: for each combination of the parameters.

## Creating a build parameter sweep

One can currently only create a parameter sweep in the ReSim CLI (as of version `0.1.27`). The syntax is similar to creating a test batch, with the exception of two alternative ways to specify the parameters.

For a one-dimensional grid search, the CLI offers a shortcut:

Bash

```
resim sweeps create \
    --project <PROJECT NAME> \
    --build-id <BUILD UUID> \
    --metrics-build-id <METRICS BUILD UUID> \
    --experience-tags "<TAG 1>, <TAG 2>"
    --parameter-name "my_first_parameter"
    --parameter-values "value1","value2","value3" ...
```

This creates a sweep with three batches, testing out the build with three parameterizations.

For a multi-dimensional grid search, or to allow one to persist the parameters on a local machine, the CLI offers:

Bash

```
resim sweeps create \
  --project <PROJECT NAME> \
  --build-id <BUILD UUID> \
  --metrics-build-id <METRICS BUILD UUID> \
  --experience-tags "<TAG 1>, <TAG 2>"
  --grid-search-config <path to json config file>
```

The syntax for a config file is a JSON list of sweep parameters and their values:

JSON

```
[
  {
    "name": "parameter-1",
    "values": ["parameter-1-value-1", "parameter-1-value-2"]
  },
  {
    "name": "parameter-2",
    "values": ["parameter-2-value-1", "parameter-2-value-2"]
  }
]
```

For more parameters extend the list of parameters, for more values extend each individual values list. To modify a default parameter with only one option, simply have a singleton list, which will not contribute to the growth in the number of tests. Caution: grid searches with many parameters can result in a combinatorial explosion i.e. many more tests than you expected!

## Visualizing batch parameter sweeps

The ReSim web app has an experimental page to help with visualizing sweeps. In particular the page enables:

- For each experience in the sweep: a direct comparison of any scalar metrics (seen as a bar chart) or line chart metrics (e.g. time series) superimposed.
- At the test batch level, a direct comparison of any scalar or line chart batch metrics. For more information about batch metrics, see the [metrics page](../../setup/metrics-builds/)
- A list of the test batches and tests created in the sweep to navigate quickly to the full metrics.

In the near future, we plan to support direct comparisons of all metrics types as well as the ability to create sweeps in the web app.

## Build parameter sweep mechanics

In order to support sweeps, some small modifications are required to the builds that are registered with ReSim.

Now, the `/tmp/resim` directory will, *for all batches*, now contain a file called `parameters.json` which contains the parameters and their values *as strings* that are to be used in place of defaults for this particular test.

The format of `parameters.json` is very simple: a set of fields of `parameter:value`. For example:

JSON

```
{
  "parameter-1": "parameter-1-value",
  "parameter-2": "parameter-2-value"
}
```

If the test batch is not part of a sweep (and parameter overrides are not specified), then this will be an empty JSON file.

Your startup script/entrypoint should, therefore, read this file, parse the parameter values and set arguments or environment variables as required. Since ReSim do not require any further details of how a customer's docker entrypoint operates, we can provide ad-hoc support on parsing this data in a particular application.

## Test suites and parameters

It is not yet directly possible to create a parameter suite from a test suite, but is on the development roadmap.

It is, however, possible and often extremely valuable to run a test suite with specific parameter overrides passed to the build. This means, you can run your test suite with `my-beta-feature : true` selected, for example.

To do this in the ReSim CLI, simply adapt the following command:

Bash

```
resim suites run \
    --test-suite "Perception Nightly Regression"
    --build-id "<UUID of the build>"
    --parameter "my-param : my-value", "my-param-2 : my-value-2"
```

This will create a test batch and pass the parameters in the `parameters.json` file described above.

# ReSim agent

## Introduction

An important part of testing embodied AI is hardware-in-the-loop testing or HiL. This is testing that involves communicating with real hardware. HiL testing complements simulation-based testing and is typically run less frequently such as when changes are merged to a default branch, or before a release.

The ReSim Agent is designed to facilitate HiL testing by integrating with your existing suite of tests, allowing HiL tests to be triggered and the results to be analyzed using the same tools as simulation-based testing.

The ReSim Agent works by polling ReSim for tasks so there is no need to open ports or adjust firewall rules to allow inbound traffic.

## Prerequisites

The ReSim Agent is currently tested against Linux on amd64 and requires Docker (version 27 or later).

Note

The user running the Agent needs permission to use the Docker API. In some environments this means adding the user to the `docker` group.

If you can do `docker run hello-world` you're ready to go.

## Installation

Pre-built binaries are available for linux-amd64, linux-arm64, darwin-arm64 (Mac OS):

For Linux on AMD64:

Text

```
curl -L https://github.com/resim-ai/agent/releases/latest/download/agent-linux-amd64 -o agent
chmod +x agent
```

For Linux on ARM64:

Text

```
curl -L https://github.com/resim-ai/agent/releases/latest/download/agent-linux-arm64 -o agent
chmod +x agent
```

For Mac OS on Apple Silicon/ARM:

Text

```
curl -L https://github.com/resim-ai/agent/releases/latest/download/agent-darwin-arm64 -o agent
chmod +x agent
```

The Agent is open-source so you can inspect the code before you run it.

You can also install the Agent directly using Go:

Bash

```
go install github.com/resim-ai/agent
```

## Configuration

The Agent loads configuration from `~/resim/config.yaml`.

The minimal configuration you need is as follows:

Text

```
name: robot-arm-1
pool-labels:
  - arm
username:
password:
```

The name of the agent will be used to identify which agent instance ran a particular job (although uniqueness is not currently enforced.)

Setting pool-labels as above will configure the agent to request jobs from ReSim that have been put into the pool called `arm`. This is a way to group jobs, for example based on the hardware they require. This value is a list, and the agent will receive jobs with *any* of the configured labels.

Info

Pool labels that begin with the prefix `resim` are reserved, so please ensure your pool labels do not have this prefix.

A username and password for use by your agents will be provided by ReSim.

Run the binary and it will start polling ReSim for work.

By default, the agent logs to stdout and to `~/resim/agent.log`

### Network mode

By default, your containers will be run as part of the default bridge network, which matches the Docker default.

From this network, containers can reach services on the same network (or the internet), but not contact anything else running on the host.

If you need to communicate with services running on the same host, set the following option in the agent configuration file:

Text

```
docker-network-mode: host
```

This is equivalent to `docker run --net=host`.

### Privileged mode

By default, your containers will not be run with elevated privileges, which matches the Docker default.

If you would like your containers to be run with elevated privileges, set the following option in the agent configuration file:

Text

```
privileged: true
```

This is equivalent to `docker run --privileged`.

### Passing mounts and environment variables

It is possible to provide volume or socket mounts and environment variables to your test container by providing additional entries in the config, for example:

Text

```
mounts:
 - /path/on/host:/mounted/path
 ...
environment-variables:
 - HIL=true
 ...
```

### Experience caching

As of v1.1.0, the Agent will keep downloaded experience data on disk, avoiding repeat downloads at the start of each test. The two options of importance and their defaults are:

Text

```
remove-experience-cache: false
experience-cache-dir: /tmp/resim/cache
```

You can set `experience-cache-dir` to a custom path, overriding the default path to ensure the cache folder is kept between reboots of the host system. If `remove-experience-cache` is set to `true`, the Agent will clean up any cached experiences when the agent exits, keeping disk usage down.

### Other options

See the [repo README file](https://github.com/resim-ai/agent) for a full list of configuration options.

## Build images

The ReSim Agent uses the Docker daemon on the host to pull and launch images.

If the image repository requires authentication, make sure the host is authenticated before running the agent by running [`docker login`](https://docs.docker.com/reference/cli/docker/login).

For automatic authentication, the agent supports the [AWS ECR credential helper](https://github.com/awslabs/amazon-ecr-credential-helper/).

To enable its use, ensure credentials/configuration are available in `~/.aws` and that `~/.docker/config.json` is [configured as required](https://github.com/awslabs/amazon-ecr-credential-helper/tree/main?tab=readme-ov-file#docker).

## Launching batches

You can now launch batches to be run by the agent, by specifying a matching pool label:

Text

```
resim batch create --project robot-arm --build-id 4b50f4f9-3b86-49a6-84b2-7813fd4bbcbe --experience-tags hardware-validation --pool-labels arm
```

Note that when creating a batch, the pool labels are used as an `AND`, that is, an agent will only run the batch if it is running with *all* of the labels on the batch.

This is to enable selecting agents with certain capabilities, without having to create unique labels for each permutation.

The status and results of the batch will be presented in ReSim as normal. In the near future, ReSim will highlight when a batch was run on an agent.

### Setting batch priority

When launching a batch, you can control the order in which it will be picked up by an agent by passing the `--priority` flag:

Text

```
resim batch create --project robot-arm --build-id 4b50f4f9-3b86-49a6-84b2-7813fd4bbcbe --experience-tags hardware-validation --pool-labels arm --priority 100
```

The priority is an integer from `0` to `32767`. **Lower values indicate higher priority**, so a batch with `--priority 0` will be picked up before a batch with `--priority 100`. If the flag is omitted, batches use a default priority of `1000`.

This is useful when you have several batches queued for the same pool of agents and you want to ensure that critical or time-sensitive work is run first, while lower-priority work (e.g. nightly regression runs) waits in the queue.

## Designing experiences

The workflow we anticipate is that the agent is running on the hardware to be tested, or that the hardware is accessible over the network from the container that will run your experience. The experience container will send commands to the hardware and receive data back which it will process as required and store in preparation for the metrics step.

Experience input data is currently expected to be built into the image, or accessible via S3.

## Running in the background

Here is an example systemd unit file for running the agent as a systemd service:

INI

```
[Unit]
Description=ReSim Agent
After=network-online.target

[Service]
ExecStart=/usr/local/bin/agent # Or other path as required
Restart=always
User=ubuntu # Replace with the user that will be running the agent

[Install]
WantedBy=multi-user.target
```

# ReSim SDK

## Introduction

ReSim's standard workflow runs your sim (i.e. [build](../../core-concepts/#build)) inside our platform so we can manage execution, collect outputs, and generate metrics. However, if you can't or don't want to dockerize your sim, you can still generate metrics for analysis. ReSim's SDK allows you to run tests outside of ReSim, on your own infrastructure, export the results, and generate metrics for these tests.

There are a few different reasons you may want to run tests on your own infrastructure, outside of ReSim:

- You are just getting started, and want to explore our metrics framework
- You want to run tests on actual hardware, and only use ReSim as a visualization/analysis tool
- Your system is difficult to dockerize, perhaps due to special hardware requirements

This guide walks you through how to use ReSim's Python SDK. You can run your tests wherever you like, emit structured data from each test, then upload the results to ReSim. The tests will appear in the ReSim app exactly like any other batch.

Metrics required

This feature utilizes the metrics framework. It is recommended to familiarize yourself with [Metrics](../metrics/). You need at least a valid metrics config file (`config.resim.yml`) with topics defined before running batches this way. A small, functional example is provided in the Github repo below to get you started.

## Installation

Running batches outside of ReSim is supported via our Python SDK:

Bash

```
pip install resim-sdk
# or, if you're using uv
uv add resim-sdk
```

Note: The resim-sdk package requires Python >= 3.10.

## Quickstart

The following repository contains a working example that creates a batch with a few tests and generates some metrics:

<https://github.com/resim-ai/resim-sdk-example>

The example in this repo will create a small batch, with a couple of tests, and generate some simple metrics.

## API Documentation

Basic usage involves create a `Batch`, with `Test`s inside, and `emit`-ing relevant data under each test.

Python

```
from resim.sdk.batch import Batch
from resim.sdk.test import Test

# `DeviceCodeClient` will trigger an interactive prompt to authenticate.
from resim.sdk.auth.device_code_client import DeviceCodeClient
client = UsernamePasswordClient()

# OR, if you want non-interactive authentication, like for a CI system, use `UsernamePasswordClient`
# instead, and request credentials from ReSim. The username/password below are NOT your personal
# credentials.
# from resim.sdk.auth.username_password_client import UsernamePasswordClient
# client = UsernamePasswordClient(username="*****", password="*****")

# Create a batch, and run the "my metrics" metrics set from your metrics config file
with Batch(
    client=client,
    project_name="my-project",
    branch="metrics-test-branch",
    metrics_set_name="my metrics",
    metrics_config_path="resim/config.resim.yml", # See example git repo for an example config file
) as batch:
    print(f"Created batch {batch.friendly_name} with id {batch.id}")

    # Create a test, and emit some random data for the "position" topic
    with Test(client, batch, "test 1") as test:
        for i in range(10):
            test.emit("position", {"x": float(i), "y": float(i)}, i)

    # When finished the emitted data will be uploaded to ReSim so metrics processing can begin
```

Branches and versions

The Batch constructor accepts a `branch` (required) and an optional `version` string. Common values for `version` include a commit SHA, a semver tag, or a build number.

Using these fields can enable longitudinal analysis, such as:

- How has metric X changed in my branch versus main?
- How has metric Y changed between release versions 2.0 and 3.0?

`branch` is required because ReSim performs a backwards-compatibility check when syncing a new config, ensuring you don't make destructive changes to your topics - for example, changing an string field to a integer could potentially break queries against historical data. For this reason, we recommend experimenting on a branch other than `main` when getting started.

### `Batch` parameters

| Parameter             | Type                  | Description                                                                                             |
| --------------------- | --------------------- | ------------------------------------------------------------------------------------------------------- |
| `client`              | `AuthenticatedClient` | **Required.** Authenticated API client.                                                                 |
| `project_name`        | `str`                 | **Required.** Name of the project to associate this batch with. Provide this or `project_id`, not both. |
| `project_id`          | `str`                 | **Required.** UUID of the project to associate this batch with.                                         |
| `branch`              | `str`                 | **Required.** Branch name to associate this batch with.                                                 |
| `metrics_set_name`    | `str`                 | `None`                                                                                                  |
| `name`                | `str`                 | `None`                                                                                                  |
| `version`             | `str`                 | `None`                                                                                                  |
| `metrics_config_path` | `str`                 | `None`                                                                                                  |
| `system`              | `str`                 | `None`                                                                                                  |
| `test_suite`          | `str`                 | `None`                                                                                                  |
| `test_suite_revision` | `str`                 | `None`                                                                                                  |

### `Test` parameters

| Parameter | Type                  | Description                                |
| --------- | --------------------- | ------------------------------------------ |
| `client`  | `AuthenticatedClient` | **Required.** Authenticated API client.    |
| `batch`   | `Batch`               | **Required.** The parent `Batch` instance. |
| `name`    | `str`                 | **Required.** Name of the test.            |

**Note:** Each test is tied 1-to-1 with an Experience. The name of a test comes from the experience. This means when you do `Test(name="hello world")`, the SDK will get-or-create an experience with that name. This is mainly an implementation detail, but useful to know if you want to associate any tests with pre-existing experiences.

## Emitting data

`test.emit()` writes a data point to the emissions log for that test. Topic names and schemas must match your metrics config file. You do not need to manage the emissions file yourself. The SDK handles creating and uploading it to ReSim automatically. The test object supports all emit methods. The metrics guide will cover these in greater depth.

Python

```
test.emit(topic_name, data, timestamp)
test.emit_series("odom_linear_velocity", {
 "x": [1.0, 1.1, 1.1, 1.2],
 "y": [2.0, 1.9, 1.8, 1.7],
 "z": [3.0, 3.0, 3.0, 3.0]
}, timestamps=[0, 500000000, 1000000000, 1500000000])
test.emit_event({"name": "Goal 1 Reached", ...})
```

For more details on emitting data, see [emitting your data](../metrics/#emitting-your-data)

## Attaching files

Use `test.attach_log()` to upload a file (such as an image, GIF, or other artifact) and associate it with a test result:

Python

```
test.attach_log("run_001.mcap")
```

The file is uploaded to ReSim's storage and registered under the test. You can attach any file type — images, logs, MCAPs, videos, etc.

### Image and Video metrics

For image and video metrics, attaching the file is **required**. Your metric references the image by filename, so the file must be uploaded for the image metric to render:

Python

```
test.attach_log("screen_capture_123.png")
test.emit("my-image-topic", {"img": "screen_capture_123.png"}, time.time_ns())
```

The image's filename in the emission must match the filename passed to `attach_log`. The metrics engine uses this reference to retrieve and render the image.

Here is how the metrics config would look for the above image metric:

YAML

```
topics:
  images:
    schema:
      img: image

metrics:
  images:
    type: test
    query_string: select img from images
    template_type: system
    template: image
```

## Viewing results

Once the `Batch` context exits, and all metrics processing is complete, navigate to your project in the [ReSim app](https://app.resim.ai) to view the batch, its tests, and generated metrics — the same as any batch run inside ReSim.

# MCP server

ReSim's MCP server follows the [authenticated remote MCP](https://modelcontextprotocol.io/) spec, so the server is centrally hosted and managed — no local installation or API keys required. Authentication is handled via OAuth, and compliant clients will prompt you to sign in through your browser on first use. All available tools are read-only queries for browsing projects, batches, test results, metrics, logs, and more.

LLM-friendly docs

Pair the MCP server with our LLM-friendly documentation, published as plain markdown following the [llms.txt convention](https://llmstxt.org):

- [`https://docs.resim.ai/llms.txt`](https://docs.resim.ai/llms.txt) — curated index of the docs
- [`https://docs.resim.ai/llms-full.txt`](https://docs.resim.ai/llms-full.txt) — full corpus as a single file
- Any page is also available as raw markdown by appending `.md` to its URL (e.g. [`/setup/cli.md`](https://docs.resim.ai/setup/cli.md))

Point your agent at these alongside the MCP server for full context on the platform.

## Setup

Add the ReSim MCP server to your client's configuration. The only thing you need is the server URL:

Text

```
https://bff.resim.ai/mcp
```

### Claude Desktop

Open **Settings > Developer > Edit Config** and add the `resim` entry to your `mcpServers`:

JSON

```
{
  "mcpServers": {
    "resim": {
      "url": "https://bff.resim.ai/mcp"
    }
  }
}
```

Restart Claude Desktop. You'll be prompted to authenticate through your browser on first use.

### Cursor

Open **Settings > MCP** and click **+ Add new global MCP server**. Use the following configuration:

JSON

```
{
  "mcpServers": {
    "resim": {
      "url": "https://bff.resim.ai/mcp"
    }
  }
}
```

### Claude Code

Run the following command in your terminal:

Bash

```
claude mcp add --transport http resim https://bff.resim.ai/mcp
```

Claude Code will handle OAuth automatically when the server is first used.

## Rate limiting

The MCP server enforces a rate limit of **60 requests per minute** per user. If you exceed this limit, the server responds with HTTP 429 and a JSON-RPC error. Back off and retry after a short delay.

## Available tools

The MCP server exposes read-only tools organized into the categories below. All list tools support `page_size` and `page_token` for pagination, and most support `order_by` for sorting. Tool arguments like `project_id`, `batch_id`, etc. are UUIDs.

### Projects & branches

| Tool            | Description                                                                      |
| --------------- | -------------------------------------------------------------------------------- |
| `list_projects` | List all projects the user has access to                                         |
| `list_branches` | List branches for a project. Filter by type: `MAIN`, `CHANGE_REQUEST`, `RELEASE` |

### Builds & systems

| Tool           | Description                                                   |
| -------------- | ------------------------------------------------------------- |
| `list_builds`  | List builds for a project. Supports fuzzy name/version search |
| `get_build`    | Get details of a specific build                               |
| `list_systems` | List systems for a project. Filter by name                    |
| `get_system`   | Get details of a specific system                              |

### Experiences & tags

| Tool                   | Description                                                                                                                                                             |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `list_experiences`     | List experiences for a project. Supports free-text search across name, description, and tags, plus structured filters (`tag_id`, `test_suite_id`). Can include archived |
| `get_experience`       | Get details and tags of a specific experience                                                                                                                           |
| `list_experience_tags` | List experience tags for a project. Filter by name                                                                                                                      |

### Test suites

| Tool               | Description                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------- |
| `list_test_suites` | List test suites for a project. Filter by `experience_ids`, `system_id`. Can include archived |
| `get_test_suite`   | Get details of a test suite including its experiences                                         |

### Batches

| Tool                          | Description                                                                                                                                                                  |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `list_batches`                | List batches for a project. Each batch has a `conflatedStatus` (`ERROR`, `BLOCKER`, `WARNING` = failing; `COMPLETE` = passing). Supports fuzzy search and structured filters |
| `get_batch`                   | Get details of a specific batch                                                                                                                                              |
| `list_batch_errors`           | List error details for jobs in a batch. `CUSTOMER_` prefix = user-side issue                                                                                                 |
| `list_batches_for_build`      | List batches for a specific build (requires `branch_id`)                                                                                                                     |
| `list_batches_for_test_suite` | List batches for a specific test suite across all revisions                                                                                                                  |
| `get_batch_comparison`        | Compare two batches — per-experience status diffs                                                                                                                            |
| `get_batch_suggestions`       | Get suggested comparison targets (`latestOnMain`, `lastPassingOnMain`, `latestOnBranch`, etc.)                                                                               |

### Jobs

| Tool        | Description                                                                                                                     |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `list_jobs` | List jobs within a batch. Includes inline `executionErrors` and `conflatedStatus`. Filter by status, experience name, tag UUIDs |
| `get_job`   | Get job details including execution errors and optional AI-generated `log_analysis`                                             |

### Metrics

| Tool                        | Description                                                   |
| --------------------------- | ------------------------------------------------------------- |
| `get_batch_metrics_summary` | Get metrics summary for a batch. Filter by metric name        |
| `get_job_metrics_summary`   | Get metrics summary for a specific job. Filter by metric name |
| `list_job_events`           | List structured events emitted during job execution           |

### Reports

| Tool           | Description                                            |
| -------------- | ------------------------------------------------------ |
| `list_reports` | List reports for a project. Supports fuzzy name search |
| `get_report`   | Get details of a specific report                       |

### Logs

| Tool              | Description                                                                                     |
| ----------------- | ----------------------------------------------------------------------------------------------- |
| `list_batch_logs` | List log file metadata for a batch                                                              |
| `get_batch_log`   | Get a batch log file with presigned download URL                                                |
| `list_job_logs`   | List log file metadata for a job. Filter by type: `CONTAINER_LOG`, `ERROR_LOG`, `EXECUTION_LOG` |
| `get_job_log`     | Get a job log file with presigned download URL                                                  |

### Datalake introspection

| Tool                 | Description                                                                          |
| -------------------- | ------------------------------------------------------------------------------------ |
| `list_topics`        | List available datalake topics and column schemas. Use before writing metric queries |
| `get_topic_schema`   | Get detailed schema for a topic including all metadata columns for joins             |
| `preview_topic_data` | Preview up to 10 sample rows from a topic (may take 10-30s)                          |

# Reeves

Reeves is the AI assistant built into the ReSim web app. It's powered by Claude and is available throughout the UI to help you triage failures, explore your data, draft metrics, and assemble dashboards — without leaving the page you're on.

## Where to find it

Reeves lives in a chat panel inside the ReSim web app. Open it from any page — batches, jobs, dashboards, test suites, project overview — and it picks up the context of what you're looking at automatically. You can ask "why did this batch fail?" or "what does this metric mean?" without restating which batch or metric you mean.

Your conversation is persistent and per-user. History, page context, and any in-flight work (for example, a dashboard being drafted) survive page navigation and reloads, so you can keep working as you click around the app.

## What Reeves can do

### Triage failures

- Summarize errors across a batch and surface common failure modes.
- Pull up the [agentic log summary](../log-summary/) for a failed job and reason over it.
- Compare jobs and batches to narrow down regressions.

### Explore your data

- List and inspect projects, branches, builds, systems, experiences, and test suites.
- Query batches and jobs with status filtering (passing, failing, in-progress, etc.).
- Navigate you directly to the relevant page when that's the fastest answer.

### Draft metrics

- List available metric topics and inspect their schemas.
- Preview sample rows against your real data.
- Co-write SQL with you and preview what the draft query would return before saving.
- Push a pre-filled draft into the metric editor for you to review and save. Reeves never saves metrics on your behalf.
- Edit existing metrics the same way — Reeves opens them in the editor with the proposed changes staged.

### Build dashboards

The dashboard-creation workflow is a multi-phase agent: it researches your project's data, proposes a layout, and assembles the dashboard with appropriate metrics. It pauses at checkpoints for your approval before moving on, so you stay in control of the result.

### Capture team knowledge

Reeves can save org-level notes — project conventions, naming, gotchas it has learned from you — so they're available to everyone on your team in future sessions. You can ask Reeves to update or add a note at any time.

## Scope and limitations

- **No destructive actions.** Reeves proposes metrics, dashboards, and notes as drafts. It does not delete or destructively edit anything on its own.
- **Web app only.** Reeves runs inside the ReSim web app. For read-only programmatic access from your IDE or other tools, use the [MCP server](../mcp/) instead.
- **Time and iteration budgets.** Long-running subagents (dashboard creation, metric drafting) enforce per-phase iteration and wall-clock budgets to prevent runaway loops. If a phase hits its budget, Reeves stops and reports back rather than continuing silently.
- **Best with page context.** Reeves is most useful when you ask it questions from a relevant page — the batch you're triaging, the dashboard you're building, the test suite you're reviewing. Open-ended questions with no page anchor work too, but may need more prompting.

## Feature availability

Reeves is enabled on a per-organization basis. If you don't see the chat panel in the web app, contact [info@resim.ai](mailto:info@resim.ai) to have it enabled for your organization.

# Foxglove primary sites

If you produce large simulation results, you may benefit from streaming them from a Foxglove [Primary Site](https://docs.foxglove.dev/docs/primary-sites/introduction) when visualizing. Once set up, any `.mcap` files saved to the `/tmp/resim/outputs/foxglove` directory at the end of a job will be automatically uploaded to your inbox and processed by your primary site for streaming. Clicking the `Open in Foxglove` button in your job results within the ReSim app will still take you to Foxglove, but will now stream your results. This will mean your visualizations will load faster and play back more smoothly. Your results will still be uploaded for storage with ReSim - they’re safe with us and still available to open from our cloud if you’d like.

Foxglove-hosted Primary Sites

We don't currently support [Foxglove-hosted Primary Sites](https://docs.foxglove.dev/docs/primary-sites/introduction/#foxglove-hosted), but they're on our roadmap. Please get in touch if you're interested in using them.

### Self-hosted primary sites

If you have a [self-hosted primary site](https://docs.foxglove.dev/docs/primary-sites/introduction#self-hosted) in S3, ReSim can upload your MCAPs to your inbox bucket so they’re ready to stream when your simulations complete. To give us access, extend your bucket’s access policy to add the following statement. Make sure you replace the `inbox-bucket-name` placeholder value with the name of your bucket and check whether you are using a generic role or a customer-specific role as described [here](../../setup/resim-data-access/#iam-roles).

JSON

```
"Statement": [
  {
      "Sid": "AllowReSimUploads",
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::083585393365:role/resim-customer-prod"
      },
      "Action": [
          "s3:PutObject"
      ],
      "Resource": [
          "arn:aws:s3:::inbox-bucket-name",
          "arn:aws:s3:::inbox-bucket-name/*"
      ]
  }
]
```

Contact us to let us know what S3 URI your MCAPs should be uploaded to, i.e. `s3://inbox-bucket-name` and your Foxglove organization slug (see the top of your [organization settings page](https://app.foxglove.dev/~/settings/general)). We'll get you set up, and you'll be streaming your results in no time.

# ReSim log ingestion

## Introduction

An important part of testing embodied AI is field testing and gathering examples from real-world deployments. Historically this type of testing requires manual inspection of the generated logs and is decoupled from the simulation testing workflows.

The ReSim platform, however, makes it possible to ingest these logs and compute metrics automatically as part of a ReSim metrics flow. One can then further process these logs to generate replay tests for interesting subsets.

This guide explains how log ingestion works with the ReSim CLI.

## Prerequisites

The ReSim CLI requires that any logs to be ingested have already been uploaded to a cloud bucket and that ReSim has been granted access. This is described in the [setup guide](../../setup/resim-data-access/).

## Core concepts

The log ingestion process in ReSim will:

1. Store the logs as experiences, each with a user-supplied name, tagged as `ingested-via-resim`
1. Create a test batch that will ingest the logs and run a given metrics build on them. The batch will contain a test for every log ingested and will, by default, attempt to also calculate batch metrics on the logs
1. Once completed, the test results will be available to visualize in the usual way, in the results page

## Configuration

The ReSim CLI functionality takes several required and optional parameters. To see full documentation of each, please refer to `resim ingest help`.

### Required parameters

- `project`: the name of the project to ingest the log to (can be omitted if you have used `resim project select` to set your default project)
- `system`: the name of the system that this log belongs to e.g. `full-stack`
- `log`: the name and location of the logs you would like to ingest, specified as `log-name=log-location`. This accepts multiple log name and location pairs to ingest multiple logs. For ingesting large numbers of logs, it is recommended to use the [config file](#yaml-log-specification-file) approach
- `metrics-build-id`: the unique ID for the metrics build to run, which can be found via `resim metrics-builds list` or in the ReSim app.

### Optional (and recommended) parameters

- `branch`: the name of the branch that you would like this log ingestion to be related to (e.g. what branch of your system created it). If no branch name is supplied, it will automatically be ingested under `log-ingest-branch`.
- `version`: what version of your code you would like this log ingestion to be related to (e.g. what release or commit sha). If no version is supplied, then `latest` is used.
- `tags`: a comma-separated list of tags to be added to the ingested log; by default ReSim applies `ingested-via-resim`, but additional metadata tags can be provided here.

### Alternative approaches

#### YAML log specification file

The ReSim CLI supports providing a location to a YAML file with the log names and locations via `--config-file <path-to-file>`. The syntax of the YAML input is:

YAML

```
# logs.yaml
logs:
  - name: "log1"
    location: "s3://bucket/path1"
  - name: "log2"
    location: "s3://bucket/path2"
```

#### Deprecated single-log ingestion

It is possible to use two flags to ingest a single log, rather than the `--log "name=location` syntax:

- `log-name`: the unique name that you would like to give to this log (often a field test id or timestamped)
- `log-location`: a cloud storage path to find the log (currently only `s3` is supported)

### Example

Bash

```
resim ingest \
    --project "my-project" \
    --system "My Drone Stack" \
    --log "Drone-Test-42=s3://my-bucket/prefix" \
    --log "Drone-Test-43=s3://my-bucket/prefix2" \
    --metrics-build-id "<UUID of the metrics build to run>" \
    --branch "main" \
    --version "1.1.0" \
    --tags "field-test,hardware-v2"
```

## Using a custom ingestion build

If you would like to pre-process a log before ingestion and cannot do that within a metrics build (due to wishing to reuse it in non-ingestion settings), the ReSim CLI allows you to run the `resim ingest` command with your own **build**. In this case, you would simply use:

Bash

```
resim ingest \
    --project "my-project" \
    --build-id "<UUID of the build to use>" \
    --log "Drone-Test-42=s3://my-bucket/prefix" \
    --metrics-build-id "<UUID of the metrics build to run>" \
    --tags "field-test,hardware-v2"
```

In this case, one cannot supply the `--system`, `--branch`, or `--version` flags as that information is derived from the build.

# Agentic Log Summary

When a test fails, ReSim automatically generates a log summary to help you understand why. Rather than manually digging through your test container's logs, you get a concise explanation of the test failure mode alongside specific points of interest — each annotated with the file name, line number, and a brief note on its relevance.

## Feature availability

This feature is available under the **Logs** tab for any test with the `error` state.

Log summaries are enabled on a per-organization basis. If you don't see the Log Summary panel for failed jobs, please contact [info@resim.ai](mailto:info@resim.ai) to have it enabled for your organization.

# Open in IDE

When a test fails, you can send the failure context directly to your IDE for AI-assisted debugging. ReSim packages the log summary, relevant log lines, and test identifiers into a prompt and stages it in your IDE's AI chat panel — ready for you to review and submit.

## Supported IDEs

| IDE     | Requirements                                                                                          |
| ------- | ----------------------------------------------------------------------------------------------------- |
| VS Code | [ReSim VS Code extension](https://marketplace.visualstudio.com/items?itemName=resim-ai.resim) v0.1.3+ |
| Cursor  | [ReSim Cursor extension](https://open-vsx.org/extension/resim-ai/resim) v0.1.3+                       |

## Using the feature

1. Navigate to a test with an **Error** status.
1. Click the **Open in IDE** button near the top of the page.
1. Select your IDE from the dropdown menu.
1. Your IDE will open with a pre-filled prompt in the AI chat panel.
1. Review the prompt, then submit it to start investigating the failure.

Note

This prompt is staged as a draft — it won't be sent to the AI assistant until you review and submit it.

### Copy Prompt

If you use an AI tool that isn't listed in the dropdown (Claude Code, Codex, etc) - select **Copy Prompt** from the dropdown instead. This copies the failure context to your clipboard as a ready-to-use prompt that you can paste into any AI assistant.

## Troubleshooting

**I don't see the "Open in IDE" button**

The button only appears on tests in the Error state. This feature is also enabled on a per-organization basis — contact [info@resim.ai](mailto:info@resim.ai) if you don't see it on failed tests in the Error state.

**My IDE didn't open**

For VS Code and Cursor, make sure the ReSim extension is installed and active. If the link expires (it's valid for 30 seconds), click **Open in IDE** again.

# Debug mode

## Introduction

Debugging why a batch has failed by making changes to the configuration and rerunning can be fiddly. To help with debugging in certain situations, ReSim has developed a debug mode which can be used to interact with your image in a live environment.

## Prerequisites

- The [ReSim CLI](https://github.com/resim-ai/api-client) must be installed on your local device
- Your image must include a shell (e.g. `sh` or `bash`) and the `sleep` command.
- You need an experience ID or name
- You also need a batch name or a build ID

## Running a debug session

Run the debug command:

Bash

```
resim debug --project <project-id> --build <build-id> --experience my-test-data
```

In the background, ReSim will launch a new ephemeral environment and run your build image in it. The experience data will be available (read-only) in `/tmp/resim/inputs` as in a standard batch.

If your image is very large, it can take a few minutes to set up the debug environment.

By default, the command that runs in your container will be `sh`, but you can override this by setting the `--command` flag on the `debug` command.

Once the batch is running, you will be dropped into a shell. From this shell, you can run commands as normal to inspect the environment and configuration.

For example, you could:

- check dependencies are installed
- check the input data in `/tmp/resim/inputs` is present and structured correctly
- run entrypoint scripts to see immediate output and debug errors

The debug batch will run for 1 hour or your experience's timeout setting, whichever is longer. After the timeout or when you exit the shell by running `exit` or pressing Ctrl-D, ReSim will terminate the debug batch.

Any files you write to `/tmp/resim/outputs` will be uploaded as in a standard batch for later inspection. Nested output directories are archived by default after the run, except for `/tmp/resim/outputs/logs/`, whose contents remain unarchived.

### Debug mode for multi-container builds

If you have a multi-container build setup, which is described by a docker compose file, you can specify the container you want to bash in using the `--container` flag, which should be the `container_name` parameter that you specify in the compose file for a service.

# Multi-container builds

## Introduction

A truly single-node system is hard to find in the real world. Many robots are made up of multiple compute nodes working together, and even a robot that had just a single node onboard would typically interact with an offboard system to some degree.

The ReSim platform supports running multiple containers together in a single experience, to better simulate your real world systems. This may be useful as a representation of your multi-node robot, or multiple robots working together, or your robot communicating with an offboard cloud service, or similar.

## Docker compose

We support defining your experience's multiple containers via a [docker-compose](https://docs.docker.com/reference/compose-file/) file. You can see an example that we use for testing [here](https://github.com/resim-ai/api-client/blob/main/testing/data/test_build_spec.yaml).

Our execution environment supports a subset of the configuration options of docker-compose. The expectation is that you can pass ReSim the same compose file that you would use for running your system locally on Docker, without any changes. However, not all options in the file will take effect in the ReSim platform. Most unsupported options will be ignored, but a small number that are strictly incompatible will surface as errors that will prevent your batch from running.

### Volumes

We support the definition of named volumes in your docker-compose file. Each volume specified in this way will be created ephemerally for the duration of your running experience. It is available to mount as read/write into any or all services in your experience as per normal docker-compose behavior.

In addition to any named volumes, the standard [/tmp/resim/](../../setup/build-images/#inputs-and-outputs) input/output directories will be automatically mounted into every container.

### Networks

Custom networks are not supported. All services will be run in a single default network. Your experience will not expose any ports to the outside world. Each service that listens on a port must listen on a unique port, as containers may share the same host and port space, and thus conflict.

### Services

Each service is run as a separate container. Services may use the same image, or different images. See [build image setup](../../setup/build-images/) for more information on how to create a build image for your service and make it available to ReSim.

#### Startup dependencies

Unlike Docker, `depends_on` does not control start up ordering. If your containers have startup dependencies they must fail healthchecks (or exit) and expect to be restarted continuously until their dependencies are running and they can start up correctly.

See [Container Lifecycle](#container-lifecycle) below for how `depends_on` works with container restarts.

#### Restart

See [Container Lifecycle](#container-lifecycle) below for `restart`.

#### Resources

Service resource requests are supported. The resource requests of all containers must sum up to less than the total amount of resources requested in your [system](../../setup/systems/). If per-container resource requests are not specified, your containers will share the available resources without any resource guarantees - they may, for example, be OOM-killed.

You can request GPUs for a given service.

YAML

```
services:
  my-sim:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              count: 1
```

Services will not share GPUs; each service reserving a GPU will receive the requested number of dedicated GPUs.

#### Other

Other compose service blocks that should function as expected:

- `entrypoint` / `command`
- `workingdir`
- `healthcheck`

### Environment variables

Environment variables can be passed in from your compose file. They may also be set at the experience level. Environment variables set at the experience level will take precedence over those set in your compose file. Environment variables set at the experience level will be set across all services.

### Profiles

[Profiles](https://docs.docker.com/reference/compose-file/profiles/) can be used for services. The specific profile to use can be set at the experience level.

Bash

```
resim experiences create [...] --profile {name}
```

## Container lifecycle

When testing a multi-node system, it's typical to have a number of nodes representing the various parts of the system, plus a single node that actually controls the running of the test, often referred to as the "test orchestrator". Nodes representing parts of a robot, for example, may not have things like "graceful shutdown" handling. So if we wanted to wait for all containers to exit to finish a test, that might not be possible.

Our system focuses on the "test orchestrator" for lifecycle management, such that you do not need to add complex startup or shutdown logic to other containers just for test purposes.

At least one container **must** have a label of `resim.ai/isTestOrchestrator` = `true`.

When monitoring a test, a non-zero exit code in any orchestrator container will fail the test. When all orchestrator containers have exited with a zero exit code, the test is considered successful.

For non-orchestrator containers, their lifecycle is handled in two phases.

Initially, if a container `depends_on` another container, any failure of that container will be ignored until after their dependency is satisfied - when the other container is running.

After a container has satisfied dependencies (if any), any failure of that container will fail the test, **unless** it has `restart` defined, in which case it will happily restart without affecting the test. An orchestrator will not restart.

Non-orchestrator containers may exit successfully at any time.

**Summary of capabilities**

|                                      | Orchestrator Containers                             | Non-Orchestrator Containers                         |
| ------------------------------------ | --------------------------------------------------- | --------------------------------------------------- |
| `depends_on`                         | Exits are ignored before dependencies are satisfied | Exits are ignored before dependencies are satisfied |
| exit code non-zero with `restart`    | Not possible                                        | Container restarts                                  |
| exit code non-zero without `restart` | Fails the test                                      | Fails the test                                      |
| exit code zero                       | Succeeds the test when all exit                     | Ignored, not restarted                              |

### Termination on failure

When the test enters a failed state due to an inappropriate container exit or timeout, any other running containers will be terminated.

## Runtime considerations

### Container logs

Logs from each container will be available in the ReSim web app, as `experience-CONTAINERNAME-container.log`

### Container state

Resource metrics (CPU, memory) are automatically collected for each container. They can be viewed in the ReSim web app.

### Cross-container communication

Your containers will be able to communicate with each other over the default network, by using the service name as the hostname. For example, if you have a service defined as `my-service`, you can communicate with port `8080` on that container with by connecting to `my-service:8080`.

## Creating a multi-container build

Once you have a compose file created, it is as simple as passing it to `resim builds create` in place of where you would use `---image` to pass in a single image URI:

Bash

```
resim builds create --branch=my-branch --system=my-system --version=1.0.0 --build-spec=my-docker-compose.yml
```

Once created, you can define and run test suites and batches as normal using this build ID. The ReSim web app will show the full compose file for each experience defined this way.

### Direct API usage

If you use the API directly, the compose file is passed in as the `buildSpecification` field as part of the `createBuildForSystem` or `createBuildForBranch` inputs. It can be passed in as either YAML or JSON string content.

## Helpful tips

### Image tags

Every image in the compose file should be fully specified with the tag for the desired version of that image to be pulled and run. Make sure the compose file specifies these, they are not parameterized later.

In some testing scenarios, you may not need all images in the environment to be freshly built for each test, and you might find it more efficient to use different tags across images, and only change some of the image tags on each test.

YAML

```
services:
  onboard-node-a:
    image: my-image-a:test-build-1-sha
  onboard-node-b:
    image: my-image-b:test-build-1-sha
  cloud-service-node:
    image: my-image-c:stable-v1
```

## Archive

The **Archive** feature in ReSim enables soft deletion of artifacts such as [Experiences](../../core-concepts/#experience) and [Test Suites](../../core-concepts/#test-suite), allowing users to manage their workspace without permanently losing data. Archiving helps reduce clutter and enforce data hygiene, especially as autonomy development scales, while preserving traceability and compatibility metadata.

Archiving is *non-destructive* — the data remains available for inspection and can be restored at any time. However, archived items are excluded from default workflows and cannot be used in most UI-based actions.

Test results (Batch/Test details, Reports, etc) will continue to reflect their association with any archived entity. However these entities will no longer appear in experience lists in either the CLI or web UI and cannot be selected in the UI for running tests. They can still be invoked directly for experimentation or validation through the CLI.

Archived test suites and experiences retain their associated [System](../../core-concepts/#system) compatibility.

### Archiving experiences

Archiving an **Experience** removes it from active use and visibility in the ReSim platform’s UI and default CLI queries. Upon archiving, the experience is automatically removed from all [Test Suites](../../core-concepts/#test-suite) it belongs to and results in a new version of the test suite. Restoration does *not* re-add it to these test suites — this must be done manually. Since active experiences must have unique names, restoring an archived experience with a duplicate name will result in a numeric suffix being added to the restored experience (e.g. `Experience Name (1)`).

### Archiving test suites

Archiving a **Test Suite** disables its use in future workflows but retains all historical versions and associations. If a Test Suite is linked to an **Executive Dashboard** on the Overview tab, the UI prevents archiving.

# Experience caching

Fetching [experience data](../../setup/adding-experiences/) from cloud storage can potentially involve a large amount of data transfer depending on the size of the experience. In a lot of cases the experience data either doesn't change or at least doesn't change very often. Thus it is not particularly efficient or cost-effective to pull identical experience data for every test that requires it. For this reason we have introduced the experience caching feature.

## How does it work?

When we run your workloads, they are supervised by a worker. That worker does jobs like pulling the experience data, collecting container metrics and collating and storing your logs and metrics. For the experience data, we mount a shared storage location that is unique and isolated to your organization and create a directory with the ID of the target experience. We then sync the contents of your experience location with that directory. That directory is then mounted in your workload container(s) at `/tmp/resim/inputs` in the case of your experience build and `/tmp/resim/inputs/experience` for metrics builds.

When we do this for the first time, we record a timestamp for when it was last updated. For subsequent uses of that experience, when we come to the sync job, we check each file in the cloud storage location's `lastModified` timestamp against our timestamp and only update or download files that are newer than that. Any files that do not exist in the cloud storage location are then deleted from our cache as well.

This means we're pulling the minimum subset of data every time, but ensuring we check the status of every file individually for changes.

## What do I need to do?

Absolutely nothing. We'll let you know if this feature is available to you, but please reach out if you need more info.

## What do I need to know?

### The experience data is read-only

Because we share the experience data with potentially concurrent running jobs, we have to ensure its integrity. Thus the directory is mounted read-only. To clarify, in an experience job, the `/tmp/resim/inputs` directory is read-only, and in a metrics job, the `/tmp/resim/inputs/experience` directory is read-only.

### How do I invalidate the cache?

The easiest way to ensure the cached copy of a particular experience is pulled fresh is to "touch" (i.e. modify in some way) every file in the remote location. As we check the `lastModified` timestamp for every file individually, this will ensure it is definitely newer than the cached copy.

It should be noted that this shouldn't be necessary under normal circumstances, only if you experience any unexpected behaviour in your tests.

If you've done this and you're still experiencing unexpected behaviour, reach out to us and we'll help.

# Asset caching

Fetching [asset data](../../core-concepts/#asset) from cloud storage can involve a large amount of data transfer, especially for assets like HD maps, 3D meshes, or ML model weights. In most cases asset data changes infrequently, often much less frequently than your code. To avoid redundant downloads, assets are cached by default.

This works in the same way as [experience caching](../experience-caching/), so if you are already familiar with that feature, the same principles apply.

## How does it work?

When we run your workloads, the ReSim worker mounts a shared storage location that is unique and isolated to your organization. For each asset, we create a directory identified by the asset ID and revision. We then sync the contents of your asset's cloud storage location into that directory, which is mounted read-only in your workload container(s) at `/tmp/resim/assets/<mountFolder>`.

On the first sync, all files are downloaded and we record a timestamp. On subsequent uses of the same asset revision:

1. We check each file in the cloud storage location's `lastModified` timestamp against our recorded timestamp and the file size.
1. Only files that do not match.
1. Files that no longer exist in the cloud storage location are removed from the cache.

This means we always pull the minimum subset of data while ensuring every file is individually checked for changes.

## Cache-exempt assets

Some assets change frequently enough that caching is counterproductive. When creating an asset, you can mark it as **cache-exempt**:

Bash

```
resim assets create \
    --project "my-project" \
    --name "Rapidly Changing Data" \
    --description "Data that updates every run" \
    --locations "s3://my-bucket/volatile-data/" \
    --mount-folder "volatile" \
    --version "1.0.0" \
    --cache-exempt
```

Cache-exempt assets are always fetched fresh from cloud storage, bypassing the caching mechanism entirely.

## Asset data is read-only

Because cached asset data is shared across potentially concurrent running jobs, we must ensure its integrity. The `/tmp/resim/assets/<mountFolder>` directory is mounted **read-only**.

If your workload needs to modify asset data at runtime, copy the relevant files to a writable location (such as `/tmp/resim/outputs/` or any other writable directory) before modifying them.

## How do I invalidate the cache?

The easiest way to ensure a cached copy of a particular asset is pulled fresh is to "touch" (i.e. modify in some way) every file in the remote location. Since we check the `lastModified` timestamp for every file individually, this ensures it is definitely newer than the cached copy.

Under normal circumstances this should not be necessary as the sync mechanism handles changes automatically. If you have touched the files and are still experiencing unexpected behavior, reach out to us and we'll help.

# Experience syncing

For larger projects, it's often convenient to store a registry of [experiences](/core-concepts/#experience) and their properties locally for a number of reasons. For example:

- Other tools besides ReSim may leverage this configuration to run local (or HiL) sims or manage test sets.
- It may be useful for additional organization-specific metadata to be attached to experiences, even if said data isn't as relevant within the ReSim app.
- Experience history can be stored in version control alongside code which can be useful when inspecting historical sim behavior.

In order support these sorts of use cases, and many more, ReSim supports *Experience Syncing*. We define a schema which contains a list of experiences, their tags, systems, and test suites. Users can then produce config files matching that schema in a number of ways (more on this below). The key to making this config useful, however, is the [ReSim CLI](https://github.com/resim-ai/api-client), which can be used to mirror the config state to the ReSim project of your choice. This makes it very easy for users to make bulk programmatic updates to experiences (e.g. renaming, switching s3 paths, retagging) without having to manually navigate all the api endpoints needed for the specific category of update they're trying to accomplish.

## A simple syncing example

The config schema is relatively simple and looks like this in practice:

YAML

```
experiences:
    - name: scenario-survey-alpha
      description: Aerial survey over test zone
      locations:
        - s3://drone-missions/surveys/alpha-test-zone
      tags:
        - regression
        - my_custom_tag

    - name: new-scenario-system-check
      description: Regression validation run
      locations:
        - s3://drone-missions/system-checks/regression-1
      tags:
        - progression
      systems:
        - demo_system
      customFields:
        - name: weather
          type: text
          values:
            - cloudy
            - rainy
        - name: version id
          type: number
          values:
            - 1.23

managedTestSuites:
    - name: Basic Suite
      experiences:
        - scenario-survey-alpha
        - new-scenario-system-check

managedExperienceTags:
    - regression
    - progression
```

The full schema for this file is provided below in the [Appendix](#appendix-config-schema). This config can be mirrored onto an *existing* ReSim project like so:

Warning

This command should be run with care. It *will* affect other users of whatever project you run it on, so it's advisable to set up a test project if you're just testing out the tool. You can do this using the `resim projects create` command in the CLI.

Bash

```
resim experiences sync \
    --project <project-name> \
    --experiences-config <config/file/path.yaml>
```

Let's walk through what this command will do. We hope that this is relatively intuitive, but there are a lot of edge cases so we break it down in detail:

1. The experiences in the project will be updated so that every experience in the config will exist with the specified parameters. Existing experiences will be updated and new experiences will be created. Experiences that *aren't* in the config will be archived, and any archived experiences that *are* in the config will be restored as they are updated. Experiences are typically identified in the config by unique name, but their `experience_id` can also be provided as a field to enable renaming of existing experiences so long as doing so doesn't violate the uniqueness constraint on names. Note that swapping or permuting names is not currently allowed even if it preserves uniqueness because computing a valid ordering for the updates would be complex.
1. The experiences tags are updated in the following manner: All experience tags listed for each experience are added to that experience if they don't already exist on that experience. All managed experience tags that aren't listed on each experience are removed from that experience if it does currently have them. The managed experience tags are the only tags that are ever removed from any experience. None of these changes apply to experiences we're archiving. **All of the tags listed must already exist in the project**.
1. The systems are updated similarly to the experience tags, but they're always additive. We never remove systems from experiences. **All of the systems listed must already exist in the project**.
1. All test suites in the `managed_test_suites` are revised to contain exactly the experiences listed in the config by their (possibly new) names. Other test suites may be revised, but this will only happen if experiences they currently contain get archived according to the earlier rules, forcing a revision. **All of the test suites listed must already exist in the project**.

## Where should I get a config?

If you'd like to use this feature, there are a number of ways to obtain a useful config. First, you can just write one manually. That's probably not feasible for large projects however. In practice, you will likely want to generate your config using python or another language of your choice. The source data for this could be:

- A directory containing all of your experiences. In this case, your config-generating script can walk through the directory parsing each experience to compute the right tags and systems for it, and then write out the config to a yaml file for you.
- Another config. If you have another config describing your tests, the translation into the ReSim config should be a pretty straightforward script to convert one schema to another.
- An s3 bucket containing your experiences. This is similar to the first option, although you'll likely be using `boto3` to get the information you need. As outlined in the [experience docs](/setup/adding-experiences/), experiences in s3 buckets work very well in ReSim.
- An existing project with your experiences already in them. For this see the `--clone` option below.

## Other options

Some other useful options bear mentioning:

### The `--update-config` flag

When syncing a config to the app, you can update the provided config to include the `experience_id`s and other information for the existing and new experiences by passing the Boolean `--update-config` flag when calling `resim experiences sync`. This is useful if you want to check the config into version control and rename experiences over time.

### The `--clone` flag

Similarly, you can also clone the existing state of the experiences in the app to the passed-in config using the Boolean `--clone` flag. E.g.

Bash

```
resim experiences sync \
    --project <project-name> \
    --experiences-config <new/cloned/config/path.yaml> \
    --clone
```

If the config path exists already, the experiences field in it will be replaced while the managed test suites and tags are unaffected. This is useful if you want to clone the experiences from an existing project to add them to another, or simply as a convenient way to get a list of all experiences with their tags and systems.

Note

This currently only populates the experiences. It does not try to determine which experience tags you'd like to be `managed_experience_tags` or which test suites you'd like to be in `managed_test_suites`. Furthermore, if the config exists and has the `managed_test_suites` field, the test suites therein will not be updated to reflect their current membership. This is because we don't currently have to fetch the current test suite membership during the normal sync operation, so we'd have to add it specifically for this. This is a feature we hope to add in the near future.

### The `--no-archive` flag

You may not necessarily want to archive experiences that aren't listed in the config. This can be useful when batch adding a large number of experiences or when you don't generate an exhaustive list of them when generating the config. In this case, the `--no-archive` flag can be used to prevent unlisted experiences from being archived as they otherwise would. Note that such experiences will still be removed from managed test suites if they aren't included in the `experiences` field of such test suites.

## Appendix: Config schema

### ExperienceSyncConfig

| Field                 | Type                              | Notes    |
| --------------------- | --------------------------------- | -------- |
| experiences           | list\[[Experience](#experience)\] | optional |
| managedTestSuites     | list\[[TestSuite](#testsuite)\]   | optional |
| managedExperienceTags | list[string]                      | optional |

### Experience

| Field                   | Type                                | Notes                                |
| ----------------------- | ----------------------------------- | ------------------------------------ |
| name                    | string                              | **required, must be unique**         |
| description             | string                              | **required**                         |
| locations               | list[string]                        | optional                             |
| tags                    | list[string]                        | optional                             |
| systems                 | list[string]                        | optional                             |
| profile                 | string                              | optional                             |
| experienceID            | string                              | optional, must be unique if provided |
| environmentVariables    | list[object]                        | optional (fields: `name`, `value`)   |
| customFields            | list\[[CustomField](#customfield)\] | optional                             |
| cacheExempt             | bool                                | optional                             |
| containerTimeoutSeconds | int                                 | optional                             |

______________________________________________________________________

### CustomField

| Field  | Type         | Notes                                                   |
| ------ | ------------ | ------------------------------------------------------- |
| name   | string       | **required**                                            |
| type   | string       | **required** (`text`, `number`, `timestamp`, or `json`) |
| values | list[string] | optional                                                |

______________________________________________________________________

### TestSuite

| Field       | Type         | Notes    |
| ----------- | ------------ | -------- |
| name        | string       | optional |
| experiences | list[string] | optional |

______________________________________________________________________

# Custom fields

Experiences support adding **custom fields**. Custom fields are similar to tags, but allow specifying both keys and values. This allows you to add flexible metadata to your experiences. These can be used for organizational purposes when launching batches or managing your test suites. For example, you could add a custom field for "version", along with a version number, to experiences. You could then launch a batch containing all experiences where version is greater than 10.

In general, we recommend working with custom fields via [experience syncing](../experience-syncing/), as it prevents the need to maintain the fields via manually-run CLI commands.

Custom fields support various **types** of data:

- text
- numbers, e.g. 2, 4, or 3.283
- timestamps e.g. 2025-10-01T12:24:00Z
- raw JSON e.g. {"hello": "world"}

Types are supported in order to allow for more advanced filtering. For example, numbers and timestamps can be filtered using greater than/less than operators, in addition to the usual equals/not equals. When adding a custom field you will specify the type as well.

**Note:** JSON fields do not support filtering. These are mainly useful in the context of tests or metrics builds, where you could store arbitrary metadata on the experience and use it at runtime.

## Adding custom fields to experiences

You can add custom fields via our CLI or when using [experience syncing](../experience-syncing/). We do not support adding custom fields via our webapp yet.

You can add a Custom Field to an experience when creating or editing an experience:

Bash

```
resim experiences update \
    --experience 2fa0453f-749f-4803-bfcc-91690a831c12 \
    --custom-field weather=sunny \
    --custom-field weather=hot \ # a field can have multiple values
    --custom-field age=23 \
    --custom-field "run time=2025-06-01T12:23:45Z" \ # must be a RFC3339 formatted timestamp
    --custom-field 'raw json={"foo": "bar"}'
```

You will see the Custom Fields listed under the relevant experience in the webapp.

Custom fields can have a type of **text, number, timestamp**, or **JSON**. The specific type changes the filtering options available to you in the UI, see the filtering section for more details.

In certain cases, you may want to explicitly mark the type of a custom field, instead of letting the CLI infer it for you. You can do this by using a colon : in the custom field:

Bash

```
resim experiences update \
    --experience 2fa0453f-749f-4803-bfcc-91690a831c12 \
    --custom-field weather:text=cloudy \
    --custom-field version:number=239.1 \
    --custom-field import_time:timestamp=2025-10-05T12:15:00Z
```

## Removing custom fields from experiences

When updating custom fields on an experience, it replaces all existing custom fields. To remove a field, simple leave it out when performing an update:

Bash

```
resim experiences update \
    --experience 2fa0453f-749f-4803-bfcc-91690a831c12 \
    --custom-field weather=sunny \
    --custom-field weather=hot \
    --custom-field age=23 # "run time" and "raw json" fields will be removed
```

## Filtering

You can filter experiences by their custom values on the Experiences page, as well as when launching a batch or creating/editing a Test Suite.

**number** and **timestamp** fields support additional filtering options:

| Type      | Supported filters                    |
| --------- | ------------------------------------ |
| text      | =, !=                                |
| number    | =, !=, \<, >                         |
| timestamp | =, !=, \<, >                         |
| json      | none, json values are not filterable |

# Workflows

## Introduction

ReSim workflows provide a powerful way to organize and manage collections of test suites for CI/CD pipelines. Instead of managing individual test suites separately, workflows allow you to group related test suites together and run them as a single unit.

### Why use workflows?

ReSim is frequently used with CI/CD platforms like GitHub Actions and GitLab CI to trigger test suites on pull requests, releases, and scheduled runs (e.g., nightly tests). While you can trigger individual test suites directly, this approach has limitations:

- **Configuration Management**: Adding new test suites or temporarily disabling tests requires updating CI workflow files and pushing changes
- **Complexity**: Managing multiple test suite triggers across different CI events becomes unwieldy
- **Flexibility**: Making runtime changes to test configurations requires code changes

### How workflows help

Workflows solve these problems by providing:

- **Named Collections**: Group related test suites (e.g., "nightly", "regression", "smoke tests") under meaningful names
- **Runtime Configuration**: Enable/disable test suites or add new ones through the ReSim UI without code changes
- **Simplified CI**: Trigger entire test collections with a single command
- **Flexible Management**: Update test suite configurations independently of your CI pipeline

### Example use case

Consider a nightly CI job that runs progression, regression, and smoke tests. Instead of managing three separate test suite triggers, you can:

1. Create a "Nightly Tests" workflow containing all three test suites
1. Configure your CI to run: `resim workflows runs create --workflow "Nightly Tests"`
1. Later, add new test suites or disable progression tests entirely through the ReSim UI
1. Your CI pipeline remains unchanged while test configurations evolve

## Workflow management

ReSim provides comprehensive CLI commands for managing workflows. You can create, update, list, and retrieve workflows, as well as manage workflow runs.

### Creating workflows

Create a new workflow with the `create` command:

Bash

```
resim workflows create \
  --project "my-project" \
  --name "My Nightly Workflow" \
  --description "Nightly regression and smoke tests" \
  --ci-link "https://github.com/myorg/myrepo/actions/workflows/nightly.yml" \
  --suites '[{"testSuite": "suite-uuid-1", "enabled": true}, {"testSuite": "suite-uuid-2", "enabled": false}]'
```

**Required Parameters:**

- `--project`: The name or ID of the project to create the workflow in
- `--name`: The name of the workflow
- `--description`: A description of the workflow
- `--suites` OR `--suites-file`: JSON array of test suites (**exactly one required**)

**Optional Parameters:**

- `--ci-link`: A link to the CI workflow (e.g., GitHub Actions URL)

**Test Suite Configuration:**

You can specify test suites in two ways:

1. **Inline JSON** (using `--suites`):

   Bash

   ```
   --suites '[{"testSuite": "suite-uuid-1", "enabled": true}, {"testSuite": "My Test Suite", "enabled": false}]'
   ```

**Note:** You can use either test suite UUIDs or names for the `testSuite` field. Names are often more readable and maintainable.

1. **JSON File** (using `--suites-file`):

   Bash

   ```
   --suites-file ./workflow-suites.json
   ```

The JSON format supports both UUIDs and test suite names for the `testSuite` field. Test suite names are unique identifiers, so you can use either the UUID or the human-readable name.

### Listing workflows

List all workflows in a project:

Bash

```
resim workflows list --project "my-project"
```

This command returns a summary of each workflow including:

- Workflow ID and name
- Description
- CI workflow link (if set)
- Associated test suites with their enabled/disabled status

### Getting workflow details

Retrieve detailed information about a specific workflow:

Bash

```
resim workflows get --project "my-project" --workflow "My Nightly Workflow"
```

You can specify the workflow by either name or UUID.

### Updating workflows

Update an existing workflow:

Bash

```
resim workflows update \
  --project "my-project" \
  --workflow "My Nightly Workflow" \
  --name "Updated Nightly Workflow" \
  --description "Updated description" \
  --ci-link "https://new-ci-link.com"
```

**Optional Parameters:**

- `--name`: New name for the workflow
- `--description`: New description
- `--ci-link`: New CI workflow link
- `--suites` OR `--suites-file`: Replace all test suites (see create command for format)

**Note:** You must provide at least one update parameter. The `--suites` and `--suites-file` flags are mutually exclusive.

## Workflow runs

Once you have a workflow defined, you can execute it to run all enabled test suites.

### Creating workflow runs

Execute a workflow:

Bash

```
resim workflows runs create \
  --project "my-project" \
  --workflow "My Nightly Workflow" \
  --build-id "build-uuid-here" \
  --parameter "env=production" \
  --parameter "debug=false" \
  --pool-labels "gpu" \
  --pool-labels "high-memory" \
  --account "ci-user" \
  --allowable-failure-percent 5
```

**Required Parameters:**

- `--project`: The name or ID of the project
- `--workflow`: The name or ID of the workflow to run
- `--build-id`: The ID of the build to use in this workflow run

**Optional Parameters:**

- `--parameter`: Parameter overrides (format: `name=value` or `name:value`)
- `--pool-labels`: Pool labels to determine execution environment (logical AND)
- `--account`: Username for CI/CD platform account association
- `--allowable-failure-percent`: Maximum percentage of test failures allowed. If 100, metrics will run on tests, even if the tests have failed. (0-100, default: 0)

**Parameter Format:** Parameters can be specified multiple times or comma-separated:

Bash

```
--parameter "env=production,debug=false"
# or
--parameter "env=production" --parameter "debug=false"
```

### Listing workflow runs

List all runs for a specific workflow:

Bash

```
resim workflows runs list --project "my-project" --workflow "My Nightly Workflow"
```

### Getting workflow run details

Retrieve details about a specific workflow run:

Bash

```
resim workflows runs get \
  --project "my-project" \
  --workflow "My Nightly Workflow" \
  --run-id "run-uuid-here"
```

This command returns information about the workflow run and lists all associated test suite runs.

## Examples

### Example 1: Creating a nightly test workflow

Bash

```
# Create a workflow for nightly testing
resim workflows create \
  --project "my-robot-project" \
  --name "Nightly Tests" \
  --description "Comprehensive nightly test suite including regression and smoke tests" \
  --ci-link "https://github.com/myorg/robot-repo/actions/workflows/nightly.yml" \
  --suites-file ./nightly-suites.json
```

Where `nightly-suites.json` contains:

JSON

```
[
  {"testSuite": "regression-tests", "enabled": true},
  {"testSuite": "smoke-tests", "enabled": true},
  {"testSuite": "performance-tests", "enabled": false}
]
```

**Note:** This example uses test suite names instead of UUIDs, which are often more readable and easier to maintain.

### Example 2: Running a workflow in CI

Bash

```
# In your CI pipeline
resim workflows runs create \
  --project "my-robot-project" \
  --workflow "Nightly Tests" \
  --build-id "$BUILD_ID" \
  --parameter "test_environment=staging" \
  --pool-labels "gpu" \
  --allowable-failure-percent 10
```

### Example 3: Updating workflow configuration

Bash

```
# Disable performance tests and update description
resim workflows update \
  --project "my-robot-project" \
  --workflow "Nightly Tests" \
  --description "Nightly regression and smoke tests (performance tests moved to weekly)" \
  --suites '[{"testSuite": "regression-tests", "enabled": true}, {"testSuite": "smoke-tests", "enabled": true}]'
```

## Best practices

### Workflow organization

- **Use Descriptive Names**: Choose workflow names that clearly indicate their purpose (e.g., "Nightly Regression", "PR Smoke Tests", "Release Validation")
- **Group Related Tests**: Organize test suites logically by functionality, environment, or execution frequency
- **Document with Descriptions**: Provide clear descriptions explaining what each workflow tests and when it should be used

### CI/CD integration

- **Link CI Workflows**: Use the `--ci-link` parameter to connect ReSim workflows with your CI pipeline for easy navigation
- **Use Environment Variables**: Leverage CI environment variables for dynamic parameters like build IDs and test environments
- **Set Appropriate Failure Thresholds**: Use `--allowable-failure-percent` to handle expected test failures in non-critical workflows

### Test suite management

- **Start with Core Tests**: Begin with essential test suites enabled, then add optional tests as needed
- **Use Test Suite Names**: Prefer test suite names over UUIDs for better readability and maintainability
- **Use JSON Files**: For complex configurations, use `--suites-file` instead of inline JSON for better maintainability
- **Regular Review**: Periodically review and update workflow configurations to ensure they remain relevant

### Parameter management

- **Consistent Naming**: Use consistent parameter naming conventions across your workflows
- **Environment-Specific Values**: Use parameters to customize test behavior for different environments
- **Document Parameters**: Keep track of what parameters each workflow expects and their valid values

### Monitoring and maintenance

- **Track Workflow Runs**: Regularly review workflow run results to identify patterns and issues
- **Update Descriptions**: Keep workflow descriptions current as test suites evolve
- **Archive Unused Workflows**: Remove or archive workflows that are no longer needed to keep your project organized

# Optimizing build images

## Introduction

Running your tests at scale in ReSim involves running the image as quickly as possible on multiple cloud instances.

This document is intended to explain the way images are handled in ReSim and to assist customers in optimising their images. Spending some time to make your images smaller will save time and money, not only when running tests in ReSim but also when storing and distributing images in your own environments.

N.B.: This document refers to single images, however note that if you are using our [multi-container builds](../multi-container-builds/) feature, images are processed in the same way.

### ReSim's image handling architecture

When you register a build image with ReSim, our Mirror service will pull the image from your registry and store it in ReSim's platform. Then once a batch is launched with that build image, the Mirror service checks it has an up-to-date copy of the image, and the ReSim platform launches cloud compute **instances** to run the image with the experiences configured in the batch.

This means that your image is transferred from one place to another at the following times:

1. When a batch is launched, the Mirror service checks that it has an image in its storage that matches the tag and digest. If not, it authenticates and pulls the image.

   - If the Mirror service determines that it needs to update its copy of the image, it transfers only the layers of the image that have changed. This indicates a significant potential optimisation, discussed below (see [Order layers](#order-layers) below).
   - If your image is hosted in a registry other than AWS ECR in `us-east-1`, for example in AWS ECR in `us-west-2` or Google Artifact Registry, this will incur egress costs for you according to the size of the image, meaning that reducing the size of your image will save time and money here. This cost is also potentially incurred in other cases, for example when you transfer images into hardware in the field.

1. When tests are running, the instances running the test each pull the image from our mirror service.

   - The number of instances we launch is carefully optimized - once you are running more than a small number of tests in a batch, we launch fewer instances and re-use them for tests in the same batch. This means that the image is pulled a number of times equal to the number of instances launched, and possibly not equal to the number of tests. Because pulling and extracting images - even from our Mirror service which is closely co-located with test instances - takes time, smaller images will reduce the time it takes for your batches to run.

```
architecture-beta
    group customer_cloud(cloud)[Customer Cloud]

    service customer_registry(disk)[Image Registry] in customer_cloud


    group resim(cloud)[ReSim]
    service mirror(disk)[Image Mirror] in resim
    service instance1(server)[Instance] in resim
    service instance2(server)[Instance] in resim
    service instance3(server)[Instance] in resim

    mirror:R -- L:instance1
    mirror:T -- L:instance2
    mirror:B -- L:instance3

    customer_registry:R -- L:mirror
```

## Potential optimizations

There is a lot of discussion online about optimizing container images (we've provided some links to further reading below). Here we have outlined some of the more impactful changes you can make, in our experience. Some of these are trade-offs, where you may be trading complexity or maintenance overhead for smaller images that are cheaper and quicker to handle, and some are outright improvements.

### Use a lean base image

Generic base images can be very large, as they include packages to suit the potential needs of many users (text editors, SSH servers, compilers etc.). Consider starting with a minimal image and adding only the packages and dependencies you need.

Many images publish a `slim` variant which is useful for this purpose.

### Order layers

In your Dockerfile, commands that modify the filesystem (such as `ADD`, `COPY` and `RUN`) cause a new layer to be created. Those layers are cached "in order", so if a command at the top of the file means that next time the build is run, the layer it produces has changed, all layers "below" that one in the Dockerfile will be invalidated and rebuilt.

When using the image in ReSim, this means that our Mirror service will need to pull all of the newly-invalidated layers, and not just the most recently changed layer.

As a concrete example, suppose you had a Dockerfile like this:

Dockerfile

```
FROM ubuntu

# let's record which commit this is (argument value provided by build command)
ARG GIT_SHA
RUN echo $GIT_SHA > /.build-version

# install dependencies and build our application
RUN apt-get update && apt-get install -y build-essential
COPY main.c Makefile /src/
WORKDIR /src/
RUN make build
```

This is a contrived example, but in this case every time the image is built with a different `GIT_SHA` value, the `RUN echo $GIT_SHA > /.build-version` layer will change, and all layers that run after it (the apt-get install, ADD, and make build commands) will be invalidated and rebuilt, even though those dependencies and source files may not have changed.

This means that all of: running the build, uploading the image to long-term storage (because `docker push` and similar commands are layer-aware), and transferring the image to ReSim will take longer than they need to.

If you really wanted that `.build-version` file, putting it at the end of the Dockerfile (or at least after the expensive operations) would mean that the `RUN apt-get...` and later commands would not be invalidated by it.

### Don't install unnecessary packages

You should only install the packages you need to run your application. Development tools and other utilities make images less secure, slower to build, and larger. See [Consider using multi-stage builds](#consider-using-multi-stage-builds) below for an example of how to build a smaller image that only contains your application and its runtime dependencies.

### Install packages in one command

If you are installing multiple packages, install them in one command. Splitting the installations into separate commands leads to inefficient layer caching, affecting both build time and overall image size.

For example:

Dockerfile

```
RUN apt-get update && apt-get install -y build-essential \
    python3 \
    python3-pip \
    libcudnn8 \
    <...other required packages> \
    && rm -rf /var/lib/apt/lists/* # See below
```

### Clear or disable package manager caches

Package managers cache information about the packages available in configured repositories. This can take up a surprising amount of space.

For `apt`:

Dockerfile

```
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    <...other required packages> \
    && rm -rf /var/lib/apt/lists/*
```

Note that if you split the installation and cache removal into separate `RUN` commands the cache will appear to be removed in the final image, but it will still be cached in the prior layers, taking up space.

For `apk` (Alpine's package manager):

Dockerfile

```
RUN apk add --no-cache <packages>
```

This both updates package information and prevents `apk` storing any.

### Consider using multi-stage builds

This is an advanced topic, but for image size optimization and security it can be best to produce an image that only contains your compiled binary, if it suits your application.

For example:

Dockerfile

```
FROM golang:1.25 AS builder
WORKDIR /src
COPY main.go .

RUN go build -o /bin/hello ./main.go

FROM scratch
COPY --from=builder /bin/hello /bin/hello
CMD ["/bin/hello"]
```

## Conclusion

As you can see, it's possible to spend a small amount of time working on your Dockerfile and save significant time and money in return - not just when running in ReSim, but also when building, storing and distributing images for your own use. In addition, images with only the necessary packages installed present a smaller attack surface and therefore can be more secure.

## Further reading

- [Dockerfile best practices](https://docs.docker.com/build/building/best-practices/)
- [Layer caching](https://docs.docker.com/build/cache/)
- [Multi-stage Builds](https://docs.docker.com/build/building/best-practices/#use-multi-stage-builds)
# Reference

# Reference

Reference material is **information-oriented**: precise, complete, and authoritative. Use this section when you need to look something up — not to learn a concept, but to verify a detail.

This section contains:

- **Core Concepts** — the ReSim data model: projects, systems, experiences, builds, batches, and metrics
- **Metrics Library** — the full API reference for the ReSim open-core metrics library (Python and C++)
- **Open Core Libraries** — API references for transforms, curves, math, time, visualization, and autonomy-log utilities

In order to easily manage the execution of your code on the cloud, ReSim introduces a few basic concepts that are useful to understand.

## Project

A **Project** is a flexible container for all related concepts within the ReSim Ecosystem. All data within the ReSim platform is isolated across projects, enabling different teams or embodied AI applications to be developed within a dedicated space. Projects can be used to represent a full embodied AI system, for an individual subsystem or as a personal sandbox.

Projects typically map to repositories, in which case it is helpful to use the same name for the project as the repository so that CI services associated with repositories (e.g. GitHub, GitLab) have easy ways to reference repository names inside CI workflows and commands can be parameterised. It is not mandatory to maintain this distinction.

We recommend that a **Project** be used to reflect an autonomy stack as a whole, and that [systems](#system) be used to partition individual components of that autonomy stack e.g. perception, localization, etc. In this way, the reporting functionalities of the ReSim app can be used to offer an overview of the state of your stack.

Projects have a name and an ID.

- [Create a project](../setup/projects/)

## Experience

An **experience** is a set of interchangeable input files for a test you want to execute in the ReSim app. For a simulation test, an experience will likely include a scenario description (e.g. where the ego is in the world and what its goals are), autonomy stack parameters, and any other assets the simulation test needs to run.

Experiences can be stored in cloud blob storage (such as AWS S3 or Google Cloud Storage), in the Foxglove Data Platform, or at a local path in the [build container](#build). You will need to [configure ReSim to access your experience data](../setup/experience-data-sources/) in order for us to consume it. Experiences can be stored in multiple locations, all of which is fetched by ReSim.

Experiences can be tagged to help filter and search them.

- [Add experiences](../setup/adding-experiences/)

Experiences also support important metadata for when they are executed as a test. In particular, you can set:

- A set of environment variables that can be injected into your [build](#build)
- A timeout (in seconds) that can be used as an upper bound on your expected experience runtime, to guard against infinite loops
- A [docker compose profile](https://docs.docker.com/compose/how-tos/profiles/) that should be used to run this experience if you are using a multiple container build. Note: we do not support multiple profiles right now, please contact us if this is important.

## Asset

An **asset** is a set of shared data files that a [build](#build) needs at runtime but that are not specific to any single [experience](#experience). Common examples include HD maps, 3D environment meshes, ML model weights, physics parameters, and calibration data -- anything that remains constant across many experiences.

While experiences represent per-test inputs (mounted at `/tmp/resim/inputs/`), assets represent shared data that is associated with a **build** and mounted read-only at `/tmp/resim/assets/<mountFolder>/`. This separation means you don't need to duplicate large shared files across experience locations or bake them into your Docker image.

Assets support **revisions**: each time the source data changes, a new revision is created with a version string, preserving an immutable history. Assets are [cached](../guides/asset-caching/) by default, so only changed files are synchronized between runs.

Assets are explicitly linked to builds, and we recommend managing these associations in your CI/CD pipeline to ensure reproducibility.

- [Set up assets](../setup/assets/)

## System

A **System** is an abstraction of a software component that can be run on a specific set of data with its own resource needs, such as an embodied AI solution. This solution may be composed of several subsystems that can be tested individually or collectively at scale. Examples might include perception system, motion planning, full stack.

A **system** definition includes the hardware resources requirements for a particular component, such as CPUs, memory, and GPUs. It can then be run against a suite of [experiences](#experience) using a [build](#build) definition and a [metrics build](#metrics-build).

You may wish to use experiences and metrics builds with multiple systems. ReSim allows you to tag experiences and metrics builds with one or more systems to show that a particular system is compatible with them. This flexible representation enables users to filter individual test results based on which system is being tested, or get a unified view of the status of the systems that make up a given [project](#project).

Note

We have a feature on the roadmap to introduce interactive prompts and compatibility validation to the UI and CLI, but for now, you'll need to visually check

- [Define a system](../setup/systems/)

## Build

A **Build** is an instantiation of a system in time. It is a snapshot of the code, represented by a container image, that can actually run the [system](#system) under test with inputs provided by an [experience](#experience).

Builds are typically versioned, as the executable test is modified over time. For instance, in continuous integration workflows, you might generate a new build for every new commit to your autonomy code repository so that you can run tests on the newest software. The name of a build usually references the relevant system (e.g. "Perception Build") whereas its description contains a descriptive commit message or SHA.

The executable and dependencies for the build is packaged up in a [container image](https://opencontainers.org/) which is pushed to a container registry and [registered with the ReSim app](../setup/build-images/). ReSim supports images from AWS ECR, Docker Hub, GitHub Container Registry, and Google Artifact Registry. You will need to [grant us access](../setup/container-registry-access/) to any non-public container registry you use.

- [Build a system image](../setup/build-images/)
- [Push a system image from CI](../setup/ci/)

### Multi-container builds

In place of a single container image, a **Build** may also be a collection of container images tied together by a [Docker Compose](https://docs.docker.com/reference/compose-file/) file. Defining a multi-container build like this allows you to have multiple containers executed together during the **Experience** phase. These containers can communicate with each other, share data, etc, and may allow you to better simulate a more complex system.

## Metrics build

A **Metrics Build Image** is a Docker image that contains the code for evaluating the performance of your system in the test.

As one tests an autonomy application at increasing scale in a virtual environment, it becomes ever more critical to have good *metrics* to quantitatively evaluate the performance of your system.

The ReSim platform provides an open-source, [general purpose metrics SDK](https://github.com/resim-ai/open-core/tree/main/resim/metrics) and framework to enable the calculation of metrics on the data output from the execution of your autonomy application.

ReSim enables metrics to be specified and computed at two different levels of granularity: at the [*test*](#test) level, where metrics are evaluating the performance across a single experience, and, at the [*test batch*](#test-batch) level, where one can compute aggregate metrics across many experiences.

For example, you may compute *mean localization error magnitude* for your application for each experience in a *test* metric. However, if you run your application on hundreds of experiences, rather than checking that value for each experience, a *test batch* metric could provide *batch mean localization error magnitude*, which aggregates the individual localization error metrics.

The batch metrics framework is flexible enough to enable one to create a *weighted mean*, which takes into account that some experiences may have more frames than others.

- [Build a metrics image](/open-core/metrics/metrics_builds/)

## Branch

A **branch** is a collection of builds. Branches in the ReSim app are similar to the idea of a branch in [git](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell), although there is no automatic association between branches in the two systems. Branches are useful for associating builds that represent different iterations of the same work in the same way that git branches associate a graph of commits. In a continuous integration workflow, each pull request could therefore have its own branch in the ReSim app to contain associated builds.

- [Create a branch](../setup/build-images/#creating-a-branch)

## Test

A **test** is the fundamental unit of cloud-based work at ReSim. It is a single execution of a given build with a given experience.

It is canonical terminology in software engineering to refer to a **Test** as a single function that checks some functionality of your system and a **Test Suite** as a... well, a suite of tests.

A test returns either a pass or a fail, and a test suite passes if and only if all of its tests pass.

At ReSim, we adopt this terminology with a small modification:

- A **Test** is the execution of a single [*experience*](#experience) against a [*build*](#build), where performance is evaluated by a [*metrics build*](#metrics-build)
- A **Test Suite** is a *specification* of a number of tests to run and evaluate with the same metrics build. The aggregate performance of the test suite is also determined by the *metrics build*.

A test suite can then be run with a build to generate tests. Running a test suite will generate a test for each experience in the test suite and then generate aggregate metrics.

To distinguish between the test suite *qua* specification and group of tests it creates, we use the term [**Test Batch**](#test-batch). A test batch is a group of tests that have been run in the ReSim platform. As has been discussed in the [Metrics Build](../setup/metrics-builds/) page, the test suite's metrics build can also be instrumented to compute so-called batch metrics, which aggregate the metrics from each individual test to display, for example, the average performances of the build or a list of the worst performing experiences.

## Test suite

A **test suite** is a named template for a set of [tests](#test). It is an *organizational* component to allow you to specify a commonly used set of [experiences](#experience) to run. A common example is to define the set of regression tests for a given [system](#system).

A test suite has five main pieces of data:

- The **name** of the test suite e.g. Nightly Regression Test.
- An optional **description** to provide some more detail about the intent and purpose of this test suite.
- The **system** that this test suite is intended to be run with. The ReSim app will warn if you attempt to run the test suite with a build that is not from that system, since it is likely that there will be compatibility issues and it will fail.
- The **metrics build** that will be used to compute the metrics for this test suite.
- A list of **experiences** that will be run against the build you choose.

In order to provide traceability of previous results, test suites are *versioned*. Any time a change is made to a test suite (updating the metrics build, adding or removing experiences), a new *revision* is created. Test suites start at `revision 0` and increment.

One of the main benefits for using test suites -- other than the time saving from having to choose what experiences and metrics you want repeatedly -- is that it makes it very easy to:

- Filter and Compare your tests. If there are many test batches created in your project, it can be helpful to filter a list of results by test suite to find what you are looking for and to then compare like for like e.g. compare the last two times a test suite has been run against main

- Control your tests. If a systems and test engineering team exists separately to engineering and have responsibility for defining acceptance criteria, test suites can be used to define what tests the engineering team should be passing for a given feature.

- Construct longitudinal reports. If a test suite is run regularly against a particular branch, it can be helpful to generate a longitudinal report that shows how performance is improving over time i.e. is your system better. Using test suites makes this easy in the ReSim app, as [Reports](../setup/reports/) can be used to generate this longitudinal analysis.

- [Create test suites](../setup/test-suites/)

## Test batch

A **test batch**, often referred to as simply a **batch**, is a collection of tests that have been triggered simultaneously using the same build. In a continuous integration workflow, a batch of simulation tests could be kicked off on each change to the code under development. These would all use the same build (based on the latest iteration of the software), but each could use a different experience.

- [Run a batch](../setup/test-suites/#running-a-test-suite)

## Test suite report

A **test suite report** is a longitudinal report, displaying *evolving performance of a [system](#system) over time*. We have already seen how the ReSim platform supports analysis of an atomic test via [metrics](#metrics-build), then the aggregate analysis of a group of those tests in a test batch via batch metrics. It can also be useful to construct a longitudinal report, displaying evolving performance of a system over time.

In ReSim, a **Test Suite Report** offers such a capability, by enabling users to generate completely custom reports that display how a System has performed for a given test suite over time, for a particular [**branch**](#branch).

This means, for example, that it is possible to generate a report displaying how the `main` branch has performed in *Nightly Progression Tests* over the past month (ideally ever better). One could also use it to generate a report on the performance of a *PR* against a defined set of tests *My Acceptance Tests* that need to be passed to merge the PR.

Since the philosophy at ReSim is to support maximum customizability for customers, reports can be calculated using a **metrics build** of your own construction. ReSim does, however, offer an open-source generic report metrics build that creates sensible default reports.

The default ReSim report metrics build generates aggregate statistics such as:

- The total number of tests and batches run over the report period.

- The list of experiences in the test suite, ordered by the number of failures.

- The list of all metrics computed for the batches, ordered by number of failures.

- A line chart showing how the number of tests passing per batch has evolved over the time period

- For any scalar batch metrics, a line chart displaying that scalar value's evolution over time e.g. mean average precision over time.

- [Test suite reports](../setup/reports/)

## View

A **view** is a set of visualizable 3D content generated by ReSim View. This content can be visualized using Foxglove Studio by clicking the **Open in Foxglove** link provided on the view's page in the ReSim app. ReSim View is a C++ library that users can link to their code to make it easy to visualize 3D transforms in their code.

- [Visualization](../open-core/visualization/)

# Overview

One of ReSim's core products is a comprehensive and growing **metrics framework** for evaluating the performance of Embodied AI applications, such as robots. The purpose of this product is to allow users to display information from their simulation runs of their embodied AI system in an online dashboard such that judgments can be rendered about its performance. To this end, such information can be compared between different versions of the system on different software branches or longitudinally along a single software branch.

Metrics exist at three levels:

1. Test Metrics: Metrics are computed per-test (i.e. per simulation), based off outputs from the simulation (such as log data and other artifacts) - an example would be a *precision-recall* curve across a set of input data.
1. Batch Metrics: Multiple test metrics are aggregated to result in batch metrics. For example, an *average accuracy* across all the tests in your batch.
1. Test Suite Report Metrics: These metrics help you assess how the performance of your system has evolved over time. For example *average accuracy over time*.

These metrics are written using the same system, which is the [ResimMetricsWriter](https://github.com/resim-ai/open-core/blob/main/resim/metrics/python/metrics_writer.py), although the "input data" is of course different between the three.

The basic mechanism in the ReSim app is a `metrics build`, which is a docker image that wraps any metrics code you write. The image is run by the ReSim app after every simulation, every batch, and on demand as part of report generation.

It is ReSim's core philosophy to allow maximum flexibility for engineers to display the analysis that is most relevant to them. We aim to achieve this in three ways:

1. Open Source: The entirety of our metrics framework is open source to ensure transparency for those using ReSim to test and evaluate their embodied AI system.
1. Determine your own aggregations: Rather than the ReSim web app providing fixed, limited aggregations of the results of a set of tests (batch metrics), or fixed longitudinal reports (test suite report metrics), the ReSim metrics framework lets you write code to decide how to aggregate. ReSim also provides some sensible default metrics to get users started.
1. Standard plotting libraries: While ReSim supports a range of custom metrics styles, described in [Metrics Types](metric_types/), any Plotly chart can also be wrapped with our metrics metadata and displayed using [Plotly.JS](https://github.com/plotly/plotly.js).

To allow you to use this powerful framework, we recommend working through the docs in this order:

# Contents

1. [Metrics Builds](metrics_builds/): How to make a metrics build to run your metrics and what data contract it expects.
1. [Metrics Data](metrics_data/): How to extract and represent data from your logs, ready to visualize in a metric.
1. [Metric Types](metric_types/): A summary of the types of metrics supported in the dashboard, and how to use them.
1. [Metrics Writer](metrics_writer/): How to write metrics in a way such that they appear in the dashboard.
1. [Events](events/): A summary of events, the assignment of metrics to events, and how to use them.
1. [Batch Metrics](batch_metrics/): How to compute batch metrics.
1. [Test Suite Reports](report_metrics/): How to compute test suite reports and their metrics.

# Metrics Builds

## What are Metrics Builds?

Metrics builds are code which is run *after* your test runs, taking experience and log data, and writing it to a format that the ReSim app is able to plot in app dashboard. Our current supported language for metrics is **Python**.

## How to Register Metrics Builds

Just like regular system builds, you need to make a docker image and register metrics builds with the ReSim CLI before you can run them. When registered, they should appear in the ReSim app when creating test suites or running ad hoc tests. For more on registering metrics builds, see [Run batches with metrics](https://docs.resim.ai/setup/metrics-builds/) in the ReRun docs. For the purpose of this doc, we'll assume you have such a docker image registered, and just talk about what it should do.

## Test Mode vs Batch Mode

Metrics Builds have three modes, two of which *must implemented within the same metrics build*.

1. [test mode](#test-mode-computation) - in this mode, we compute metrics associated with a single test, based off output data from the test.
1. [batch mode](#batch-mode-computation) - in this mode, we compute metrics associated with a batch of tests, based off output metrics data from `test mode`.

Whether we are in `test mode` or `batch mode` is determined by the files present in `/tmp/resim/inputs/` when the metrics build container runs.

- If the file `/tmp/resim/inputs/batch_metrics_config.json` is present, we are in batch mode, and our build should compute batch metrics using this config.
- If the file `/tmp/resim/inputs/batch_metrics_config.json` is *not* present, then we are in test mode, and our build should compute test metrics. In this case, `/tmp/resim/inputs/` will contain an `experience` and `logs` directory. The `experience` directory will contain the experience data from the test (e.g. the `/tmp/resim/inputs` folder from the experience stage), and the `logs` directory will contain the log data (e.g. the `/tmp/resim/outputs` folder from the experience stage).

## Test Suite Report Mode

The third mode that a metrics build can operate in exists to compute **test suite reports**. A test suite report is a longitudinal comparison of a given [test suite](https://docs.resim.ai/setup/test-suites) over time. A report is expected to compute how performance of a given branch of the system has changed over time. For more information, please see the main [reports documentation](https://docs.resim.ai/setup/reports).

For a metrics build to determine that it is in `report mode`, it must check that a file called `report_config.json` exists in the `/tmp/resim/inputs` directory. If this file is present, we are in report mode and we should compute reports.

# Output

Regardless of the mode, in order to have metrics show up your metrics build is required to write to `/tmp/resim/outputs/metrics.binproto`, in a format complying with the [JobMetrics object](https://github.com/resim-ai/open-core/blob/main/resim/metrics/proto/metrics.proto). This protobuf should validate with the [Python validator](https://github.com/resim-ai/open-core/blob/main/resim/metrics/proto/validate_metrics_proto.py) we provide.

> WARNING: It is highly recommended you do not interact with this proto directly, and instead use the metrics SDK documented in [Metrics Data](../metrics_data/), [Metrics Writer](../metrics_writer/), [Metric Types](../metric_types/), and [Events](../events/). We reserve the right to modify this proto at any point, and do not guarantee substantial backwards compatibiliity support (such as field names remaining the same).

## Test Mode Computation

Assuming the file `/tmp/resim/inputs/batch_metrics_config.json` is not present, we are in `test mode`! We then have two relevant directories:

1. `/tmp/resim/inputs/logs/` - this contains any log data output by the simulation.
1. `/tmp/resim/inputs/experience/` - this contains all the experience data associated with the simulation.

We would read these in, and as always, we'd output any metrics we want to `/tmp/resim/outputs/metrics.binproto`. For more on how to actually write these metrics see [Metrics Data](../metrics_data/), [Metrics Writer](../metrics_writer/), [Metric Types](../metric_types/), and [Events](../events/).

## Batch Mode computation

If the file `/tmp/resim/inputs/batch_metrics_config.json` is present, we are in `batch mode`! This file should look something like this:

JSON

```
{
  "authToken" : "...",
  "apiURL" : "https://api.resim.ai/v1",
  "batchID" : "7579affb-3e5b-4f02-871b-bf275aef67ee",
  "projectID": "6133c1a9-b6d8-41fb-b7d7-44f1889511dd"
}
```

These four things will be used to retrieve the metrics data associated from tests with the API. This is made easier by various tooling we provide in this repo. We then compute metrics based off this, just as we'd do for test metrics. See [Batch Metrics](../batch_metrics/) for more on this.

Note

The ReSim API is generic and supports other cloud workloads, so the raw API endpoint for fetching test data is called the `jobs` endpoint. We will use the term test in this documentation unless referencing the API.

## Report Mode Computation

If the file `/tmp/resim/inputs/report_config.json` is present, we are in `report mode`! This file should look something like this:

JSON

```
{
  "authToken" : "...",
  "apiURL" : "https://api.resim.ai/v1",
  "reportID" : "7579affb-3e5b-4f02-871b-bf275aef67ee",
  "projectID": "a7a6f4fd-7852-4992-b073-4c89773ef953"
}
```

These four things will be used to retrieve the metrics data associated from batches with the API. This is made easier by various tooling we provide in this repository. See [Report Metrics](../report_metrics/) for more on this.

# Metrics Data

Metrics data is series-like data extracted from logs, in a way that can be referenced by metrics, and represented in the app.

Within the metrics SDK, the basic object is a `SeriesMetricsData`, which is fundamentally a numpy `array` of arbitrary length, with a unique `name` associated to it. This numpy `array` can contain arbitrary types, but as of right now, only four types of data can actually be written to the metrics output:

1. `float`
1. `string`
1. `resim.python.metrics_utils.Timestamp`
1. `resim.python.metrics_utils.MetricStatus`

Attempts to write any other types will fail.

> (To store `int` types, it is currently recommended to convert to `float`. To store `enum`, it is recommended to convert to `string`.)

Here is an example of code making a `SeriesMetricsData`

Python

```
from resim.metrics.python.metrics import SeriesMetricsData
from resim.metrics.python.metrics_utils import Timestamp

# Example timestamp data - in practice this would be extracted from log
timestamps = np.array(
  [Timestamp(secs=i, nanos=0) for i in range(100)]
) 

# Make SeriesMetricsData from array
timestamps_data = SeriesMetricsData(
  name="Log timestamps",
  series = timestamps,
  unit = "Timestamp" # optional
)
```

We also support a "fluent API" style for declaring `MetricsData`, which becomes more useful when trying to make new metrics and data and write them at the same time. The above would be equivalent to:

Python

```
timestamps_data = (
  SeriesMetricsData(name = "Log timestamps")
    .with_series(timestamps)
    .with_unit("Timestamp")
)
```

# Indexing data

In order to provide a "table-like" functionality, we support the idea of one array `index`-ing another. This means that one array is considered the index of another.

As an example, we could add the following code to the above:

Python

```
floats = np.array([random.random() for i in range(100)])

floats_data = SeriesMetricsData(
  name = "floats",
  series = floats,
  unit = "/ 1.0",
  index_data = timestamps_data,
)
```

This is now an array "indexed" by the timestamps we made earlier, so can be thought of as a "dictionary" of the form:

Python

```
{
  timestamps[0]: floats[0],
  timestamps[1]: floats[1],
  ...
  timestamps[100]: floats[100]
}
```

By indexing multiple series with the same single index, tables can easily be converted into our format.

> Note: There are a few key details here:
>
> 1. The index array and data array must have the **same length**.
> 1. Index arrays should **not** themselves be indexed.
> 1. If the index array contains repeats, then the first matching index is considered the relevant one when trying to retrieve variables by index.

# Grouping metrics data

We support the notion of `grouped` metrics data, which is effectively the output of a `GROUP BY` operation. That is, it is one array "grouped" by the values in the another array of the same length. These arrays should generally share the same index, if one is present.

For example:

Python

```
state_set = ['ACCELERATING', 'BRAKING', 'STOPPED']

states = np.array(
  [random.choice(state_set) for i in range(100)]
)

states_data = SeriesMetricsData(
  name = "states",
  series = states,
  index = timestamps
)

grouped_floats_data = floats_data.group_by(states_data)
```

The output of this can be thought of as dictionary of the form:

Python

```
grouped_floats_data = {
  "ACCELERATING": [floats],
  "BRAKING": [floats],
  "STOPPED": [floats]
}
```

> NB: When *writing* grouped data to ResimMetricsWriter, we currently only support grouping by `string`.
>
> NB: GroupedMetricsData can be indexed, and should be indexed by other GroupedMetricsData, with the same groups and lengths!
>
> Tip: Generally, if you put GroupedMetricsData into a chart, it will make a "list" of charts - one for each group present. You can pick which chart you want using a dropdown menu. Some of these are not yet fully implemented in the UI yet. See the [Metrics types docs](../metric_types/) for more on this.

## Using metrics data

Metrics data are *referenced* by metrics, in order to create charts that are then shown in the UI. You can think of this as passing an array into `plt.plot(...)` or similar. To see the types of metrics, and what data they use, see [the Metric Types docs](../metric_types/). To see how to write these metrics, see [the Metrics Writer docs](../metrics_writer/).

# Events

The ReSim metrics SDK and web app support the creation of so-called **events** to aid the analysis of an embodied AI system's performance in a test.

An event is a timestamped occurrence of some importance to an engineer, which can have metrics specifically associated with it. In comparison to the usual ReSim metrics types e.g. line chart of velocity over time, or scalar precision/recall values, an event is centered around a timestamp and allows the engineer to highlight interesting parts of the simulation and metrics relevant to them.

As a simple example, consider an autonomous mower processing a large field. A metrics build could generate an event every time the mower got within a safe distance to a pedestrian *and* show snapshots from the system state as metrics.

In this way, we can generate a timeline of important parts of a test with rich metrics information at all the key moments.

# Registering Events

In the metrics SDK, events can be created in the same way as normal metrics. Assuming you already have a `MetricsWriter`:

Python

```
# Example event write
event = metrics_writer
        .add_event("My Unique Event Name") # Event specified here
        .with_description("An optional description of an event")
        .with_tags(["collision", "pedestrian"]) # A list of tags that can be used to categorize the event 
        .with_relative_timestamp(Timestamp(secs=10)) # Specify the timestamp in your simulation that the event occurs
        .with_status(MetricStatus.PASSED_METRIC_STATUS)
        .with_importance(MetricImportance.HIGH_IMPORTANCE)
```

In addition, a `with_metrics(...)` function expects you to pass a list of metrics objects that you have previously created to associate with the event. The metrics will not be displayed outside the context of the event and need to be registered as event metrics. For example:

Python

```
# Create two event metrics
metric_1 = metrics_writer
            .add_double_over_time_metric("Localization error")
            # ...
            .is_event_metric()

metric_2 = metrics_writer
            .add_scalar_metric("Total performance")
            # ...
            .is_event_metric()

# Now associate with the event.
event.with_metrics([metric_1, metric_2])
```

The [ReSim Metrics Validator](https://github.com/resim-ai/open-core/blob/main/resim/metrics/proto/validate_metrics_proto.py) checks that all metrics associated with an event have been flagged as event metrics via the `is_event_metric()` function. The metrics validator should be called on the metrics proto before it is written to `metrics.binproto` as illustrated in [The Metrics Writer docs](./metrics_writer).

## Available Parameters

Aside from the list of metrics, the available parameters for an event are described below:

- `name: str` - A required, immutable name. The same name cannot be used twice, so they are uniquely identifying.
- `description: str` - A string description of the event.
- `timestamp: Timestamp` - A seconds and nanos representation of the timestamp that the event occurs at. In the SDK, this can be interpreted as either a `relative` (to the start of the simulation) or an `absolute` timestamp via `with_absolute_timestamp()` and `with_relative_timestamp()`. The most common use case is for relative times e.g. 3s into the simulation, but for processing real-world logs, the absolute timestamp is more relevant.
- `tags: list[str]` - An optional list of tags associated with the event, that can be used to filter and organize on the web app.
- `status: MetricStatus` - An overall status (e.g. PASSED, FAIL_BLOCK) for the event itself. Note that this is not, by default, computed from the statuses of any associated metrics: it needs to be computed by the user
- `importance: MetricImportance` - An overall importance (e.g. CRITICAL, HIGH, LOW, ZERO).

# Batch Metrics

Batch metrics are very similar to test metrics, but are computed after all the test metrics for a given batch are computed, using the test-level metrics and metrics data as the "input."

> Example Batch Metric 1: a weighted average of a test-level scalar metric, across all the tests present.
>
> Example Batch Metric 2: a histogram of a single series that is computed by all tests, after merging these series into one by appending.

## Batch metrics mode

As mentioned in [the Metrics Builds docs](../metrics_builds/), batch metrics use the same single Docker image as test metrics.

This image can differentiate whether it's computing in `test mode` or `batch mode`, based off whether the file `/tmp/resim/inputs/batch_metrics_config.json` is present. The app's interface guarantees that this file will be present whenever it expects the metrics build to compute batch metrics. Note that the metrics build *must* generally handle the batch metrics case (even if it does nothing in such a case) since the normal test outputs will not be present and a failure will therefore usually result if normal test metric logic is employed.

## Computing batch metrics

The batch metrics config (as provided to the batch metrics run on launch) is a simple json with three fields: an auth token, an API URL, and a batch ID.

JSON

```
{
  "authToken" : "...",
  "apiURL" : "https://api.resim.ai/v1",
  "batchID" : "7579affb-3e5b-4f02-871b-bf275aef67ee",
  "projectID": "6133c1a9-b6d8-41fb-b7d7-44f1889511dd"
}
```

> NOTE: If you're wanting to develop locally, you can retrieve your authToken from <https://app.resim.ai/debug> in the `auth.bearer` field.

These fields should be used to retrieve the test-level metrics and metrics data associated with a batch, and these should be used to compute batch-level metrics. We provide code to do this in [open-core](https://github.com/resim-ai/open-core/tree/main/resim/metrics), in combination with some code snippets below.

First you can read the config in using the following snippet:

Python

```
import json 

BATCH_METRICS_CONFIG_PATH = "/tmp/resim/inputs/batch_metrics_config.json"

with open(BATCH_METRICS_CONFIG_PATH, "r", encoding="utf-8") as metrics_config_file:
    metrics_config = json.load(metrics_config_file)

token=metrics_config["authToken"]
api_url=metrics_config["apiURL"]
batch_id=metrics_config["batchID"]
project_id=metrics_config["projectID"]
```

Once these are loaded, you can download the metrics using our `fetch_job_metrics` Python package.

Python

```
import resim.metrics.fetch_job_metrics as fjm

job_to_metrics: Dict[uuid.UUID, UnpackedMetrics] = fjm.fetch_job_metrics_by_batch(token=token, api_url=api_url, project_id=uuid.UUID(project_id), batch_id=uuid.UUID(batchID))
```

The result maps job IDs to `UnpackedMetrics` - this is a simple `dataclass` with three fields:

- `metrics: List[Metric]` - a list of all the metrics in that test
- `metrics_data: List[MetricsData]` - a list of all the metrics data in that test
- `names: Set[str]` - a set of all the names of Metrics and MetricsData present

In other words, it very simply gives you all the metrics and metrics data associated with each test. You then use these metrics and data in order to compute and write your batch metrics - (just as you did for test metrics!) - by following the instructions in the [Metrics writer docs](../metrics_writer/). You write the output to the exact same place as before: `/tmp/resim/outputs/metrics.binproto`.

> NOTE: A common pattern is to write MetricsData in the test metrics *without an associated Metric*, and then use this data when it comes to computing batch metrics. This allows you to make new batch metrics without having associated test metrics.

# Test Suite Report Metrics

Test suite report metrics are quite distinct from test and batch-level metrics in the sense that they are designed to offer a longitudinal perspective on a test suite.

> Example Report Metric 1: a line chart of a scalar batch metric, over each time it has been run
>
> Example Report Metric 2: a list of the experiences that have failed most frequently

## Report metrics mode

As mentioned in [the Metrics Builds docs](../metrics_builds/), report metrics use a similar (potentially identical) Docker image as job and batch metrics.

This image can differentiate whether it's computing in `test mode`, `batch mode`, or `report mode` , based off whether the file `/tmp/resim/inputs/report_config.json` is present. If it is present, it should be computing report metrics.

## Computing report metrics

The report metrics config (as provided to the report metrics run on launch) is a simple json with four fields: an auth token, an API URL, a project ID, and a report ID.

JSON

```
{
  "authToken" : "...",
  "apiURL" : "https://api.resim.ai/v1",
  "projectID" : "7579affb-3e5b-4f02-871b-bf275aef67ee",
  "reportID" : "9328c806-4f2a-41ca-abbc-7e28e1741384"
}
```

These fields should be used to retrieve the report and therefore the list of batches to be used to compute the report. We provide code to do this in [open-core](https://github.com/resim-ai/open-core/tree/main/resim/metrics/default_report_metrics.py), in combination with some code snippets below.

First you can read the config in using the following snippet:

Python

```
import json 

with open(REPORT_METRICS_CONFIG_PATH, "r", encoding="utf-8") as metrics_config_file:
    metrics_config = json.load(metrics_config_file)

token=metrics_config["authToken"],
api_url=metrics_config["apiURL"],
project_id=metrics_config["projectID"],
report_id=metrics_config["reportID"],
```

Once these are loaded, you can download the batches associated with the report using our `fetch_report_metrics` Python package.

Python

```
import asyncio

from resim.sdk.client import AuthenticatedClient
import resim.metrics.fetch_report_metrics as frm

async def main():
    client = AuthenticatedClient(base_url=api_url, token=token)
    batches = await frm.fetch_batches_for_report(client, report_id, project_id)

if __name__ == '__main__':
    asyncio.run(main())
```

# ReSim Open Libraries

ReSim's `open-core` repository contains the open source subset of ReSim's C++ code intended to accelerate robotics development.

## Getting Started

A number of our libraries have Python bindings or implementations, such as a subset of the `transforms` and `metrics` libraries as well as a Python client for the ReSim API. These are distributed as part of our [Python package](https://pypi.org/project/resim-open-core/) which can be easily installed using `pip`.

Bash

```
pip install resim-open-core
```

Note

Currently, this package is only built for Python 3.10, supporting Linux for x86_64 and aarch64 architectures.

If you're using C++ and Bazel, you can follow the development instructions below to build some examples and explore our libraries.

## Development

### Install and Setup Docker

These instructions vary by host platform:

- [Mac](https://docs.docker.com/desktop/mac/install/)
- [Windows](https://docs.docker.com/desktop/windows/install/)
- [Linux](https://docs.docker.com/engine/install/ubuntu/)
  - [post-install](https://docs.docker.com/engine/install/linux-postinstall/)

### Get the Code

Bash

```
git clone git@github.com:resim-ai/open-core.git
```

### Setting up the Environment

In order to easily build and interact with ReSim's code, you need our development docker image. To fetch this image, navigate to the open source repo and call the following script.

Bash

```
cd open-core
./.devcontainer/pull.sh
```

If you have any trouble pulling here, please see [this section](#building-the-docker-image-locally) below which shows how to build your own image locally. Next, we can start up a development docker container locally:

Bash

```
./.devcontainer/run.sh
```

### Building the Code

We use [Bazel](https://bazel.build/) as our build system. Therefore, to build all of the example binaries you simply have to execute:

Bash

```
bazel build //resim/examples/...
```

Now, all of the binaries will be present in `bazel-bin/resim/examples/`. Individual binaries can be built and run in a single step using `bazel run`. For example:

Bash

```
bazel run //resim/examples:liegroups
```

### Autocomplete via `clangd`, `clang-tidy`, and `compile_commands.json`

Many developer tools (such as `clangd` and `clang-tidy`) require a `compile_commands.json` file in the root of the tree. This file is large and prone to change, so we don't check it in and each developer is responsible for generating it themselves. You will need to regenerate it whenever you change your BUILD files.

We use [Hedron's Compile Commands Extractor](https://github.com/hedronvision/bazel-compile-commands-extractor) to generate `compile_commands.json`. It can be invoked with:

Bash

```
bazel run @hedron_compile_commands//:refresh_all
```

### Using Dotfiles

If you'd like to use your own dotfiles for customization inside the container, our recommendation is to create a dotfiles repository and write an `install.sh` script to automatically move those dotfiles to the appropriate location (and perform whatever other workspace customization you'd like). Note: do not put credentials of any kind in this repository.

If you'd like to streamline this system a bit, you can check out your dotfiles repo to `$HOME/dotfiles` and use `.devcontainer/run_dotfiles.sh` (instead of `.devcontainer/run.sh`) to have it automatically mounted and installed in your container.

### Building the Docker Image Locally

We also provide scripts to build the development Docker image locally. To do this, simply do:

Bash

```
./.devcontainer/build.sh
```

And then once this is complete simply use `.devcontainer/run_local.sh` or `.devcontainer/run_dotfiles_local.sh` instead of `.devcontainer/run.sh` or `.devcontainer/run_dotfiles.sh`.

# Lie Groups

## Introduction

Disclaimer

*This is not written from the perspective of a mathematician or physicist, but rather from the perspective of an engineer that has found Lie groups to be an indespensible tool for robotics. The knowledge here is primarily practical knowledge built up over experience, so it is very likely that some of the mathematical details are not precisely correct as written. Please don't hesitate to reach out and let us know if you find any errors!*

[**Lie groups**](https://en.wikipedia.org/wiki/Lie_group) are mathematical objects with a rich history of study and application. They are named for the Norwegian mathematician [Sophus Lie](https://en.wikipedia.org/wiki/Sophus_Lie) who pioneered the mathematics of continuous transformation groups.

Why should we use Lie groups in robotics? The answer is that robotics is replete with examples of continuous transform groups. One example is the fundamental concept of pose. A pose describes a rigid-body transformation from one set of coordinates to another. Most commonly, a robot's location and orientation relative to the world is represented by the pose (or rigid transformation) relating the "world" and "robot" frames.

There are, of course, many ways of representing a robot's pose. Orientation alone can be represented by Euler angles, quaternions, rotation matrices, etc. Using Lie groups affords us two primary advantages:

- First, the set of all orientations and the set of all poses are both Lie groups. Representing them as such allows us to write generic algorithms (like Hermite spline construction or Newtonian dynamics) that operate equally well on both since both are equipped with the same basic Lie group properties. This would be impossible to do, for example, if we represented orientations as quaternions (which otherwise have very nice properties) and poses as tuples of (quaternion, translation vector).

- Second, every Lie group has a corresponding Lie algebra (a vector space), which can be used to describe its structure locally (i.e. near any given element in the group). This means that we can do a lot of things on Lie groups that we normally know how to do on vector spaces. For instance, we can:

  - Construct Gaussian distributions on the Lie group of rigid body transformations (SE(3)) and sample from them.
  - Interpolate between recorded actor poses to produce a trajectory without worrying about wrap-around issues that might result if we interpolated poses with Euler angles describing their orientations.
  - Write loss functions on orientations and use them for optimal control, once again without worrying about wrap-around or singularities.
  - Take the derivatives of our poses and represent them using elements of the corresponding Lie algebra.

We've glossed over everything exceptionally briefly here, so I would encourage the curious reader to peruse the [external links](#external-links) below.

## LieGroup Interface

In an effort to make Lie-groups-related code more reusable, we require each implementation of a Lie group object (that is an object representing an element of a Lie group) to satify a few constraints.

First, the implementation must inherit from the LieGroup template. This base class is templated on the dimensionality of the input to the transform (e.g. 3 for 3D rotations), and the number of degrees of freedom for this group (e.g. 6 = {3 rotational} + {3 translational} for rigid body transformations). This base template also defines some helpful aliases for tangent vectors (i.e. members of the Lie algebra).

Second, the implementation should satisfy the LieGroupType concept which enforces a number of the group axioms (e.g. group action, invertability, identity, etc.). We implement these checks as a concept since different groups need different signatures so inheritance doesn't allow us to enforce them.

## Lie Groups Implemented

**SO(3)** - The Special Orthogonal Group in 3D. The set of all rotations.

**SE(3)** - The Special Euclidian Group in 3D. The set of all rigid transformations.

## Framed Groups

As discussed above, Lie group elements are frequently used to represent transformations between coordinate frames. In practice, this can often lead to mistakes when users of Lie group libraries compose group elements in the wrong order (e.g. multiplying `robot_from_sensor` times `scene_from_robot`). These small mistakes are simple enough that they should be possible to catch, at least at runtime. We therefore assign a unique id to each coordinate frame. Group elements thus have two frame ids that they track (`into` and `from`). When two elements are multiplied, consistency is enforced between them. More precisely, if a pose that transforms points in frame B's coordinates to frame A's coordinates (call the transform `A_from_B`) and a similar transform `C_from_D` then we should fail if the user ever tries to multiply `A_from_B * C_from_D`. However `A_from_B * B_from_D` is fine. Unframed groups are supported, by setting the frame to a null (0) id, in which case they are not checked. Be aware that multiplying an unframed group by a framed group will always result in an unframed group, so unframed groups can propagate quickly!

## More Information

For more information on the features our Lie group classes provide for working with the derivatives of functions involving Lie groups, please refer to [this guide](../liegroup_derivatives/).

## External Links

For more information, please take a look at the following links:

- [A micro Lie theory for state estimation in robotics](https://arxiv.org/pdf/1812.01537)
- [ethaneade.com](https://ethaneade.com/)
- Jakob Schwichtenberg:
  - [Lie Group Theory - A Completely Naive Introduction](https://jakobschwichtenberg.com/naive-introduction-lie-theory/)
  - [How is a Lie Algebra able to describe a group](https://jakobschwichtenberg.com/lie-algebra-able-describe-group/)
  - [What's so special about the adjoint representation of a Lie group?](https://jakobschwichtenberg.com/adjoint-representation/)

# Using SO(3) and SE(3)

## Introduction

This document is intended to breifly outline the basics of using ReSim's main Lie group libraries. For an introduction to Lie groups and the reasons why we think they're worth using, the reader is highly encouraged to read [Lie Groups](../liegroups/).

As a motivating example, we imagine that we have the pose of a robot in 3D space given by Tait-Bryan [Euler Angles](https://en.wikipedia.org/wiki/Euler_angles) (yaw, pitch, and roll) (\\psi), (\\theta), and (\\varphi) and the actor's location coordinates (x), (y), and (z) expressed in some stationary scene coordinate frame. We will show how to create an SO3 describing the orientation of this robot and an SE3 describing its full pose.

## Orientation / SO3

Before we get to the full pose, it's simpler to take on the orientation part separately. Our Euler angles are defined as the composition of three rotations. If we have a vector (v\_{\\text{robot}}) in our robot's coordinates and we want to express it in our scene coordinates, we have:

[ v\_\\text{scene} = R_x(\\varphi) R_y(\\theta) R_z(\\psi) v\_\\text{robot} ]

Where (R_x(\\varphi)) is a rotation matrix for a rotation around the (x) axis by (\\varphi) radians. Hence the overall rotation matrix for our orientation is given by:

[ R = R_x(\\varphi) R_y(\\theta) R_z(\\psi) ]

One can identify the colums of this matrix as the axes of the robot coordinate frame expressed in scene coordinates. Since rotation matrices are a *representation* of (\\text{SO(3)}), we can use this equation to compute an `SO3` for our robot pose like so:

C++

```
#include <cmath>

#include "resim/transforms/so3.hh"
#include "resim/visualization/view.hh"

using namespace resim::transforms;

// ...

const double psi = M_PI_4;
const double theta = 0.5;
const double phi = 0.1;

const SO3 scene_from_robot_rotation = SO3(phi, {1., 0., 0.}) *
                                      SO3(theta, {0., 1., 0.}) *
                                      SO3(psi, {0., 0., 1.});

// Visualize with ReSim View
VIEW(scene_from_robot_rotation) << "My rotation";
```

Here, we're constructing `SO3`s representing the rotations around each axis. Note that we have to treat each rotation in order if we're using Euler angles. If we use the exponential map on `SO3` (which converts angle/axis representations of an orientation into an `SO3` object) with a vector of the angles, we will get a different result:

C++

```
const SO3 scene_from_robot_rotation_wrong = SO3::exp({phi, theta, psi});

// The following assertion will not fail except in the degenerate case of two of
// the angles being zero.
REASSERT(
    not scene_from_robot_rotation.is_approx(scene_from_robot_rotation_wrong));
```

The naming here (i.e. `scene_from_robot`) is conventional in ReSim's libraries as it reflects the fact that the SO3 is a transformation from robot coordinates into scene coordinates. In fact, one can use the `SO3` to directly transform vectors.

C++

```
using Vec3 = Eigen::Vector3d;

const Vec3 robot_forward_in_robot_coordinates{1.0, 0.0, 0.0};
Vec3 robot_forward_in_scene_coordinates =
    scene_from_robot_rotation * robot_forward_in_robot_coordinates;

// Can also be more explicit. This is equivalent to the above.
robot_forward_in_scene_coordinates =
    scene_from_robot_rotation.rotate(robot_forward_in_robot_coordinates);
```

Quaternions are also often used to represent orientations, and have many benefits over Euler angles. Fortunately, `SO3` objects can also be converted to and from quaternions:

C++

```
const Eigen::Quaterniond scene_from_robot_quat{
    scene_from_robot_rotation.quaternion()};
REASSERT(SO3(scene_from_robot_quat).is_approx(scene_from_robot_rotation));
```

Furthermore, like quaternions, SO3 affords smooth interpolation without wrap-around issues. To do this, we use the `SO3::interp()` member function:

C++

```
constexpr double EPSILON = 1e-2;
const Vec3 axis = Vec3(0.1, 0.2, 0.3).normalized();
const SO3 scene_from_a(M_PI - EPSILON, axis);
const SO3 scene_from_b(-M_PI + EPSILON, axis);

constexpr double FRACTION = 0.5;
const SO3 scene_from_interped_rotation{
    scene_from_a *
    (scene_from_a.inverse() * scene_from_b).interp(FRACTION)};

const SO3 expected(M_PI, axis);
REASSERT(scene_from_interped_rotation.is_approx(expected));
```

## Pose / SE3

Now, if we want to express the robot's whole pose, most of the work is already done:

C++

```
#include "resim/transforms/se3.hh"
// ...
const Vec3 scene_from_robot_translation{x, y, z};
const SE3 scene_from_robot{
    scene_from_robot_rotation,
    scene_from_robot_translation};
```

This can still be used to transform points as above, but as one might expect, the translation is now included:

C++

```
const Vec3 point_in_robot_coordinates{Vec3::Random()};
const Vec3 point_in_scene_coordinates{
    scene_from_robot * point_in_robot_coordinates};
REASSERT(
    point_in_scene_coordinates ==
    scene_from_robot_rotation * point_in_robot_coordinates +
        scene_from_robot_translation);
```

Sometimes, we don't want to transform a point, but instead we want to transform a vector (e.g. robot velocity). In this case, we instead use:

C++

```
const Vec3 vector_in_robot_coordinates{Vec3::Random()};
const Vec3 vector_in_scene_coordinates{
    scene_from_robot.rotate(vector_in_robot_coordinates)};
REASSERT(
    vector_in_scene_coordinates ==
    scene_from_robot_rotation * vector_in_robot_coordinates);
```

Note that the `rotate()` member function is different than `operator*()` for `SE3` even though it is equivalent for `SO3`.

Like `SO3`, `SE3` also has an `interp()` member function which can be used to interpolate elements of the group:

C++

```
const SE3 scene_from_interped{scene_from_robot.interp(0.5)};
REASSERT(
    scene_from_robot.is_approx(scene_from_interped * scene_from_interped));
```

Note that interpolation of the Lie group (\\text{SE(3)}) follows a smooth geodesic curve between the two frames. This property can confer some advantages

- especially in robotics applications - but, it's important the caller is aware of this.

## Derivatives

So far, we have dealt with the static pose of an example robot relative to the scene. A natural next step is to discuss how to work with time derivatives of poses since robots typically have components that change their poses over time. We very frequently use objects called *tangent vectors* to express this velocity information. We'll define these vectors here without diving deeply into the math (for more see [Lie Group Derivatives](../liegroup_derivatives/)). For (\\text{SE(3)}), we can represent any element/object (g) with a 4x4 matrix of the form:

[ g = \\begin{bmatrix} R & t \\ 0 & 1 \\ \\end{bmatrix} ]

Where (R) is a 3x3 [rotation matrix](https://en.wikipedia.org/wiki/Rotation_matrix), and (t) is the 3x1 translation vector. To operate on a point (p), we simply append 1 to it and we can see:

[ g * p= \\begin{bmatrix} R & t \\ 0 & 1 \\ \\end{bmatrix} \\begin{bmatrix} p \\ 1 \\ \\end{bmatrix} = \\begin{bmatrix} Rp + t \\ 1 \\end{bmatrix} ]

Because we can represent our poses using a matrix, we now have a way to represent their derivatives. We can just store the ordinary time derivatives of the matrix. This will involve us storing 12 floating point numbers: 9 for the rotation matrix and 3 for the translation. We can do better however. After all, we know that the rotation time derivatives (a.k.a. the angular velocity) should be expressible with only three numbers. There's a trick that we can use here. If we multiply our ordinary time derivative (\\dot g) by (g^{-1}), we get a simpler form:

[ g^{-1} \\dot g = \\begin{bmatrix} 0& -\\omega_3 & \\omega_2 & v_1\\ \\omega_3 & 0& -\\omega_1 & v_2\\ -\\omega_2 & \\omega_1 & 0& v_3\\ 0 & 0 & 0 & 0& \\ \\end{bmatrix} ]

Now we only have to store six number in what we call a **right** tangent vector:

[ \\begin{bmatrix} \\omega_1 & \\omega_2 & \\omega_3 & v_1 & v_2 & v_3 \\end{bmatrix} ^T ]

This is a "right" tangent vector because the ordinary time derivative (\\dot g) appears on the right side of the expression (g^{-1} \\dot g). As you might imagine, there are also "left" tangent vectors because the expression (\\dot g g^{-1}) happens to also produce matrices of exactly the above form. However, the left and right tangent vector representations of the time derivative are in general different for a given robot trajectory, so care must be taken when dealing with them. For simplicity, we only deal with right tangent vectors here.

One can verify that if (g) as written above is `scene_from_robot`, then (\\omega_1), (\\omega_2), and (\\omega_3) in the right tangent vector represent the components of the robot's angular velocity expressed in its own coordinates. If, for example, they are ([1, 0, 0]), then the robot is rolling to its right (assuming the x axis is forward). One can further verify that (v_1), (v_2), and (v_3) are the robot's linear velocity expressed in its own coordinates. If they are ([1, 0, 0]) the actor is moving along its own x axis.

When working with these tangent vectors in code, we often want to easily access the components from them. For example, let's say we want to know what the velocity (v) and angular velocity (\\omega) are in **scene** coordinates. The paragraph above tells us how to get these velocities in robot coordinates, and then we need to rotate them to get them into scene coordinates. The variable `d_scene_from_robot` holds a right tangent vector representing our time derivative (e.g. (g^{-1}\\dot g) if (g) represents `scene_from_robot`). `SE3` is equipped with helpers to get these components from a tangent vector:

C++

```
const SE3::TangentVector d_scene_from_robot = SE3::TangentVector::Random();

const Vec3 robot_angular_velocity_in_robot_coordinates{
    SE3::tangent_vector_rotation_part(d_scene_from_robot)};
const Vec3 robot_velocity_in_robot_coordinates{
    SE3::tangent_vector_translation_part(d_scene_from_robot)};

const Vec3 robot_angular_velocity_in_scene_coordinates{
    scene_from_robot.rotation() *
    robot_angular_velocity_in_robot_coordinates};

const Vec3 robot_velocity_in_scene_coordinates{
    scene_from_robot.rotation() * robot_velocity_in_robot_coordinates};
```

If we want to go in the opposite direcion, we can also do that:

C++

```
REASSERT(
    d_scene_from_robot == SE3::tangent_vector_from_parts(
                              robot_angular_velocity_in_robot_coordinates,
                              robot_velocity_in_robot_coordinates));
```

Note

For (\\text{SO(3)}), the tangent vector matrices have a simpler form:

[ g^{-1} \\dot g = \\begin{bmatrix} & -\\omega_3 & \\omega_2 \\ \\omega_3 & & -\\omega_1 \\ -\\omega_2 & \\omega_1 & \\ \\end{bmatrix} ]

Where (\\omega) is once again the angular velocity as described above.

## Exp and Log

We touched breifly above on the fact that the exponential on (\\text{SO(3)}) converts angle axis representations to elements of (\\text{SO(3)}), but we did not fully explain this exponential operation that the `SE3` and `SO3` classes both implement. This operation takes a tangent vector (i.e. a time derivative) and tells you where you end up in the group following that constant time derivative for one unit of time. More explicitly, if we think about the idea of the the matrix representation of the right tangent vector (g^{-1} \\dot g) (as described above) being constant along a trajectory, we realize that this statement is a matrix ordinary differential equation (ODE):

[ g^{-1} \\dot g = \\text{Constant} = X ]

[ \\dot g = g X ]

This matrix ODE has a known solution given by the [Matrix exponential](https://en.wikipedia.org/wiki/Matrix_exponential):

[ g = g_0\\text{Exp}(Xt) = g_0\\left( \\sum\_{k = 0}^{\\infty} \\frac{1}{k!} X^k t^k\\right) ]

Where (g_0) is some initial condition for (g). The exponential map (\\exp()) defined for `SE3` and `SO3` is the same as this matrix exponential up to the input formatting. For instance, for (\\text{SO(3)}):

[ \\exp\\left(\\begin{bmatrix} \\omega_1 \\ \\omega_2 \\ \\omega_3 \\end{bmatrix}\\right) = \\text{Exp}\\left(\\begin{bmatrix} & -\\omega_3 & \\omega_2 \\ \\omega_3 & & -\\omega_1 \\ -\\omega_2 & \\omega_1 & \\ \\end{bmatrix}\\right) ]

In practice, we don't have to compute an infinite series for (\\exp()) because there are closed forms for (\\text{SO(3)}) (the [Rodrigues formula](https://en.wikipedia.org/wiki/Rodrigues%27_rotation_formula)) and (\\text{SE(3)}).

We also provide the logarithm function, which is the inverse of (\\exp()). In other words, it gives the constant velocity needed to arrive at the given element from the identity in one time unit. For their complex definitions, using these in code is quite simple:

C++

```
  REASSERT(my_tangent_vector.isApprox(SE3::exp(my_tangent_vector).log()));
```

## Frame Checking

So far we've ignored the fact that our `SO3` and `SE3` classes come equipped with frame checking capabilities (as alluded to in [Lie Groups](../liegroups/)). In brief, each `SO3` and `SE3` object can be assigned two coordinate frames, represented by `Frame` objects. This is done by passing the coordinate frames into the constructor or into the exponential or identity member functions. When assigned, these objects ensure that frame consistency is maintained when composing Lie groups. Here's an example:

C++

```
#include "resim/assert/assert.hh"
#include "resim/transforms/frame.hh"
#include "resim/transforms/se3.hh"
#include "resim/transforms/so3.hh"
#include "resim/visualization/view.hh"

using resim::transforms::SE3;
using resim::transforms::SO3;
using Frame = resim::transforms::Frame<SE3::DIMS>; /* SE3::DIMS == 3 */

// ...

const Frame world{Frame::new_frame()};
const Frame robot{Frame::new_frame()};
const Frame sensor{Frame::new_frame()};

// The pose of the robot in the world
const SE3 world_from_robot{SO3::identity(), {5., 5., 0.}, world, robot};

// Visualize with ReSim View
VIEW(world) << "World frame";
VIEW(robot) << "Robot frame";
VIEW(world_from_robot) << "World from robot";

// The pose of a sensor mounted on the robot
const SE3 robot_from_sensor{
    SO3{M_PI_2, {0., 0., 1.}},
    {0., 0., 1.},
    robot,
    sensor};

// Visualize with ReSim View
VIEW(sensor) << "Sensor frame";
VIEW(robot_from_sensor) << "Robot from sensor";

const SE3 world_from_sensor{world_from_robot * robot_from_sensor};
REASSERT(world_from_sensor.is_framed());
REASSERT(world_from_sensor.into() == world);
REASSERT(world_from_sensor.from() == sensor);

// Whoops! This fails at run time because we forgot to invert
// robot_from_sensor!
// const SE3 robot_from_world{world_from_sensor * robot_from_sensor};

// We should have done:
const SE3 robot_from_world{world_from_sensor * robot_from_sensor.inverse()};
```

Using frame checking is generally good practice as it makes it less likely for silly bugs to occur. In the future, we plan on making it possible to deactivate frame checking as an optional performance optimization. Note that composition with an unframed `SO3` or `SE3` *always* results in an unframed `SO3`/`SE3`. Consequently, unframed objects can propogate rapidly if one is not deliberate about using framed objects.

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/liegroups.cc) for the examples above.

# Lie Group Derivatives

## Introduction

This document provides some mathematical background on Lie groups and Lie algebras. In particular, this document covers how differentiation works on Lie groups, the Lie group exponential, left and right tangent spaces, the adjoint representations of a Lie group and its algebra, and the chain rule for Lie groups.

## Differentiating Curves on Lie Groups

By definition, a Lie group is any differentiable manifold which is also a group. A differentiable manifold is not generally [diffeomorphic](https://en.wikipedia.org/wiki/Diffeomorphism) to Euclidean space (e.g. (\\mathbb{R}^N)), but it is locally diffeomorphic to Euclidean space in the sense that we can define a diffeomorphism (\\phi_p) with (\\mathbb{R}^N) in the neighborhood of any point (p) in our group (G). We don't know how to take derivatives directly on the group (G), but we *do* know how to do this on Euclidean space. So we define the derivative with respect to a **particular** choice of (\\phi_p) in the neighborhood of the point (p) where we're taking the derivative. For a trajectory (g:\\mathbb{R}\\rightarrow G) with (g(t) = p), this looks like:

[ \\frac{d^{\\phi_p}g(t)}{dt} \\equiv \\frac{d}{dt}(\\phi_p \\circ g)(t) ]

Since (\\phi_p \\circ g) is just a function from (\\mathbb{R}) to (\\mathbb{R}^N), we can take its derivative in the standard way, and the typical properties of derivatives (e.g. the chain rule) all apply. We just need to come up with a consistent way of picking (\\phi_p) for whatever point (p) we're at. For a Lie group, one way we can do this is to first define (\\phi_e) for the identity element (e\\in G) and then we can define (\\phi_p) in the neighborhood of any point (p), since Lie group elements are guaranteed to have an inverse. We can do this like so:

[ \\phi_p(g) = \\phi_e(p^{-1} g) ]

So our derivative at a point (p = g(t)) along our trajectory is given by:

[ \\frac{d^{\\phi_p}g(t)}{dt} = \\frac{d}{d\\epsilon} \\phi_e (g(t)^{-1} g (t + \\epsilon)) ]

This happens to be a nice choice for (\\phi_p) for each point (p) because it has a property called left invariance. This means that the derivative is the same for a curve that is left multiplied by a **constant** element of (G). Say that we're differentiating (f(t) = hg(t)) for constant (h\\in G) and (g:\\mathbb{R}\\rightarrow G). In this case, (p = f(t) = h g(t)) and we have:

[ \\begin{aligned} \\frac{d^{\\phi_p} f}{dt} &= \\frac{d}{dt} \\phi_p(h g(t)) \\ &= \\frac{d}{dt} \\phi_e(p^{-1} h g(t)) \\ &= \\frac{d}{d\\epsilon} \\phi_e((g(t)^{-1} h^{-1}) h g(t + \\epsilon)) \\ &= \\frac{d}{d\\epsilon} \\phi_e(g(t)^{-1} g(t + \\epsilon)) \\ &= \\frac{d^{\\phi\_{g(t)}} g}{dt} \\end{aligned} ]

Which demonstrates the left invariance.

## Lie Group Exponential

Now, let's define a curve (\\gamma: \\mathbb{R} \\rightarrow G) with the properties that ((d^{\\phi_p}\\gamma / dt)) as defined above is **constant** at all points along the curve and that (\\gamma(0) = e). We claim without proof that it's possible to uniquely define such a curve. If we define (\\gamma_2(t) = g^{-1} \\gamma(t)) for some (g) in the range of (\\gamma), then we can conclude that (\\gamma_2(t) = \\gamma(t - \\gamma^{-1}(p))) because (\\gamma_2) also goes through the origin (at (t = \\gamma^{-1}(p))) and has the same constant value for ((d^{\\phi_p} \\gamma_2/dt)). Therefore, the curves have the same range, and (\\gamma_2(0) = p^{-1}) is also a member of the range of (\\gamma). We can do this for any (p) in the range of (\\gamma). We can conclude that the range of (\\gamma) is a subgroup of (G) because it contains the identity, every element in the subgroup has an inverse, and it is closed under composition with other members of the subgroup. This is called a **one parameter subgroup** of (G).

Now, we can go further by defining the *Lie Group Exponential* as the function (\\exp: \\mathbb{R}^N \\rightarrow G) satisfying (\\exp(X) = \\gamma(1)) where (\\gamma) is defined as above with (X = (d^{\\phi_p}\\gamma /dt)). In other terms, the exponential tells you where you end up if you follow the same constant derivative for one unit of time through the Lie group. The standard exponential on real numbers falls out of this definition if you consider curves with constant derivatives that intersect the multiplicative identity:

\[ \\begin{aligned} \\frac{d^{\\phi_p}y}{dt} &= \\frac{d}{d\\epsilon} \\left[y(t)^{-1} y(t + \\epsilon)\\right] = k \\ y(0) &= 1 \\ \\end{aligned} \]

With some rearrangement and variable substitution:

[ \\begin{aligned} y'(t) &= ky \\ y(0) &= 1 \\ \\end{aligned} ]

which has a known solution (y = \\exp(t)). Therefore, the Lie group exponential can be interpreted as a natural extension of the familiar concept of the exponential. In fact, for matrix Lie groups (Lie groups which have a matrix representation), the Lie group exponential is identical to the matrix exponential. Furthermore, just as the ordinary exponential has an inverse, the logarithm, there is a Lie group logarithm, which is the inverse of the Lie group exponential. Conceptually, the logarithm tells you "what constant velocity would I have to move at to arrive at this point in one unit of time from the origin."

It's worth noting that (\\mathbb{R}^N) is not formally the input to the exponential. The domain of the exponential is actually called the **Lie Algebra** (\\mathfrak{g}) corresponding to the Lie group (G). It is a vector space and hence it is homeomorphic to (\\mathbb{R}^N), so we casually treat them as interchangeable.

The exponential and logarithm define a correspondence between the Lie group and a vector space (\\mathbb{R}^N) called the Lie algebra. This connection allows us to leverage much of the mathematics derived in vector spaces on Lie groups which can be incredibly powerful. For instance, one can define a Gaussian probability distribution on the Lie algebra (\\mathfrak{so}(3)) and take the exponential of samples from it to sample orientations in the Lie group (\\text{SO(3)}). Because they are commonly useful, we provide implementations of the exponential and logarithm for each Lie group we implement.

## Left and Right Tangents

So far, we have been working with the left invariant definition for the derivative:

[ \\frac{d^{\\phi_p}g(t)}{dt} = \\frac{d}{d\\epsilon} \\phi_e (g(t)^{-1} g (t + \\epsilon)) ]

However, we've been very vague about what we may pick for (\\phi_e). We know it needs to be a diffeomorphism defined in an open set around (e). It turns out that the Lie group logarithm is a reasonable choice for this, so this is what we use. Hence:

[ \\phi_p(g) = \\log(p^{-1} g) ]

As noted before, this definition is left invariant since we can multiply on the left by any member of the group without changing the value of the derivative. This is practically quite useful because if (g) represents a transform of `world_from_robot` as a function of time, it doesn't actually matter where we decide the "world" frame is when we're computing derivatives as long as our choice does not move relative to some agreed-upon world frame (remember that (h) must be constant for the above to work. Confusingly, although this choice for (\\phi_p) yields *left* invariance, ((d^{\\phi_p}g/dt)) is often referred to as the "right tangent" or "right tangent space derivative" of (g). This is because it can also be defined in terms of the *right* perturbation to (g) that the trajectory is "following" at that moment.

\[ g(t + \\epsilon) = g(t) \\exp\\left[\\frac{d^{\\phi_p}g}{dt} \\epsilon\\right] \]

which when rearranged gives:

\[ \\frac{d^{\\phi_p}g}{dt} = \\frac{\\log\\left[g(t)^{-1} g(t + \\epsilon )\\right]}{\\epsilon} \]

In the limit as (\\epsilon \\rightarrow 0), we get:

[ \\frac{d^{\\phi_p}g}{dt} = \\frac{d}{d\\epsilon} \\log(g(t)^{-1} g(t + \\epsilon)) ]

which is equivalent to what we had before. For this reason, we henceforth refer to this definition of (\\phi_p) as (\\phi_p^R). The reason for the special notation, is that we could equally well have chosen:

[ \\phi_p(g) = \\phi_p^L(g) = \\log(g p^{-1}) ]

As before, (g p^{-1}) is in the neighborhood of the identity (e) when (g) is in the neighborhood of (p), so this works. With this, we get:

[ \\begin{aligned} \\frac{d^{\\phi_p^L} g}{dt} &= \\frac{d}{dt} \\log(gp^{-1}) \\ &= \\frac{d}{d\\epsilon} \\log(g(t + \\epsilon) g(t)^{-1}) \\end{aligned} ]

((d^{\\phi_p}g/dt)) is often referred to as the "left tangent" or "left tangent space derivative" of (g), and one can verify that it can be defined in terms of a left perturbation to (g) that the trajectory is following and that it has a **right** invariance property such that multiplying the trajectory on the **right** by a constant element of the group does not affect its value.

In summary, we have left and right tangent space derivatives that have right and left invariance respectively.

## The Adjoint

In practice both the left and right tangent space derivatives happen to be useful in particular cases, so one might ask if it's possible to easily convert between the right tangent of a trajectory and the left tangent of the trajectory. If we want the left tangent space derivative, we can see:

\[ \\begin{aligned} \\frac{d^{\\phi_p^L}g}{dt} &= \\frac{d}{dt}\\log(g p^{-1}) \\ &= \\frac{d}{dt}\\log((p p^{-1}) g p^{-1}) \\ &= \\frac{d}{dt}\\log(p (p^{-1} g) p^{-1}) \\ &= \\frac{d}{dt}\\log(p \\exp[\\log(p^{-1} g)] p^{-1}) \\ \\end{aligned} \]

Applying the chain rule to (h(k(t))) where (k(t) = \\log(p^{-1} g)) and (h(k) = \\log(p \\exp(k) p^{-1})) are both functions to and from Euclidean space, we have:

\[ \\begin{aligned} \\frac{d^{\\phi_p^L}g}{dt} &= \\left[\\frac{d}{dk} \\log(p \\exp(k) p^{-1})\\right] \\frac{d^{\\phi_p^R}g}{dt} \\end{aligned} \]

so we can convert from right to left tangent space by multiplying by this Jacobian matrix:

\[ \\text{Ad}\_g \\equiv \\left[\\frac{d}{dk} \\log(g \\exp(k) g^{-1})\\right] \]

where we define this Jacobian to be the adjoint representation of the element (g\\in G). Technically, the adjoint representation is the map that produces such matrices given inputs from (G). This map can also be defined as the derivative of a curve (f(g) = p g p^{-1}) at the identity. Since this is a map from (G) to (G), one can show this by assuming that (g) is a curve passing through the identity and using the chain rule as above to find what ((df/dg) = \\text{Ad}\_p) is. There are a number of properties that the adjoint has that are worth noting:

[ \\text{Ad}\_{g^{-1}} = \\text{Ad}\_g^{-1} ]

\[ \\text{Ad}_{gh} = \\text{Ad}_{g} \\text{Ad}\_{h} \]

which is basically just a reminder that the adjoint is a **representation** of G. In the case where the Lie group has a matrix representation (which is true for all the Lie groups we use), one can simplify our definition to be:

[ \\text{Ad}\_g X = g X g^{-1};\\quad\\quad g\\in G,,,X\\in \\mathfrak{g} ]

for any (X) in the Lie Algebra (\\mathfrak{g}) of (G). Note that this is a matrix representation of the Lie algebra element, not just a vector in (\\mathbb{R}^N). Expressions for (\\text{Ad}\_g) are derived for all the Lie groups we implement since it is so commonly needed.

It's often the case that we need to take the derivative of (\\text{Ad}\_g) with respect to time. We define the adjoint representation of the algebra (\\text{ad}\_X) based on a curve (g(t)) going through the identity.

\[ \\left.\\frac{d \\text{Ad}_g}{dt}\\right|_{g = e} = \\text{ad}_X;\\quad\\quad X = \\left.\\frac{dg}{dt}\\right|_{g = e} \]

It doesn't matter whether we use the right or left tangent space derivative here for (g) since they are equivalent at the identity, as one can verify by inspecting their definitions. The algebra adjoint is related to the Lie bracket:

\[ \\text{ad}\_X Y = [X, Y] \]

For matrix Lie groups, the bracket is the commutator on the matrix representation of algebra elements.

\[ \\text{ad}\_X Y = [X, Y] = XY - YX \]

The algebra adjoint is somewhat commonly used, so we provide it as a static member function in our Lie group objects.

## The Chain Rule

Let's look at how one might differentiate the composition of two Lie group elements in right tangent space, as an example. In other words, take the time derivative of (f(t) = g(t)h(t)):

\[ \\begin{aligned} \\frac{d^Rf}{dt} &= \\frac{d^R}{dt} [g(t)h(t)] \\ &= \\frac{d}{d \\epsilon} \\log [ h(t)^{-1} g(t)^{-1} g(t + \\epsilon) h(t + \\epsilon)] \\end{aligned} \]

To help ourselves, let's define a function (c(\\delta_1, \\delta_2)) like so:

\[ c(\\delta_1, \\delta_2) = \\log [ h(t)^{-1} g(t)^{-1} g(t + \\delta_1) h(t + \\delta_2)] \]

where (\\delta_1) and (\\delta_2) are functions of (\\epsilon). Taking the derivative with respect to (\\epsilon) with the multivariable chain rule gives:

[ \\frac{d}{d\\epsilon}c(\\delta_1, \\delta_2) = \\text{Ad}\_{h^{-1}} \\frac{d^R g}{d t} \\frac{d\\delta_1}{d\\epsilon} + \\frac{d^R h}{dt} \\frac{d\\delta_2}{d\\epsilon} ]

Of course, the derivative of (c) is useful to us if (\\delta_1 = \\delta_2 = \\epsilon) so:

[ \\frac{d^R f}{dt} = \\text{Ad}\_{h^{-1}} \\frac{d^R g}{dt} + \\frac{d^R h}{dt} ]

which is the chain rule in the right tangent space. There is also a chain rule for the left tangent space:

[ \\frac{d^L f}{dt} = \\frac{d^L g}{dt} + \\text{Ad}\_g \\frac{d^L h}{dt} ]

One can verify the left and right invariance properties using these expressions and assuming one of the group elements is constant.

# Curves

Curves are functions from a single real scalar variable (such as time) to another manifold (such as 3D space or a [Lie group](../transforms/liegroups/)). Consequently, they end up being very useful for representing the trajectory of a rigid body over time (among other things), so we provide two primary classes to represent curves through Lie groups:

- **DCurve**: This class represents an arc-length-parameterized curve through `SE3`. It gives an `SE3` object as a function of the distance travelled along the curve.
- **TCurve**: This class gives a twice continuously differentiable time-parameterized curve built up from a sequence of quintic Hermite spline segments.

Each of these has its own peculiarities and uses which we will outline in detail.

## DCurve

A `DCurve` is a sequence of geodesic segments that interpolate between a vector of `SE3` objects passed to its constructor. More precisely we can make a piecewise geodesic curve connecting the frames described by each `SE3` object we passed in order. We can query the curve as a function of the arc length along it using the `DCurve::point_at()` function. As the arc length increases from zero, the curve will move through each of the `SE3`s we passed into the constructor creating a continuous curve. `DCurve`s are useful whenever one has a set of points with headings (or full orientations) and they want to produce a reasonable continuous geometry connecting them. This could be the case if a user wants to represent static geometry, such as the center-line of a road, or a boundary around an area. However, one should note that the derivatives of the motion along a `DCurve` are not continuous when switching from one segment to the next. Furthermore, they do not work well if they have zero-length segments. Here's an example of how to construct and use a `DCurve`:

C++

```
#include "resim/assert/assert.hh"
#include "resim/curves/d_curve.hh" 
#include "resim/transforms/se3.hh"
#include "resim/transforms/so3.hh"
#include "resim/visualization/view.hh"

// ...

using resim::transforms::SE3;
using resim::transforms::SO3;
using Frame = resim::transforms::Frame<3>;

const Frame world_frame{Frame::new_frame()};
VIEW(world_frame) << "World frame";
const Frame d_curve_frame{Frame::new_frame()};
VIEW(d_curve_frame) << "DCurve";

// Make a DCurve that goes straight for one unit, turns left along a unit
// circle for pi/2 radians, continues straight in the y direction for one
// unit, and then turns to the right back to its original heading along a unit
// circle.
const resim::curves::DCurve<SE3> d_curve{
    SE3::identity(world_frame, d_curve_frame),
    SE3{{1., 0., 0.}, world_frame, d_curve_frame},
    SE3{SO3{M_PI_2, {0., 0., 1.}}, {2., 1., 0.}, world_frame, d_curve_frame},
    SE3{SO3{M_PI_2, {0., 0., 1.}}, {2., 2., 0.}, world_frame, d_curve_frame},
    SE3{{3., 3., 0.}, world_frame, d_curve_frame},
};

// Query a point halfway along the third edge:
const double QUERY_ARC_LENTH = 1.0 + M_PI_2 + 0.5;
const SE3 reference_from_queried{d_curve.point_at(QUERY_ARC_LENTH)};

REASSERT(
    reference_from_queried.rotation().is_approx(SO3{M_PI_2, {0., 0., 1.}}));
REASSERT(reference_from_queried.translation().isApprox(
    Eigen::Vector3d{2., 1.5, 0.}));

VIEW(d_curve) << "My DCurve";
```

This produces a curve that looks like this:

## Two Jet

Before we address `TCurve` objects, we need to discuss the `TwoJet` object. Briefly, a `TwoJet` is a struct containing a Lie group element along with its first and second derivatives. There are two varieties of `TwoJet`, namely `TwoJetR` and `TwoJetL`. The distinction between them is that `TwoJetR` represents its first and second derivatives as *right tangent vectors* (described fully in [Using SO(3) and SE(3)](../transforms/using_liegroups/) and [Lie Group Derivatives](../transforms/liegroup_derivatives/)), whereas `TwoJetL` uses *left tangent vectors*. For simplicity, we'll use the `actor::state::RigidBodyState` abstraction in our example to make defining these easy. That way, we can write clearer code that doesn't require us reason about tangent vectors.

## TCurve

A `TCurve` represents a time-based trajectory using a sequence of quintic [Hermite splines](https://en.wikipedia.org/wiki/Hermite_interpolation) on the group (\\text{SE(3)}) (or (\\text{SO(3)})). The details of this operation are outlined [here](https://ethaneade.com/lie_spline.pdf), but the main take-away is that this creates a curve through (\\text{SE(3)}) which is twice continuously differentiable ((\\text{C}^2)) as it proceeds through a set of user-provided control points. All we need to create a `TCurve` is a sequence of control points which each have a time along with a `TwoJetL` that describes the desired pose and derivatives at that time. One major use of `TCurve`s is to replay observed actor trajectories in a simulation. One simply has to take the observed states of the actors from an autonomy log and convert them into control points from which they can make a `TCurve`. Then, the simulation just has to query this `TCurve` over time to replay the trajectory. Usually, it is recommended to use the `actor::state::Trajectory` class, which is a higher-level wrapper for `TCurve` in such cases. Note that the control points must be monotonically increasing in time for the curve to be valid. Here's an example of how we can create one:

C++

```
#include "resim/actor/state/rigid_body_state.hh"
#include "resim/assert/assert.hh"
#include "resim/curves/t_curve.hh"
#include "resim/transforms/frame.hh"
#include "resim/transforms/se3.hh"
#include "resim/transforms/so3.hh"
#include "resim/visualization/view.hh"

// ...

using resim::transforms::SE3;
using resim::transforms::SO3;
using RigidBodyState = resim::actor::state::RigidBodyState<SE3>;
using TCurve = resim::curves::TCurve<SE3>;
using Frame = resim::transforms::Frame<3>;

const Frame t_curve_frame{Frame::new_frame()};
VIEW(t_curve_frame) << "TCurve";

// Define some states that we want to pass through
// A state at the origin with a small forward velocity
RigidBodyState state_a{SE3::identity(world_frame, t_curve_frame)};
state_a.set_body_linear_velocity_mps({0.1, 0., 0.});

// A state at (1, 1, 0) oriented along the x axis with a small forward
// velocity and a small angular velocity about the x axis.
RigidBodyState state_b{SE3{{1., 1., 0.}, world_frame, t_curve_frame}};
state_b.set_body_linear_velocity_mps({0.1, 0., 0.});
state_b.set_body_angular_velocity_radps({0.1, 0., 0.});

// A state at (3, 0, 0.5) with a rotation of pi/4 about the z axis
RigidBodyState state_c{
    SE3{SO3{M_PI_4, {0., 0., 1.}},
        {3., 0., 0.5},
        world_frame,
        t_curve_frame}};
state_c.set_body_linear_velocity_mps({0.1, 0., 0.});

// Create a t_curve by getting left two jets from the states:
const TCurve t_curve{{
    TCurve::Control{
        .time = 0.,
        .point = state_a.ref_from_body_two_jet().left_two_jet(),
    },
    TCurve::Control{
        .time = 10.,
        .point = state_b.ref_from_body_two_jet().left_two_jet(),
    },
    TCurve::Control{
        .time = 20.,
        .point = state_c.ref_from_body_two_jet().left_two_jet(),
    },
}};

// Visualize
VIEW(t_curve) << "My TCurve";
```

The resulting curve looks like this:

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/curves.cc) for the examples above.

# Rigid Body State & Trajectory

Two very common tasks in robotics are:

- Representing the position and orientation (i.e. pose) of a body along with its derivatives (velocity & acceleration) at a particular point in time. We call this a "rigid body state".
- Representing rigid body states over a bounded period of time. In other words: a trajectory.

Rigid body states and trajectories can be adequately represented by the `resim::curves::TwoJetR` and `resim::curves::TCurve` classes respectively. However, these are low-level mathematical representations that do not provide all the functionality we would like when reasoning about rigid body states and trajectories. Therefore, we provide two higher-level utility classes called `RigidBodyState` and `Trajectory`. These wrap `TwoJetR` and `TCurve` respectively and provide that functionality. In particular, using `RigidBodyState` objects, one can work directly with linear and angular velocities and accelerations expressed in the body's frame without having to reason directly about `SE3` tangent vectors. If we have a time series of `RigidBodyState` objects representing a trajectory, we can bundle these into a `Trajectory` object. The `Trajectory` object merely contains a `TCurve` under the hood, so the interpolation is performed in exactly the same way (i.e. using quintic Hermite splines). As an example of how to use these objects, lets make a unit circle trajectory:

C++

```
#include <Eigen/Dense>
#include <chrono>

#include "resim/actor/state/rigid_body_state.hh"
#include "resim/actor/state/trajectory.hh"
#include "resim/time/timestamp.hh"
#include "resim/transforms/se3.hh"
#include "resim/transforms/so3.hh"
#include "resim/visualization/view.hh"

// ...

using resim::transforms::SE3;
using resim::transforms::SO3;
using RigidBodyState = resim::actor::state::RigidBodyState<SE3>;
using Trajectory = resim::actor::state::Trajectory;
using Vec3 = Eigen::Vector3d;

// ...

// We will be orbiting the unit circle at a rate of one radian per second.
// Derivatives are constant along the unit circle:

// Moving forward at one unit per second
const Vec3 linear_velocity_mps = {1., 0., 0.};

// Yawing to the left at one unit per second
const Vec3 angular_velocity_radps = {0., 0., 1.};

// Importantly, the acceleration here is the derivative of the velocity
// expressed in the body's coordinate frame. Because the velocity expressed
// in the body's frame is constant in this case, the acceleration is zero.
// Because this coordinate frame is itself accelerating, the centripetal
// acceleration (i.e. mv^2 / r) *does not appear*.
const Vec3 linear_acceleration_mpss = {0., 0., 0.};
const Vec3 angular_acceleration_radpss = {0., 0., 0.};

const RigidBodyState::StateDerivatives derivatives{
    .velocity =
        {
            .linear_mps = linear_velocity_mps,
            .angular_radps = angular_velocity_radps,
        },
    .acceleration =
        {
            .linear_mpss = linear_acceleration_mpss,
            .angular_radpss = angular_acceleration_radpss,
        },
};

// We want to divide the circle into 4 segments, so we need to compute the
// amount of time it should take per second. Since we're orbiting at 1 radian
// per second, this time is just a quarter of the unit circle's circumference:
constexpr double TIME_PER_SEGMENT = M_PI_2;
constexpr resim::time::Timestamp START_TIME;
const Vec3 z_axis = {0., 0., 1.};

const Trajectory::Control control_a{
    .at_time = START_TIME,
    .state =
        RigidBodyState{SE3{SO3{M_PI_2, z_axis}, {1., 0., 0.}}, derivatives},
};
const Trajectory::Control control_b{
    .at_time = START_TIME + resim::time::as_duration(TIME_PER_SEGMENT),
    .state =
        RigidBodyState{SE3{SO3{M_PI, z_axis}, {0., 1., 0.}}, derivatives},
};
const Trajectory::Control control_c{
    .at_time = START_TIME + resim::time::as_duration(2. * TIME_PER_SEGMENT),
    .state =
        RigidBodyState{SE3{SO3{-M_PI_2, z_axis}, {-1., 0., 0.}}, derivatives},
};
const Trajectory::Control control_d{
    .at_time = START_TIME + resim::time::as_duration(3. * TIME_PER_SEGMENT),
    .state = RigidBodyState{SE3{SO3::identity(), {0., -1., 0.}}, derivatives},
};
const Trajectory::Control control_e{
    .at_time = START_TIME + resim::time::as_duration(4. * TIME_PER_SEGMENT),
    .state =
        RigidBodyState{SE3{SO3{M_PI_2, z_axis}, {1., 0., 0.}}, derivatives},
};

// Construct the trajectory
Trajectory unit_circle_trajectory{{
    control_a,
    control_b,
    control_c,
    control_d,
    control_e,
}};

// Query our unit circle and make sure we always are one unit from the origin:
const auto is_one = [](double x) {
  constexpr double TOLERANCE = 1e-12;
  return std::fabs(x - 1.) < TOLERANCE;
};
for (resim::time::Timestamp t = unit_circle_trajectory.start_time();
     t < unit_circle_trajectory.end_time();
     t += std::chrono::milliseconds(100)) {
  REASSERT(is_one(unit_circle_trajectory.point_at(t)
                      .ref_from_body()
                      .translation()
                      .norm()));
}

// Visualize the trajectory
VIEW(unit_circle_trajectory) << "My trajectory";
```

Running this code shows us our unit circle trajectory which looks like this:

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/trajectory.cc) for the example above.

# Multivariate Gaussian

[Gaussian (i.e. normal) distributions](https://en.wikipedia.org/wiki/Normal_distribution) are very useful in robotics (and in engineering more broadly) because physical quantities are quite often distributed in this way. This is because the sample mean of a collection of independent and identically distributed (iid) random variables is normally distributed assuming the underlying distribution has finite variance. Gaussian assumptions underlie the frequently-used [Kalman filter](https://en.wikipedia.org/wiki/Kalman_filter) algorithm, [optimal control theory](https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic%E2%80%93Gaussian_control), and many other useful mathematical tools for robotics.

Accordingly, it can be quite useful to use a Gaussian distribution when generating plausible physical values for use in a simulation. Of course, it is often the case that the values we might like to generate for a simulation are not independently distributed. For example, let's say we're trying to generate a realistic and representative road-using vehicle agent for our robot to interact with, and we want to generate values for the agent's height as well as a parameter representing their average speed. Intuitively we may expect that these parameters are negatively correlated since short sports cars tend to drive faster than tall class 8 trucks. Therefore, we shouldn't independently sample each parameter from a scalar Gaussian distribution. If we did, we would end up with as many fast-moving trucks as fast-moving sports cars! We instead want to sample from a realistic joint distribution for the two quantities which we approximate as a Gaussian with mean and covariance:

[ \\mu = \\begin{bmatrix} 20 \\ 2 \\end{bmatrix}\\quad\\quad \\Sigma = \\begin{bmatrix} 4.8 & -0.96 \\ -0.96 & 0.25 \\ \\end{bmatrix} ]

To sample from this distribution, we could do the following:

C++

```
#include <fmt/core.h>

#include <Eigen/Dense>
#include <fstream>

#include "resim/math/multivariate_gaussian.hh"

// ...

using resim::math::Gaussian;
using Vec = Gaussian::Vec;
using Mat = Gaussian::Mat;

// Set up our Gaussian sampler
Vec mean = Vec::Zero(2);
mean << 20., 2.;

Mat covariance = Mat::Zero(2, 2);
covariance << 4.8, -0.96, -0.96, 0.25;

Gaussian gaussian{mean, covariance};

// Sample the distribution:
Gaussian::SamplesMat samples = gaussian.samples(1000);

// Write our data out
std::ofstream output;
output.open("gaussian_samples.csv");
for (const auto &sample : samples.rowwise()) {
  output << fmt::format("{0}, {1}", sample.x(), sample.y()) << std::endl;
}
output.close();
```

Which gives the following samples:

As you can see, we've produced a qualitatively reasonable set of values where agents with smaller heights tend to have a higher average speed. We could now use these to generate a set of simulation scenarios with agents like this. In reality we would likely want to use sample means and covariances based on real world data rather than making up a qualitatively nice mean and covariance like we did for this example, but hopefully this gives a taste of how this library can be used in production code.

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/gaussian.cc) for the example above.

# Time

## Basic Types

Within the ReSim libraries, we generally represent durations using `resim::time::Duration`, which is an alias for `std::chrono::nanoseconds` and timestamps as `resim::time::Timestamp` which is an alias for `std::chrono::sys_time<resim::time::Duration>`.

## Converters

### Floating Point Representation

While we generally use the aforementioned integer-based timestamps and durations where possible, it is very often convenient to store times as double-precision floating point values, especially when using them in physical computations. If the values are sufficiently small (e.g. elapsed time in a sim), this can be done with no loss of accuracy. To facilitate, this, we have convenience converters:

C++

```
#include "resim/time/timestamp.hh"
#include "resim/assert/assert.hh"

// ...

using namespace resim::time;
Duration my_duration{std::chrono::nanoseconds(10000U)};

double my_duration_s = as_seconds(my_duration);

// Should pass since my_duration is small
REASSERT(as_duration(my_duration_s) == my_duration);

my_duration += std::chrono::system_clock::now().time_since_epoch();

my_duration_s = as_seconds(my_duration);

// Will likely not pass because my_duration is big and precision is lost
// converting it to a double.
REASSERT(as_duration(my_duration_s) == my_duration);
```

### Seconds & Nanoseconds Struct Representation

It's common in time serialization formats (e.g. [google::protobuf::Timestamp](https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/timestamp.proto) and ROS2's [builtin_interfaces/msg/Time.msg](https://github.com/ros2/rcl_interfaces/blob/master/builtin_interfaces/msg/Time.msg)) for the seconds and nanoseconds to be stored as separate integer counts. For `google::protobuf::Timestamp`, we provide standard packers and unpackers:

C++

```
const time::Timestamp time{};
google::protobuf::Timestamp time_msg{};

pack(time, &time_msg);

const time::Timestamp unpacked_time = unpack(time_msg);
```

To facilitate other conversions to and from such serialization types, we have a time representation called `SecsAndNanos`:

C++

```
struct SecsAndNanos {
  int64_t secs = 0;
  int32_t nanos = 0;
};
```

And converters to and from it:

C++

```
const SecsAndNanos my_secs_and_nanos = to_seconds_and_nanos(my_duration);
REASSERT(from_seconds_and_nanos(my_secs_and_nanos) == my_duration);
```

## Event Scheduling

Running simulations or processing logs often requires iterating through an ordered set of times in increasing order. These sets of interesting timestamps are not always uniformly spaced, and sometimes we want to add future times to this set as we are iterating through it. For instance, when modeling message transmission latency in a simulated system (i.e. one where we are explicitly controlling a simulated time), we may want to schedule a future time when we expect the message to arrive so we can properly transmit it to its receiver and keep the order of messages consistent with the real world system. To support this functionality, we define the `EventSchedule` class template which can store a time-ordered queue of events with arbitrary payloads. Here's a simple example of how to use it:

C++

```
#include <iostream>

#include "resim/time/event_schedule.hh"

// ...

EventSchedule<std::string> my_messages;
my_messages.schedule_event(Timestamp(std::chrono::nanoseconds(200)), "ReSim");
my_messages.schedule_event(Timestamp(std::chrono::nanoseconds(100)), "Hello");
my_messages.schedule_event(Timestamp(std::chrono::nanoseconds(200)), "user!");

std::cout.precision(9);
std::cout << std::fixed;
while (my_messages.size() > 0U) {
  const auto event = my_messages.top_event();
  std::cout << "[" << event.time.time_since_epoch().count() / 1e9 << "] "
            << event.payload << std::endl;
  my_messages.pop_event();
}
```

Note that the event schedule is "stable" in the sense that it's first-in-first-out for events with the same timestamp. Hence the code above will print out:

Text

```
[0.000000100] Hello
[0.000000200] ReSim
[0.000000200] user!
```

## Interval Sampling

Another common operation in running simulations and collecing log metrics is sampling an interval uniformly. For example, one may want to numerically compute a robot's average deviation from its desired pose over the course of a simulation. If a user wants to compute this average with Reimann rectangles, they could pick a maximum rectangle width (dt) that they are willing to accept (depending on what accuracy they desire) and then compute the integral using `resim::time::sample_interval()` and the associated `resim::time::num_samples()` function like so:

C++

```
#include <cmath>

#include "resim/time/sample_interval.hh"

// ...

const Timestamp start_time;
const Timestamp end_time{start_time + std::chrono::seconds(30)};
const Duration max_dt = std::chrono::microseconds(100);

// A stand in function for illustration purposes.
const auto deviation = [](const Timestamp &t) {
  return std::cos(std::sqrt(as_seconds(t.time_since_epoch())));
};

double integral = 0.;
sample_interval(start_time, end_time, max_dt, [&](const Timestamp &t) {
  // Left rectangles so we leave off the value at the end time
  if (t != end_time) {
    integral += deviation(t);
  }
});

// The actual dt per rectangle is the total interval divided by (N - 1)
const int N = num_samples(start_time, end_time, max_dt);
const double dt_s = as_seconds(end_time - start_time) / (N - 1);
integral *= dt_s;

// Make sure we match the analytical result
const double analytical_result = 2. * (-1. + std::cos(std::sqrt(30.)) +
                                       std::sqrt(30.) * std::sin(std::sqrt(30.)));
REASSERT(std::fabs(integral - analytical_result) < 1e-4);
```

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/time.cc) for the examples above.

# Error Handling

## Introduction

It is practically inevitable that errors will be encountered in any sufficiently complex software. In order to handle errors when they occur, we divide errors into the following categories:

- **Recoverable Errors:** This is the class of errors where this is some possibility of the error being handled gracefully by the calling code. An example might be the case where a client is trying to connect to a server and it should re-try if the connection fails.
- **Unrecoverable Errors:** This is the class of errors where there is (and should be) no way for the program to continue if encountered. Such an error means that the program is fundamentally invalid in some way. An example might be the case where I insert something into a `std::set` and then `std::map::contains()` returns false for that element. In this case, the contract of a specific interface has failed and there's no way for the caller to gracefully handle that.

Within ReSim's libraries, we have specific ways of handling each of these error types outlined below.

## Unrecoverable Errors: REASSERT()

For unrecoverable errors we use the `REASSERT()` macro. Using this macro is very simple:

C++

```
#include "resim/assert/assert.hh"

// ...

REASSERT(some_condition);
REASSERT(some_other_condition, "Some failure message.");
```

If `some_condition` and `some_other_condition` are both true, then this code runs with no problem. If `some_condition` is false, the program will exit immediately with:

Bash

```
terminate called after throwing an instance of 'resim::AssertException'
  what():  <path/to/source.cc:NN> - ReAssertion failed: (some_condition). Message: 
Aborted
```

If `some_other_condition` is false, then we'll get the Message field populated:

Bash

```
terminate called after throwing an instance of 'resim::AssertException'
  what():  <path/to/source.cc:NN> - ReAssertion failed: (some_other_condition). Message: Some failure message.
Aborted
```

Note that the text of the condition is always copied verbatim into the output. E.g.:

Text

```
REASSERT(2 + 2 == 5);
```

Outputs:

Bash

```
terminate called after throwing an instance of 'resim::AssertException'
  what():  <path/to/source.cc:NN> - ReAssertion failed: (2 + 2 == 5). Message: 
Aborted
```

Under the hood, `REASSERT()` throws a `resim::AssertException` if the condition is false. As noted below, we never catch this exception in production code, so it always terminates the program. The reason we use an exception at all is so that we can verify in unit tests that the assertion would terminate the program when it should do so. We use Google Test's `EXPECT_THROW()` macro for this:

C++

```
// my_library.cc
// ...
void my_failing_subroutine() { REASSERT(false); }

// my_library_test.cc
// ...
EXPECT_THROW(my_failing_subroutine(), AssertException);
```

## Recoverable Errors: Status & StatusValue

### Why not Exception Handling?

C++ has a built in language feature to handle this sort of error. Namely [exception handling](https://en.cppreference.com/w/cpp/language/exceptions). Developers can indicate that such an error has occurred by `throw`ing exceptions which can then be caught in a `try` block and gracefully handled. The [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html#Exceptions) has a detailed description of why using exceptions in this way is less than ideal. In brief, they can lead to very non-linear control flow which makes the code harder to reason about. Consequently, to the extent possible, we **never catch exceptions in our production libraries**.

### The Alternative

Instead, we use function return values to pass error information up to the caller. This is a well-trodden path with languages like [Rust](https://doc.rust-lang.org/reference/expressions/operator-expr.html#the-question-mark-operator) even adding language features to make this sort of error handling syntactically nice. At ReSim we use `resim::Status` for this. For example:

C++

```
#include "resim/assert/assert.hh"
#include "resim/utils/status.hh"

using namespace resim;

enum class Arg {
  GOOD_ARGUMENT = 0,
  BAD_ARGUMENT,
};

Status my_subroutine(const Arg arg) {
  if (arg == Arg::BAD_ARGUMENT) {
    // Use this to make a new Status object with line number information.
    return MAKE_STATUS("Oh no! We failed!");
  }
  return OKAY_STATUS;
}

// ...

Status good_status = my_subroutine(Arg::GOOD_ARGUMENT);
REASSERT(good_status.ok());
REASSERT(good_status.what() == "OKAY");

Status bad_status = my_subroutine(Arg::BAD_ARGUMENT);

REASSERT(not bad_status.ok());

// Call this macro if we decide we want to exit on bad status. I.e. if we
// don't want to gracefully handle the error, although that might be
// possible.
CHECK_STATUS_OK(bad_status);
```

When run, this code outputs the following:

Bash

```
terminate called after throwing an instance of 'resim::AssertException'
  what():  <path/to/source.cc:NN> - ReAssertion failed: ((bad_status).ok()). Message: {bad_status.what() == <path/to/source.cc:MM> Oh no! We failed!}
Aborted
```

Where `NN` is the line number where `CHECK_STATUS_OK()` is, and `MM` is the line number where `MAKE_STATUS()` is. Note that these might not generally be in the same source file, and we include both so that a user debugging the failure can more easily see where things began to go wrong and also where the issue became unrecoverable.

It's commonly the case that a function may want to handle bad statuses by "passing the buck" up to its caller. Assuming that the enclosing function also returns a status, we can utilize the `RETURN_IF_NOT_OK()` macro to make this easy:

C++

```
Status my_wrapping_subroutine(const Arg arg) {
  // This is equivalent to:
  // Status s = my_subroutine(arg);
  // if (not s.ok()) {
  //     return s;
  // }
  RETURN_IF_NOT_OK(my_subroutine(arg));
  return OKAY_STATUS;
}

// ...

Status good_status = my_wrapping_subroutine(Arg::GOOD_ARGUMENT);
REASSERT(good_status.ok());
REASSERT(good_status.what() == "OKAY");

Status bad_status = my_wrapping_subroutine(Arg::BAD_ARGUMENT);

REASSERT(not bad_status.ok());
```

### Working with Values

Sometimes, we also want to return a value from a function that can fail. To handle this we use the `StatusValue<T>` template where `T` is the type that we want to return. `StatusValue<T>` is designed and tested to preserve `const`ness and `ref`ness of the contained value (i.e. you can do `StatusValue<const int &>` and return a wrapped reference). It also implements the "buck-passing" behavior that `Status` does via the `RETURN_OR_ASSIGN()` macro which works in functions returning `Status` or `StatusValue<T>` for any `T`. Here's an example of `StatusValue<T>` in action:

C++

```
#include "resim/utils/status.hh"
#include "resim/utils/status_value.hh"

// ...

StatusValue<int> my_returning_subroutine(const Arg arg) {
  if (arg == Arg::BAD_ARGUMENT) {
    return MAKE_STATUS("Oh no! We failed!");
  }
  return 3;
}

StatusValue<double> my_returning_wrapper(const Arg arg) {
  int val = RETURN_OR_ASSIGN(my_returning_subroutine(arg));
  return 2.0 * val;
}

Status my_outer_wrapper(const Arg arg) {
  double val = RETURN_OR_ASSIGN(my_returning_wrapper(arg));
  std::cout << val << std::endl;
  return OKAY_STATUS;
}

// ...

Status good_status = my_outer_wrapper(Arg::GOOD_ARGUMENT);
REASSERT(good_status.ok());
REASSERT(good_status.what() == "OKAY");

Status bad_status = my_outer_wrapper(Arg::BAD_ARGUMENT);
CHECK_STATUS_OK(bad_status);
```

This code first prints `6` (for the `GOOD_ARGUMENT` case) and then outputs:

Bash

```
terminate called after throwing an instance of 'resim::AssertException'
  what():  <path/to/source.cc:NN> - ReAssertion failed: ((bad_status).ok()). Message: {bad_status.what() == <path/to/source.cc:MM> Oh no! We failed!}
Aborted
```

As the error propogates out of the wrappers to the `CHECK_STATUS_OK()` macro which terminates the program.

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/assert_and_status.cc) for the examples above.

# Variant Matching

## Overview

When working with variants, one very frequently encounters cases of branching logic depending on what's present in the variant. Typically this looks like this:

C++

```
std::variant<int, char, double, bool> my_variant = 'm';

// ...

if (std::holds_alternative<char>(my_variant)) {
  const char my_char = std::get<char>(my_variant);
  //
  // <Do something with my_char>
  //
} else if (std::holds_alternative<int>(my_variant)) {
  const int my_int = std::get<int>(my_variant);
  //
  // <Do something with my_int>
  //
} else {
  //
  // <Handle default case>
  //
}
```

It can be a bit annoying to set up this branching logic every time, and such an approach can sometimes be clunkier when multiple variant cases can be supported with the same logic (e.g. if we just want to convert all cases in `std::variant<int, float, double>` to `double`). To address such cases, we include an implementation of pattern matching that allows users to provide a list of functors (referred to as branches) that can be applied to the current variant to the `match()` function. The branch which best matches the current case of the variant according to C++'s overload resolution rules will be selected and executed on the current case of the variant. The `match()` function then returns any value returned by the selected functor. Note that all functors must have the same return type and all variant cases must match at least one of the functors.

## Example

Here's how you can perform pattern matching with the `match()` function:

C++

```
std::variant<int, char, double, bool> my_variant = 'm';

// ...

std::cout
    << match(
           my_variant,
           // case: char
           [](const char c) { return "This variant contains a char!"; },
           // case: int
           [](const int i) { return "This variant contains an int!"; },
           // default:
           [](const auto x) {
             return "This variant contains a double or bool!";
           })
    << std::endl;
```

Gotchas / Pitfalls

Due to the current implementation of this functionality, users should be aware of a few limitations. TL;DR just wrap every branch in a non-`mutable` lambda to be safe if you aren't sure.

- The functors passed must be *functors*. Function pointers don't work. As a simple work-around, users can wrap these in a non-`mutable` lambda.
- The functors should not define non-`const` `operator()`. This admonition includes any lambdas marked `mutable`. Because non-`const` member functions are preferred over `const` in overload resolution, this can cause ambiguities when other branches can potentially bind the current type (e.g. a `int` branch binding a `char`). We considered disabling all non-const `operator()` overloads, but decided that this would be more likely to cause silent errors in most cases (i.e. a default case might simply be used when a user didn't expect it). If a user must use a functor with a non-`const` `operator()`, they can work around it again by wrapping in a non-`mutable` lambda.
- Functors that are passed as lvalues will end up being copied. If a user wishes to avoid this as an optimization, they can do so by reference capturing it in a lambda that they then pass in instead.

Development Opportunities

- It's not all that hard to add some functionality to convert any function pointers passed into `match()` into lambdas, but we haven't done it yet since that use case is very rare.
- It's possible to avoid copying functors passed as lvalues, but we believe this would require a much more complex implementation than is warranted for this edge case.

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/match.cc) for the example above.

# Output Parameters

## InOut Wrappers

Although the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html#Inputs_and_Outputs) and consequently our style guide have a preference for return values over output parameters, there are still cases where having a function output data using a parameter is reasonable (e.g. when there are many outputs from a function and making a custom struct to return them all is inconvenient). Typically one could do this using pass-by-reference. Consider the following simple example:

C++

```
#include <iostream>

void set_to_three(int &x) {
  x = 3;
}

int main(int argc, char **argv) {
  int val;
  set_to_three(val);
  std::cout << val << std::endl;
  return 0;
}
```

This code as written works perfectly well. However, in practice such code can be bugprone because developers cannot tell whether the call to `set_to_three()` will modify its arguments. To improve code clarity, we therefore prefer the use of our `InOut` wrapper when passing in arguments that may be modified by a function. Using it works like so:

C++

```
void set_to_three(resim::InOut<int> x) { 
  *x = 3; 
}

int main(int argc, char **argv) {
  int val;
  set_to_three(resim::InOut{val});
  std::cout << val << std::endl;
  return 0;
}
```

In this case, the reader immediately knows that `val` may be changed by `set_to_three()`. Note that you can also use the arrow operator with `InOut`:

C++

```
#include <iostream>

#include "resim/utils/inout.hh"

struct Foo {
  int x = 0;
};

void set_to_three(resim::InOut<Foo> f) { 
  f->x = 3; 
}

int main(int argc, char **argv) {
  Foo f;
  set_to_three(resim::InOut{f});
  std::cout << f.x << std::endl;
  return 0;
}
```

Under the hood, `InOut` is simply a pointer to the object passed in, so it should be treated with care to avoid dangling references. It's only use is to annotate the code so that users know which arguments they pass may be modified. Typically, there is no danger of memory issues if, as in this example, we're simply wrapping and passing a variable from the stack into our function.

## NullableReference

Sometimes we only want a function to use an output parameter conditionally. Normally, one could use a raw pointer to do this. If the pointer is not `nullptr`, the function populates it, but otherwise leaves it alone:

C++

```
#include <iostream>

void maybe_set_to_three(int *x) {
  std::cout << "Ran maybe_set_to_three()!" << std::endl;
  if (x) {
    *x = 3;
  }
}

int main(int argc, char **argv) {
  int val;
  maybe_set_to_three(&val);
  std::cout << val << std::endl;
  maybe_set_to_three(nullptr);
  return 0;
}
```

As above, however, we would like to make things more explicit. In this case, we want to make it clear that the pointer is being used as a nullable reference and that no memory ownership is being passed from the caller to the function. To do this, we use the `NullableReference` template:

C++

```
#include <iostream>

#include "resim/utils/nullable_reference.hh"

void maybe_set_to_three(resim::NullableReference<int> x) {
  std::cout << "Ran maybe_set_to_three()!" << std::endl;
  if (x.has_value()) {
    *x = 3;
  }
}

int main(int argc, char **argv) {
  int val;
  maybe_set_to_three(resim::NullableReference{val});
  std::cout << val << std::endl;
  maybe_set_to_three(resim::null_reference);
  return 0;
}
```

In some ways, this wrapper is almost the same as `InOut`, and one could use `NullableReference` anywhere that an `InOut` could be used. However, it is important to have these as separate wrappers because a function can express more about how it treats its parameters by its choice of `InOut` (where its clear that the function **will** output something there) or `NullableReference` (where it's clear the function **won't** if you don't ask it to).

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/output_parameters.cc) for the examples above.

# ReSim View

Visualization is a critical accelerator for robotics development. When developing and debugging robotics code there is simply no substitute for a good picture of what is going on in 3D space. Objects like vectors, transforms, geometries, and curves are very difficult to reason about up until the moment they are visualized. Then, the human brain can bring to bear its innate abilities to reason about 3D space and gain insights much more quickly and correctly. It is for this reason that visualization is of prime importance to our mission.

The main visualization tool at the code level is ReSim View. In a single phrase, this tool is "print statements for 3D objects". In the same way that a print or log statement can help one reason about the behavior of non-robotics code, a View statement allows users a quick and lightweight way to see their 3D objects. Here's how it works:

First, a user adds a `VIEW()` statement to their code, enclosing one of a set of supported types. The types currently include:

1. Named coordinate frames (`Frame`).
1. 3D rotations (`SO3`).
1. 3D poses (`SE3`).
1. Arc length parameterized curves (`DCurve`).
1. Time parameterized curves (`TCurve` and `Trajectory`).
1. Vectors in a coordinate frame (`FramedVector`).

It's also possible to assign a name to each object being viewed which can make it much easier when viewing many different objects at once. This is done with the `<<` operator to pipe a name into the view statment. Here's an example of how one might visualize a transform with named coordinate frames:

C++

```
#include "resim/transforms/frame.hh"
#include "resim/transforms/se3.hh"
#include "resim/transforms/so3.hh"
#include "resim/visualization/view.hh"

// ...

using resim::transforms::SE3;
using resim::transforms::SO3;
using Frame = resim::transforms::Frame<SE3::DIMS /* 3 */>;

// Generate new frames with unique ids:
const Frame scene = Frame::new_frame();
const Frame robot = Frame::new_frame();

const SE3 scene_from_robot{
    SO3::identity(),  // Rotation
    {1., 0., 1.},     // Translation
    scene,            // Destination/Into Frame
    robot             // Source/From Frame
};

// We're using the << operator to add a name to our frame. If we don't use
// it, the frame id is used instead.
VIEW(scene) << "Scene Frame";
VIEW(robot);
VIEW(scene_from_robot) << "My Transform";
```

When this statement is executed at run time, it will prompt the user to authenticate (if they have not previously):

Text

```
Authenticate at the following URL: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX
```

And once authenticated the user will be given a link which they can follow to observe their visualization in Foxglove Studio.

Text

```
View your data visualization in the ReSim App: http://app.resim.ai/view?bucket=resim-mcap&path=sessions/059787c0-3800-47e7-b95b-9d2e593d4908/view.mcap
```

Following this link opens Foxglove Studio in your browser with the visualization:

If you don't initially see anything, you may have to enter the 3D panel settings and set its display frame to one of the frames in your scene (e.g. "Scene Frame" for this example):

Foxglove Studio will automatically build a scene graph for you from the frames you provide. Consider the following example:

C++

```
#include "resim/transforms/frame.hh"
#include "resim/transforms/se3.hh"
#include "resim/transforms/so3.hh"
#include "resim/visualization/view.hh"

// ...

using resim::transforms::SE3;
using resim::transforms::SO3;
using Frame = resim::transforms::Frame<SE3::DIMS /* 3 */>;

// Generate new frames with unique ids:
const Frame scene = Frame::new_frame();
const Frame robot = Frame::new_frame();
const Frame sensor = Frame::new_frame();

const SE3 scene_from_robot{
    SO3::identity(),  // Rotation
    {1., 0., 1.},     // Translation
    scene,            // Destination/Into Frame
    robot             // Source/From Frame
};

const SE3 robot_from_sensor{
    SO3(M_PI, {0., 0., 1}),  // Rotation
    {-0.25, 0.25, 0.25},     // Translation
    robot,                   // Destination/Into Frame
    sensor                   // Source/From Frame
};

// We're using the << operator to add a name to our frame. If we don't use
// it, the id is used instead.
VIEW(scene) << "Scene Frame";
VIEW(robot) << "Robot";
VIEW(scene_from_robot) << "Scene from Robot";

VIEW(sensor) << "Sensor";
VIEW(robot_from_sensor) << "Robot from Sensor";
```

As mentioned above, ReSim View is capable of visualizing many more data types. Rather than go through these in detail here, we refer the reader to the examples in `resim/visualization/examples/` to explore and play around with the full capabilities of the tool. Don't forget to play around with the "play" button in Foxglove Studio when working with the time parameterized `TCurve` or `Trajectory` objects!

Note

Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/view.cc) for the examples above.

# ReSim Log Format

The ReSim app is designed to work with any log format. This way, you can ensure that your simulation jobs produce the same types of logs that you might have from real-world data collection. This is often essential for tool or library re-use. In ReSim's open source library, however, we are working to provide a rich set of tools for computing and visualizing salient [metrics](https://github.com/resim-ai/open-core/tree/main/resim/metrics). We further aim to continue producing tools to analyze and leverage data from both simulation and real-world logs in the open source library. As we cannot write such tools to work with any log format, we maintain our own **ReSim Log Format**. We want this format to be as simple and portable as possible so that converters from other common robotics log formats are as trivial as possible. To this end we define the format like so:

The ReSim Log Format is based on Foxglove's [MCAP](https://foxglove.dev/blog/introducing-the-mcap-file-format) file format. Using this file format furnishes us with a pre-existing set of open source tools and libraries for reading, writing, and visualizing our logs. In particular, [Foxglove Studio](https://foxglove.dev/) natively visualizes this format efficiently.

Those familiar with the MCAP format will be aware that this format can hold arbitrary serialized robotics data with ROS1, ROS2, JSON, Flatbuffers, and Protobuf recieving first-class support from Foxglove's open-source libraries. Therefore, we define a set of protobuf message schemas in [open-core/resim/msg](https://github.com/resim-ai/open-core/tree/main/resim/msg) which our tools are designed to recognize and work with efficiently. This allows all of ReSim's metrics libraries to be leveraged for the small cost of writing log converters. For the ubiquitous [ROS2](https://ros.org/) middleware log format (CDR), such converters are [provided](../ros2/) for the defined ReSim message types. With that said, we have deliberately chosen to promulgate our own log format rather than using ROS's so that our users are not forced to depend on this large software ecosystem if they would prefer not to. Simultaneously, our provided converters eliminate integration friction for ROS users.

A ReSim log therefore consists of an MCAP file containing ReSim messages for topics of interest. Other topics may be present and may have any schema/serialization. These topics will generally be ignored by ReSim tools, or copied unmodified by tools that input and output mcap files. There may be exceptions to this in the future.

# ROS2

The [Robot Operating System](https://www.ros.org/) or ROS and its associated libraries represent one of the most ubiquitous collections of robotics software in use today. We therefore provide first class support for using ROS2 with ReSim's tools. To enable this without making ReSim's tools explicitly depend on ROS2 libraries, we provide a converter binary that can be used to convert ROS2 bags to the [ReSim Log Format](../msg) which our other tools and libraries natively consume.

## Basic Conversion

If you're using common ROS2 message types (e.g. those in `std_msgs` or `geometry_msgs`) to log your topics of interest the conversion is as simple as cloning [resim/open-core](https://github.com/resim-ai/open-core/) and running the following commands.

Bash

```
bazel run //resim/ros2:convert_log -- \
    --log </path/to/my/rosbag_dir/>   \
    --output </path/to/my/converted/log.mcap>
```

If there are common public types that you need which we don't yet support, feel free to reach out and we can add them.

A pybound version of this converter is also provided in the `resim_ros2` package distributed through our open-core [Releases](https://github.com/resim-ai/open-core/releases) and can be used like so:

Python

```
import resim.ros2.resim_log_from_ros2_python as rlr2
from resim.ros2 import RESIM_DIR

# ...

rlr2.resim_log_from_ros2(
    str(RESIM_DIR / "ros2" / "default_converter_plugin.so"), # See plugin info below
    input_log_path,
    converted_log_path)
```

## Custom Message Types & Converter Plugins

Occasionally, it is necessary or just convenient to convert a custom message type to a serialized ReSim protobuf type so that its contents can be used with ReSim's open source metrics and analysis tools. To facilitate this, the aforementioned converter tools are designed with a plugin architecture to allow custom converters to be written. The core of the plugin interface is the following function:

C

```
extern "C" ReSimConverterPluginStatus resim_convert_ros2_to_resim(
    const char *const ros2_message_type,
    const rcutils_uint8_array_t *const ros2_message,
    rcutils_uint8_array_t *const resim_message);
```

This essentially converts serialized ros2 message bytes to serialized resim message bytes based on the message type given as a string. There are also other functions that the binary needs to interrogate the plugin for:

- Which types are supported for conversion.
- Descriptions of the serialization format for the converted type (as required by Foxglove/MCAP tools).

The best practice here is to simply copy [resim/ros2/default_converter_plugin.cc](https://github.com/resim-ai/open-core/blob/main/resim/ros2/default_converter_plugin.cc) and add the messages you care about to it, following its example. One can also create a minimal plugin which dynamically loads the default converter plugin and uses it to handle all the common message types. Once you've compiled such a plugin, it can be provided using the `--plugin` flag for `convert_log`:

Bash

```
bazel run //resim/ros2:convert_log --          \
    --plugin </path/to/my/custom/converter.so> \
    --log </path/to/my/rosbag_dir/>            \
    --output </path/to/my/converted/log.mcap>
```

or in Python:

Python

```
import resim.ros2.resim_log_from_ros2_python as rlr2
from resim.ros2 import RESIM_DIR

# ...

rlr2.resim_log_from_ros2(
    my_custom_converter_plugin_path,
    input_log_path,
    converted_log_path,
)
```
# Explanation

# Explanation

Explanation is **understanding-oriented**: it illuminates the *why* behind ReSim's design. Read these articles when you want to build a mental model, not when you need step-by-step instructions.

## Articles

- [Understanding Batches & Tests](../guides/batches-and-tests/) — how ReSim structures test runs, what a batch is, and how individual tests relate to it

## ReSim U

ReSim U is a collection of deeper technical articles on the mathematics underlying autonomous systems:

- [Introduction to ReSim U](../resimu/introduction/)
- [Lie Group Dynamics](../resimu/lie_group_dynamics/)
- [Dual Quaternions for SE(3)](../resimu/dual_quaternions_for_se3/)

More explanation articles are coming, covering topics like the ReSim testing model, the metrics pipeline, and how to design good test experiences.

# Batches and tests

In this document, we will cover two fundamental concepts in the ReSim infrastructure that enable customers to run a large number of tests and aggregate the results: [tests](../../core-concepts/#test) and [test batches](../../core-concepts/#test-batch). This guide will help you understand what is happening when you are running thousands of tests.

## Understanding batches

You can trigger a batch in many ways in the ReSim platform, such as:

- the ReSim CLI tool
- ReSim App
- GitHub Action

If you need to run a thousand tests, you will initiate one batch that will encompass all those tasks. The Batch is the unit of work that processes an entire set of experiences.

## Understanding tests

Within each batch, individual work items are broken down into tests. When running experiences, Each test corresponds to a specific experience that runs.

## Detailed workflow

The following sections will illustrate each of the phases in the overall batch execution process defined here:

### Batch submission

When you submit a batch, you define:

- **Experiences**: the set of experiences you wish to run against your version of code.
- **Build**: Contains your simulator and autonomy code.
- **Metrics Build**: Used for analyzing test outputs.

For each experience a separate, parallel, test will be created that runs that experience through the build and then computes metrics on the outputs from the experience using a metrics build.

### Test execution and metrics

The execution of a test is split into two separate stages: an experience execution stage and a metrics stage:

1. **Experience Execution**: Each test runs a specific experience against that build, generating some outputs. The output of this stage is processed in the following metrics stage.
1. **Metrics Stage**: After execution, the test's output is analyzed using the metrics build. This is where your pass fail signal is configured and computed.

### Batch metrics and aggregation

Once all tests within a batch are completed, ReSim moves to the batch metrics stage. During this stage, you can define aggregation methods such as weighted averages in whatever way is best suited to your system.

## Conclusion

Running thousands of tests efficiently in the cloud is a challenge. Our use of batches and tests helps us organize the work and make it clear what state your test is in while it is running.

# Introduction

Welcome to our collection of ReSimU articles! As the name may suggest, this section of our documentation is intended to cover topics of a more academic interest relative to the rest of our documentation. More plainly, it's a place for us to share simulation-related derivations, demonstrations, and learnings which we feel to be intrinsically interesting and potentially useful for the typical robotics engineer to be aware of. Please enjoy and don't hesitate to reach out with any questions, observations, or errata. We're excited by the opportunity to improve these pages over time.

So far, we have the following articles:

- [Rigid Body Dynamics on Lie Groups](../lie_group_dynamics/)
- [Dual Quaternions for SE(3)](../dual_quaternions_for_se3/)

# Rigid body dynamics on Lie groups

*October 11, 2024*

*Michael Bauer*

## Introduction

While one can often get a lot of value from a simple kinematic model of a system (e.g. the bicycle model for wheeled vehicles), there are many cases in robotics where a simulation must adopt an accurate Newtonian dynamics model in order to usefully test the system in question. The bicycle model, for instance, may lose fidelity in extreme maneuvers with high lateral acceleration. This is why many robotics simulation tools (e.g. [Dartsim](https://dartsim.github.io/), [Drake](https://drake.mit.edu/), [Gazebo](https://gazebosim.org/docs)) provide rich rigid body dynamics solvers.

As we discuss and motivate in our [open core documentation](https://docs.resim.ai/open-core/transforms/liegroups/), ReSim uses Lie groups and their derivatives to describe the trajectories of simulated actors through the world. Given our extensive use of Lie groups for kinematic applications ([example](https://github.com/resim-ai/open-core/blob/85bb030069425bc5f446825be29afef2d5ddfcd5/resim/curves/t_curve_segment.hh#L9)), one may wonder if Lie groups can be used for dynamics problems as well. We can describe the pose and velocity of a rigid body using elements of the Lie group (\\text{SE}(3)) and the Lie algebra (\\mathfrak{se}(3)). However, having chosen these coordinates, is it feasible to predict how the system's state will evolve over time under the influence of external forces and torques? The answer is, of course, yes.

## Definitions

Lets start by defining some parameters. We begin by describing a rigid body's position and orientation. We do this by attaching a Cartesian coordinate frame to our body. For convenience, we place the origin of this coordinate frame at its [center of mass](https://en.wikipedia.org/wiki/Center_of_mass) and align its axes with the [principal axes of rotation](https://en.wikipedia.org/wiki/Moment_of_inertia#Principal_axes) of our body. This coordinate frame is attached to our rigid body such that every part of the body is always stationary in this coordinate frame. We then associate a group element (g\\in \\text{SE}(3)) with the pose of the rigid body. Specifically, the group action of (g) on a point's coordinates in our body coordinate frame gives that same point's coordinates in an inertial reference frame. For example, applying (g) to ((0, 0, 0)\\in \\mathbb{R}^3) (the origin in body coordinates) gives the position of the rigid body our inertial reference frame.

Now, lets define our generalized velocity (\\xi\\in\\mathfrak{se}(3)) as the time derivative of (g) in [right tangent space](https://docs.resim.ai/open-core/transforms/liegroup_derivatives/#left-and-right-tangents). Specifically, we define it as the real derivative:

\[ \\xi(t) = \\frac{d}{d\\epsilon}\\log\\left[g(t)^{-1}g(t + \\epsilon)\\right] \]

Although we often write this in shorthand:

[ \\xi(t) = \\frac{dg(t)}{dt} ]

See [Lie Group Derivatives](https://docs.resim.ai/open-core/transforms/liegroup_derivatives/) from our open core documentation for information on where the full expression comes from. The Chain Rule section of that document will be relevant as well. We choose a basis for our Lie algebra such that the first three entries of (\\xi) correspond to the angular velocity of the body in its own coordinate frame, and the last three entries of (\\xi) correspond to the linear velocity of the body in its own coordinate frame. In other words (\\xi^T=\\begin{bmatrix}\\omega^T & v^T\\end{bmatrix}) where (\\omega\\in\\mathbb{R}^3) is the angular velocity and (v\\in\\mathbb{R}^3) is the linear velocity.

Finally, we define our generalized inertia tensor (I\\in\\mathbb{R}^{6\\times 6}) like so:

[ \\mathcal{I} = \\begin{bmatrix}I\_{xx} \\ & I\_{yy}\\ & & I\_{zz} \\ & & & m \\ & & & & m \\ & & & & & m\\end{bmatrix} ]

Where (I\_{xx}), (I\_{yy}), and (I\_{zz}) are the moments of inertia about the principle axes and (m) is the mass of the body. The inertia tensor has this nice diagonal form due to our choice of body coordinate frame (origin at the center of mass and axes aligned with the principal axes of rotation), but it can in general be a dense symmetric matrix if we choose another coordinate frame attached to our body.

The important thing about these definitions is that we can now write the kinetic energy of our rigid body system like so:

[ \\begin{aligned}T = \\frac{1}{2} \\xi^T \\mathcal{I} \\xi &= \\frac{1}{2}\\omega^T I_R\\omega + \\frac{1}{2}m||v||^2 \\&= KE_R + KE_T\\end{aligned} ]

Where we can identify the terms as the rotational kinetic energy (KE_R) and the translational kinetic energy (KE_T).

## Forward dynamics

Using this expression for kinetic energy, we can derive an ordinary differential equation for the forward dynamics of the system. This ordinary differential equation will relate the system's generalized acceleration (\\dot \\xi) to its generalized velocity (\\xi). To do this, we will utilize a variational approach. In particular, we define the Lagrangian without any conservative forces:

[ L(g, \\xi, t) = T - V = \\frac{1}{2} \\xi^T \\mathcal{I} \\xi ]

and then apply the [Lagrange-D'Alembert Principle](https://en.wikipedia.org/wiki/D%27Alembert%27s_principle#Formulation_using_the_Lagrangian) which states:

\[ 0 = \\delta\\int\_{t_1}^{t_2} L(g,\\xi, t)dt

- \\int\_{t_1}^{t_2} (F \\cdot \\delta g) dt \]

Where (F\\in \\mathfrak{dse}(3)) is the generalized force on our rigid body (\\delta g\\in\\mathfrak{se}(3)) is an arbitrary infinitesimal variation of our system's six degree of freedom state. (\\mathfrak{dse}(3)) is the [*dual space*](https://en.wikipedia.org/wiki/Dual_space) of (\\mathfrak{se}(3)) and is also isomorphic to (\\mathbb{R}^6). One might guess that the first three components of (F) represent the torque on the system in body coordinates and the last three represent the linear force on the system in body coordinates and this is in fact correct. (F\\cdot \\delta g) is the virtual work done by the forces on our system applied along the virtual displacements (\\delta g). Now, let's start by plugging in the definition of (\\xi):

\[ 0 = \\delta\\int\_{t_1}^{t_2} \\left[ \\frac{1}{2}\\left( \\frac{dg}{dt}\\right)^T \\mathcal{I} \\left(\\frac{dg}{dt} \\right) \\right]dt

- \\int\_{t_1}^{t_2} (F \\cdot \\delta g) dt \]

Now, let's take the functional derivative (\\delta) on the first term. Translating this into more familiar terms, we define the derivative like so:

\[ 0 = \\frac{d}{d\\epsilon}\\left\[ \\int\_{t_1}^{t_2} \\left[ \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) \\right]dt \\right\]\_{\\epsilon = 0}

- \\int\_{t_1}^{t_2} (F \\cdot \\delta g) dt \]

Where we have substituted in the varied version of (g):

[ g\_\\epsilon := g\\exp(\\epsilon \\delta g) ]

Using the chain rule in right tangent space, we have:

\[ \\frac{dg\_{\\epsilon}}{dt} = \\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt}

- \\text{Dexp}(\\epsilon \\delta g) \\epsilon \\frac{d\\delta g}{dt} \]

Where (\\text{Dexp}) is the differential of the exponential map in right tangent space and (\\text{Ad}\_{\\exp(-\\epsilon \\delta g)}) is the [adjoint representation](https://docs.resim.ai/open-core/transforms/liegroup_derivatives/#the-adjoint) of (\\exp(-\\epsilon\\delta g)). Plugging this in and expanding the first term gives:

\[ \\begin{aligned} \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) =& \\frac{1}{2}\\left( \\text{Ad}_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\text{Ad}_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right) \\ &+ \\left( \\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\text{Dexp}(\\epsilon \\delta g) \\epsilon \\frac{d\\delta g}{dt} \\right) + O(\\epsilon^2) \\end{aligned} \]

Differentiating this with respect to (\\epsilon) and setting (\\epsilon\\rightarrow0) we get:

\[ \\frac{d}{d\\epsilon}\\left[ \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) \\right]_{\\epsilon=0} = -\\left( \\text{ad}_{ \\delta g} \\frac{dg}{dt} \\right)^T \\mathcal{I} \\frac{dg}{dt} + \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \]

This step merits some explanation. Looking at the first term, we recognize it as a quadratic form and compute the derivative of (\\left(\\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt}\\right)) with respect to (\\epsilon) first:

# \[ \\frac{d}{d\\epsilon}\\left( \\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right)

-\\text{Ad}_{\\exp(-\\epsilon \\delta g)} \\text{ad}_{\\delta g}\\frac{dg}{dt} \]

Then (\\text{Ad}\_{\\exp(-\\epsilon \\delta g)}=\\text{Ad}\_e = I_6) when (\\epsilon\\rightarrow 0) so this goes away. (\\text{ad}) is the adjoint representation of the Lie *algebra* which we get because we differentiated the group adjoint representation (\\text{Ad}). In the second term, we see that the whole term is multiplied by (\\epsilon) to begin with, so only one term resulting from the product rule will survive after (\\epsilon\\rightarrow 0). We then use the fact that the adjoints go to the identity and (\\text{Dexp}(0)=\\text{Identity}) also to arrive at the final expression above. Finally, one more manipulation is useful. Using the fact that (\\text{ad}\_uv = -\\text{ad}\_vu) we have:

\[ \\frac{d}{d\\epsilon}\\left[ \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) \\right]_{\\epsilon=0} = \\left( \\text{ad}_{\\frac{dg}{dt}}\\delta g \\right)^T \\mathcal{I} \\frac{dg}{dt} + \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \]

Plugging this into the integral and grouping terms yields:

\[ \\begin{aligned} 0 &= \\int\_{t_1}^{t_2} \\left\[ \\left( \\text{ad}_{\\frac{dg}{dt}}\\delta g \\right)^T \\mathcal{I} \\frac{dg}{dt} + \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \\right\]dt + \\int_{t_1}^{t_2} (F \\cdot \\delta g) dt \\ 0&= \\int\_{t_1}^{t_2} \\left[ \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \\right]dt + \\int\_{t_1}^{t_2} \\left[ \\left( F + \\text{ad}\_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} \\right)\\cdot \\delta g \\right] dt \\end{aligned} \]

Integrating the first term by parts, we get:

\[ \\begin{aligned} 0 &= -\\int\_{t_1}^{t_2} \\left\[ \\left( \\frac{d}{dt}\\left[ \\mathcal{I}\\frac{dg}{dt} \\right] \\right)\\cdot \\delta g \\right\]dt + \\int\_{t_1}^{t_2} \\left\[ \\left( F + \\text{ad}_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} \\right)\\cdot \\delta g \\right\] dt \\ 0 &= \\int_{t_1}^{t_2} \\left\[ \\left( F + \\text{ad}\_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} - \\frac{d}{dt}\\left[ \\mathcal{I}\\frac{dg}{dt} \\right] \\right)\\cdot \\delta g \\right\] dt \\end{aligned} \]

Note that the boundary terms are zero when integrating by parts since (\\delta g) is defined to vanish at (t_1) and (t_2). Using the fundamental theorem of variational calculus, we note that, because this must be true for any variations (\\delta g), we have:

\[ \\begin{aligned} 0 &= F + \\text{ad}_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} - \\frac{d}{dt}\\left[ \\mathcal{I}\\frac{dg}{dt} \\right] \\ 0&= F + \\text{ad}_\\xi^T\\mathcal{I}\\xi - \\mathcal{I} \\dot \\xi \\end{aligned} \]

So we have our forward dynamics:

[ \\boxed{\\mathcal{I}\\dot \\xi = F + \\text{ad}\_\\xi^T\\mathcal{I}\\xi} ]

As a sanity check let's look at what this expression looks like on (\\text{SE}(3)). For this group we can express elements of the algebra in a basis such that:

# \[ \\xi

\\begin{bmatrix} \\omega \\ v \\end{bmatrix} \]

Where (\\omega) and (v) are the angular and linear velocity expressed in body coordinates. We know:

# \[ \\text{ad}\_\\xi

\\begin{bmatrix} \\omega\_\\times & \\ v\_\\times & \\omega\_\\times \\end{bmatrix} \]

Assuming that our inertia tensor (I) has the diagonal form shown above, plugging these in yields:

# \[ \\begin{bmatrix} I\_\\text{rot} \\ & m \\end{bmatrix} \\begin{bmatrix} \\dot \\omega \\ \\dot v \\end{bmatrix}

## \\begin{bmatrix} \\tau \\ f \\end{bmatrix}

\\begin{bmatrix} \\omega\_\\times & v\_\\times\\ & \\omega\_\\times \\end{bmatrix} \\begin{bmatrix} I\_\\text{rot} \\ & m \\end{bmatrix} \\begin{bmatrix} \\omega \\ v \\end{bmatrix} \]

[ \\begin{aligned} I\_\\text{rot} \\dot \\omega &= \\tau - \\omega \\times I\_\\text{rot}\\omega \\ m\\dot v &= f - m\\omega\\times v \\end{aligned} ]

The first equation is identifiable as the vector form of [Euler's equations](<https://en.wikipedia.org/wiki/Euler%27s_equations_(rigid_body_dynamics)>) for rotation, which is exactly what we expect! Now, if we look at the second equation we have for the linear velocity in body-attached coordinates:

[ m\\dot v = f - m\\omega\\times v ]

Now, we make a substitution into inertial coordinates. The velocity in inertial coordinates is related to the body-attached velocity by the rotation: (u=Rv). From this, we can also derive:

[ \\begin{aligned} \\dot u &= \\frac{d}{dt}(Rv) = R(\\omega \\times v) + R\\dot v \\ \\dot v &= R^T\\dot u - \\omega\\times v \\end{aligned} ]

Substituting this into our translational dynamics (along with transforming the force into inertial coordinates (\\varphi=Rf)) we get:

[ \\begin{aligned} mR^T\\dot u - m\\omega\\times v &= R^T\\varphi - m\\omega \\times v \\ mR^T\\dot u &= R^T\\varphi \\ m\\dot u &= \\varphi \\end{aligned} ]

In other terms, (F = ma), which is Newton's Second Law. Therefore, we see that this single equation is in fact equivalent to the combination of Newton's Second Law and Euler's Equations of rigid body rotation. The expression (I\\dot \\xi = F + \\text{ad}_\\xi^TI\\xi) is actually a generalization of both of these equations and is a more general statement of Newton's Second Law. By choosing (\\mathbb{R}^3) under addition as a Lie group, one can derive (F=ma) immediately, and by choosing (\\text{SO}(3)) as the Lie group one can derive (I_\\text{rot} \\dot \\omega = \\tau - \\omega \\times I\_\\text{rot}\\omega) immediately.

## Simulating

We can then express our system's equations of motion like so:

[ \\begin{aligned} \\frac{dg}{dt}&=\\xi \\ \\frac{d\\xi}{dt} &= \\mathcal{I}^{-1}(F + \\text{ad}\_\\xi^T \\mathcal{I}\\xi) \\end{aligned} ]

This can be integrated using Forward Euler integration by using the Lie group exponential the (\\xi\\Delta t) step to (g):

[ \\begin{aligned} g(t\_{i+1})&=g(t_i)\\exp(\\xi_i \\Delta t) \\ \\xi(t\_{i+1})&=\\xi(t_i) + \\mathcal{I}^{-1}(F + \\text{ad}\_{\\xi_i}^T \\mathcal{I})\\Delta t \\end{aligned} ]

This will give first order accuracy as one would expect from Forward Euler. In the case of (\\text{SE}(3)), it happens to be the case that integrating (\\xi) in the above manner is not stable, although it's usually not a big deal if small enough timestamps are used and appropriate dissipative forces are present (e.g. friction or air resistance). Higher-order integration schemes (e.g. Ralston's Method, RK4) can also be employed to mitigate this, although we will leave the exploration of such integrators for another time as some care should be taken in applying them to Lie groups in general.

## Handling gravity

Gravity is a very common force that's relevant when modeling rigid body dynamics, so it's worth taking a quick look on how to handle it. Due to our choice of body coordinates (i.e. coordinates whose origin is the center of mass), this is relatively doable. We know that the force of gravity is a linear force applied to the center of mass. In *reference* coordinates:

[ f\_{g,\\text{ref}} = \\begin{bmatrix} 0 \\ 0 \\ -9.8\\text{m/s}^2 \\cdot m \\ \\end{bmatrix} ]

However, to use this force in the equation above, we need the rotation (R) associated with (g). In particular (R) can be defined as the rotation that satisfies (Rx=gx-g\\begin{bmatrix} 0 & 0 & 0\\end{bmatrix}^T). Then we can write the generalized force due to gravity as:

[ F= \\begin{bmatrix} 0 \\ 0 \\ 0 \\ R^{-1}\\begin{bmatrix} 0 \\ 0 \\ -9.8\\text{m/s}^2\\cdot m \\ \\end{bmatrix} \\end{bmatrix} ]

Where (R^T = R^{-1}) can also be used if a matrix representation is employed for (R). Note that the first three entries are zero since gravity does not apply a torque to the center of mass. The final three entries are simply the gravitational force rotated into the coordinates of the body.

## Bringing it all together

As a quick illustration of this formulation of rigid body dynamics, here's a quick sim that you can use to explore how such dynamics behave. We're using forward Euler to simulate the rotational degrees of freedom of the above equations of motion, and we've tweaked the translational dynamics integration to enhance stability.

#### Controls

- W/S: Pitch
- A/D: Yaw
- Q/E: Roll
- G: Thrust
- Space: Reset
- T: Hold to point the engine downwards
- Arrow Keys: Camera

Acknowledgement

This work is heavily inspired by [Lie Group Formulation of Articulated Rigid Body Dynamics](https://www.cs.cmu.edu/~junggon/tools/liegroupdynamics.pdf) by Junggon Kim. I highly recommend this work as a particularly lucid exposition of the topic.

# Dual quaternions for SE(3)

*February 25, 2025*

*Michael Bauer*

## Introduction

At ReSim we use Lie groups for a [variety of reasons](https://docs.resim.ai/open-core/transforms/liegroups/). One good reason for choosing to express our robotics algorithms and libraries in terms of Lie groups is that they are an abstraction for rigid body transformations, rotations, and other relevant transformations, meaning we can write common algorithms to solve a variety of problems. This, of course, is very appealing to software engineers, who always love a good abstraction. As an example, the [Sophus](https://github.com/strasdat/Sophus) library includes a single header which provides a template implementation of `ceres::Manifold` for any Lie group defined in a similar manner to the ones implemented in that library. For those unfamiliar with CERES, the upshot of this is that Lie-group-valued parameters (e.g. camera extrinsics) can be easily optimized based on logged robotics data if those parameters are expressed using Sophus's library.

This abstraction goes even deeper: even for a particular Lie group such as (\\text{SO}(3)) or (\\text{SE}(3)), we have multiple choices of how to implement it. You may be familiar with how unit quaternions can be used to represent 3D orientation (if not I highly recomment [this video](https://www.youtube.com/watch?v=d4EgbgTm0Bg) and [this follow-up video](https://www.youtube.com/watch?v=zjMuIxRvygQ) from 3Blue1Brown). Unit Quaternions are one possible way of representing (\\text{SO}(3)) and they have some advantages, including the ability to compose transforms rapidly, compared to the matrix representation of (\\text{SO}(3)) (i.e. rotation matrices). However, rotation matrices are able to act on (i.e. rotate) points/vectors in fewer floating-point operations (FLOPs) than unit quaternions. The point is that one may have software design reasons for preferring one implementation of (\\text{SO}(3)) over another in certain contexts, and this is true for other groups as well.

In this post, we explore an alternative way to represent (\\text{SE}(3)) using unit dual quaternions, which is different from our approach in the [open-core](https://github.com/resim-ai/open-core/blob/main/resim/transforms/se3.hh) repository. Rather than rigorously proving every claim or covering all possible operations, we aim to provide a high-level overview. We'll discuss how to compose transformations, express algebra elements as dual quaternions, and derive the exponential map. Our goal is to build intuition that may be useful for other representations of (\\text{SE}(3)) and beyond.

## Quaternions

[Quaternions](https://en.wikipedia.org/wiki/Quaternion) are a number system introduced by William Hamilton that can be seen as an extension of the complex numbers. They have gained popularity in 3D graphics in the 21st Century due to their ability to store orientations efficiently (with just 4 floating point values) and their ability to be composed efficiently (16 multiplications vs 27 for composing rotation matrices), although they do have some disadvantages such as their inability to uniquely represent a given orientation (there are exactly two unit quaternions that represent a given orientation) and their lower efficiency at transforming points (15 multiplications vs 9 for rotation matrices). Here’s how they work in a nutshell. Analogous to how complex numbers are built off the definition of a constant (i) such that (i^2=-1), quaternions are defined based on three such values: (i), (j), and (k) such that (i^2=j^2=k^2=ijk=-1). From these rules, one can also infer:

[ \\begin{aligned}jk&=i&kj&=-i\\ki&=j&ik&=-j\\ ij&=k&ji&=-k\\end{aligned} ]

Note how these values *do not commute*. Similar to complex numbers, quaternions also have a real part, so in general, the set of all quaternions can be written as:

[ \\mathbb{Q}=\\left{w+xi+yj+zk|(w,x,y,z)\\in\\mathbb{R}^4\\right} ]

Where (w) is the real part of the quaternion. As a notational matter, we often refer to the non-real part of the quaternion (xi+yj+zk) as the “vector part” and write the whole quaternion as a tuple of it’s real part and vector part:

\[ q=w+xi+yj+zk=(w,\\vec \\nu),\\quad\\text{where},,,\\vec \\nu=[x,y,z]^T \]

Doing so allows us to express multiplication between two quaternions according to the above rules like so:

[ q_1 q_2=(w_1,\\vec \\nu_1)(w_2,\\vec \\nu_2)=(w_1 w_2 - \\vec \\nu_1\\cdot \\vec \\nu_2, w_1\\vec \\nu_2 + w_2 \\vec \\nu_1 + \\vec \\nu_1 \\times \\vec \\nu_2) ]

This product is referred to as the "Hamilton product". Verifying this formula based on the above rules is left as an exercise to the reader. From this expression, we can also note the fact that quaternion multiplication is **not** commutative due to the fact that (i), (j), and (k) do not commute.

Like complex numbers, quaternions also have the notion of a *conjugate* defined such that:

[ q^\* = (w,\\nu)^\*=(w,-\\nu) ]

And the concept of a norm (a.k.a. modulus) defined by:

[ |q|=\\sqrt{qq^\*}=\\sqrt{q^\*q} ]

And the concept of a multiplicative inverse (q^{-1}) such that:

[ qq^{-1}=q^{-1}q=1 ]

Unit quaternions (i.e. those for which (|q| = 1)) can be used to [represent 3D rotations](https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation) as elucidated by the aforementioned videos.

## Dual numbers

Just like quaternions, the [Dual Number](https://en.wikipedia.org/wiki/Dual_number) system is another number system that is similar to the complex numbers. Instead of defining a constant (i) such that (i^2=-1), dual numbers define a constant (\\varepsilon) such that (\\varepsilon^2 = 0). In other words, dual numbers look like (a + \\varepsilon b) for (a,b\\in\\mathbb{R}) and muliplying them works like so:

[ (a + \\varepsilon b)(c + \\varepsilon d) = ac + \\varepsilon (bc + ad) ]

Where the (\\varepsilon^2) drops out since it's zero. Doing this becomes useful for numerous applications including [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation#Automatic_differentiation_using_dual_numbers) where the rules of dual multiplication effectively enforce the product rule. This is not actually the way we leverage dual numbers when representing (\\text{SE}(3)), however. To see how we do, we'll have to take a deeper look at dual quaternions.

## Dual quaternions

Even more interesting things begin to happen when we combine quaternions and dual numbers together. We do this by imagining a dual number whose real and dual parts (q) and (r) are themselves quaternions.

[ Q = q + \\varepsilon r =(s,u)+\\varepsilon (t, \\nu) ]

Multiplying such objects together works as one would expect. That is to say you multiply them using the distributive property and apply the rules outlined above regarding the quaternion constants (i), (j), and (k) and the dual constant (\\varepsilon). Importantly, (\\varepsilon) is allowed to commute with (i), (j), and (k) even though they don’t commute with each other. This ends up looking like so:

[ \\begin{aligned}Q_1 Q_2 &= (q_1 + \\varepsilon r_1)(q_2 + \\varepsilon r_2) \\ &= q_1q_2 + \\varepsilon(r_1q_2 + q_1 r_2)\\end{aligned} ]

Define the conjugate (Q^\*) such that:

[ Q^\* = (s, -u) + \\varepsilon (t, -\\nu) ]

And define the norm of a dual quaternion like so:

\[ \\begin{aligned} ||Q||^2 &= Q Q^\* \\ &= \\left[(s, u) + \\varepsilon(t, \\nu)\\right]\\left[(s, -u) + \\varepsilon(t, -\\nu)\\right] \\ &= s^2 + u\\cdot u + 2\\varepsilon(ts + \\nu\\cdot u) \\end{aligned} \]

With this definition we define unit dual quaternions as those with unit norm:

[ ||Q||^2 = s^2 + u\\cdot u + 2\\varepsilon(ts + \\nu\\cdot u) = 1 ]

If we define (q_r := (s, u)) as our real (i.e. non-dual) part, this implies that unit dual quaternions must have (|q_r| = 1). We also must satisfy (ts + \\nu\\cdot u = 0), and it turns out we can guarantee this by re-writing our dual part as ((t, \\nu) = (0, d)q_r) (with (d\\in\\mathbb{R}^3)), where we can compute (d) by using the fact that ((0, d) = (t, \\nu)q_r^\*). Therefore, we can write any unit dual quaternion in the form:

[ Q = q_r + \\epsilon (0, d)q_r ]

For some unit quaternion (q_r) and some three vector (d).

We claim that the set of *unit dual quaternions* under the above product is a group. Notably it has the following properties:

1. It is closed under multiplication. One can relatively easily show that for dual quaternions (Q) and (U), ((QU)^\* = U^\*Q^\*). From here, it's easy to see that the norm of a product is a product of norms for dual quaternions by directly substituting (QU) into the definition of the norm. This implies closure for the set of unit dual quaternions under multiplication.
1. The product operation is associative. This follows from the associativity of the Hamilton and dual products, although we don't explicitly demonstrate this here.
1. Every unit dual quaternion has an inverse element, given by (Q^{-1} = Q^\*) and an identity element given by (1) such that (Q\\cdot 1 = Q). By definition, we see that (Q Q^{\*}=1) and since (Q = \\left(Q^\*\\right)^\*), we also know (Q^\*Q = 1)

The unit dual quaternions are also a differentiable manifold. A rough argument for this could use the fact that we have a differentiable mapping (f) from (\\mathbb{Q}\_1\\times \\mathbb{R}^3) (where (\\mathbb{Q}\_1) is the set of unit quaternions) to the unit dual quaternions (\\mathbb{D}\_1). This mapping is given by (f(q_r, d) = q_r + \\varepsilon(0, d)q_r). Since (\\mathbb{Q}\_1) and (\\mathbb{R}^3) are differentiable manifolds, you can use their charts to define appropriate charts on (\\mathbb{D}\_1) to prove that it is as well.

Since (\\mathbb{D}\_1) is a group and a differentiable manifold, we can say that it is a Lie group.

## Tangent vectors

Let (\\mathbb{D}\_1) be the set of unit dual quaternions. Now, let's assume that we have a curve (\\gamma: \\mathbb{R} \\rightarrow \\mathbb{D}\_1) that has (\\gamma(0) = 1). Describe this curve by its components like so:

[ \\gamma(\\tau) = (s(\\tau), u(\\tau)) + \\varepsilon(t(\\tau), \\nu(\\tau)) ]

Where (s, t: \\mathbb{R}\\rightarrow\\mathbb{R}) are real curves and (u, \\nu: \\mathbb{R}\\rightarrow\\mathbb{R}^3) are vector curves.

We know that the curve satisfies the unit constraint:

\[ 1 = \\left[(s, u) + \\varepsilon(t, \\nu)\\right]\\left[(s, -u) + \\varepsilon(t, -\\nu)\\right] \]

Differentiating this, we see:

\[ 0 = \\left[(s', u') + \\varepsilon(t', \\nu')\\right]\\left[(s, -u) + \\varepsilon(t, -\\nu)\\right]

- \\left[(s, u) + \\varepsilon(t, \\nu)\\right]\\left[(s', -u') + \\varepsilon(t', -\\nu')\\right] \]

Evaluating this at the identity where ((s, u, t, \\nu) = (1, 0, 0, 0)) we see:

\[ 0 = \\left[(s', u') + \\varepsilon(t', \\nu')\\right] + \\left[(s', -u') + \\varepsilon(t', -\\nu')\\right] \]

By inspection of this expression, we can see that it is satisfied whenver (s' = t' = 0). These two constraints reduce the eight degrees of freedom inherent in the dual quaternion number system to six, which intuitively matches the number of degrees of freedom of (\\text{SE}(3)). We can therefore write our tangent vectors at the identity (i.e. Lie algebra elements) like so:

[ \\gamma'(0) = \\frac{1}{2}(0, \\omega) + \\frac{1}{2}\\varepsilon(0, v) ]

where (v) is unrelated to the (\\nu) used above. We've used (\\omega) and (v) here as these will end up corresponding to angular and linear velocities. The reason for the factor of (1/2) will be elucidated later.

## Correspondence to SE(3)

So far, we've claimed that unit dual quaternions are a Lie group and shown what the derivatives at the identity should look like. We also claim that unit dual quaternions are related to (\\text{SE}(3)). To explore this, we first designate the tangent space at the identity as (T\\mathbb{D}\_1). The elements of this space are dual quaternions of the form ((1/2)(0, \\omega) + (1/2)\\varepsilon (0, v)). We claim that this space is isomorphic to (\\mathfrak{se}(3)) when equipped with an appropriate bracket operation. To do this, we define the bracket very simply like so. For

[ \\begin{aligned} k_1 &= (1/2)(0, \\omega_1) + (1/2)\\varepsilon (0, v_1) \\ k_2 &= (1/2)(0, \\omega_2) + (1/2)\\varepsilon (0, v_2) \\end{aligned} ]

Define the bracket as the commutator:

\[ [k_1, k_2] = k_1 k_2 - k_2 k_1 \]

One can relatively easily verify that this satisfies, closure, bilinearity, the alternating property, and the Jacobi identity, so it is a valid choice. This can be seen when one works out the algebra:

\[ [k_1, k_2] = \\frac{1}{2}(0, \\omega_1 \\times \\omega_2) + \\frac{1}{2}\\varepsilon(0, \\omega_1 \\times v_2 - \\omega_2 \\times v_1) \]

Note that the Lie algebra is not closed under the Hamilton/dual product, but is under the bracket. If we define (k_3) by (k_3 \\equiv [k_1, k_2]) and define (\\omega_3) and (v_3) as its angular and linear velocity respectively, then we can read from the above expression:

[ \\begin{aligned} \\omega_3 &= \\omega_1 \\times \\omega_2 \\ v_3 &= \\omega_1 \\times v_2 - \\omega_2 \\times v_1 \\end{aligned} ]

If we look at the common matrix form for (\\mathfrak{se}(3)), its members are usually expressed as matrices of the following form:

[ X = \\begin{bmatrix} \\omega\_{\\times} & v \\ 0 & 0 \\end{bmatrix} ]

And if one computes the bracket for (\\mathfrak{se}(3)) they'll get:

\[ [X_1, X_2] = \\begin{bmatrix} (\\omega_1 \\times \\omega_2)\_\\times & [\\omega_1 \\times v_2 - \\omega_2 \\times v_1] \\ 0 & 0 \\end{bmatrix} \]

By comparing this with the bracket we defined on (T\\mathbb{D}\_1), we see that both algebras have the same structure (i.e. you could compute the [structure constants](https://en.wikipedia.org/wiki/Structure_constants) for each and they would be identical). Note that the choice of ((1/2)) prefactors in the form of our tangent vectors allows the structure constants to match (rather than differing by a constant factor) and for us to interpret the vector components of our dual quaternion tangents as angular velocity and linear velocity directly.

This all means there is a linear isomorphism (\\phi) that maps (\\mathfrak{se}(3)) to (T\\mathbb{D}\_1). The upshot of this is that we can apply the Lie group [homomorphisms theorem](https://en.wikipedia.org/wiki/Lie_group%E2%80%93Lie_algebra_correspondence#The_correspondence).

Lie Homomorphisms Theorem (Hall Theorem 5.6)

If (\\phi: Lie(G) \\rightarrow Lie(H)) is a Lie algebra homomorphism and if (G) is simply connected, then there exists a (unique) Lie group homomorphism (f: G\\rightarrow H) such that (\\phi = df).

The proof for this is quite fascinating and I recommend having a look at *Lie Groups, Lie Algebras, and Representations: An Elementary Introduction* by Brian Hall. The Baker-Campbell-Hausdorff formula is used to establish a local homomorphism near the identity which is extended to the entirety of the domain employing the fact that it's simply connected.

It turns out that the set of unit dual quaternions (\\mathbb{D}\_1) is simply connected. As we sketched above, the Lie algebra of (\\mathbb{D}\_1) is homomorphic to the Lie algebra of (\\text{SE}(3)), therefore there exists a homomorphism (\\phi: \\mathbb{D}\_1\\rightarrow \\text{SE}(3)). Recall the defining property of homomorphisms. For (Q_1, Q_2\\in\\mathbb{D}\_1):

[ \\phi(Q_1 Q_2) = \\phi(Q_1)\\phi(Q_2) ]

This means that we can work with dual quaternions in our code, composing them and such as we see fit, while always getting the same result we would have gotten using elements of (\\text{SE}(3))

Warning

We do have to be careful when taking the unit dual quaternion logarithm as its codomain is larger than the (\\text{SE}(3)) logarithm. This could result in the logarithm giving us values that the (\\text{SE}(3)) logarithm could not. In practice, we just define it so it's equivalent to ((\\log \\circ, \\phi)).

On a related note, it turns out that (\\text{SE}(3)) is *not* simply connected and this homomorphism is not invertible (i.e. we can't apply the theorem in the opposite direction). The reason for this is that, as we'll see below, (Q\\in \\mathbb{D}\_1) and (-Q) actually map to the same element of (\\text{SE}(3)) by (\\phi).

To get a sense for what (\\phi) looks like, lets compute the quaternion exponential for tangent vectors of the above form. To do this, we simply plug them into the power series definition of the exponential and grind out the algebra (see the appendix). The expression you end up with looks like this:

\[ \\begin{array}{rcl}X&:=&(0,\\omega/2)+\\varepsilon(0,v/2) \\ \\omega' &:=& \\omega/2 \\ v' &:=& v/2 \\ \\theta' &:=& ||\\omega'|| = ||\\omega||/2 \\ a\_\\theta &:=& \\frac{\\sin\\theta}{\\theta} \\ b\_\\theta &:=& \\frac{\\cos\\theta - a(\\theta)}{\\theta^2} \\ \\exp(X)&=& \\left(\\cos\\theta',\\omega'a\_{\\theta'}\\right) + \\varepsilon\\left[-(\\omega'\\cdot v')a\_{\\theta'}, v'a\_{\\theta'}+\\omega'(w'\\cdot v')b\_{\\theta'}\\right]\\end{array} \]

If we plug a pure rotation ((v = 0)) into this, we get:

[ Q_R = \\left(\\cos \\tfrac{\\theta}{2}, \\tfrac{\\omega}{\\theta}\\sin\\tfrac{\\theta}{2}\\right):= q_r ]

Note that there's no dual part. You may recognize this as a unit quaternion that one might use for three dimensional rotation, which is very encouraging, and we define this as the non-dual quaternion (q_r) for the sake of brevity.

Now if we plug in a pure translation ((\\omega = 0)) of (t = v \\cdot 1), we get:

[ Q_t = 1 + \\frac{1}{2}\\varepsilon (0, t) ]

Typically, we like to think of an (\\text{SE}(3)) as a translation composed with a rotation:

[ Q = Q_tQ_R = (1 + \\varepsilon(0, t/2))q_r = q_r + \\varepsilon (0, t/2)q_r ]

This expression tells us that we can get the rotation quaternion from our dual quaternion just by taking the real (i.e. non-dual) part of it. Then we can get the translation by taking the dual part, multiplying it by two, and right multiplying by (q_r^{-1} = q_r^\*). Quite encouragingly, it also matches the form we derived for unit dual quaternions above.

Feel free to use the following calculator to familiarize yourself with the behavior of the dual quaternion exponential. It shows how the corresponding (\\text{SE}(3)) transform relates two frames.

## Seeing double

Those familiar with (\\text{SO}(3)) may know that its exponential has the following property:

\[ \\exp(\\omega) = \\exp\\left[\\frac{\\omega}{||\\omega||}(||\\omega|| + 2\\pi n )\\right] \\quad\\forall n\\in\\mathbb{Z} \]

This is just a fancy way of stating that if you keep rotating around the same axis, you'll periodically end up back at the same orientation. Conventionally this multivaluedness leads to a branch cut at (||\\omega|| = \\pi) when defining the logarithm so that the bijective region of (\\exp)/(\\log) is (||\\omega|| < \\pi)

However, if we plug such tangent vectors into our expression for (q_r), we'll see:

[ \\begin{aligned} q\_{rn} &= \\left(\\cos\\tfrac{\\theta + 2\\pi n}{2}, \\tfrac{\\omega}{||\\omega||} \\sin\\tfrac{\\theta + 2\\pi n}{2}\\right) \\ &= \\left(\\cos\\left(\\tfrac{\\theta}{2} + \\pi n\\right), \\tfrac{\\omega}{||\\omega||} \\sin\\left(\\tfrac{\\theta}{2} + \\pi n\\right)\\right) \\ &= \\left{ \\begin{array}{rl} q_r & n ,,\\text{ even} \\ -q_r & n ,,\\text{ odd}\\end{array}\\right. \\end{aligned} ]

What this tells us is that (q_r) and (-q_r) represent exactly the same element of (\\text{SO}(3)) making the unit quaternions a double cover of (\\text{SO}(3)). Since a factor of (q_r) appears in both terms of our expression above for (Q), it happens to be the case that (Q) and (-Q) represent exactly the same element of (\\text{SE}(3)) making the unit dual quaternions a double cover of (\\text{SE}(3)). This shows that there isn't an inverse of our homomorphism (\\phi) because there are two elements in (\\mathbb{D}\_1) which (\\phi) maps to each element of (\\text{SE}(3)) so (\\phi) is not injective.

## Conclusion

In summary, dual quaternions with unit norm are a Lie group and a double cover of (\\text{SE}(3)) so we can use them to represent 3D rigid body transformations in our robotics software. In certain cases, this may be advantagous to save space or computation time. We showed that (\\mathbb{D}\_1) and (\\text{SE}(3)) share the same Lie algebra so we don't need to change our handling of tangent vectors in any way. In fact the dynamics simulation from our [prior ReSim U post](../lie_group_dynamics/) was formulated using unit dual quaternions. The only reason why these Lie groups aren't wholly isomorphic is due to the fact that (\\text{SE}(3)) is not simply connected whereas the unit dual quaternions are. This relationship is in many ways analogous to the relationshp between (\\text{SO}(3)) and the ordinary unit quaternions.

## Appendix: Dual quaternion exponential derivation

To keep this from being more painful than it has to, we'll remove the factor of (2) from (\\omega) and (v), only re-introducing it in the final expressions.

\[ \\begin{aligned}\\exp(X) &= \\sum\_{n=0}^\\infty \\frac{1}{n!} [(0, \\omega) + \\varepsilon(0, v)]^n \\&=\\sum\_{n=0}^\\infty \\frac{1}{n!}(0,\\omega)^n+\\varepsilon \\left((0, v) + \\frac{1}{2}((0, \\omega)(0, v) + (0,v)(0, \\omega)) + \\cdots\\right)\\end{aligned} \]

[ \\begin{aligned}\\exp(X) &=\\sum\_{n=0}^\\infty \\frac{1}{n!}(0,\\omega)^n + \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(n+k+1)!}(0,\\omega)^n(0,v)(0,\\omega)^k\\end{aligned} ]

Let’s start by tackling the first term:

[ \\begin{aligned}\\exp(\\omega) &= \\sum\_{n=0}^\\infty\\frac{1}{n!}(0,\\omega)^n\\ (0,\\omega)^0 &= (1,0) \\ (0, \\omega)^1 &= (0, \\omega) \\ (0, \\omega)^2 &= -||\\omega||^2(1, 0) \\\\end{aligned} ]

So:

[ \\begin{aligned}\\exp(\\omega) &= \\left(\\sum\_{n=0}^\\infty \\frac{(-1)^n}{(2n)!}||\\omega||^{2n},\\frac{\\omega }{||\\omega||}\\sum\_{n=0}^\\infty \\frac{(-1)^n}{(2n+1)!}||\\omega||^{2n+1}\\right) \\ &=(\\cos\\tfrac{\\theta}{2}, n\\sin\\tfrac \\theta 2)\\end{aligned} ]

Where (n = \\omega/||\\omega||) and we’ve re-introduced the factor of (1/2) we removed from (\\omega) at the beginning of this derivation to show that we get exactly the form we expect for the rotation part (q_r=(\\cos \\tfrac \\theta 2,n\\sin \\tfrac \\theta 2)) of the dual quaternion. Now we look at the translation part:

[ \\begin{aligned}T :=& \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(n+k+1)!}(0,\\omega)^{n}(0,v)(0,\\omega)^{k} \\ =&\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+1)!}(0,\\omega)^{2n}(0,v)(0,\\omega)^{2k} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,\\omega)^{2n + 1}(0,v)(0,\\omega)^{2k} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,\\omega)^{2n}(0,v)(0,\\omega)^{2k+1} \\ &+ \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+3)!}(0,\\omega)^{2n+1}(0,v)(0,\\omega)^{2k+1} \\ =&\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+1)!}(0,v)(0,\\omega)^{2(n+k)} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,\\omega)(0,v)(0,\\omega)^{2(n+k)} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,v)(0,\\omega)^{2(n+k)+1} \\ &+ \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+3)!}(0,\\omega)(0,v)(0,\\omega)^{2(n + k)+1} \\ =&\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+1)!}(0,v)(0,\\omega)^{2n} \\ &+\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+2)!}(0,\\omega)(0,v)(0,\\omega)^{2n} \\ &+\\sum\_{n=0}^\\infty \\frac{n+1}{(2n+2)!}(0,v)(0,\\omega)^{2n+1} \\ &+ \\sum\_{n=0}^\\infty\\frac{n+1}{(2n+3)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1}\\end{aligned} ]

Where we’ve used the fact that ((0,\\omega)^{2n}) is real and hence commutes with ((0,v)). Now break this down term-by-term:

[ \\begin{aligned}\\text{I} :=&\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+1)!}(0,v)(0,\\omega)^{2n} \\=& \\frac{1}{2}\\sum\_{n=0}^\\infty\\frac{1}{(2n)!}(0,v)(0,\\omega)^{2n} + \\frac{1}{2}\\sum\_{n=0}^\\infty\\frac{1}{(2n + 1)!}(0,v)(0,\\omega)^{2n} \\ =& \\frac{1}{2}(0,v)\\cosh(0,\\omega) - \\frac{1}{2||\\omega||^2}(0,v)(0,\\omega)\\sinh(0,\\omega) \\ =& \\frac 1 2 (0, v) \\cos||\\omega|| - \\frac 1 {2||\\omega||^3}(0,v)(0,\\omega)(0,\\omega)\\sin||\\omega|| \\ =&\\frac 1 2 (0, v) \\cos||\\omega|| + \\frac 1 {2||\\omega||}(0,v)\\sin||\\omega|| \\ \\text{II} :=& \\sum\_{n=0}^\\infty\\frac{n+1}{(2n+2)!}(0,\\omega)(0,v)(0,\\omega)^{2n} \\ =& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)\\sinh(0,\\omega) \\ =& \\frac{1}{2||\\omega||}(0,\\omega)(0,v)\\sin ||\\omega|| \\ \\text{III} :=& \\sum\_{n=0}^\\infty \\frac{n+1}{(2n+2)!}(0,v)(0,\\omega)^{2n+1} \\=&\\frac{1}{2}(0,v)\\sinh(0,\\omega) \\ =&\\frac{1}{2||\\omega||}(0,v)(0,\\omega)\\sin||\\omega|| \\ \\text{IV}:=&\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+3)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1} \\ =&\\frac{1}{2}\\sum\_{n=0}^\\infty \\frac{1}{(2n + 2)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1} - \\frac{1}{2}\\sum\_{n=0}^\\infty \\frac{1}{(2n + 3)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1} \\ =& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)(\\cosh(0,\\omega) - 1) \\ &+\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(\\sinh(0,\\omega) - (0,\\omega)) \\ =& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)\\cosh(0,\\omega) \\ &+\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)\\sinh(0,\\omega)\\=& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)\\cos ||\\omega|| \\ &+\\frac{1}{2||\\omega||^3}(0,\\omega)(0,v)(0, \\omega)\\sin ||\\omega||\\ \\end{aligned} ]

So:

[ \\begin{aligned}T=&,,\\text{I}+\\text{II}+\\text{III}+\\text{IV}\\ =&\\left(-\\omega\\cdot v\\tfrac{\\sin||\\omega||}{||\\omega||}, v\\tfrac{\\sin||\\omega||}{||\\omega||} + \\tfrac{\\omega}{||\\omega||^2}(\\omega\\cdot v)(\\cos||\\omega|| - \\tfrac{\\sin||\\omega||}{||\\omega||})\\right)\\end{aligned} ]

And:

\[ \\begin{aligned}\\exp[(0,\\omega)+\\varepsilon(0,v)] =& \\left(\\cos||\\omega||, \\omega \\tfrac {\\sin||\\omega||}{||\\omega||}\\right) \\ &+ \\varepsilon\\left(-\\omega\\cdot v\\tfrac{\\sin||\\omega||}{||\\omega||}, v\\tfrac{\\sin||\\omega||}{||\\omega||} + \\tfrac{\\omega}{||\\omega||^2}(\\omega\\cdot v)(\\cos||\\omega|| - \\tfrac{\\sin||\\omega||}{||\\omega||})\\right)\\end{aligned} \]

Re-introducing the factor of (1/2) into the basis we’re using for our algebra, we have:

\[ \\boxed{\\begin{array}{rcl}X&:=&(0,\\omega/2)+\\varepsilon(0,v/2) \\ \\omega' &:=& \\omega/2 \\ v' &:=& v/2 \\ \\theta' &:=& ||\\omega'|| = ||\\omega||/2 \\ a\_\\theta &:=& \\frac{\\sin\\theta}{\\theta} \\ b\_\\theta &:=& \\frac{\\cos\\theta - a(\\theta)}{\\theta^2} \\ \\exp(X)&=& \\left(\\cos\\theta',\\omega'a\_{\\theta'}\\right) + \\varepsilon\\left[-(\\omega'\\cdot v')a\_{\\theta'}, v'a\_{\\theta'}+\\omega'(w'\\cdot v')b\_{\\theta'}\\right]\\end{array}} \]

For small (\\theta'), we should use small angle approximations:

[ \\begin{array}{rcrlrlrlrl}a\_\\theta &\\approx& &1 &-& x^2/6 &+& x^4/120 &+&O(\\theta^6)\\ b\_\\theta &\\approx& -& 1/3 &+& x^2/30 &-& x^4/840 &+&O(\\theta^6)\\end{array} ]