Skip to content

See ReSim Working in Five Minutes

This tutorial runs a demo that fills a ReSim project with real test results, so you can see what the platform does before connecting anything of your own. No Docker, no container registry, no config to write. One command.

Time: about 5 minutes, plus a few minutes of waiting for metrics.

Before you start:

  • You have a ReSim account. Sign up at app.resim.ai if you don't.
  • You have Python 3.10 or newer.
  • You can open a browser to authenticate.

What you're building

The demo replays test data from a real navigation project into your own ReSim project. When it finishes you'll have:

  • Two batches of tests, one per version of the navigation stack. The same scenarios ran in both, so every test has a counterpart in the other batch.
  • Metrics on every test: speed and localization error over time, goal state timelines, camera footage, pass/fail checks, and more.
  • The files each run produced, attached to the test they came from, so the raw material sits next to the charts drawn from it.
  • An A/B comparison between the two batches, showing which scenarios regressed and which improved.
  • A dashboard comparing the two versions across every run of them.

That's the navigation demo. There's a second one built from MuJoCo policy evaluations — see Choosing a demo below.

Step 1: Install the SDK

Shell
pip install resim-sdk

Step 2: Run the demo

Shell
resim-demo --demo navigation

Run resim-demo with no arguments and it lists the demos it ships rather than picking one for you — see Choosing a demo below.

The first time you run it, you'll be asked to authenticate in a browser:

Output
Authenticating by Device Code

Please navigate to: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX

Visit that URL, sign in, and the demo continues on its own. Subsequent runs reuse the cached token.

Before it starts, the demo downloads the test data it replays and reports progress as it goes. That happens once — later runs reuse the cached copy.

By default the demo works in a project called ReSim SDK Demo, creating it if it doesn't exist, so it won't touch your real projects. To use a different one:

Shell
resim-demo --demo navigation --project-name "my project"

You can also call it from Python, which is useful if you want the IDs back rather than reading them off the terminal:

Python
from resim.demo import run

result = run(demo="navigation")
print(result.batch_ids, result.urls)

Choosing a demo

--demo picks which demo to run. There's no default — running resim-demo on its own lists the ones that ship:

Shell
resim-demo
--demo What it replays
navigation A robot running errands around a hospital. Dense telemetry across a suite of scenarios, covering every chart type ReSim renders.
mujoco An ALOHA bimanual manipulation policy evaluated in MuJoCo, one test per cube placement, compared across two attention configurations.

Each lands in its own project and on its own branch, so running both leaves you with two independent sets of results to look at.

Every MuJoCo test also carries the run's MCAP, which opens in the viewer, alongside its episode animation.

The rest of this tutorial follows the navigation demo. The MuJoCo one works the same way with different metrics, described in What the MuJoCo demo shows below.

When the demo finishes it prints where to go:

Output
ReSim SDK demo complete.

  Batch A (baseline)    https://app.resim.ai/projects/.../batches/...
  Batch B (candidate)   https://app.resim.ai/projects/.../batches/...
  A/B comparison        https://app.resim.ai/projects/.../batches/.../compare-batch/batch/...
  Trends dashboard      https://app.resim.ai/projects/.../dashboards/...

Metrics are computed after each batch closes, which takes a few minutes.
Reload these pages when it finishes.

Metrics are computed once each batch closes, so the pages look empty at first. Give it a couple of minutes and reload.

Batch A and Batch B

Each batch page lists its tests with a pass, warning, or blocker status.

Open any test to see its metrics: speed over time, distance to each goal, what the navigation stack was doing as a state timeline, a table judging whether the localizer's confidence was honest, and on some tests the onboard camera footage. The Events tab shows the moments the robot reached each goal.

The Logs tab holds the files that run produced. On the tests that recorded one, that is the camera clip and the still frame drawn from it.

Each file is uploaded as the kind of file it is, and that decides what ReSim can do with it. The MuJoCo demo makes the point better: every one of its tests carries the run's MCAP, which arrives as robotics data and opens in a viewer inline, where the same bytes uploaded as a plain file would only offer a download. Your own tests attach files the same way — see attach_log.

Batch-level metrics sit on the batch page itself, aggregating across every test.

The A/B comparison

This is the page that answers "did this version make things better or worse". Metrics are matched by name between the two runs, so each chart from Batch A sits next to the same chart from Batch B. Merged Metrics goes further and draws both runs on the same axes.

The Tests tab is the fastest read: it groups the scenarios by how their outcome differs between the two batches, so the regressions and the fixes are each their own list. See the A/B comparison guide for the full picture.

The dashboard

The batches show you two runs. The dashboard aggregates across them, grouped by build version: how each version's tests turned out, and its mean localization error. Run the demo again and those bars take in the new runs too.

What the MuJoCo demo shows

--demo mujoco gives you the same four links over a different problem: two ALOHA arms passing a cube between them. It is worth a look once you have toured the navigation demo, because the shape of its data is different.

Each test is one cube placement on the table. That placement is the only thing that differs between tests, so each test is named for it: Cube start (0.099, 0.446) seed 50. The same placements run on both sides of the comparison, which is what lets every test pair with its counterpart.

The two batches are the same policy built two ways, differing in one hyperparameter: the number of attention heads. One matches the value the policy was trained with and the other does not, which is enough to change how often the handover completes. That is the question the A/B page answers here.

The task has stages rather than a single outcome: the cube gets touched, lifted, carried across, then released. Most of the metrics follow from that:

  • A state timeline of which stage the episode was in, so a run that grasps and then drops the cube looks different from one that never grasps it.
  • Line charts of the staged reward, the cube's height, and how far past the handover line it has traveled. Zero on that last one is the line itself.
  • A top-down plot of the cube's path, drawn by the evaluation itself rather than by one of ReSim's chart templates. It is a custom template reading a figure the test emitted, which is how you chart something the built-in templates do not cover.
  • Events for the moment each stage is first reached, carrying the time it took and how many times the episode had slipped backwards by then.
  • A table per test, and a pass/fail check: completing the handover passes, anything short of it warns. Nothing blocks, because a cube placement the policy cannot solve is not a broken build.

At batch level the same stage data aggregates into how far the whole set of placements got, how long each stage took on average, and the spread of episode reward.

Step 4: Look at how it's put together

Everything the demo displays comes from one config file, shipped inside the package. Nothing is hardcoded into the platform, and nothing here is special to the demo — this is the same file you would write for your own system.

Three pieces, in the order they matter:

Topics describe the shape of the data your tests emit:

config.resim.yml
topics:
  odom_linear_velocity:
    schema:
      x: float
      y: float
      z: float

Metrics are SQL queries over that data, each rendered by a template:

config.resim.yml
metrics:
  Robot Speed:
    type: test
    description: Ground speed of the robot over the course of the run.
    query_string: |
      SELECT
        'Speed' AS group_name,
        timestamp / 1E9 AS "Time (s)",
        SQRT(POWER(x, 2) + POWER(y, 2)) AS "Speed (m/s)"
      FROM odom_linear_velocity;
    template_type: system
    template: line

A metric's type decides what it runs against: test per test, batch across a batch, dashboard across every batch in the dashboard's window. The dashboard's charts join against the built-in metadata table, which is where the build version, experience name and status of each test live:

config.resim.yml
  Localization Error by Build Version:
    type: dashboard
    query_string: |
      SELECT
        'Mean' AS group_name,
        m.build_version AS "Build Version",
        AVG(pd.position_diff_m) AS "Mean Localization Error (m)"
      FROM pose_difference pd
      JOIN metadata m ON pd.job_id = m.job_id
      WHERE m.build_version IS NOT NULL
      GROUP BY m.build_version
      ORDER BY m.build_version;
    template_type: system
    template: bar

Metrics sets group metrics so a batch or dashboard can run them.

To read the whole file:

Python
from resim.demo import config_path

print(config_path("navigation").read_text())

config_path("mujoco") gives you the other demo's config.

The metrics guide covers topics, every template, status checks, and custom templates in full.

Next steps

  • Emit your own data. Get Your First Metrics in ReSim walks through the same API the demo uses, starting from an empty script.
  • Write your own metrics. Start with the metrics guide, and develop them against a debug dashboard before running a batch.
  • Run tests in ReSim's infrastructure. Run Your First Test Batch sets up builds and experiences so ReSim executes your system rather than you running it locally.
  • Reference. The ReSim SDK guide covers the SDK's API surface in full.