Skip to content

Run your first test batch

In this tutorial you'll set up a complete ReSim project from scratch and run your first batch of tests. By the end you'll have three tests running in parallel against a drone flight simulator and be looking at real metrics in the ReSim dashboard.

The demo uses pre-built public Docker images, so you don't need AWS ECR, Docker, or any build tooling to follow along.

About metrics in this tutorial

This tutorial uses a pre-built metrics build image from the demo repo — a Docker container that runs after each test and computes metrics. In your own projects you likely won't need a metrics build at all: ReSim's Metrics framework lets you emit data directly from your experience build and define visualizations in a config file, with no separate image required.

Time: about 20 minutes

Before you start:

  • You have a ReSim account. Sign up at app.resim.ai if you don't have one.
  • You have the ReSim CLI installed and authenticated. See Install the CLI if not.

What you're building

The getting-started-demo repo contains a simple drone flight simulator. The "system under test" reads a flight log (a JSON file with position, speed, and state data sampled once per second) and produces a processed output. The metrics build then reads that output and computes things like maximum speed, altitude over time, and whether any warning conditions were triggered.

Three flight logs are bundled into the demo image as experiences:

Experience What it represents
Maiden Flight Voyage A normal, nominal flight
Drone Flight Fast The same route flown at higher speed
Drone Flight with Warning A flight that trips warning thresholds

You'll register these experiences with ReSim, point ReSim at the pre-built images, create a test suite, and run it. ReSim will launch all three tests in parallel and aggregate the results.

Step 1: Create a project

A project is the top-level container for your work. Create one for this tutorial:

Shell
resim projects create \
  --name "drone-demo" \
  --description "Getting started tutorial"

Then select it so you don't have to pass --project on every subsequent command:

Shell
resim projects select "drone-demo"

Step 2: Create a system

A system defines the software stack you're testing and the compute resources it needs. The demo is lightweight, so defaults are fine:

Shell
resim systems create \
  --name "drone-flight" \
  --description "Demo drone flight system"

Step 3: Register the experiences

Experiences in this demo are bundled inside the Docker image at /app/experiences/. When ReSim runs a test, it writes the experience location into /tmp/resim/test_config.json, and the sim reads from there. Since the files live inside the container (not in S3), the --location is just the path inside the image.

Register all three:

Shell
resim experiences create \
  --name "Maiden Flight Voyage" \
  --description "A nominal drone flight" \
  --location "experiences/maiden_drone_flight/" \
  --systems "drone-flight"

resim experiences create \
  --name "Drone Flight Fast" \
  --description "A faster version of the nominal flight" \
  --location "experiences/fast_drone_flight/" \
  --systems "drone-flight"

resim experiences create \
  --name "Drone Flight with Warning" \
  --description "A flight that triggers warning conditions" \
  --location "experiences/warning_drone_flight/" \
  --systems "drone-flight"

Each command returns the UUID of the created experience and you can verify them in app.resim.ai under Experiences.

Step 4: Register the build

A build is the Docker image that contains your system under test. The demo image is hosted on a public AWS ECR registry — no auth needed. Register it with ReSim:

Shell
resim builds create \
  --name "Demo Build v25" \
  --description "Getting started demo build v25" \
  --version "v25" \
  --image "public.ecr.aws/resim/open-builds/getting-started-demo:experience-build-v25" \
  --branch "demo" \
  --system "drone-flight" \
  --auto-create-branch

Note the build UUID in the output. You'll need it in Step 6. You can also look it up later:

Shell
resim builds list --system "drone-flight"

Step 5: Register the metrics build

The metrics build is a second Docker image that runs after each test. It reads the sim's output and produces structured metrics that ReSim can store, display, and aggregate across batches.

Register the demo metrics image:

Shell
resim metrics-builds create \
  --name "Demo Metrics v25" \
  --version "v25" \
  --image "public.ecr.aws/resim/open-builds/getting-started-demo:metrics-build-v25" \
  --systems "drone-flight"

Note the metrics-build UUID from the output, or retrieve it:

Shell
resim metrics-builds list

Step 6: Create a test suite

A test suite pairs a system with a fixed set of experiences and a metrics build. Every time you run the suite, ReSim launches one test per experience and then runs the metrics build against each result.

Shell
resim suites create \
  --name "Demo Regression Suite" \
  --description "All three demo experiences" \
  --system "drone-flight" \
  --metrics-build-id "<UUID from Step 5>" \
  --experiences "Maiden Flight Voyage" "Drone Flight Fast" "Drone Flight with Warning"

Step 7: Run the suite

Run the suite against the build you registered in Step 4:

Shell
resim suites run \
  --test-suite "Demo Regression Suite" \
  --build-id "<UUID from Step 4>"

ReSim returns a batch name (something like rejoicing-aquamarine-starfish) and a batch ID. Keep these handy.

Step 8: Watch it run

Check the status of the batch:

Shell
resim batches get --batch-name <your-batch-name>

The batch goes through these states: Submitted → Running → Succeeded. With three lightweight tests, it typically completes in a few minutes.

To watch individual tests:

Shell
resim batches tests --batch-name <your-batch-name>

If anything fails, the per-test container logs are useful for debugging:

Shell
resim logs list \
  --batch-id <batch-id> \
  --test-id <test-id>

The output includes pre-signed URLs so you can download experience-container.log (your sim's stdout) and metrics-container.log (the metrics stage output) directly.

Step 9: View results in the dashboard

Once the batch completes, open app.resim.ai, navigate to your drone-demo project, and click into the batch.

The metrics build computes the following for each test:

  • Maximum Speed — scalar pass/fail metric
  • Speed Over Time — line chart
  • Altitude Over Time — line chart with a warning threshold drawn in
  • Flight States Over Time — state transitions plotted over time
  • X Position Over Time — position trace
  • Speed Distribution — histogram
  • 3D Flight Path — interactive Plotly chart
  • Flight Summary — text summary of the run

At the batch level you'll also see:

  • Highest Recorded Speed across all three flights
  • Average Max Speed
  • Overall Success Rate
  • Altitude Comparison — all three flights overlaid

The "Drone Flight with Warning" experience is designed to trip warning thresholds, so you should see it come up differently in the batch-level pass/fail summary compared to the other two.

What's next

You've now completed a full ReSim cycle: register experiences and images, create a test suite, run a batch, and inspect metrics.

The next step is to swap out the demo images for your own system:

  • Build your own system image — packaging your code into a Docker image with the right inputs/outputs contract
  • Add your own experiences — pointing ReSim at your own data in S3 or bundled in your image
  • Write a metrics build — if you need to do post processing on your emitted data from the experience build you can register a metrics build
  • Set up CI — automatically triggering batches on pull requests