Skip to content

Add Metrics to Your Batch

In this tutorial you'll add Metrics to the drone flight demo from Run Your First Test Batch. By the end you'll have replaced the legacy metrics build with direct data emission, and you'll be able to see speed, altitude, and flight state visualizations in the dashboard — plus a batch-level summary comparing all three flights.

Time: about 30 minutes

Before you start:

  • You've completed Run Your First Test Batch and have the drone-demo project set up with the drone-flight system and three experiences registered.
  • You have Docker installed and access to a container registry (AWS ECR, Docker Hub, or similar).
  • You have Python 3.8+ installed.

What you're changing

The first tutorial used a pre-built metrics build image — a separate Docker container that ran after each test and computed metrics using the legacy protobuf-based SDK. In this tutorial you'll:

  1. Clone the demo repo and modify the experience build to emit flight data directly using the Emitter class.
  2. Write a config.resim.yaml that defines the data schema and SQL metrics.
  3. Create a new test suite that uses the Metrics framework instead of a metrics build image.
  4. Run a batch and see the results.

Step 1: Clone the repo

Shell
git clone https://github.com/resim-ai/getting-started-demo.git
cd getting-started-demo

The experience build lives in experience-build/. The key file is sim_run.py — it reads the flight log JSON for each experience and writes processed_flight_log.json to /tmp/resim/outputs/. You'll add emission calls alongside that existing logic.

Step 2: Install the Emitter library

Shell
pip install resim-open-core

The resim-open-core package provides the Emitter class used to write data to the ReSim data lake.

Step 3: Define your data schema

Create the directory and config file:

Shell
mkdir -p .resim/metrics
touch .resim/metrics/config.resim.yaml

The flight log has one sample per second with speed, position (x, y, z in meters), state (Idle, Takeoff, Hovering, Moving, Landing), and status (OK, WARNING, Error). Define two topics to capture this:

.resim/metrics/config.resim.yaml
version: 1

topics:
  flight_data:
    schema:
      speed: float
      x: float
      y: float
      altitude: float

  flight_state:
    schema:
      state: string
      status: string

flight_data captures the numeric values you'll plot over time. flight_state captures the categorical state and status strings you'll use for the state timeline and status checks.

Step 4: Add emission to the experience build

Open experience-build/sim_run.py. The file reads the flight log and writes the processed output. You'll add emission calls that run alongside that logic.

Add the import near the top of the file:

experience-build/sim_run.py
from datetime import datetime
from resim.metrics.python.emissions import Emitter

Then, after the flight log is loaded into flight_data (the dict parsed from flight_log.json), add the emission block:

experience-build/sim_run.py
with Emitter(config_path=".resim/metrics/config.resim.yaml") as emitter:
    first_ts = None

    for sample in flight_data["samples"]:
        ts = datetime.fromisoformat(sample["timestamp"])
        if first_ts is None:
            first_ts = ts

        # Elapsed time in nanoseconds from the start of the flight
        elapsed_ns = int((ts - first_ts).total_seconds() * 1e9)

        emitter.emit("flight_data", {
            "speed":    sample["speed"],
            "x":        sample["position"]["x"],
            "y":        sample["position"]["y"],
            "altitude": sample["position"]["z"],
        }, timestamp=elapsed_ns)

        emitter.emit("flight_state", {
            "state":  sample["state"],
            "status": sample["status"],
        }, timestamp=elapsed_ns)

The emitter writes to /tmp/resim/outputs/emissions.resim.jsonl by default — ReSim picks this file up automatically at the end of each test.

Step 5: Write your metrics

Add a metrics and metrics sets section to config.resim.yaml. Start with four test-level metrics and two batch-level metrics.

Speed over time

A line chart of drone speed for each test. Timestamps are in nanoseconds, so divide by 1E9 to show seconds on the x-axis.

.resim/metrics/config.resim.yaml
metrics:
  Speed Over Time:
    type: test
    description: Drone speed over the course of the flight.
    query_string: |
      SELECT
        'Speed',
        timestamp / 1E9 AS "Time (s)",
        speed AS "Speed (m/s)"
      FROM flight_data;
    template_type: system
    template: line

Maximum speed — with a status check

A scalar metric that also flags flights that exceed speed thresholds. The warning_drone_flight experience reaches ~89 m/s and the fast_drone_flight reaches ~45 m/s, so thresholds of 50 m/s (block) and 20 m/s (warn) will produce the expected pass/warn/block split across the three experiences.

.resim/metrics/config.resim.yaml
  Maximum Speed:
    type: test
    description: Highest speed recorded during the flight.
    query_string: |
      SELECT MAX(speed) FROM flight_data;
    template_type: system
    template: scalar
    units: "m/s"
    status:
      query_string: SELECT '1' FROM flight_data HAVING MAX(speed) > ?
      block: 50.0
      warn: 20.0

Altitude over time

.resim/metrics/config.resim.yaml
  Altitude Over Time:
    type: test
    description: Drone altitude over the course of the flight.
    query_string: |
      SELECT
        'Altitude',
        timestamp / 1E9 AS "Time (s)",
        altitude AS "Altitude (m)"
      FROM flight_data;
    template_type: system
    template: line

Flight state timeline

The state timeline template expects three columns: [system_name, timestamp, state_name]. Consecutive rows with the same state are merged into segments, making it easy to see when the drone was hovering vs. moving vs. landing.

.resim/metrics/config.resim.yaml
  Flight States:
    type: test
    description: Drone state over the course of the flight.
    query_string: |
      SELECT
        'Drone' AS "System",
        timestamp,
        state
      FROM flight_state;
    template_type: system
    template: state_timeline

Batch metrics

Batch metrics aggregate across all jobs in a batch. Add these to compare the three flights at the batch level.

.resim/metrics/config.resim.yaml
  Average Max Speed:
    type: batch
    description: Average of the highest speed recorded across all flights in the batch.
    query_string: |
      SELECT AVG(max_speed_ms)
      FROM (
        SELECT job_id, MAX(speed) AS max_speed_ms
        FROM flight_data
        GROUP BY job_id
      );
    template_type: system
    template: scalar
    units: "m/s"

  Peak Altitude by Experience:
    type: batch
    description: Highest altitude reached in each experience, side by side.
    query_string: |
      SELECT
        'Peak Altitude',
        m.experience_name AS "Experience",
        MAX(f.altitude) AS "Peak Altitude (m)"
      FROM flight_data f
      JOIN metadata m ON f.job_id = m.job_id
      GROUP BY m.experience_name
      ORDER BY m.experience_name;
    template_type: system
    template: bar

Wire up a metrics set

A metrics set groups metrics together and is what you attach to a test suite. Add this at the end of the config:

.resim/metrics/config.resim.yaml
metrics sets:
  Drone Flight Metrics:
    metrics:
      - Speed Over Time
      - Maximum Speed
      - Altitude Over Time
      - Flight States
      - Average Max Speed
      - Peak Altitude by Experience

Complete config

Your finished config.resim.yaml should look like this:

.resim/metrics/config.resim.yaml
version: 1

topics:
  flight_data:
    schema:
      speed: float
      x: float
      y: float
      altitude: float

  flight_state:
    schema:
      state: string
      status: string

metrics:
  Speed Over Time:
    type: test
    description: Drone speed over the course of the flight.
    query_string: |
      SELECT
        'Speed',
        timestamp / 1E9 AS "Time (s)",
        speed AS "Speed (m/s)"
      FROM flight_data;
    template_type: system
    template: line

  Maximum Speed:
    type: test
    description: Highest speed recorded during the flight.
    query_string: |
      SELECT MAX(speed) FROM flight_data;
    template_type: system
    template: scalar
    units: "m/s"
    status:
      query_string: SELECT '1' FROM flight_data HAVING MAX(speed) > ?
      block: 50.0
      warn: 20.0

  Altitude Over Time:
    type: test
    description: Drone altitude over the course of the flight.
    query_string: |
      SELECT
        'Altitude',
        timestamp / 1E9 AS "Time (s)",
        altitude AS "Altitude (m)"
      FROM flight_data;
    template_type: system
    template: line

  Flight States:
    type: test
    description: Drone state over the course of the flight.
    query_string: |
      SELECT
        'Drone' AS "System",
        timestamp,
        state
      FROM flight_state;
    template_type: system
    template: state_timeline

  Average Max Speed:
    type: batch
    description: Average of the highest speed recorded across all flights in the batch.
    query_string: |
      SELECT AVG(max_speed_ms)
      FROM (
        SELECT job_id, MAX(speed) AS max_speed_ms
        FROM flight_data
        GROUP BY job_id
      );
    template_type: system
    template: scalar
    units: "m/s"

  Peak Altitude by Experience:
    type: batch
    description: Highest altitude reached in each experience, side by side.
    query_string: |
      SELECT
        'Peak Altitude',
        m.experience_name AS "Experience",
        MAX(f.altitude) AS "Peak Altitude (m)"
      FROM flight_data f
      JOIN metadata m ON f.job_id = m.job_id
      GROUP BY m.experience_name
      ORDER BY m.experience_name;
    template_type: system
    template: bar

metrics sets:
  Drone Flight Metrics:
    metrics:
      - Speed Over Time
      - Maximum Speed
      - Altitude Over Time
      - Flight States
      - Average Max Speed
      - Peak Altitude by Experience

Step 6: Add the dependency to the Dockerfile

Open experience-build/Dockerfile and add resim-open-core to the pip install step. It will look something like:

experience-build/Dockerfile
RUN pip install resim-open-core

Also ensure the config file gets copied into the image:

experience-build/Dockerfile
COPY .resim/metrics/config.resim.yaml .resim/metrics/config.resim.yaml

Step 7: Build and push your image

Build the modified experience image and push it to your container registry:

Shell
docker build -f experience-build/Dockerfile -t <YOUR_REGISTRY>/drone-demo:metrics-v1 .
docker push <YOUR_REGISTRY>/drone-demo:metrics-v1

Step 8: Register the new build

Register your updated image with ReSim. Make sure your project is still selected:

Shell
resim projects select "drone-demo"

resim builds create \
  --name "Demo Build — Metrics" \
  --description "Drone demo build with metrics instrumentation" \
  --version "metrics-v1" \
  --image "<YOUR_REGISTRY>/drone-demo:metrics-v1" \
  --branch "main" \
  --system "drone-flight" \
  --auto-create-branch

Note the build UUID from the output.

Step 9: Sync your metrics config

Push your config.resim.yaml to the ReSim platform so it knows about your topics and metrics:

Shell
resim metrics sync \
  --project "drone-demo" \
  --branch "main"

Step 10: Create a test suite with your metrics set

Create a new test suite that uses your metrics set. This suite doesn't need a --metrics-build-id — the Metrics framework handles results directly from the emitted data.

Shell
resim suites create \
  --name "Drone Metrics Suite" \
  --description "Test suite for drone flight metrics" \
  --system "drone-flight" \
  --metrics-set "Drone Flight Metrics" \
  --experiences "Maiden Flight Voyage" "Drone Flight Fast" "Drone Flight with Warning"

Step 11: Run the suite

Shell
resim suites run \
  --test-suite "Drone Metrics Suite" \
  --build-id "<UUID from Step 8>" \
  --sync-metrics-config

The --sync-metrics-config flag automatically syncs your config before triggering the batch, so you don't have to run resim metrics sync separately on future runs.

Step 12: View results in the dashboard

Once the batch completes, open app.resim.ai, navigate to drone-demo, and click into the batch.

For each test you'll see:

  • Speed Over Time — a line chart of drone speed in m/s
  • Maximum Speed — a scalar with a pass/warn/block status indicator
  • Altitude Over Time — a line chart of altitude in meters
  • Flight States — a state timeline showing Idle → Takeoff → Hovering → Moving → Landing segments

At the batch level:

  • Average Max Speed — a single number aggregated across all three flights
  • Peak Altitude by Experience — a bar chart comparing the three experiences side by side

The status checks will produce the expected split:

Experience Max speed Status
Maiden Flight Voyage ~2 m/s PASSED
Drone Flight Fast ~45 m/s WARNING
Drone Flight with Warning ~89 m/s BLOCKER

What's next

You've now completed the full Metrics workflow: define a schema, emit data from your system, write SQL metrics, and view the results in the dashboard.

From here you can: