Skip to content

Bootstrap a Demo Project with the CLI

One command fills a SignalFlag project with real test results — batches of tests, metrics computed on them, and a trends dashboard. The demos replay data captured from evaluations that actually ran, so what you get is a working project to explore rather than a screenshot tour. There is nothing to configure, no Docker, and no build to register.

Three demos ship today. Two compare two versions of the same system head to head — an A/B pair, with a comparison page between the batches. The third, session, is a different shape: real field sessions from one system, replayed one batch per session and trended over time rather than paired off. Each demo lands in its own project, so you can run all three and leave them side by side. This page walks through the command, then details exactly what each demo puts in your account.

Time: about 5 minutes per demo, plus a few minutes of waiting for metrics.

Before you start:

  • You have a SignalFlag account. Sign up at app.signalflag.ai/signup if you don't.
  • You have Python 3.10 or newer.
  • You can open a browser to authenticate.

Looking for the short version?

See SignalFlag Working in Five Minutes is install, run, look — the fastest path to something on screen. This page is the same command documented in full, plus a section on each demo's contents.

What you're building

The navigation and MuJoCo demos give you the same shape of results:

  • Two batches of tests, run against two versions of the same system. Both batches run the same experiences, which is what pairs every test with its counterpart in the other batch.
  • Metrics on every test, computed from the replayed data by a metrics config that ships inside the package.
  • The files the runs produced, attached to the tests that made them, so the raw material sits next to the charts drawn from it — on every MuJoCo test, and on the navigation tests that recorded camera footage.
  • An A/B comparison between the two batches, showing which experiences regressed and which improved.
  • A trends dashboard that groups results by build version and takes in every run of the demo, not just this one.

The session demo gives you a different shape, because it is answering a different question — not "did build A beat build B" but "how did today's session compare to the last several":

  • Four batches, one per field session, each holding exactly one test — the whole session, not one scenario cut out of it.
  • Every session's raw recording attached to its test — the mcap snippets and clip GIFs a real robotics session produces, not synthetic stand-ins.
  • A system, and every session's experience tagged, so they show up together in the project's Sessions library — this demo's real trend view, so there is no metrics dashboard.

Step 1: Install the CLI

Shell
pip install signalflag

The wheel installs the demo as a console script under two names, signalflag-demo and resim-demo. They are the same command — resim-demo is kept so anything already scripted against the old package name keeps working. python -m signalflag.demo runs it too.

The SDK used to be published as resim-sdk. That name still installs and works; signalflag is the same package and the one to use from here on.

Step 2: See which demos ship

--demo has no default. The demos are peers, so running one you didn't ask for would be a surprise. Run the command with no arguments and it lists what is on offer instead:

Shell
signalflag-demo
Output
signalflag-demo replays real test data into your own ReSim project.

Pick one with --demo:

  mujoco       An ALOHA bimanual manipulation policy in MuJoCo: one test per
               cube placement, compared across two policy builds.
  navigation   A hospital navigation suite: dense telemetry across 34
               scenarios, covering every chart type ReSim ships.
  session      Four real field sessions from a legged robot: GNSS, IMU and
               gait telemetry, one batch per session on a shared branch.

For example:
  signalflag-demo --demo navigation

Each demo has its own project, branch and metrics config, so running all three leaves you with three independent sets of results, and none goes near your real projects:

--demo Project Branch Batches Data to download
navigation SignalFlag SDK Demo sdk-demo 2 × 34 tests ~7 MB
mujoco SignalFlag SDK Demo (MuJoCo) sdk-demo-mujoco 2 × 20 tests ~110 MB
session SignalFlag SDK Demo (Logs to Insights) sdk-demo-logs-to-insights 4 × 1 test ~950 MB

The navigation demo, the MuJoCo demo and the session demo below go through what each one contains, metric by metric.

Step 3: Bootstrap the demo

Shell
signalflag-demo --demo navigation

Four things happen, in this order.

It fetches the data it replays. Each demo's data is a single tarball in a public bucket, pinned by checksum and cached after the first run. The download comes before the login on purpose: there is no point sending you through a browser only to fail on a download afterwards.

Output
Downloading demo data (7 MB)
  7 / 7 MB (100%)
Checking it downloaded intact

It authenticates you. The first run opens a device-code login. Later runs reuse the cached token.

Output
Authenticating by Device Code

Please navigate to: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX

Visit that URL, sign in, and the demo continues on its own.

It creates the project, the branch and the dashboard. The project is created only if it doesn't already exist. Syncing the demo's metrics config is what creates the branch and the dashboard along with it.

Output
Creating project 'SignalFlag SDK Demo'

It replays the two batches. Every test is created, its emissions and events are pushed through the same typed SDK calls you would use for your own data, its files are attached, and it is closed.

Output
Running batch A: Nav stack v2 (baseline) (34 tests)
  Hospital East Hallway Navigation From North Hallway - Bright with Pedestrians
  Hospital East Hallway Navigation From North Hallway - Bright without Pedestrians
  ...
Running batch B: Nav stack v3 (candidate) (34 tests)
  ...

Both batches land on one branch. A dashboard is scoped to a single branch, so sharing one is what lets a single dashboard trend across both; the batches are told apart by their build version instead.

When it finishes, the demo prints where to go:

Output
SignalFlag SDK demo complete.

  Batch A (baseline, nav-v2.0.0)    https://app.signalflag.ai/projects/.../batches/...
  Batch B (candidate, nav-v3.0.0)   https://app.signalflag.ai/projects/.../batches/...
  A/B comparison                    https://app.signalflag.ai/projects/.../batches/.../compare-batch/batch/...
  Trends dashboard                  https://app.signalflag.ai/projects/.../dashboards/...

Metrics are computed after each batch closes, which takes a few minutes.
Reload these pages when it finishes.

Metrics are computed once each batch closes, so the pages look empty at first. Give it a couple of minutes and reload.

  • Batch A and Batch B list their tests with a pass, warning or blocker status. Open any test for its metrics, plus an Events tab and a Logs tab wherever that run recorded something for them.
  • The A/B comparison matches metrics by name between the two runs, so each chart from Batch A sits next to the same chart from Batch B. The Tests tab groups experiences by how their outcome differs, which puts the regressions and the fixes in their own lists. See the A/B comparison guide.
  • The dashboard aggregates across runs rather than showing one, grouped by build version. Run the demo again and the same bars take in the new runs too.

Command reference

Flags

Flag Default What it does
--demo {mujoco,navigation,session} none Which demo to run. Required.
--project-name the demo's own Project to run in, created if it does not exist.
--branch the demo's own Branch to create every batch on.
--data-dir none Replay an already-extracted bundle from this directory instead of downloading one.
--quiet off Only report failures.

Passing any of the other flags without --demo is an error rather than a listing, because dropping your argument and exiting successfully would read as though the run had happened:

Output
signalflag-demo: error: --demo is required when passing --project-name

To keep a demo out of the default project, name your own:

Shell
signalflag-demo --demo navigation --project-name "my project"

Where the data is cached

Each demo's bundle is downloaded once and cached under its own key, so switching demos does not evict the other's data. The cache root honours XDG_CACHE_HOME and otherwise sits under ~/.cache:

  • ~/.cache/resim/sdk-demo/navigation-v1/
  • ~/.cache/resim/sdk-demo/mujoco-v4/
  • ~/.cache/resim/sdk-demo/session-v1/

Every bundle is pinned by sha256. A download that does not match its expected digest is refused rather than replayed, and the archive is checked for unsafe member paths before anything is extracted. Delete a cache directory to force a fresh download.

Running without a download

--data-dir points the demo at an already-extracted bundle — useful on a restricted network, or to re-run repeatedly without touching the cache:

Shell
signalflag-demo --demo mujoco --data-dir ./extracted-bundle

The directory must contain the bundle's manifest.json at its root.

Exit codes

Code Meaning
0 The demo ran, or a bare invocation listed the demos.
1 The replay data could not be fetched or used, or contact with SignalFlag was lost. Batches already created stay in the app; re-run to start a fresh pair.
2 Bad arguments — an unknown --demo, or another flag passed without --demo.

Calling it from Python

Useful when you want the IDs back rather than reading them off the terminal:

Python
from signalflag.demo import DEMOS, run

result = run(demo="navigation")

result.project_id     # the project it used or created
result.batch_ids      # {"a": ..., "b": ...}
result.dashboard_id   # None if the dashboard could not be resolved
result.urls           # {"batch_a": ..., "batch_b": ..., "compare": ..., "dashboard": ...}

DEMOS.keys()          # dict_keys(['navigation', 'mujoco', 'session'])

session has four batches rather than two, so its result.batch_ids has four keys — the four session dates. It has no dashboard, so result.dashboard_id is always None and result.urls has no "compare" or "dashboard" key; it has a "sessions" entry instead, pointing at the project's Sessions library.

run() takes the same options as the command line — project_name positionally, then demo, branch, data_dir and quiet as keywords, plus a client if you would rather authenticate yourself than be sent to a browser. It raises signalflag.demo.DemoDataError when the replay data cannot be fetched.

The navigation demo

Shell
signalflag-demo --demo navigation

A mobile robot running errands around a hospital. This is the fuller tour of the two: the source data is dense telemetry, and the config that reads it draws nine of the ten system templates — everything but artifact — plus two custom ones. Start here if you want to see what the platform renders.

What lands in your project
Project SignalFlag SDK Demo
Branch sdk-demo
Batch A Nav stack v2 (baseline), build version nav-v2.0.0, 34 tests
Batch B Nav stack v3 (candidate), build version nav-v3.0.0, 34 tests
Metrics set Demo Metrics — 13 test metrics and 5 batch metrics
Dashboard SDK Demo Trends, from the Demo Trends set, over a rolling 30 days

Each test is one scenario: a route through the hospital, at one of three lighting levels, with or without pedestrians — Hospital East Hallway Navigation From North Hallway - Bright with Pedestrians, and so on. The same 34 run in both batches, so every test pairs with its counterpart in the A/B view. Seventeen of them have pedestrians in them, and eleven carry camera footage — the same eleven on both sides, so the media metrics appear on both halves of those pairs.

Pass and fail are not copied from the source runs. They are recomputed from the config's status checks against the replayed data, which works out as 16 passed, 9 warned and 9 blocked in Batch A against 15, 8 and 11 in Batch B. Seven scenarios change outcome between the two — four get worse, three get better — and those are the ones that land in the comparison's Failing on just one batch group. The rest divide between the groups either side of it, so none of the four comes up empty.

The data it replays

Thirteen topics, declared in the demo's config and emitted per test:

Topic Fields What it carries
odom_linear_velocity x, y, z Robot velocity from the drive stack.
goal_distance goal_name, distance_m Distance to each named goal, over the run.
time_to_goal goal_name, time_s Time taken to reach each goal. One row per goal actually reached.
goal_count count How many goals the run was asked to reach.
goal_status state What the navigation stack was doing, as a discrete state.
nearest_human_distance person_name, distance_m Distance to the nearest tracked person. Only the 17 scenarios with pedestrians emit it.
pose_difference position_diff_m Gap between wheel odometry and the localizer's pose estimate.
localization_uncertainty cov_x, cov_xy, cov_y, cov_yaw, position_uncertainty_m, yaw_uncertainty_rad The localizer's own confidence in that estimate.
covariance_accuracy mahalanobis_distance, normalized_innovation_squared, normalized_x, normalized_y, normalized_yaw, position_error_m, within_1_sigma, within_2_sigma, within_3_sigma, yaw_error_rad Whether that confidence is honest — true error against reported covariance.
robot_trajectory raw_metric A Plotly figure the test rendered itself.
camera_video camera_name, filename (video) The onboard camera clip.
camera_frame camera_name, filename (image) A still pulled from that clip. Separate from camera_video because image and video columns cannot share a topic.
goal_reached event: name, description, status, tags One event each time the robot reaches a goal.

What you see on a test

Metric Template What it shows
Robot Speed line Ground speed in m/s over the run.
Speed Distribution histogram How much of the run was spent at each speed.
Time in Speed Band pie Share of samples stopped (under 0.05 m/s), creeping (under 0.3 m/s) or cruising.
Distance to Goal line Distance to each goal over time, one line per goal.
Goal Status state timeline The navigation stack's state, as bands along the run.
Localization Error line Odometry against the localizer's estimate, in metres.
Uncertainty vs Actual Error line Predicted uncertainty, observed error and Mahalanobis distance on one set of axes. A localizer whose uncertainty tracks its error is one you can trust.
Covariance Calibration table Sample count, the share of samples inside 1σ, 2σ and 3σ against their targets, and median NIS.
Nearest-Human Distance line Distance to the nearest person. Warns inside 1 m, blocks inside 0.5 m. Skipped on the scenarios with no pedestrians.
Time to Reach Final Goal scalar (seconds) Time taken to reach the run's goals, added up. Blocks when the run does not reach every goal it was given, and charges 75 s instead.
Camera Feed video The onboard clip, on the 11 tests that carry one.
Camera Frame image The still from that clip.
Robot Trajectory custom (raw.liquid) Top-down view of the path the robot took, from the figure the test emitted.

Those two status checks are the whole of pass and fail here: everything else is there to be read. The config is written for runs with any number of goals, but every scenario in this data sets exactly one, so the per-goal charts draw a single line and a run either reaches its goal in 35 to 85 seconds or blocks.

The Events tab carries a Goal 1 Reached event pinned to the moment the robot got there, with a navigation tag — on 27 of the 34 tests in Batch A and 24 in Batch B. The Logs tab holds the camera clip and the still frame on the eleven tests that recorded them.

What you see on the batch

Metric Template What it shows
Mean Localization Error by Experience bar Average localizer error in each scenario, across the batch.
Localization Error Spread custom (strip.liquid) Worst error per scenario, as a strip plot.
Covariance Calibration Summary table Sample count, the 1σ and 2σ shares and median NIS, across every test in the batch.
Pedestrian Proximity Incidents bar Per tracked person, how many scenarios came within 0.5 m (a hit) or 0.5–1 m (too close). At most one count per scenario.
Total Time to Reach Final Goal scalar (seconds) Every scenario's time added up, with the same 75 s charged for a missed goal.

What you see on the dashboard

SDK Demo Trends looks at every test on the branch inside a rolling 30-day window and groups by build version, so it compares versions of the stack rather than snapshotting one run:

Metric Template What it shows
Localization Error by Build Version bar Mean localizer error per version. A taller bar than the version before it is a regression.
Test Outcomes by Build Version bar How each version's tests turned out, one bar per outcome. Click a bar to open a batch that version ran in.
Tests Run scalar Tests on this branch inside the window.

Run the demo again and those bars take in the new runs rather than multiplying into new points, which is what you want from a dashboard pinned to a branch.

The MuJoCo demo

Shell
signalflag-demo --demo mujoco

Two ALOHA robot arms passing a cube between them, evaluated in MuJoCo. Worth a look once you have toured the navigation demo, because the shape of the data is different: a staged task with a reward ladder rather than dense telemetry, and a question — did this policy configuration get better or worse — rather than a survey of chart types.

What lands in your project
Project SignalFlag SDK Demo (MuJoCo)
Branch sdk-demo-mujoco
Batch A n_heads=16, attention re-partitioned, build version aloha-n16, 20 tests
Batch B n_heads=8, checkpoint default, build version aloha-n8, 20 tests
Metrics set ALOHA Metrics — 9 test metrics and 5 batch metrics
Dashboard ALOHA Policy Trends, from the ALOHA Trends set, over a rolling 30 days

Each test is one starting placement of the cube on the table. That placement is the only thing that differs between tests, so each is named for it: Cube start (0.099, 0.446) seed 50. The same 20 placements run on both sides, which is what pairs every test with its counterpart.

The two batches are the same policy checkpoint built two ways, differing in one hyperparameter: the number of attention heads. One matches the value the policy was trained with and the other does not, which is enough to change how often the handover completes — 8 of 20 placements solved on the re-partitioned build against 14 on the checkpoint default, with 8 placements changing outcome between them. That is the question the A/B page answers here.

The task has stages rather than a single outcome. The cube gets touched, lifted, carried across, then released, and most of the metrics follow from that ladder: no contact, right gripper contact, lifted, transfer attempted, handover complete.

The data it replays

Eight topics:

Topic Fields What it carries
summary_metrics overall_success_rate, overall_average_sum_reward, overall_average_max_reward, total_evaluation_time, total_regressions One row per test, summarising the episode.
step_reward reward The staged reward, one row per simulation step.
task_stage state Which rung of the ladder the episode is on, per step.
cube_pose x, y, z World position of the cube. The handover counts as complete once x crosses -0.10.
stage_timing stage, seconds_from_start, seconds_from_previous, regressions One row per stage the episode reached.
cube_trajectory raw_metric A Plotly figure the evaluation drew itself.
episode_gif filename (image) The whole episode as a looping animation.
stage_reached event: name, description, status, tags, metrics Raised the first time each stage is reached, carrying the timings that explain the transition.

What you see on a test

Metric Template What it shows
Task Stage state timeline Which stage the episode was in, so a run that grasps and then drops the cube looks different from one that never grasps it.
Reward Over Time line The staged reward: 1 on contact, 2 lifted, 3 transfer attempted, 4 handover complete.
Handover Progress line How far the cube is past the handover line. Zero is the line; the handover completes once this goes positive.
Cube Height line The cube sits at 0.05 m on the table, so a rise is the lift and a drop back is a lost grasp.
Episode Summary table Handover completed, furthest stage, episode length, sum reward, ladder regressions. Warns when the handover did not complete.
Episode Replay image The looping animation of the attempt.
Cube Path custom (raw.liquid) Top-down view of the cube's path, coloured by stage. The only chart that shows the shape of the motion rather than one axis of it against time.
Stage Latency bar Simulated seconds from the start of the episode to each stage it reached. A bar rather than a table column so an A/B comparison can overlay two runs on one set of axes.
Stage Timeline table Each stage reached, when, how long the stage before it took, and how many times the episode had fallen back by then.

Completing the handover passes and anything short of it warns. Nothing blocks: a cube placement the policy cannot solve is not a broken build.

Every test carries two files on its Logs tab. transfer_cube.mcap goes up as robotics data, and uploading it as the kind of file it is rather than as bytes is what lets it open in the viewer inline instead of only offering a download. episode.gif goes up as a generic log, and reaches the page by a different route: the Episode Replay metric finds it by the filename the episode_gif topic emits. Your own tests attach files the same way; see Attaching files.

The Events tab carries one event per stage the episode first reached — Cube Contacted, Cube Lifted, Transfer Attempted, Handover Complete — each one carrying the time to that stage, the time spent in the previous one, and the regressions so far.

What you see on the batch

Metric Template What it shows
Mean Time Per Stage bar Average simulated seconds each stage took, across the placements that reached it.
Seeds Reaching Each Stage bar How many placements got as far as each stage.
Furthest Stage Distribution pie How far the policy got, as a share of the batch.
Run Summary table Placements evaluated and solved, mean sum reward, mean episode length, mean ladder regressions.
Reward Distribution histogram Spread of total episode reward across the batch.

What you see on the dashboard

ALOHA Policy Trends groups by build version, so each policy configuration becomes one category. Every bar here is click-through: select one to open a batch that build ran in.

Metric Template What it shows
Success Rate by Build Version bar Share of placements where the handover completed. The headline number a policy change is trying to move.
Test Outcomes by Build Version bar How each build's tests turned out, one bar per outcome.
Furthest Stage by Build Version bar Where each build's placements ended up on the ladder. A build can hold its pass count while its failures move between stages, and that shift is the more informative signal.
Time to Handover by Build Version bar Mean seconds to complete the handover, counting only the placements that completed it. Separates getting there from getting there quickly.
Reward by Build Version bar Mean total episode reward. A build that drops here has regressed, whatever its pass count says.
Trend Summary table Builds compared, tests run, batches, inside the window.

The session demo

Shell
signalflag-demo --demo session

Four real field sessions from a legged robot walking outdoors, captured on four different days. This demo is not an A/B pair — there is no build to compare against another — so it answers a different question than the other two: not "did this change help," but "how did today's session compare to the last several." Each session becomes its own batch, all four land on one branch, and the demo tags every session's experience so it shows up in the project's Sessions library alongside its own evaluation.

What lands in your project
Project SignalFlag SDK Demo (Logs to Insights)
Branch sdk-demo-logs-to-insights
System Session Evaluations, created on first run
Experience tag resim-session, applied to every session's experience
Batches Four, one per session date — 2024-11-04, 2024-11-14, 2024-11-15, 2024-11-18 — each with exactly one test
Metrics set Session Metrics — 6 test metrics and 3 batch metrics

There is no dashboard here — the Sessions library is this demo's trend view, grouping by session rather than by build version, so a metrics dashboard alongside it would be empty by construction.

Every batch's one test is the whole session, not a scenario cut out of it — the same mcap snippets, gif clips and telemetry a real ingested recording carries. Because each batch has exactly one test, the demo prints a direct link to that test's page rather than to the batch as a whole, opened on its log viewer tab.

The data it replays

Thirteen topics per session:

Topic Fields What it carries
mission mission_idx, start_s, end_s, duration_s, distance_m, net_displacement_m, mean_speed_mps, max_speed_mps, moving_share, dlio_distance_m, odom_dlio_gap_m, gnss_distance_m, gnss_odom_gap_m, has_gnss, gnss_coverage, gnss_fix_count, altitude_min_m, altitude_max_m, altitude_gain_m, start_lat, start_lon, foot_slip_share, odom_gnss_fit_rmse_m One row per mission ReSim's own boundary detector found in the session.
motion_phase mission_idx, phase_idx, kind, start_s, end_s, duration_s, distance_m, mean_speed_mps, max_speed_mps Contiguous stopped/moving segments within a mission.
movement_summary seq, mission_idx, t_s, kind, label, clip (image), snippet, phase_duration_s, phase_distance_m, phase_mean_speed_mps One row per start/stop transition, with the clip that shows it and the raw mcap snippet it was cut from.
robot_pose mission_idx, series, t_s, x, y, z, east_m, north_m, has_enu Robot position, in both the local and the ENU frame where available.
robot_state mission_idx, series, t_s, feet_in_contact, feet_slipping, max_foot_force_n, max_joint_effort_nm, max_joint_velocity_rps, speed_mps Gait state — how many feet are down, and whether any are slipping.
speed_sample mission_idx, series, t_s, speed_mps Ground speed.
imu_sample mission_idx, series, t_s, accel_mag_mps2, gyro_mag_rps IMU acceleration and gyro magnitude.
gnss_track mission_idx, series, t_s, latitude, longitude, altitude_m, east_m, north_m Raw GNSS fixes.
command_tracking mission_idx, series, t_s, cmd_speed_mps, act_speed_mps, speed_error_mps, cmd_yaw_rps Commanded against actual speed.
battery_sample mission_idx, series, t_s, state_of_charge, cell_temperature_c Battery state over the session.
session_events event: name, description, status, tags, recording_key, mission_idx, t_s, metrics Mission boundaries and clip references, one per moment worth marking.
session_summary n_missions_detected, n_missions_ground_truth, detection_correct, boundary_error_s, total_duration_s, active_duration_s, total_distance_m, total_gnss_distance_m, mean_speed_mps Headline numbers for the whole session.
session_inventory start_ns, end_ns, files, sizes_bytes, formats What the session's mcap snippets were cut from.

What you see on a test

Metric Template What it shows
Speed Over Time custom (custom_line.liquid) Ground speed over the session.
IMU Acceleration line IMU acceleration magnitude over the session.
Foot Contact and Slip table Gait state over the session — feet down, feet slipping, peak foot force.
Motion Phases table Every stopped/moving segment the session was cut into.
Movement Snapshots image A frame from every start/stop transition.
GNSS Track table Raw GNSS fixes over the session.

The Events tab carries one event per mission boundary and transition the session was cut on. The Logs tab holds every mcap snippet and gif clip the session produced.

What you see on the batch

Metric Template What it shows
Session Summary table Missions detected against ground truth, total distance, total duration, mean speed.
GNSS Coverage scalar Mean share of the session with a GNSS fix, across missions. Warns below 0.5.
Raw Source Inventory table The source recording this session's snippets were cut from.

How the three compare

navigation mujoco session
Shape A/B pair A/B pair Trended over time, no pair
System under test A hospital delivery robot's navigation stack An ALOHA bimanual manipulation policy A legged robot's field sessions
What varies between batches Two versions of the stack One hyperparameter of the same checkpoint The day the session was recorded
Batches 2 2 4
Tests per batch 34 scenarios 20 cube placements 1 — the whole session
Topics 13 8 13
Metrics 21 — 13 test, 5 batch, 3 dashboard 20 — 9 test, 5 batch, 6 dashboard 9 — 6 test, 3 batch
Dashboard Yes, by build version Yes, by build version None — the Sessions library is the trend view instead
System templates covered Nine of ten — all but artifact — plus two custom Liquid templates Seven of ten — no scalar, video or artifact — plus one custom template Four of ten, plus one custom Liquid template
Status checks Two, both of which can block One, warn only One, warn only
Files per test Camera clip and still, on 11 of 34 MCAP and episode animation, on all 20 Every mcap snippet and gif clip the session produced
Data to download ~7 MB ~110 MB ~950 MB

None is a subset of the others. The navigation demo is the wider tour of what SignalFlag renders; the MuJoCo demo is the closer look at comparing two builds of one policy; the session demo is the one to run if what you actually have is field recordings rather than a build to compare against another.

Read the config that drives it

Everything either demo displays comes from a metrics config and its templates, shipped inside the package. Nothing is hardcoded into the platform, and nothing about these files is special to the demo — this is the same config you would write for your own system:

Python
from signalflag.demo import config_path, templates_path

print(config_path("navigation").read_text())
print(templates_path("navigation"))   # directory of the .liquid templates it references

config_path("mujoco") and config_path("session") give you the other two. Three pieces matter, in this order.

Topics describe the shape of the data your tests emit:

config.resim.yml
topics:
  odom_linear_velocity:
    schema:
      x: float
      y: float
      z: float

Metrics are SQL queries over that data, each rendered by a template. A metric's type decides what it runs against: test per test, batch across a batch, dashboard across every batch in the dashboard's window.

config.resim.yml
metrics:
  Robot Speed:
    type: test
    description: Ground speed of the robot over the course of the run.
    query_string: |
      SELECT
        'Speed' AS group_name,
        timestamp / 1E9 AS "Time (s)",
        SQRT(POWER(x, 2) + POWER(y, 2)) AS "Speed (m/s)"
      FROM odom_linear_velocity;
    template_type: system
    template: line

Batch and dashboard metrics can join against the built-in metadata table, which is where the build version, experience name and status of each test live:

config.resim.yml
metrics:
  Localization Error by Build Version:
    type: dashboard
    query_string: |
      SELECT
        'Mean' AS group_name,
        m.build_version AS "Build Version",
        AVG(pd.position_diff_m) AS "Mean Localization Error (m)"
      FROM pose_difference pd
      JOIN metadata m ON pd.job_id = m.job_id
      WHERE m.build_version IS NOT NULL
      GROUP BY m.build_version
      ORDER BY m.build_version;
    template_type: system
    template: bar

Metrics sets group metrics so that a batch or a dashboard can run them. Note the key is metrics sets, with a space:

config.resim.yml
metrics sets:
  Demo Metrics:            # run against each batch
    metrics:
      - Robot Speed
      - Speed Distribution
      # ...
  Demo Trends:             # powers the dashboard, which spans batches
    metrics:
      - Localization Error by Build Version
      # ...

dashboards:
  SDK Demo Trends:
    metrics_set: Demo Trends
    refresh: auto
    day_range: 30

A metric with template_type: custom names a .liquid file instead of a system template, and that file has to be synced alongside the config — which is why templates_path() exists next to config_path(). The metrics guide covers topics, every template, status checks and custom templates in full.

Next steps