Metrics 1.0 (Legacy)

This is legacy documentation

Metrics 1.0 uses a metrics build — a Docker container image that runs after each test to compute and emit metrics as a binary protobuf file. This approach has been superseded by Metrics, which ingests data directly from your experience build and lets you define visualizations in a config file without building or maintaining a separate image. New projects should use Metrics. This page is preserved for teams already running Metrics 1.0.

Overview

The ReSim metrics framework is an open-source set of libraries for constructing metrics functions that generate metrics data the ReSim platform can parse. The libraries can be found on GitHub in resim-ai/open-core. The Python metrics API can be installed from PyPI with pip install resim-open-core and its documentation can be found here.

How metrics builds work

Similar to builds, you register a metrics build with ReSim as a container image. ReSim executes it after each test. The same image is expected to handle three modes:

Test mode — ReSim populates /tmp/resim/inputs with:

experience/ — the files that made up the experience
logs/ — any files generated as outputs from the experience build

The image must write a file called metrics.binproto to /tmp/resim/outputs. ReSim uses this file to register metrics. Any additional files in the outputs directory are uploaded as logs.

Batch mode — ReSim populates /tmp/resim/inputs with a configuration file the Metrics SDK can use to fetch per-test metrics for aggregation. A metrics.binproto is expected in the outputs directory.

Report mode — See Test Suite Reports.

Creating a metrics build

Bash

resim metrics-builds create \
    --project "my-project" \
    --name "My Metrics Build" \
    --version <commit sha or version> \
    --image <Docker image URI> \
    --systems "perception" "localization"

Associating a metrics build with a system

Bash

resim metrics-builds add-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"

To remove the association:

Bash

resim metrics-builds remove-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"

Debugging

The ReSim app has a metrics debugger page where you can upload a metrics.binproto file and see how it would render in the dashboard, without running a full test. See Metrics Debugger.

SDK reference

For full documentation on constructing metrics using the Python SDK, see the Metrics Build SDK reference.