Skip to content

Metrics Images

This document introduces our open source metrics SDK and describes how the ReSim platform supports executing metrics.

ReSim Metrics Framework

The ReSim metrics framework is an open-source set of libraries to support constructing metrics functions that generate metrics data that the ReSim platform can parse. The libraries can be found on GitHub. We are currently writing more comprehensive documentation for our metrics framework.

Metrics in the ReSim Platform

Similar to builds, customers can register a Metrics Build with ReSim, which will be executed after each test. A metrics build is also a container image, with a similar contract to builds.

For convenience, the same metrics build is expected to be able to act in three modes: test mode, batch mode, and report mode. Using the same metrics build for all three related tasks means we have less code repetition, fewer metrics builds, and a lower chance of compatibility errors.

In test mode, the ReSim platform will populate the /tmp/resim/inputs directory with:

  • a subdirectory, experience, which contains any files as part of the experience
  • a subdirectory, logs, which contains any files generated as logs from the execution of the simulation.

The ReSim platform expects to find a file called metrics.binproto in the /tmp/resim/outputs directory and will use this to register metrics with the system. It is highly recommended to follow the ReSim Metrics SDK for constructing this protobuf binary, since it provides an abstraction from the raw schema.

Any additional files placed in the outputs directory will also be uploaded to the ReSim platform as logs.

In batch mode, the ReSim platform will only populate the /tmp/resim/inputs directory with a configuration file that the Metrics SDK can use to fetch the metrics for each individual test. For more information, see the open source documentation. It is expected that a metrics.binproto exists in the outputs directory.

For more details about report mode, see Test Suite Reports.

Creating Metrics Builds

It is possible to create metrics builds in CI/CD or with the CLI tool:

resim metrics-builds create \
    --project "my-project" \
    --name "My Metrics Build" \
    --version <commit sha or version> \
    --image <Docker image URI> \
    --systems "perception", "localization"

Where, again, if one has a project selected, that parameter can be omitted. Similarly to builds, ReSim recommend the set-up of CI/CD workflows to generate metrics builds when the metrics generation code is updated.

Systems and Metrics Builds

Since metrics are closely tied to the test application which runs, the ReSim app enables metrics builds to be aligned to a given system, to enable compatibility checking at test creation time. One can register an existing metrics build against a given system:

resim metrics-builds add-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"

and remove then with:

resim metrics-builds remove-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"

Debugging Metrics

The Resim app provides a page which can be used for debugging Metrics output by uploading a binproto. The page will render what the app would render given the proto. See: Metrics Debugger