Skip to content

ReSim Log Ingestion

Introduction

An important part of testing embodied AI is field testing and gathering examples from real-world deployments. Historically this type of testing requires manual inspection of the generated logs and is decoupled from the simulation testing workflows.

The ReSim platform, however, makes it possible to ingest these logs and compute metrics automatically as part of a ReSim metrics flow. One can then further process these logs to generate replay tests for interesting subsets.

This guide explains how log ingestion works with the ReSim CLI.

Prerequisites

The ReSim CLI requires that any logs to be ingested have already been uploaded to a cloud bucket and that ReSim has been granted access. This is described in the setup guide.

Core Concepts

The log ingestion process in ReSim will:

  1. Store the log as an experience, with a user-supplied name, tagged as ingested-via-resim
  2. Create a test batch with the name Ingested Log: <Log Name> that will ingest the log and run a given metrics build on it. The batch will contain a single test, reflecting the log ingested
  3. Once completed, the test results will be available to visualize in the usual way, in the results page

Configuration

The ReSim CLI functionality takes several required and optional parameters. To see full documentation of each, please refer to resim ingest help.

Required Parameters

  • project: the name of the project to ingest the log to (can be omitted if you have used resim project select to set your default project)
  • system: the name of the system that this log belongs to e.g. full-stack
  • log-name: the unique name that you would like to give to this log (often a field test id or timestamped)
  • log-location: a cloud storage path to find the log (currently only s3 is supported)
  • metrics-build-id: the unique ID for the metrics build to run, which can be found via resim metrics-builds list or in the ReSim app.
  • branch: the name of the branch that you would like this log ingestion to be related to (e.g. what branch of your system created it). If no branch name is supplied, it will automatically be ingested under log-ingest-branch.
  • version: what version of your code you would like this log ingestion to be related to (e.g. what release or commit sha). If no version is supplied, then latest is used.
  • tags: a comma-separated list of tags to be added to the ingested log; by default ReSim applies ingested-via-resim, but additional metadata tags can be provided here.

Example

resim ingest \
    --system "My Drone Stack" \
    --log-name "Drone-Test-42" \
    --log-location "s3://my-bucket/prefix" \
    --metrics-build-id "<UUID of the metrics build to run>" \
    --branch "main" \
    --version "1.1.0" \
    --tags "field-test,hardware-v2"

Using a custom ingestion build

If you would like to pre-process a log before ingestion and cannot do that within a metrics build (due to wishing to reuse it in non-ingestion settings), the ReSim CLI allows you to run the resim ingest command with your own build. In this case, you would simply use:

resim ingest \
    --build-id "<UUID of the build to use>" \
    --log-name "Drone-Test-42" \
    --log-location "s3://my-bucket/prefix" \
    --metrics-build-id "<UUID of the metrics build to run>" \
    --tags "field-test,hardware-v2"

In this case, one cannot supply the --system, --branch, or --version flags as that information is derived from the build.