Skip to content

ReSim Log Ingestion

Introduction

An important part of testing embodied AI is field testing and gathering examples from real-world deployments. Historically this type of testing requires manual inspection of the generated logs and is decoupled from the simulation testing workflows.

The ReSim platform, however, makes it possible to ingest these logs and compute metrics automatically as part of a ReSim metrics flow. One can then further process these logs to generate replay tests for interesting subsets.

This guide explains how log ingestion works with the ReSim CLI.

Prerequisites

The ReSim CLI requires that any logs to be ingested have already been uploaded to a cloud bucket and that ReSim has been granted access. This is described in the setup guide.

Core Concepts

The log ingestion process in ReSim will:

  1. Store the logs as experiences, each with a user-supplied name, tagged as ingested-via-resim
  2. Create a test batch that will ingest the logs and run a given metrics build on them. The batch will contain a test for every log ingested and will, by default, attempt to also calculate batch metrics on the logs
  3. Once completed, the test results will be available to visualize in the usual way, in the results page

Configuration

The ReSim CLI functionality takes several required and optional parameters. To see full documentation of each, please refer to resim ingest help.

Required Parameters

  • project: the name of the project to ingest the log to (can be omitted if you have used resim project select to set your default project)
  • system: the name of the system that this log belongs to e.g. full-stack
  • log: the name and location of the logs you would like to ingest, specified as log-name=log-location. This accepts multiple log name and location pairs to ingest multiple logs. For ingesting large numbers of logs, it is recommended to use the config file approach
  • metrics-build-id: the unique ID for the metrics build to run, which can be found via resim metrics-builds list or in the ReSim app.
  • branch: the name of the branch that you would like this log ingestion to be related to (e.g. what branch of your system created it). If no branch name is supplied, it will automatically be ingested under log-ingest-branch.
  • version: what version of your code you would like this log ingestion to be related to (e.g. what release or commit sha). If no version is supplied, then latest is used.
  • tags: a comma-separated list of tags to be added to the ingested log; by default ReSim applies ingested-via-resim, but additional metadata tags can be provided here.

Alternative Approaches

YAML Log Specification File

The ReSim CLI supports providing a location to a YAML file with the log names and locations via --config-file <path-to-file>. The syntax of the YAML input is:

# logs.yaml
logs:
  - name: "log1"
    location: "s3://bucket/path1"
  - name: "log2"
    location: "s3://bucket/path2"

Deprecated Single-log Ingestion

It is possible to use two flags to ingest a single log, rather than the --log "name=location syntax:

  • log-name: the unique name that you would like to give to this log (often a field test id or timestamped)
  • log-location: a cloud storage path to find the log (currently only s3 is supported)

Example

resim ingest \
    --system "My Drone Stack" \
    --log "Drone-Test-42=s3://my-bucket/prefix" \
    --log "Drone-Test-43=s3://my-bucket/prefix2" \
    --metrics-build-id "<UUID of the metrics build to run>" \
    --branch "main" \
    --version "1.1.0" \
    --tags "field-test,hardware-v2"

Using a custom ingestion build

If you would like to pre-process a log before ingestion and cannot do that within a metrics build (due to wishing to reuse it in non-ingestion settings), the ReSim CLI allows you to run the resim ingest command with your own build. In this case, you would simply use:

resim ingest \
    --build-id "<UUID of the build to use>" \
    --log "Drone-Test-42=s3://my-bucket/prefix" \
    --metrics-build-id "<UUID of the metrics build to run>" \
    --tags "field-test,hardware-v2"

In this case, one cannot supply the --system, --branch, or --version flags as that information is derived from the build.