ReSim Log Ingestion
Introduction
An important part of testing embodied AI is field testing and gathering examples from real-world deployments. Historically this type of testing requires manual inspection of the generated logs and is decoupled from the simulation testing workflows.
The ReSim platform, however, makes it possible to ingest these logs and compute metrics automatically as part of a ReSim metrics flow. One can then further process these logs to generate replay tests for interesting subsets.
This guide explains how log ingestion works with the ReSim CLI.
Prerequisites
The ReSim CLI requires that any logs to be ingested have already been uploaded to a cloud bucket and that ReSim has been granted access. This is described in the setup guide.
Core Concepts
The log ingestion process in ReSim will:
- Store the logs as experiences, each with a user-supplied name, tagged as
ingested-via-resim
- Create a test batch that will ingest the logs and run a given metrics build on them. The batch will contain a test for every log ingested and will, by default, attempt to also calculate batch metrics on the logs
- Once completed, the test results will be available to visualize in the usual way, in the results page
Configuration
The ReSim CLI functionality takes several required and optional parameters. To see full documentation of each, please refer to resim ingest help
.
Required Parameters
project
: the name of the project to ingest the log to (can be omitted if you have usedresim project select
to set your default project)system
: the name of the system that this log belongs to e.g.full-stack
log
: the name and location of the logs you would like to ingest, specified aslog-name=log-location
. This accepts multiple log name and location pairs to ingest multiple logs. For ingesting large numbers of logs, it is recommended to use the config file approachmetrics-build-id
: the unique ID for the metrics build to run, which can be found viaresim metrics-builds list
or in the ReSim app.
Optional (and recommended) Parameters
branch
: the name of the branch that you would like this log ingestion to be related to (e.g. what branch of your system created it). If no branch name is supplied, it will automatically be ingested underlog-ingest-branch
.version
: what version of your code you would like this log ingestion to be related to (e.g. what release or commit sha). If no version is supplied, thenlatest
is used.tags
: a comma-separated list of tags to be added to the ingested log; by default ReSim appliesingested-via-resim
, but additional metadata tags can be provided here.
Alternative Approaches
YAML Log Specification File
The ReSim CLI supports providing a location to a YAML file with the log names and locations via --config-file <path-to-file>
. The syntax of the YAML input is:
# logs.yaml
logs:
- name: "log1"
location: "s3://bucket/path1"
- name: "log2"
location: "s3://bucket/path2"
Deprecated Single-log Ingestion
It is possible to use two flags to ingest a single log, rather than the --log "name=location
syntax:
log-name
: the unique name that you would like to give to this log (often a field test id or timestamped)log-location
: a cloud storage path to find the log (currently onlys3
is supported)
Example
resim ingest \
--system "My Drone Stack" \
--log "Drone-Test-42=s3://my-bucket/prefix" \
--log "Drone-Test-43=s3://my-bucket/prefix2" \
--metrics-build-id "<UUID of the metrics build to run>" \
--branch "main" \
--version "1.1.0" \
--tags "field-test,hardware-v2"
Using a custom ingestion build
If you would like to pre-process a log before ingestion and cannot do that within a metrics build (due to wishing to reuse it in non-ingestion settings), the ReSim CLI allows you to run the resim ingest
command with your own build. In this case, you would simply use:
resim ingest \
--build-id "<UUID of the build to use>" \
--log "Drone-Test-42=s3://my-bucket/prefix" \
--metrics-build-id "<UUID of the metrics build to run>" \
--tags "field-test,hardware-v2"
In this case, one cannot supply the --system
, --branch
, or --version
flags as that information is derived from the build.