Adding Experiences

To test our system we need some experiences.

Prerequisites

Before creating an S3 experience, you need to have an experiences AWS S3 bucket set up as outlined in ReSim Data Access. You will also need access to this S3 bucket.

Storing files

Each experience should be stored in a unique path containing the input files. So for an S3 bucket, each experience should have a unique prefix, such as s3://my-experiences-bucket/project_hal9000/experiences/experience_56ccce99/. Anything contained in that prefix will be treated as the same experience. We recommend using a directory structure that represents your ReSim projects. For example:

📂 .
├── 📂 project_a
│  └── 📂 experiences
│     └── 📂 experience_56ccce99
│        └── ⚙️ test_file
│
└── 📂 project_b
   └── 📂 experiences
      └── 📂 experience_679bef80
         └── ⚙️ test_file

These experience files will be copied into /tmp/resim/inputs/ when the experience build is being created and will be available to you.

We have now implemented experience caching for some orgs. If experience caching is enabled for you, we will only copy files that have been added or changed since the last run. See the linked guide for more details.

Local experiences

It may be the case that experience data lives in the same repository as your system-under-test. For example, if it is tightly coupled to changes in the system. In order to support this use case any path may be supplied in the location field.

Providing access to local experiences in your container

In your experience build Dockerfile you should copy over the files from your repository to your running container to ensure you have access to them. For example:

COPY <path>/<to>/<experience>/<in>/<repo> /experiences/

Programmatically accessing local experiences

If you use a local experience ReSim will not perform any automatic actions on your behalf other than the creation of the /tmp/resim/test_config.json file. This file will have a json field called experienceLocation that you can access to know where the experience is located programmatically. This field comes from the --location flag that you set when creating the experience. Here is an example of accessing the experience location programmatically:

# try to read the test_config.json file, from /tmp/resim/test_config.json
try:
    with open("/tmp/resim/test_config.json", "r") as f:
        test_config = json.load(f)
except FileNotFoundError:
    raise FileNotFoundError("test_config.json not found in /tmp/resim")

# pull out the experienceLocation key from the test_config.json file
experience_location = test_config["experienceLocation"]

Experience Timeouts

Each experience has a timeout. This is the maximum amount of time that a build can run for a given experience. If the build does not finish running before the timeout is reached, it will be sent a SIGTERM and then a SIGKILL after a grace period.

The default timeout is 1 hour, but you can specify a different timeout when creating an experience.

Creating an experience

You can create new experiences like so:

resim experiences create \
    --project "my-project" \
    --name "Experience_56ccce99" \
    --description "In this experience, we test that our robot can navigate to the goal without colliding with any other agents." \
    --location "s3://my-experiences-bucket/project_hal9000/experiences/experience_56ccce99/" \
    --systems "perception" "localization" \
    --tags "navigation" "collision-avoidance"

Important

Any location that points to an S3 bucket must be prefixed with s3:// rather than the https://... form.

This command should return a message containing the UUID of the created experience, and it should now be browsable through the ReSim app.

The --systems flag enables you to specify any systems that this experience is compatible with. You can specify multiple systems by providing multiple system names or IDs.

The --tags flag allows you to apply tags to the experience for better organization and filtering. You can specify multiple tags by providing multiple tag names or IDs.

Optional flags are:

    --timeout "1h10m" # the maximum expected duration of the container \
    --environment-variable "SYSTEM_CONFIGURATION=Foo" # many allowed \
    --environment-variable "AN=other" \
    --profile "left-arm" # a docker compose profile to select the subset of compose services to run \

Updating an experience

You can update various aspects of an experience, including its systems and tags:

resim experiences update \
    --project "my-project" \
    --experience "Experience_56ccce99" \
    --name "Updated Experience Name" \
    --description "Updated description" \
    --location "s3://my-experiences-bucket/project_hal9000/experiences/updated_experience/" \
    --timeout "2h" \
    --systems "perception" "planning" \
    --tags "updated-tag" "new-functionality"

When updating systems or tags, the provided lists will replace the existing systems and tags associated with the experience.

You can also add or remove individual systems from an experience:

resim experiences add-system \
    --project "my-project" \
    --system "<system name>" \
    --experience "Experience_56ccce99"

resim experiences remove-system \
    --project "my-project" \
    --system "<system name>" \
    --experience "Experience_56ccce99"

Similarly you can add and remove individual tags:

resim experiences tag \
    --project "my-project" \
    --tag "<tag string>" \
    --experience "Experience_56ccce99"

resim experiences untag \
    --project "my-project" \
    --tag "<tag string>" \
    --experience "Experience_56ccce99"