Experience Syncing

For larger projects, it's often convenient to store a registry of experiences and their properties locally for a number of reasons. For example:

• Other tools besides ReSim may leverage this configuration to run local (or HiL) sims or manage test sets.
• It may be useful for additional organization-specific metadata to be attached to experiences, even if said data isn't as relevant within the ReSim app.
• Experience history can be stored in version control alongside code which can be useful when inspecting historical sim behavior.

In order support these sorts of use cases, and many more, ReSim supports Experience Syncing. We define a schema which contains a list of experiences, their tags, systems, and test suites. Users can then produce config files matching that schema in a number of ways (more on this below). The key to making this config useful, however, is the ReSim CLI, which can be used to mirror the config state to the ReSim project of your choice. This makes it very easy for users to make bulk programmatic updates to experiences (e.g. renaming, switching s3 paths, retagging) without having to manually navigate all the api endpoints needed for the specific category of update they're trying to accomplish.

A Simple Syncing Example

The config schema is relatively simple and looks like this in practice:

experiences:
    - name: scenario-survey-alpha
      description: Aerial survey over test zone
      locations:
        - s3://drone-missions/surveys/alpha-test-zone
      tags:
        - regression
        - my_custom_tag

    - name: new-scenario-system-check
      description: Regression validation run
      locations:
        - s3://drone-missions/system-checks/regression-1
      tags:
        - progression
      systems:
        - mbauer_tmp_hil_repro

managed_test_suites:
    - name: Basic Suite
      experiences:
        - scenario-survey-alpha
        - new-scenario-system-check

managed_experience_tags:
    - regression
    - progression

The full schema for this file is provided below in the Appendix. This config can be mirrored onto an existing ReSim project like so:

Warning

This command should be run with care. It will affect other users of whatever project you run it on, so it's advisable to set up a test project if you're just testing out the tool. You can do this using the resim projects create command in the CLI.

resim experiences sync \
    --project <project-name> \
    --experience-config <config/file/path.yaml>

Let's walk through what this command will do. We hope that this is relatively intuitive, but there are a lot of edge cases so we break it down in detail:

1. The experiences in the project will be updated so that every experience in the config will exist with the specified parameters. Existing experiences will be updated and new experiences will be created. Experiences that aren't in the config will be archived, and any archived experiences that are in the config will be restored as they are updated. Experiences are typically identified in the config by unique name, but their experience_id can also be provided as a field to enable renaming of existing experiences so long as doing so doesn't violate the uniqueness constraint on names. Note that swapping or permuting names is not currently allowed even if it preserves uniqueness because computing a valid ordering for the updates would be complex.

2. The experiences tags are updated in the following manner: All experience tags listed for each experience are added to that experience if they don't already exist on that experience. All managed experience tags that aren't listed on each experience are removed from that experience if it does currently have them. The managed experience tags are the only tags that are ever removed from any experience. None of these changes apply to experiences we're archiving. All of the tags listed must already exist in the project.

3. The systems are updated similarly to the experience tags, but they're always additive. We never remove systems from experiences. All of the systems listed must already exist in the project.

4. All test suites in the managed_test_suites are revised to contain exactly the experiences listed in the config by their (possibly new) names. Other test suites may be revised, but this will only happen if experiences they currently contain get archived according to the earlier rules, forcing a revision. All of the test suites listed must already exist in the project.

Where should I get a config?

If you'd like to use this feature, there are a number of ways to obtain a useful config. First, you can just write one manually. That's probably not feasible for large projects however. In practice, you will likely want to generate your config using python or another language of your choice. The source data for this could be:

• A directory containing all of your experiences. In this case, your config-generating script can walk through the directory parsing each experience to compute the right tags and systems for it, and then write out the config to a yaml file for you.

• Another config. If you have another config describing your tests, the translation into the ReSim config should be a pretty straightforward script to convert one schema to another.

• An s3 bucket containing your experiences. This is similar to the first option, although you'll likely be using boto3 to get the information you need. As outlined in the experience docs, experiences in s3 buckets work very well in ReSim.

• An existing project with your experiences already in them. For this see the --clone option below.

Other Options

Some other useful options bear mentioning:

The --update-config Flag

When syncing a config to the app, you can update the provided config to include the experience_ids and other information for the existing and new experiences by passing the Boolean --update-config flag when calling resim experiences sync. This is useful if you want to check the config into version control and rename experiences over time.

The --clone Flag

Similarly, you can also clone the existing state of the experiences in the app to the passed-in config using the Boolean --clone flag. E.g.

resim experiences sync \
    --project <project-name> \
    --experience-config <new/cloned/config/path.yaml> \
    --clone

If the config path exists already, the experiences field in it will be replaced while the managed test suites and tags are unaffected. This is useful if you want to clone the experiences from an existing project to add them to another, or simply as a convenient way to get a list of all experiences with their tags and systems.

Note

This currently only populates the experiences. It does not try to determine which experience tags you'd like to be managed_experience_tags or which test suites you'd like to be in managed_test_suites. Furthermore, if the config exists and has the managed_test_suites field, the test suites therein will not be updated to reflect their current membership. This is because we don't currently have to fetch the current test suite membership during the normal sync operation, so we'd have to add it specifically for this. This is a feature we hope to add in the near future.

Appendix: Config Schema

ExperienceSyncConfig

Field	Type	Notes
experiences	list[Experience]	optional
managed_test_suites	list[TestSuite]	optional
managed_experience_tags	list[string]	optional

Experience

Field	Type	Notes
name	string	required, must be unique
description	string	required
locations	list[string]	optional
tags	list[string]	optional
systems	list[string]	optional
profile	string	optional
experience_id	string	optional, must be unique if provided
environment_variables	list[object]	optional (fields: name, value)
cache_exempt	bool	default: false
container_timeout_seconds	int	optional

---

TestSuite

Field	Type	Notes
name	string	optional
experiences	list[string]	optional

---