Asset caching

Fetching asset data from cloud storage can involve a large amount of data transfer, especially for assets like HD maps, 3D meshes, or ML model weights. In most cases asset data changes infrequently, often much less frequently than your code. To avoid redundant downloads, assets are cached by default.

This works in the same way as experience caching, so if you are already familiar with that feature, the same principles apply.

How does it work?

When we run your workloads, the ReSim worker mounts a shared storage location that is unique and isolated to your organization. For each asset, we create a directory identified by the asset ID and revision. We then sync the contents of your asset's cloud storage location into that directory, which is mounted read-only in your workload container(s) at /tmp/resim/assets/<mountFolder>.

On the first sync, all files are downloaded and we record a timestamp. On subsequent uses of the same asset revision:

1. We check each file in the cloud storage location's lastModified timestamp against our recorded timestamp and the file size.
2. Only files that do not match.
3. Files that no longer exist in the cloud storage location are removed from the cache.

This means we always pull the minimum subset of data while ensuring every file is individually checked for changes.

Cache-exempt assets

Some assets change frequently enough that caching is counterproductive. When creating an asset, you can mark it as cache-exempt:

resim assets create \
    --project "my-project" \
    --name "Rapidly Changing Data" \
    --description "Data that updates every run" \
    --locations "s3://my-bucket/volatile-data/" \
    --mount-folder "volatile" \
    --version "1.0.0" \
    --cache-exempt

Cache-exempt assets are always fetched fresh from cloud storage, bypassing the caching mechanism entirely.

Asset data is read-only

Because cached asset data is shared across potentially concurrent running jobs, we must ensure its integrity. The /tmp/resim/assets/<mountFolder> directory is mounted read-only.

If your workload needs to modify asset data at runtime, copy the relevant files to a writable location (such as /tmp/resim/outputs/ or any other writable directory) before modifying them.

How do I invalidate the cache?

The easiest way to ensure a cached copy of a particular asset is pulled fresh is to "touch" (i.e. modify in some way) every file in the remote location. Since we check the lastModified timestamp for every file individually, this ensures it is definitely newer than the cached copy.

Under normal circumstances this should not be necessary as the sync mechanism handles changes automatically. If you have touched the files and are still experiencing unexpected behavior, reach out to us and we'll help.