Skip to content

Changelog

July 16th 2024

Refactor for flow

  • Refactors structure for better flow
  • Moves all definitions into one place to reduce duplication and ambiguity
  • Simplifies language in the setup guides to concentrate on pertinent info
  • Some tweaks to CSS for readability in dark mode
  • Some nav tweaks (next/prev on footer; ToC)
  • Add in some missing guides (docker hub)
  • Add in some missing definitions from Core Concepts
  • Add placeholder for "Getting Started" to push people to the main setup guide (ultimate intention is to run people through a fast overview in a sandbox: https://app.asana.com/0/1207526954823149/1207526954823177/f)

April 10 2024

Introducing Systems

Autonomy applications are best considered as systems of systems and often have many different ways to launch and run.

We have received a lot of customer feedback expressing a way to categorize builds based on what subsystem or testing style they may be e.g. perception log replay, localization closed loop, full-stack.

The result is the System, which acts as a container for builds: each build belongs to a given system and systems can define the resource requirements needed to run that particular build.

In addition, users are encouraged to explicitly define experiences and metrics builds as compatible with a given system, which enables the ReSim platform to validate test batches before they run.

For full details of how systems work, please see the ReSim docs.

ReSim CLI v0.3.0

Improvements 🚀

  • The ReSim CLI now has full support for creating and managing systems.

March 25 2024

ReSim CLI v0.2.1

Improvements 🚀

  • Added a GovCloud mode toggle. If you use our GovCloud deployment, you can run resim govcloud enable or set RESIM_GOVCLOUD=true to configure the CLI to point to GovCloud without having to set --url explicitly.

March 21 2024

Global Projects

For a long, long time, we have wanted to evolve the concept of Project within the ReSim app to be a flexible container for all related concepts within the ReSim ecosystem. This would enable data within the ReSim platform to be isolated across projects, enabling different teams or embodied AI applications to be developed within a dedicated space, without too much confusion for users.

This change unfortunately introduces some backwards incompatibilities for existing users of the ReSim app, since a project name or identifier is now required when referencing experiences and metrics builds as well as builds. In particular:

  • Any invocations of commands within the CLI requires a project to be selected, either explicitly or implicitly.
  • Any client code generated and built on from the ReSim Open API specification (hosted here) will need to be regenerated and revised to include the project id in requests.
  • Any github CI workflows using the CLI or action will need to be revised to include the project name or id
  • Any metrics scripts that compute batch metrics require updating to v0.0.10 of the ReSim open source Metrics SDK and require the use of the project ID. This has a further consequence of invalidating the batch metrics stage of any existing metrics build, so a new metrics build needs to be registered and used in automation.

We apologise for the inconvenience caused by this backwards incompatible change -- it is not something we undertook lightly due to the potential disruption to customers. We are, however, extremely excited for the features that we can now build on top of this foundation.

ReSim CLI v0.2.0

Improvements 🚀

  • Global Projects is now supported within the CLI. In particular, resim projects select makes for a clean and ergonomic experience when working with the CLI in one project.

ReSim Github Action v1.1.0

  • Global Projects is now supported by the Github action, which automates the creation of a build and batch for users.

March 14 2024

ReSim CLI v0.1.31

Improvements 🚀

  • Allowed the ability to persist project selection with select project foo. If set via this method, --project will no longer be a required flag for other commands.
  • Added batch wait command that will continuously poll (default: 30s) until a batch reaches a final state (success, failure, cancelled, etc)
  • Added batch logs command to list all logs associated with a batch.

February 08 2024

ReSim CLI v0.1.30

Improvements 🚀

  • Introduces the ability to specify build parameters when creating batches. For example: batch create ... --parameter "param1:foo","param2:bar". This will pass a parameters.json file upon test execution, just as with sweeps.
  • The experience create command now returns a list of files found in the storage location to help with validation.

December 21 2023

ReSim CLI v0.1.29

Improvements 🚀

  • The CLI validates image URIs when builds and metrics builds are created.

December 18 2023

Regional Replication of, and IAM Role Access to, Output Data

We now replicate job output data to the west coast of the US, and the UI can generate download links for these files served from there rather than the default region. This can make a significant difference to the download speed of large output files, so if you'd like to use this feature please get in touch.

We can now also grant customer AWS IAM roles access to that customer's output data (as opposed to using the links from the UI). If this would be helpful for your workflow, get in touch and we can work with you to set it up.

Configuration of these and other features through the UI is on our roadmap.

November 29 2023

Introducing Build Parameter Sweeps

We are pleased to announce that the ReSim platform now supports executing parameter sweeps, that afford the ability to run and compare different parameterizations of an autonomy application to optimize the values used in the field. See the docs for more details.

Introducing Batch Metrics

The ReSim platform becomes more flexible and powerful with the introducion of customizable test batch metrics. Batch metrics enable the custom and flexible aggregation of metrics from each individual test (against a single experience), so that the overall performance of your autonomy application across many experiences can be visualized in one place. See the docs for more details.

October 31 2023

Introducing ReSim Open Core

We're excited to announce that we have made our first official release of our open source core repository. This release contains a number of Python wheels which are convenient for converting ROS2 logs to ReSim MCAP format and computing metrics on such logs. These are useful in crafting metrics build images for use with the ReSim app.

ReSim App v0.0.39

Improvements 🚀

Improving stability: We are continuing to make progress on improving our app stability and error handling

Fixes 🔧

  • Display worker timeouts as errors: Jobs that time out are now correctly marked with an error.

October 26 2023

ReSim CLI v0.1.26

Improvements 🚀

  • Interactive Authentication: You can now log in to ReSim using the ReSim.ai CLI without having to provide a client ID and secret.

October 24 2023

ReSim App v0.0.38

Improvements 🚀

  • Timestamps: Both the builds and metric builds metadata now include creation timestamps.
  • Job runtime performance: Updated experience fetching on the back end has improved runtime performance of our jobs!
  • Improving stability: We have made some improvements to our application type handling, which leads towards a more type-safe application and less unexpected client-side errors.

Experimental 🧪

Experimental features and pages are released to customers when we want to release a feature that is not complete but can still provide value. We love feedback about experimental features as you use them so we can deliver high value functionality when a new feature graduates out of experimental!

  • Addition of metrics builds: You are now able to access the metadata associated with metric builds from the builds page. This page is still experimental, so if you have any feedback, please let us know!

October 20 2023

ReSim GitHub Action v1.0.1

  • We've just released our GitHub action! We hope this will make it much easier to work with ReSim from GitHub Actions. Head over to github.com/resim-ai/action or the GitHub page of our CI documentation to get started.

October 10 2023

ReSim App v0.0.36

Improvements 🚀

  • States Over Time Chart: We added a new chart that can plot different robot states over a time horizon.
  • Batch-Level Metrics: Batch level metrics will now be available in the results page of the app. You can find them, if any are computed, in a tab in the top panel of the app!
  • New Logos! Check out our cool new logo!

September 29 2023

ReSim CLI v0.1.23

Improvements 🚀

  • The experiences command now has:

    • tag and untag subcommands for tagging and untagging experiences
    • A list subcommand for listing experiences

  • There is a new experience-tags command with the following subcommands:

    • create
    • list-experiences for listing the experiences with a given tag

  • The batch create subcommand now supports specifying experiences and experience tags by name or ID, using the new --experiences and --experience-tags flags. (The existing flags are still supported.)

September 27 2023

ReSim App v0.0.34

Improvements 🚀

  • Metric Chart Improvements: We've improved the sizing and spacing of axes in multiple charts to enable a better viewing experience. In addition, the ordering of metrics cards on the page should now be consistent every time you load the page.
  • Insight page load performance: We improved the loading time for metrics on the insights page.
  • New Chart: Added support for the line plot chart.

September 26 2023

ReSim App v0.0.33

Fixes 🔧

  • Fix Crash in Test Browser: There was a bug in our new insights page that resulted in a crash when browsing through individual experiences. A fix has been landed to solve the crash error.

September 25 2023

ReSim App v0.0.32

Improvements 🚀

  • Insights Page: We're excited to introduce a new page that allows you to visualize metrics from your builds! We now support visualizing the following metrics: Scalar, Double Summary, Histogram, and Bar Chart.

Experimental 🧪

Experimental features and pages are released to customers when we want to release a feature that is not complete but can still provide value. We love feedback about experimental features as you use them so we can deliver high value functionality when a new feature graduates out of experimental!

  • Builds page: You can now access metadata associated with builds directly from the builds page. This additional information will help you better understand the context and details of each build.

ReSim CLI v0.1.22

Improvements 🚀

  • Metrics: Metrics builds can now be created and listed with metrics-build create/list (ordered by recency)
  • Batch create with metrics: The batch create subcommand now has an optional --metrics-build-id flag, used to specify a metrics build to run as part of the batch.

Fixes 🔧

  • Unified branch name and ID: Commands that accept a --branch flag can now be passed either a branch name or branch ID

September 12 2023

ReSim App v0.0.28

Improvements 🚀

  • Build select: The build select menu on the experiences page has been redesigned to organize builds by branch. It also displays the full build name and relevant metadata on hover-over.
  • Batch Details: The batch details section on the results page now includes the name of the build that was used in that test batch.

ReSim CLI v0.1.20

Improvements 🚀

  • Github flag: The batch create subcommand now has an optional --github flag. Passing this flag causes the batch ID to be output in a form suitable for use in scripts and pipelines, e.g. in GitHub Actions.

ReSim CI workflows

  • Notifications on PRs/MRs: The Github and Gitlab CI workflow configurations in our documentation have been updated. Now, whenever these CI workflows are run, a comment will be posted on the user's Pull Request (PR) or Merge Request (MR). The comment includes a link to the relevant results page for that test run on the ReSim app.

September 1 2023

ReSim App v0.0.27

Improvements 🚀

  • Select all: All displayed experiences can now be selected with a single click.
  • See your tags: Tags are now displayed on the experience cards.
  • Icons: New navigation icons.

Fixes 🔧

  • Time stamps: Results logs download displays the correct time stamp format.
  • Log Paging: Log lists of more than 10 files are now paged
  • Project ID:s The project ID shown in the app has been replaced with the project name.
  • Build ID: New formatted build image names replace the UUIDs.
  • Copyable experience IDs: These can now be easily selected and copied.

ReSim CLI v0.1.20

Improvements 🚀

  • New list commands: Use the --list flag to list available branches, builds etc.
  • Required flags: Help pages for commands disambiguate required and optional flags.
  • Styled help: Help pages for commands has clearer styling of sections.

Fixes 🔧

  • Project IDs: The new --project flag will accept either a project name or id. So no more worrying about finding the id when you have the name or vice versa.