Workflows

Introduction

ReSim workflows provide a powerful way to organize and manage collections of test suites for CI/CD pipelines. Instead of managing individual test suites separately, workflows allow you to group related test suites together and run them as a single unit.

Why Use Workflows?

ReSim is frequently used with CI/CD platforms like GitHub Actions and GitLab CI to trigger test suites on pull requests, releases, and scheduled runs (e.g., nightly tests). While you can trigger individual test suites directly, this approach has limitations:

Configuration Management: Adding new test suites or temporarily disabling tests requires updating CI workflow files and pushing changes
Complexity: Managing multiple test suite triggers across different CI events becomes unwieldy
Flexibility: Making runtime changes to test configurations requires code changes

How Workflows Help

Workflows solve these problems by providing:

Named Collections: Group related test suites (e.g., "nightly", "regression", "smoke tests") under meaningful names
Runtime Configuration: Enable/disable test suites or add new ones through the ReSim UI without code changes
Simplified CI: Trigger entire test collections with a single command
Flexible Management: Update test suite configurations independently of your CI pipeline

Example Use Case

Consider a nightly CI job that runs progression, regression, and smoke tests. Instead of managing three separate test suite triggers, you can:

Create a "Nightly Tests" workflow containing all three test suites
Configure your CI to run: resim workflows runs create --workflow "Nightly Tests"
Later, add new test suites or disable progression tests entirely through the ReSim UI
Your CI pipeline remains unchanged while test configurations evolve

Workflow Management

ReSim provides comprehensive CLI commands for managing workflows. You can create, update, list, and retrieve workflows, as well as manage workflow runs.

Creating Workflows

Create a new workflow with the create command:

resim workflows create \
  --project "my-project" \
  --name "My Nightly Workflow" \
  --description "Nightly regression and smoke tests" \
  --ci-link "https://github.com/myorg/myrepo/actions/workflows/nightly.yml" \
  --suites '[{"testSuite": "suite-uuid-1", "enabled": true}, {"testSuite": "suite-uuid-2", "enabled": false}]'

Required Parameters: - --project: The name or ID of the project to create the workflow in - --name: The name of the workflow - --description: A description of the workflow - --suites OR --suites-file: JSON array of test suites (exactly one required)

Optional Parameters: - --ci-link: A link to the CI workflow (e.g., GitHub Actions URL)

Test Suite Configuration:

You can specify test suites in two ways:

Inline JSON (using --suites):

--suites '[{"testSuite": "suite-uuid-1", "enabled": true}, {"testSuite": "My Test Suite", "enabled": false}]'

Note: You can use either test suite UUIDs or names for the testSuite field. Names are often more readable and maintainable.

JSON File (using --suites-file):
```
--suites-file ./workflow-suites.json
```

The JSON format supports both UUIDs and test suite names for the testSuite field. Test suite names are unique identifiers, so you can use either the UUID or the human-readable name.

Listing Workflows

List all workflows in a project:

resim workflows list --project "my-project"

This command returns a summary of each workflow including: - Workflow ID and name - Description - CI workflow link (if set) - Associated test suites with their enabled/disabled status

Getting Workflow Details

Retrieve detailed information about a specific workflow:

resim workflows get --project "my-project" --workflow "My Nightly Workflow"

You can specify the workflow by either name or UUID.

Updating Workflows

Update an existing workflow:

resim workflows update \
  --project "my-project" \
  --workflow "My Nightly Workflow" \
  --name "Updated Nightly Workflow" \
  --description "Updated description" \
  --ci-link "https://new-ci-link.com"

Optional Parameters: - --name: New name for the workflow - --description: New description - --ci-link: New CI workflow link - --suites OR --suites-file: Replace all test suites (see create command for format)

Note: You must provide at least one update parameter. The --suites and --suites-file flags are mutually exclusive.

Workflow Runs

Once you have a workflow defined, you can execute it to run all enabled test suites.

Creating Workflow Runs

Execute a workflow:

resim workflows runs create \
  --project "my-project" \
  --workflow "My Nightly Workflow" \
  --build-id "build-uuid-here" \
  --parameter "env=production" \
  --parameter "debug=false" \
  --pool-labels "gpu" \
  --pool-labels "high-memory" \
  --account "ci-user" \
  --allowable-failure-percent 5

Required Parameters: - --project: The name or ID of the project - --workflow: The name or ID of the workflow to run - --build-id: The ID of the build to use in this workflow run

Optional Parameters: - --parameter: Parameter overrides (format: name=value or name:value) - --pool-labels: Pool labels to determine execution environment (logical AND) - --account: Username for CI/CD platform account association - --allowable-failure-percent: Maximum percentage of test failures allowed (0-100, default: 0)

Parameter Format: Parameters can be specified multiple times or comma-separated:

--parameter "env=production,debug=false"
# or
--parameter "env=production" --parameter "debug=false"

Listing Workflow Runs

List all runs for a specific workflow:

resim workflows runs list --project "my-project" --workflow "My Nightly Workflow"

Getting Workflow Run Details

Retrieve details about a specific workflow run:

resim workflows runs get \
  --project "my-project" \
  --workflow "My Nightly Workflow" \
  --run-id "run-uuid-here"

This command returns information about the workflow run and lists all associated test suite runs.

Examples

Example 1: Creating a Nightly Test Workflow

# Create a workflow for nightly testing
resim workflows create \
  --project "my-robot-project" \
  --name "Nightly Tests" \
  --description "Comprehensive nightly test suite including regression and smoke tests" \
  --ci-link "https://github.com/myorg/robot-repo/actions/workflows/nightly.yml" \
  --suites-file ./nightly-suites.json

Where nightly-suites.json contains:

[
  {"testSuite": "regression-tests", "enabled": true},
  {"testSuite": "smoke-tests", "enabled": true},
  {"testSuite": "performance-tests", "enabled": false}
]

Note: This example uses test suite names instead of UUIDs, which are often more readable and easier to maintain.

Example 2: Running a Workflow in CI

# In your CI pipeline
resim workflows runs create \
  --project "my-robot-project" \
  --workflow "Nightly Tests" \
  --build-id "$BUILD_ID" \
  --parameter "test_environment=staging" \
  --pool-labels "gpu" \
  --allowable-failure-percent 10

Example 3: Updating Workflow Configuration

# Disable performance tests and update description
resim workflows update \
  --project "my-robot-project" \
  --workflow "Nightly Tests" \
  --description "Nightly regression and smoke tests (performance tests moved to weekly)" \
  --suites '[{"testSuite": "regression-tests", "enabled": true}, {"testSuite": "smoke-tests", "enabled": true}]'

Best Practices

Workflow Organization

Use Descriptive Names: Choose workflow names that clearly indicate their purpose (e.g., "Nightly Regression", "PR Smoke Tests", "Release Validation")
Group Related Tests: Organize test suites logically by functionality, environment, or execution frequency
Document with Descriptions: Provide clear descriptions explaining what each workflow tests and when it should be used

CI/CD Integration

Link CI Workflows: Use the --ci-link parameter to connect ReSim workflows with your CI pipeline for easy navigation
Use Environment Variables: Leverage CI environment variables for dynamic parameters like build IDs and test environments
Set Appropriate Failure Thresholds: Use --allowable-failure-percent to handle expected test failures in non-critical workflows

Test Suite Management

Start with Core Tests: Begin with essential test suites enabled, then add optional tests as needed
Use Test Suite Names: Prefer test suite names over UUIDs for better readability and maintainability
Use JSON Files: For complex configurations, use --suites-file instead of inline JSON for better maintainability
Regular Review: Periodically review and update workflow configurations to ensure they remain relevant

Parameter Management

Consistent Naming: Use consistent parameter naming conventions across your workflows
Environment-Specific Values: Use parameters to customize test behavior for different environments
Document Parameters: Keep track of what parameters each workflow expects and their valid values

Monitoring and Maintenance

Track Workflow Runs: Regularly review workflow run results to identify patterns and issues
Update Descriptions: Keep workflow descriptions current as test suites evolve
Archive Unused Workflows: Remove or archive workflows that are no longer needed to keep your project organized