DimOS on SimCloud.

Your DimOS eval cases, run on GPUs in the cloud, as many at once as you like. The same suites and cases as dimos evals run, the same graders, and a pass rate with an honest interval.

What it does

A DimOS case takes 30 to 60 seconds and wants a machine of its own: one MCP port, one set of local topics. On SimCloud each case is a dimos@1 task on the GPU pool runpod:gpu, in a fresh container. Ten cases run as ten pods, side by side.

DimOS brings the MuJoCo simulator up, your system acts, DimOS grades the recorded world with the case's own grader. The verdict is DimOS's. You get every episode's result, the trajectories, and the pass rate with its Wilson 95% interval: 17 of 20 is 64% to 95%, not 85%.

Install and sign in

The simcloud CLI and Python SDK need Python 3.10 or newer and nothing else. Make a token with the user role in Settings, API tokens; it is shown once.

pip install simcloud
export SIMCLOUD_URL=https://sim.mutational.in
export SIMCLOUD_TOKEN=sc_user_…

Your agent's key

An agent calls a model, so it needs a key. Store it as a workspace secret: it is read from stdin, never from the command line, stored encrypted, and never shown again. A task names the secret, and the case gets it as an environment variable. It is never in the task's spec, result or log. You can also add one in Settings, Secrets.

simcloud secrets set OPENAI_API_KEY < key.txt
simcloud secrets list

Run a case

A scripted plan is deterministic, which makes it the system to measure a case against first. A plan is a list of steps: pick (with lift_m, grasp_dz_m and close_to) and place_on (with release_dz_m).

echo '[{"pick": "cup", "grasp_dz_m": -0.03, "lift_m": 0.15}]' > plan.json
simcloud dimos run dimos.evals.suites.mujoco_xarm --case xarm_pick_cylinder \
    --scripted plan.json --episodes 10 --wait

An agent is named as dimos evals run --agent names it, with its settings and the secret it needs. With --wait the command prints the pass rate and its interval, and downloads results.jsonl, one line per episode. An episode that errored inside DimOS, such as a simulator that would not start, is counted apart and is not a trial.

simcloud dimos run dimos.evals.suites.mujoco_xarm --case xarm_pick_cylinder \
    --agent dimos.evals.agents.pi --set model=gpt-6-astra \
    --secret OPENAI_API_KEY --episodes 10 --wait
# c_…
# xarm_pick_cylinder: 8/10 passed (80%, 95% interval 49% to 94%)

Your own suite

Bundle your code: a suite module, blueprints, an agent. --bundle DIR zips the directory, leaving caches, virtualenvs and version control out, and uploads it once. The same code is the same upload. In the task, the zip's root is first on PYTHONPATH and is the working directory, so my_suite imports as it does at home. A bundle is at most 50 MiB zipped.

DimOS settings such as N_WORKERS or MUJOCOSIMMODULE__* go in --env. Keys never do: a key is a secret.

simcloud dimos run my_suite --case pick_mug --bundle ./my-dimos-code \
    --agent my_agent.planner --set model=gpt-6-astra --secret OPENAI_API_KEY \
    --env N_WORKERS=4 --episodes 20 --wait

From Python

run_case takes the same arguments as the command. Without wait it returns the campaign at once; with it, the episodes and the interval.

from simcloud.dimos import run_case

r = run_case("dimos.evals.suites.mujoco_xarm", "xarm_pick_cylinder",
             agent={"module": "dimos.evals.agents.pi", "settings": {"model": "gpt-6-astra"}},
             secrets=["OPENAI_API_KEY"], episodes=20, wait=True)
print(r["successes"], r["trials"], r["interval"])

Would your cases notice a regression?

A pass rate says how good the robot is today. It cannot say whether your cases would catch it getting worse. Mutational answers that. It runs each case with faults delivered inside DimOS (a lagging arm, less friction, a shifted camera, a misjudged depth) and checks that the case fails when it should.

The sweep runs on your machine, which is cheap. Every simulator run it asks for is a dimos@1 task here, --workers at once. It needs the mutational binary on your PATH and its DimOS adapter, adapters/dimos in the mutational repository.

git clone https://github.com/real2simulator/mutational
export MUTATIONAL_DIMOS_ADAPTER=$PWD/mutational/adapters/dimos
simcloud dimos sweep dimos.evals.suites.mujoco_xarm --case xarm_pick_cylinder \
    --check lifted --timeout 600 --scripted plan.json --workers 8

This writes a mutational project for the case to dimos-sweep/xarm_pick_cylinder/ and runs mutational stale on it. The SimCloud token reaches the adapter in its environment only, never in a file. To run a project of your own, use --project.

simcloud dimos sweep --project mutational.project.json --workers 8

Limits and cost

What	Limit
Episodes in one task	1 to 100, one after another
A task	up to the pool's timeout; a case's own timeout bounds each episode
Bundle	50 MiB zipped
Secrets	32 a workspace, 8 KiB each
Where it runs	`runpod:gpu`, a sandbox pool: a fresh container per task, no platform credentials in it

A task pays for the seconds RunPod billed it, at the pool's rate on the price list. It is quoted before it runs, and the unused part of the hold is released. Questions: simcloud@mutational.in.