AI agent skill · by @evolving-machines-lab

Evolve Evals

Evolve hosted evals: run any model on any agent harness against datasets of Harbor-format tasks, in cloud sandboxes, from the evolve CLI and the TypeScript and Python SDKs. Use when the user wants to: (1) start, watch, or read a hosted eval job (evolve run, evolve job, jobs()), (2) browse the dataset catalog or publish their own dataset (evolve dataset), (3) check task quality (evolve check) or analyze trial traces (evolve analyze), (4) inspect or download trials, artifacts and traces (evolve trial, trials()), (5) register their own agent, mount skills, attach secrets, or choose models and sandbox providers, (6) upload a job run elsewhere (evolve upload), (7) use the dashboard's jobs, datasets, checks, traces and trial viewer pages. For running agents in sandboxes from code (the SDK's run(), Swarm, Pipeline) install the SDK skill: `npx skills add evolving-machines-lab/evolve --skill evolve-agents`.

Archived 2026-10-23 · Source updated 2026-09-19 · License: Apache-2.0. Check the original repository for the latest version.

SKILL.md instructions

# Evolve hosted evals

Hosted evaluation for agents: datasets of Harbor-format tasks, jobs that run any model on any agent harness against them in cloud sandboxes, and the trials, checks and analyses they produce — from the `evolve` CLI and the TypeScript and Python SDKs.

The pages under `references/` are the documentation site's pages, byte for byte, at the site's paths: a site link to `/core-concepts/tasks` is `references/core-concepts/tasks.mdx`, and `evolve skills get evals core-concepts/tasks` prints it. An `import` of `/snippets/<file>` is `references/snippets/<file>`.

## How to use this skill

1. Find the topic in the index below and read that page before writing any command or code: `evolve skills get evals <page>` prints it, the page named by its site path; `evolve skills get evals --full` prints every page.
2. Every CLI verb is documented from its own `--help`; run `evolve <verb> --help` to confirm the flags of the installed version.
3. Every command and every SDK client reads `EVOLVE_API_KEY`; the Installation page says where the key comes from.


## Docs

### Getting started

| Page | What it covers |
| --- | --- |
| [Evolve documentation](references/index.mdx) | Environment-driven agent evaluations: any model, any harness, any sandbox, one API key. Every task versioned and checked, every trajectory analyzed. |
| [Installation](references/getting-started/installation.mdx) | Install the evolve command and the SDKs, and set your API key. |
| [Quick start](references/getting-started/quick-start.mdx) | Start your first job, watch it run, and download a trial. |

### Core concepts

| Page | What it covers |
| --- | --- |
| [Core concepts](references/core-concepts/index.mdx) | How a dataset becomes trials, rewards, analyses and results. |
| [Tasks](references/core-concepts/tasks.mdx) | The task format: one directory with an instruction, an environment, a verifier, and an optional reference solution. |
| [Datasets](references/core-concepts/datasets.mdx) | The catalog, and publishing your own dataset. |
| [Jobs](references/core-concepts/jobs.mdx) | Start a job, watch it, read its results, and derive new jobs from it. |
| [Trials](references/core-concepts/trials.mdx) | One agent's attempt at one task: its record, its trace, and its artifacts. |
| [Trial outputs](references/core-concepts/trial-outputs.mdx) | What a finished trial gives you, file by file, and in which formats. |
| [Agents](references/core-concepts/agents.mdx) | The built-in harnesses, how to configure an arm, and how to register your own agent. |
| [Models](references/core-concepts/models.mdx) | How a model is named on -m, and why one API key covers every provider. |
| [Sandboxes](references/core-concepts/sandboxes.mdx) | Where trials run: e2b, daytona, or modal. |
| [Analyze](references/core-concepts/analyze.mdx) | Judge every trial's trace against a rubric, on the platform. |
| [Check](references/core-concepts/check.mdx) | Check the quality of tasks against a rubric before you spend a job on them. |
| [Skills](references/core-concepts/skills.mdx) | Mount skills into every agent run, from a git source or from your uploads. |
| [Secrets](references/core-concepts/secrets.mdx) | Store env secrets once and attach them to jobs by name. |
| [Upload a job](references/core-concepts/upload.mdx) | Bring a job you ran elsewhere onto the platform as a finished job. |

### Dashboard

| Page | What it covers |
| --- | --- |
| [Datasets](references/dashboard/datasets.mdx) | Browse the catalog, and each dataset's tasks and versions, in the dashboard. |
| [Jobs](references/dashboard/jobs.mdx) | Start a job from the browser, follow it, and read its results. |
| [Trial viewer](references/dashboard/trial-viewer.mdx) | Read one trial's trace event by event, its tool calls, its spend, and download its artifacts. |
| [Checks](references/dashboard/checks.mdx) | Read a job's task quality checks in the dashboard. |
| [Traces](references/dashboard/traces.mdx) | Every recorded run in one list. |

### SDK

| Page | What it covers |
| --- | --- |
| [TypeScript SDK](references/sdk/typescript.mdx) | Start a job, wait for it, read the result, and download it from TypeScript. |
| [Python SDK](references/sdk/python.mdx) | Start a job, wait for it, read the result, and download it from Python. |

## CLI reference

### The evolve command

| Page | What it covers |
| --- | --- |
| [CLI reference](references/cli-reference/index.mdx) | Every verb of the evolve command, from its own help output. |
| [evolve run](references/cli-reference/run.mdx) | Start a job, and optionally follow it to the end. |
| [evolve analyze](references/cli-reference/analyze.mdx) | Analyze a finished job's trial traces against a rubric, on the platform. |
| [evolve check](references/cli-reference/check.mdx) | Check task quality against a rubric, and read checks back. |
| [evolve upload](references/cli-reference/upload.mdx) | Upload a job directory as a finished job. |

### Command groups

| Page | What it covers |
| --- | --- |
| [evolve job](references/cli-reference/job.mdx) | Start, follow, and derive jobs. |
| [evolve trial](references/cli-reference/trial.mdx) | Inspect, download, and act on single trials. |
| [evolve analysis](references/cli-reference/analysis.mdx) | List, inspect, and download trace-analysis runs. |
| [evolve dataset](references/cli-reference/dataset.mdx) | Browse and publish the dataset catalog. |
| [evolve skill](references/cli-reference/skill.mdx) | Upload and manage platform-stored skills. |
| [evolve skills](references/cli-reference/skills.mdx) | The skills the CLI serves to coding agents. |
| [evolve agent](references/cli-reference/agent.mdx) | Register and manage your own agents. |
| [evolve auth](references/cli-reference/auth.mdx) | Identity, API keys, and your organizations. |
| [evolve secrets](references/cli-reference/secrets.mdx) | Store and manage env secrets. Values are write-only; reads return metadata. |

## SDK reference

### Clients

| Page | What it covers |
| --- | --- |
| [Clients](references/sdk-reference/index.mdx) | How the hosted clients are built and configured. |
| [jobs](references/sdk-reference/jobs.mdx) | Start, follow, and derive jobs. |
| [trials](references/sdk-reference/trials.mdx) | Read, download, and act on single trials. |
| [datasets](references/sdk-reference/datasets.mdx) | The catalog: list, read, publish, download, activate, delete. |
| [analyses](references/sdk-reference/analyses.mdx) | Read trace-analysis runs. |
| [checks](references/sdk-reference/checks.mdx) | Start and read task quality checks. |
| [skills](references/sdk-reference/skills.mdx) | Upload and manage platform-stored skills. |
| [agents](references/sdk-reference/agents.mdx) | Register and manage your own agents. |
| [auth and orgs](references/sdk-reference/auth.mdx) | Who you are, and your organizations. |
| [meta](references/sdk-reference/meta.mdx) | The platform's capability document, no key needed. |
| [Types](references/sdk-reference/types.mdx) | Every field of the objects the clients return: name, type, meaning, and when it is null. |
| [Errors](references/sdk-reference/errors.mdx) | The error class every refused request raises, and every code it can carry. |