# JustEvals documentation

JustEvals stores the results of your eval runs and the human judgments on them. Your pipeline runs the models and uploads what came out. People compare the runs side by side in the browser. Your pipeline or your coding agent reads the labels and notes back before the next change. JustEvals never calls a model.

## What does the workflow look like?

1. Create an API key for your project.
2. Upload a suite (your test cases) and one run per model or prompt configuration.
3. Review the runs side by side and pick the better output for each case.
4. Read the feedback through the API and fix the prompt, the model choice or the cases.
5. Run the same suite again and compare the new run with the old one.

## Quickstart

### 1. Get an API key and the client

Open your project in JustEvals and go to **Settings → API keys**. A key belongs to one project. The client is one Python file without dependencies (Python 3.9 or newer).

```bash
export JUSTEVALS_API_KEY=je_...
curl -fsSL https://justevals.com/downloads/justevals.py -o justevals.py
python justevals.py check
```

### 2. Write a manifest

A manifest describes one suite and the runs you want to upload. File paths are relative to the manifest. The client uploads each file once and skips files that the project already has.

```json
{
  "suite": {
    "slug": "exterior",
    "name": "Exterior edits",
    "cases": [
      {
        "key": "O01",
        "title": "Sunny exterior",
        "instructions": "Brighten, add sunshine, remove clutter",
        "expected": "Brighter and cleaner, same building",
        "failure_conditions": "Any window, door or roof changes",
        "inputs": [{"path": "cases/O01.jpg"}]
      }
    ]
  },
  "runs": [
    {
      "name": "Current prompt",
      "model": "gpt-image-2.5",
      "batch": "2026-10-14-a",
      "is_baseline": true,
      "external_id": "2026-10-14-a:baseline",
      "outputs": [
        {"case": "O01", "files": [{"path": "out/baseline/O01.jpg"}], "cost_usd": 0.09, "latency_ms": 41000}
      ]
    },
    {
      "name": "Shorter prompt",
      "model": "gpt-image-2.5",
      "batch": "2026-10-14-a",
      "external_id": "2026-10-14-a:short",
      "outputs": [
        {"case": "O01", "files": [{"path": "out/short/O01.jpg"}], "cost_usd": 0.09, "latency_ms": 38000}
      ]
    }
  ]
}
```

### 3. Upload it

```bash
python justevals.py push manifest.json
```

The command prints one link per run. Uploading the same manifest again changes nothing: `external_id` identifies the run, and outputs are identified by case and sample.

### 4. Review

Open a link, select the runs on the **Runs** page and click **Compare**. Pick the better output for each case.

### 5. Read the feedback

```bash
python justevals.py feedback --suite exterior --batch 2026-10-14-a
```

The response lists every vote with its note, every rating, and the label of each case. The [API reference](/docs/api) describes the format.

## Concepts

### Suite

A suite is a saved test set for one task, for example "exterior edits". Each upload sends the full case list. JustEvals keeps every version: if the cases change, the upload creates a new version, and older runs keep the version they ran on.

### Case

A case is one input and a description of a good result. It has a stable `key` (for example `O01`). It can hold several input files with roles, such as `input`, `mask`, `guide` or `reference`, plus parameters and tags. Do not reuse a key for a different input.

### Run

A run is one execution of a suite version with one model and prompt configuration. Runs that you start together share a `batch`. Mark the current production setup with `is_baseline` so the run page offers "Compare with baseline".

### Output

An output is the result for one case in one run. It holds files (the result, intermediate `step` files and `diagnostic` images such as heatmaps), text, cost, latency, an error message if the call failed, and automatic `scores` from your own judges or metrics. Use `sample` for repeated outputs of the same case.

### Judgments

- **Comparison**: which output is best for a case, or a tie, or "all bad", with an optional note.
- **Review**: ratings of one output, in the shape that the suite's review settings define, with an optional note.
- **Case label**: `active`, `needs_fixing` or `irrelevant`, with a note. A label changes the suite, not the run.

People make judgments in the browser. Through the API you can import judgments from an earlier tool (`source: "import"`) or record an automatic judge (`source: "auto"`). JustEvals shows automatic judges separately from people.

## How does the review screen work?

The review screen shows the input and the output of each run for one case. By default the review is blind: the order of the outputs changes per case, and run names, costs and scores stay hidden until you show them. Zoom and pan apply to all images at the same time. Hold Space to flip between the outputs in place, which makes small differences visible.

| Key | Action |
| --- | --- |
| `1`–`9` | Pick output A, B, … as best |
| `T` | Tie |
| `X` | All bad |
| `U` | Clear your vote |
| `N` | Write a note |
| `←` `→` or `J` `K` | Previous or next case |
| `Space` (hold) | Flip between outputs |
| `Z`, `Esc` | Zoom in, reset zoom |
| `H` | Show or hide run names |
| `S` | Show or hide the summary |

The summary counts wins, losses, ties and "all bad" per run, with failed outputs, total cost, median time, average ratings and average automatic scores. When you compare three or more runs, earlier pairwise votes between any two of them also count.

## Which review settings can I use?

Each project has review settings, and each suite can override them. Uploads can set them per suite with `review_config`. After someone changes the settings of a suite in the browser, uploads no longer overwrite them.

| Preset | What reviewers do |
| --- | --- |
| `best` | Pick the best output per case, or call a tie or "all bad". This is the default. |
| `good_bad` | Pick the best, and mark each output good or bad. |
| `score_1_5` | Pick the best, and give each output a score from 1 to 5. |
| `video` | Fidelity and motion from 1 to 5, hard failures as tags, accepted yes or no. |
| `prompt_harness` | Architecture fidelity and change execution from 1 to 5, ship yes or no. |

A custom configuration combines the "pick the best" options with up to 12 rating dimensions of the types `choice`, `scale`, `boolean` and `tags`:

```json
{
  "compare": {"enabled": true, "allowTie": true, "allowAllBad": true},
  "dimensions": [
    {"key": "realism", "label": "Realism", "type": "scale", "min": 1, "max": 5},
    {"key": "failures", "label": "Failures", "type": "tags", "options": ["warped lines", "new objects"]}
  ],
  "notes": true,
  "blind": true
}
```

## How do suite versions work?

- An upload that is identical to the latest version keeps that version.
- An upload that is identical to an older version returns the older version. This lets you upload past runs again without duplicates.
- Any other upload creates a new version. The cases of a version never change after it exists.
- Case labels apply to the latest version. If no run uses the latest version yet, a label changes it in place. Otherwise the label creates a new version.
- A case labeled `needs_fixing` becomes `active` again when its content changes in the next upload.

## How do I connect a coding agent?

Give your agent the [JustEvals skill](/skill). It explains the workflow, the manifest and the API, and it tells the agent to read the feedback before it changes a prompt or a model. For tools that do not read skills, the [API reference](/docs/api) and the [OpenAPI document](/openapi.json) describe the same API. Each documentation page is also available as Markdown: add `.md` to the address, for example [/docs.md](/docs.md).

## Where does JustEvals store my data?

The application and the database run on a server of Hetzner Online GmbH in Nuremberg, Germany. Files are stored in Cloudflare R2 with the EU jurisdiction setting, so they stay in the EU. Cloudflare also routes the traffic to the site. Files are private: the browser loads them through links that expire after one hour. JustEvals does not train models on your data, does not sell it and does not put it into analytics. The [privacy policy](/privacy) has the details. To delete a project or your account, write to [contact@justevals.com](mailto:contact@justevals.com).