---
name: justevals
description: Upload eval suites, runs and outputs to JustEvals (justevals.com) and read human feedback back before the next run. Use when running AI/image evals for this project, when asked to push eval results to JustEvals, to fetch eval feedback or labels, or to decide whether a model or prompt change is better.
---

# JustEvals

JustEvals stores eval results and the human judgments made on them. Your pipeline
runs the models (JustEvals never calls a model); you upload what came out; people
compare runs side by side in the browser; you read their labels and notes back
before the next change.

## Concepts

- **Suite**: a saved test set for one task (`core-v2`). Uploading a changed case list creates a new **version**; runs keep the version they ran on.
- **Case**: one input plus what good looks like. Its `key` (`O01`, `bath-1`) is its identity across versions. Never reuse a key for a different input.
- **Run**: one execution of a suite version with one model + prompt configuration. Baseline and candidates are separate runs that share a `batch`.
- **Output**: a case's result in a run (`sample` 0, 1, 2… for repeats), with files, cost, latency, error, and optional automatic `scores`.
- **Judgments** (made by people in the UI): which output is best per case (or tie / all bad) with a note, per-output ratings shaped by the suite's review settings, and case labels: `active`, `irrelevant`, `needs_fixing` (+ note).

## Setup

```bash
export JUSTEVALS_API_KEY=je_...             # project key: Settings → API keys
export JUSTEVALS_URL=https://justevals.com  # default
curl -fsSL https://justevals.com/downloads/justevals.py -o justevals.py   # stdlib-only client
python justevals.py check
```

## Workflow

1. **Before changing a prompt or model**, read the feedback on the latest runs:
   `python justevals.py feedback --suite core-v2 [--batch evr_x3m54f] [--since 2026-10-01T00:00:00Z]`
   - `suite.cases[].status == "needs_fixing"`: fix the case (input, expectation) before trusting results on it. Read `status_note`.
   - `status == "irrelevant"`: drop the case from the next upload.
   - `comparisons[]`: which run won per case and why (`note`). `reviews[]`: per-output ratings and notes.
2. **Run your existing eval pipeline** as usual.
3. **Upload** a manifest (see below): `python justevals.py push manifest.json`. It prints a link per run.
4. **Ask a human to review** the link (Compare view). Do not judge outputs yourself through the API unless asked; automatic judges must use `"source": "auto"`.
5. **Re-run the same suite version** after a fix and compare again.

## Manifest

```json
{
  "suite": {
    "slug": "core-v2",
    "name": "Core v2",
    "review_config": "score_1_5",
    "cases": [
      {
        "key": "O01",
        "title": "Exterior, sunny",
        "group": "Outdoor",
        "instructions": "optimize, sunshine, declutter",
        "expected": "Brighter, sunnier, cleaner",
        "failure_conditions": "Any window, door or roof changes",
        "inputs": [{"path": "cases/O01.jpg", "role": "input"}, {"path": "cases/O01-mask.png", "role": "mask"}],
        "params": {"mode": "outdoor"},
        "tags": ["outdoor"]
      }
    ]
  },
  "runs": [
    {
      "name": "Sunburst (production)",
      "model": "fal-ai/gpt-image-2.5",
      "batch": "evr_x3m54f",
      "is_baseline": true,
      "external_id": "evr_x3m54f:baseline",
      "config": {"prompt_version": "v12", "code_revision": "abc123"},
      "outputs": [
        {"case": "O01", "sample": 0, "files": [{"path": "out/O01-0.jpg"}],
         "cost_usd": 0.09, "latency_ms": 41000, "scores": {"judge_score": 4}},
        {"case": "O02", "sample": 0, "status": "error", "error": "FAL timeout after 300s"}
      ]
    }
  ]
}
```

- File refs: `{"path": ...}` (relative to the manifest), `{"url": ...}` (downloaded and re-uploaded, so expiring provider URLs are safe; the client remembers uploaded URLs in `~/.cache/justevals`), or `{"file_id": ...}`. Optional `role` (`input`, `mask`, `reference`, `output`, `step`, `diagnostic`) and `label`.
- The suite's `cases` is the **full** list. An upload identical to the latest version keeps it; one identical to an older version returns that older version (so re-uploading past runs creates no duplicates); anything else creates a new version. A version's cases never change once it exists, so a run always gets exactly the cases it was uploaded with.
- `external_id` makes re-uploads idempotent: the same run is updated (name, model, batch, baseline flag, config), not duplicated. Outputs are keyed by case + sample. A run never moves to another suite version.
- `review_config`: a preset (`best`, `good_bad`, `score_1_5`, `video`, `prompt_harness`) or a full config. Default: pick the best. Once someone changes a suite's review settings in the UI, uploads no longer overwrite them.
- `scores` are automatic only (judges, pixel metrics). Human labels never go there.

## Importing past human labels

Add `"judgments"` to the manifest (or `POST /api/v1/judgments`):

```json
{"comparisons": [{"outputs": [{"run": "evr_x3m54f:baseline", "case": "O01"}, {"run": "evr_x3m54f:nb2", "case": "O01"}],
                  "verdict": "winner", "winner": {"run": "evr_x3m54f:nb2", "case": "O01"},
                  "note": "keeps the window frames", "source": "import", "reviewer": "simon",
                  "created_at": "2026-09-12T10:00:00Z"}],
 "reviews": [{"output": {"run": "evr_x3m54f:nb2", "case": "O01"}, "values": {"score": 4},
              "source": "import", "reviewer": "simon"}]}
```

`verdict` is `winner`, `tie` or `all_bad`. Review `values` must match the suite's review settings. Re-importing an
unchanged judgment leaves it untouched (its `updated_at` stays). To remove imported judgments (e.g. a vote undone at
the source), `DELETE /api/v1/judgments` with `{"comparisons": [{"outputs": [...], "source": "import", "reviewer": "simon"}], "reviews": [{"output": ..., "source": "import", "reviewer": "simon"}]}`.

## API (Bearer key, JSON, snake_case)

| Method | Path | Purpose |
| --- | --- | --- |
| GET | `/api/v1/project` | Check the key |
| POST | `/api/v1/files` | Declare a file by `sha256`, `size`, `content_type`; returns `upload` (PUT URL) unless already stored |
| POST | `/api/v1/files/{id}/complete` | Confirm an upload |
| PUT | `/api/v1/suites/{slug}` | Create/update a suite from its full case list |
| GET | `/api/v1/suites/{slug}[?version=N]` | Cases with labels |
| GET | `/api/v1/runs[?suite=…][&batch=…]` | List runs (newest first) |
| POST | `/api/v1/runs` | Create or update a run (`suite`, `name`, `model`, `batch`, `is_baseline`, `config`, `external_id`) |
| POST | `/api/v1/runs/{id}/outputs` | Add/replace up to 500 outputs |
| POST | `/api/v1/runs/{id}/complete` | Mark complete or failed |
| GET | `/api/v1/runs/{id}` | Run (with `suite`, `suite_version`) and its outputs including `meta` |
| GET | `/api/v1/feedback?suite=…[&runs=…][&batch=…][&since=…]` | Labels and notes (`since` filters on `updated_at`) |
| POST | `/api/v1/judgments` | Import past labels / automatic judges |
| DELETE | `/api/v1/judgments` | Remove imported / automatic judgments |

`{id}` for runs also accepts the run's `external_id`.