# JustEvals API reference

The JustEvals API is a JSON API over HTTPS at `https://justevals.com/api/v1`. It uploads suites, runs and outputs, and it returns the human feedback on them. Field names use snake_case. The [OpenAPI document](/openapi.json) describes the same endpoints for code generators and agents. The [Python client](/downloads/justevals.py) wraps all of them.

## Authentication

Send the project API key as a bearer token. A key belongs to one project and can only see that project.

```bash
curl -H "Authorization: Bearer $JUSTEVALS_API_KEY" https://justevals.com/api/v1/project
```

## Errors and limits

Errors return a JSON body with `error` and, for invalid input, `details` with the path of each problem.

```json
{"error": "invalid request", "details": [{"path": "cases.0.key", "message": "Required"}]}
```

| Status | Meaning |
| --- | --- |
| 400 | The request body is not valid. |
| 401 | The API key is missing, wrong or revoked. |
| 404 | The object does not exist in this project. |
| 409 | A conflict, for example a file that does not match its hash. |

Limits per request: 5,000 cases in a suite, 500 outputs, 1,000 comparisons, 5,000 reviews. Files can be up to 200 MB. Allowed file types: JPEG, PNG, WebP, GIF, AVIF, MP4, WebM, JSON, plain text and PDF.

## Files

Files are stored once per project and identified by their SHA-256 hash. An upload has three steps.

1. `POST /api/v1/files` with `sha256`, `size` and `content_type`. If the project already has the file, `upload` is `null`.
2. `PUT` the bytes to `upload.url` with the headers in `upload.headers`. The URL expires after one hour.
3. `POST /api/v1/files/{id}/complete`. JustEvals checks the size and the hash.

```json
{"id": "fil_8k2…", "sha256": "9f86d0…", "status": "pending",
 "upload": {"url": "https://…", "method": "PUT", "headers": {"Content-Type": "image/jpeg"}}}
```

Anywhere a file is referenced, use `{"file_id": "fil_…"}` or `{"sha256": "…"}`, with an optional `role` and `label`. `GET /api/v1/files/{id}` returns a download link.

## Suites

`PUT /api/v1/suites/{slug}` creates or updates a suite. The body holds the full case list.

```json
{
  "name": "Exterior edits",
  "review_config": "score_1_5",
  "cases": [
    {"key": "O01", "title": "Sunny exterior", "group": "Outdoor",
     "instructions": "Brighten, add sunshine", "expected": "Same building, brighter",
     "failure_conditions": "Windows change", "inputs": [{"file_id": "fil_…", "role": "input"}],
     "params": {"mode": "outdoor"}, "tags": ["outdoor"]}
  ]
}
```

The response gives the version that holds these cases: `{"slug": "exterior", "version": 3, "created": true, "cases": 1}`. `GET /api/v1/suites` lists the suites. `GET /api/v1/suites/{slug}[?version=N]` returns the cases with their labels and the effective review settings.

## Runs and outputs

`POST /api/v1/runs` creates a run on a suite version (the latest if you leave out `suite_version`).

```json
{"suite": "exterior", "suite_version": 3, "name": "Shorter prompt", "model": "gpt-image-2.5",
 "batch": "2026-10-14-a", "is_baseline": false, "external_id": "2026-10-14-a:short",
 "config": {"prompt_version": "v13"}}
```

With a known `external_id`, the call updates the run's name, model, batch, baseline flag and configuration instead of creating a second run. A run never moves to another suite version.

`POST /api/v1/runs/{id}/outputs` adds or replaces up to 500 outputs. `{id}` can also be the run's `external_id`.

```json
{"outputs": [
  {"case": "O01", "sample": 0, "status": "ok", "files": [{"file_id": "fil_…"}],
   "cost_usd": 0.09, "latency_ms": 41000, "scores": {"judge_score": 4}, "meta": {"request_id": "…"}},
  {"case": "O02", "status": "error", "error": "Timeout after 300 s"}
]}
```

`POST /api/v1/runs/{id}/complete` with `{"status": "complete"}` or `{"status": "failed"}` closes the run. `GET /api/v1/runs[?suite=…][&batch=…]` lists runs, and `GET /api/v1/runs/{id}` returns a run with its outputs.

## Feedback

`GET /api/v1/feedback?suite={slug}` returns everything that people and judges said about the runs of a suite. Filter with `runs` (IDs or external IDs, comma-separated), `batch` and `since` (ISO date; it compares with `updated_at`).

```json
{
  "suite": {"slug": "exterior", "version": 3,
            "cases": [{"key": "O01", "status": "needs_fixing", "status_note": "The mask covers the door"}]},
  "runs": [{"id": "run_…", "external_id": "2026-10-14-a:short", "name": "Shorter prompt"}],
  "comparisons": [{"case": "O01", "verdict": "winner",
                   "winner": {"run": "run_…", "external_id": "2026-10-14-a:short", "sample": 0},
                   "note": "Keeps the window frames", "reviewer": "Simon", "source": "human",
                   "created_at": "…", "updated_at": "…"}],
  "reviews": [{"case": "O01", "run": "run_…", "sample": 0, "values": {"score": 4},
               "note": null, "reviewer": "Simon", "source": "human"}]
}
```

## Judgments from other tools

`POST /api/v1/judgments` imports judgments made outside JustEvals, or records an automatic judge. An output is identified by its ID or by `{"run": …, "case": …, "sample": …}`.

```json
{"comparisons": [{"outputs": [{"run": "2026-10-14-a:baseline", "case": "O01"},
                              {"run": "2026-10-14-a:short", "case": "O01"}],
                  "verdict": "winner", "winner": {"run": "2026-10-14-a:short", "case": "O01"},
                  "note": "Sharper", "source": "import", "reviewer": "simon",
                  "created_at": "2026-09-12T10:00:00Z"}],
 "reviews": [{"output": {"run": "2026-10-14-a:short", "case": "O01"}, "values": {"score": 4},
              "source": "auto", "reviewer": "gemini-judge"}]}
```

`verdict` is `winner`, `tie` or `all_bad`. Review values must match the suite's review settings. Importing the same judgment again changes nothing. `DELETE /api/v1/judgments` with the same identifiers (outputs, `source`, `reviewer`) removes imported or automatic judgments. The API never writes judgments in the name of a person.

## Endpoint list

| Method | Path | Purpose |
| --- | --- | --- |
| GET | `/api/v1/project` | Check the key and show the project |
| POST | `/api/v1/files` | Declare a file and get an upload URL |
| POST | `/api/v1/files/{id}/complete` | Confirm an upload |
| GET | `/api/v1/files/{id}` | File details and a download link |
| GET | `/api/v1/suites` | List suites |
| PUT | `/api/v1/suites/{slug}` | Create or update a suite |
| GET | `/api/v1/suites/{slug}` | Cases, labels and review settings |
| GET | `/api/v1/runs` | List runs |
| POST | `/api/v1/runs` | Create or update a run |
| GET | `/api/v1/runs/{id}` | A run with its outputs |
| POST | `/api/v1/runs/{id}/outputs` | Add or replace outputs |
| POST | `/api/v1/runs/{id}/complete` | Mark a run complete or failed |
| GET | `/api/v1/feedback` | Labels, verdicts, ratings and notes |
| POST | `/api/v1/judgments` | Import judgments or record a judge |
| DELETE | `/api/v1/judgments` | Remove imported or automatic judgments |