API reference for AI eval uploads

The JustEvals API is a JSON API over HTTPS at https://justevals.com/api/v1. It uploads suites, runs and outputs, and it returns the human feedback on them. Field names use snake_case. The OpenAPI document describes the same endpoints for code generators and agents. The Python client wraps all of them.

Authentication

Send the project API key as a bearer token. A key belongs to one project and can only see that project.

curl -H "Authorization: Bearer $JUSTEVALS_API_KEY" https://justevals.com/api/v1/project

Errors and limits

Errors return a JSON body with error and, for invalid input, details with the path of each problem.

{"error": "invalid request", "details": [{"path": "cases.0.key", "message": "Required"}]}

Status	Meaning
400	The request body is not valid.
401	The API key is missing, wrong or revoked.
404	The object does not exist in this project.
409	A conflict, for example a file that does not match its hash.

Limits per request: 5,000 cases in a suite, 500 outputs, 1,000 comparisons, 5,000 reviews. Files can be up to 200 MB. Allowed file types: JPEG, PNG, WebP, GIF, AVIF, MP4, WebM, JSON, plain text and PDF.

Files

Files are stored once per project and identified by their SHA-256 hash. An upload has three steps.

POST /api/v1/files with sha256, size and content_type. If the project already has the file, upload is null.
PUT the bytes to upload.url with the headers in upload.headers. The URL expires after one hour.
POST /api/v1/files/{id}/complete. JustEvals checks the size and the hash.

{"id": "fil_8k2…", "sha256": "9f86d0…", "status": "pending",
 "upload": {"url": "https://…", "method": "PUT", "headers": {"Content-Type": "image/jpeg"}}}

Anywhere a file is referenced, use {"file_id": "fil_…"} or {"sha256": "…"}, with an optional role and label. GET /api/v1/files/{id} returns a download link.

Suites

PUT /api/v1/suites/{slug} creates or updates a suite. The body holds the full case list.

{
  "name": "Exterior edits",
  "review_config": "score_1_5",
  "cases": [
    {"key": "O01", "title": "Sunny exterior", "group": "Outdoor",
     "instructions": "Brighten, add sunshine", "expected": "Same building, brighter",
     "failure_conditions": "Windows change", "inputs": [{"file_id": "fil_…", "role": "input"}],
     "params": {"mode": "outdoor"}, "tags": ["outdoor"]}
  ]
}

The response gives the version that holds these cases: {"slug": "exterior", "version": 3, "created": true, "cases": 1}. GET /api/v1/suites lists the suites. GET /api/v1/suites/{slug}[?version=N] returns the cases with their labels and the effective review settings.

Runs and outputs

POST /api/v1/runs creates a run on a suite version (the latest if you leave out suite_version).

{"suite": "exterior", "suite_version": 3, "name": "Shorter prompt", "model": "gpt-image-2.5",
 "batch": "2026-10-14-a", "is_baseline": false, "external_id": "2026-10-14-a:short",
 "config": {"prompt_version": "v13"}}

With a known external_id, the call updates the run's name, model, batch, baseline flag and configuration instead of creating a second run. A run never moves to another suite version.

POST /api/v1/runs/{id}/outputs adds or replaces up to 500 outputs. {id} can also be the run's external_id.

{"outputs": [
  {"case": "O01", "sample": 0, "status": "ok", "files": [{"file_id": "fil_…"}],
   "cost_usd": 0.09, "latency_ms": 41000, "scores": {"judge_score": 4}, "meta": {"request_id": "…"}},
  {"case": "O02", "status": "error", "error": "Timeout after 300 s"}
]}

POST /api/v1/runs/{id}/complete with {"status": "complete"} or {"status": "failed"} closes the run. GET /api/v1/runs[?suite=…][&batch=…] lists runs, and GET /api/v1/runs/{id} returns a run with its outputs.

Feedback

GET /api/v1/feedback?suite={slug} returns everything that people and judges said about the runs of a suite. Filter with runs (IDs or external IDs, comma-separated), batch and since (ISO date; it compares with updated_at).

{
  "suite": {"slug": "exterior", "version": 3,
            "cases": [{"key": "O01", "status": "needs_fixing", "status_note": "The mask covers the door"}]},
  "runs": [{"id": "run_…", "external_id": "2026-10-14-a:short", "name": "Shorter prompt"}],
  "comparisons": [{"case": "O01", "verdict": "winner",
                   "winner": {"run": "run_…", "external_id": "2026-10-14-a:short", "sample": 0},
                   "note": "Keeps the window frames", "reviewer": "Simon", "source": "human",
                   "created_at": "…", "updated_at": "…"}],
  "reviews": [{"case": "O01", "run": "run_…", "sample": 0, "values": {"score": 4},
               "note": null, "reviewer": "Simon", "source": "human"}]
}

Judgments from other tools

POST /api/v1/judgments imports judgments made outside JustEvals, or records an automatic judge. An output is identified by its ID or by {"run": …, "case": …, "sample": …}.

{"comparisons": [{"outputs": [{"run": "2026-10-14-a:baseline", "case": "O01"},
                              {"run": "2026-10-14-a:short", "case": "O01"}],
                  "verdict": "winner", "winner": {"run": "2026-10-14-a:short", "case": "O01"},
                  "note": "Sharper", "source": "import", "reviewer": "simon",
                  "created_at": "2026-09-12T10:00:00Z"}],
 "reviews": [{"output": {"run": "2026-10-14-a:short", "case": "O01"}, "values": {"score": 4},
              "source": "auto", "reviewer": "gemini-judge"}]}

verdict is winner, tie or all_bad. Review values must match the suite's review settings. Importing the same judgment again changes nothing. DELETE /api/v1/judgments with the same identifiers (outputs, source, reviewer) removes imported or automatic judgments. The API never writes judgments in the name of a person.

Endpoint list

Method	Path	Purpose
GET	`/api/v1/project`	Check the key and show the project
POST	`/api/v1/files`	Declare a file and get an upload URL
POST	`/api/v1/files/{id}/complete`	Confirm an upload
GET	`/api/v1/files/{id}`	File details and a download link
GET	`/api/v1/suites`	List suites
PUT	`/api/v1/suites/{slug}`	Create or update a suite
GET	`/api/v1/suites/{slug}`	Cases, labels and review settings
GET	`/api/v1/runs`	List runs
POST	`/api/v1/runs`	Create or update a run
GET	`/api/v1/runs/{id}`	A run with its outputs
POST	`/api/v1/runs/{id}/outputs`	Add or replace outputs
POST	`/api/v1/runs/{id}/complete`	Mark a run complete or failed
GET	`/api/v1/feedback`	Labels, verdicts, ratings and notes
POST	`/api/v1/judgments`	Import judgments or record a judge
DELETE	`/api/v1/judgments`	Remove imported or automatic judgments

JustEvals API reference