Skip to content
JustEvals
DocsSign in

JustEvals documentation

Updated View as Markdown

JustEvals stores the results of your eval runs and the human judgments on them. Your pipeline runs the models and uploads what came out. People compare the runs side by side in the browser. Your pipeline or your coding agent reads the labels and notes back before the next change. JustEvals never calls a model.

What does the workflow look like?

  1. Create an API key for your project.
  2. Upload a suite (your test cases) and one run per model or prompt configuration.
  3. Review the runs side by side and pick the better output for each case.
  4. Read the feedback through the API and fix the prompt, the model choice or the cases.
  5. Run the same suite again and compare the new run with the old one.

Quickstart

1. Get an API key and the client

Open your project in JustEvals and go to Settings → API keys. A key belongs to one project. The client is one Python file without dependencies (Python 3.9 or newer).

export JUSTEVALS_API_KEY=je_...
curl -fsSL https://justevals.com/downloads/justevals.py -o justevals.py
python justevals.py check

2. Write a manifest

A manifest describes one suite and the runs you want to upload. File paths are relative to the manifest. The client uploads each file once and skips files that the project already has.

{
  "suite": {
    "slug": "exterior",
    "name": "Exterior edits",
    "cases": [
      {
        "key": "O01",
        "title": "Sunny exterior",
        "instructions": "Brighten, add sunshine, remove clutter",
        "expected": "Brighter and cleaner, same building",
        "failure_conditions": "Any window, door or roof changes",
        "inputs": [{"path": "cases/O01.jpg"}]
      }
    ]
  },
  "runs": [
    {
      "name": "Current prompt",
      "model": "gpt-image-2.5",
      "batch": "2026-10-14-a",
      "is_baseline": true,
      "external_id": "2026-10-14-a:baseline",
      "outputs": [
        {"case": "O01", "files": [{"path": "out/baseline/O01.jpg"}], "cost_usd": 0.09, "latency_ms": 41000}
      ]
    },
    {
      "name": "Shorter prompt",
      "model": "gpt-image-2.5",
      "batch": "2026-10-14-a",
      "external_id": "2026-10-14-a:short",
      "outputs": [
        {"case": "O01", "files": [{"path": "out/short/O01.jpg"}], "cost_usd": 0.09, "latency_ms": 38000}
      ]
    }
  ]
}

3. Upload it

python justevals.py push manifest.json

The command prints one link per run. Uploading the same manifest again changes nothing: external_id identifies the run, and outputs are identified by case and sample.

4. Review

Open a link, select the runs on the Runs page and click Compare. Pick the better output for each case.

5. Read the feedback

python justevals.py feedback --suite exterior --batch 2026-10-14-a

The response lists every vote with its note, every rating, and the label of each case. The API reference describes the format.

Concepts

Suite

A suite is a saved test set for one task, for example "exterior edits". Each upload sends the full case list. JustEvals keeps every version: if the cases change, the upload creates a new version, and older runs keep the version they ran on.

Case

A case is one input and a description of a good result. It has a stable key (for example O01). It can hold several input files with roles, such as input, mask, guide or reference, plus parameters and tags. Do not reuse a key for a different input.

Run

A run is one execution of a suite version with one model and prompt configuration. Runs that you start together share a batch. Mark the current production setup with is_baseline so the run page offers "Compare with baseline".

Output

An output is the result for one case in one run. It holds files (the result, intermediate step files and diagnostic images such as heatmaps), text, cost, latency, an error message if the call failed, and automatic scores from your own judges or metrics. Use sample for repeated outputs of the same case.

Judgments

  • Comparison: which output is best for a case, or a tie, or "all bad", with an optional note.
  • Review: ratings of one output, in the shape that the suite's review settings define, with an optional note.
  • Case label: active, needs_fixing or irrelevant, with a note. A label changes the suite, not the run.

People make judgments in the browser. Through the API you can import judgments from an earlier tool (source: "import") or record an automatic judge (source: "auto"). JustEvals shows automatic judges separately from people.

How does the review screen work?

The review screen shows the input and the output of each run for one case. By default the review is blind: the order of the outputs changes per case, and run names, costs and scores stay hidden until you show them. Zoom and pan apply to all images at the same time. Hold Space to flip between the outputs in place, which makes small differences visible.

Key Action
1–9 Pick output A, B, … as best
T Tie
X All bad
U Clear your vote
N Write a note
← → or J K Previous or next case
Space (hold) Flip between outputs
Z, Esc Zoom in, reset zoom
H Show or hide run names
S Show or hide the summary

The summary counts wins, losses, ties and "all bad" per run, with failed outputs, total cost, median time, average ratings and average automatic scores. When you compare three or more runs, earlier pairwise votes between any two of them also count.

Which review settings can I use?

Each project has review settings, and each suite can override them. Uploads can set them per suite with review_config. After someone changes the settings of a suite in the browser, uploads no longer overwrite them.

Preset What reviewers do
best Pick the best output per case, or call a tie or "all bad". This is the default.
good_bad Pick the best, and mark each output good or bad.
score_1_5 Pick the best, and give each output a score from 1 to 5.
video Fidelity and motion from 1 to 5, hard failures as tags, accepted yes or no.
prompt_harness Architecture fidelity and change execution from 1 to 5, ship yes or no.

A custom configuration combines the "pick the best" options with up to 12 rating dimensions of the types choice, scale, boolean and tags:

{
  "compare": {"enabled": true, "allowTie": true, "allowAllBad": true},
  "dimensions": [
    {"key": "realism", "label": "Realism", "type": "scale", "min": 1, "max": 5},
    {"key": "failures", "label": "Failures", "type": "tags", "options": ["warped lines", "new objects"]}
  ],
  "notes": true,
  "blind": true
}

How do suite versions work?

  • An upload that is identical to the latest version keeps that version.
  • An upload that is identical to an older version returns the older version. This lets you upload past runs again without duplicates.
  • Any other upload creates a new version. The cases of a version never change after it exists.
  • Case labels apply to the latest version. If no run uses the latest version yet, a label changes it in place. Otherwise the label creates a new version.
  • A case labeled needs_fixing becomes active again when its content changes in the next upload.

How do I connect a coding agent?

Give your agent the JustEvals skill. It explains the workflow, the manifest and the API, and it tells the agent to read the feedback before it changes a prompt or a model. For tools that do not read skills, the API reference and the OpenAPI document describe the same API. Each documentation page is also available as Markdown: add .md to the address, for example /docs.md.

Where does JustEvals store my data?

The application and the database run on a server of Hetzner Online GmbH in Nuremberg, Germany. Files are stored in Cloudflare R2 with the EU jurisdiction setting, so they stay in the EU. Cloudflare also routes the traffic to the site. Files are private: the browser loads them through links that expire after one hour. JustEvals does not train models on your data, does not sell it and does not put it into analytics. The privacy policy has the details. To delete a project or your account, write to [email protected].