JustEvals agent skill

JustEvals stores eval results and the human judgments made on them. Your pipeline
runs the models (JustEvals never calls a model); you upload what came out; people
compare runs side by side in the browser; you read their labels and notes back
before the next change.

Concepts

• Suite: a saved test set for one task (core-v2). Uploading a changed case list creates a new version; runs keep the version they ran on.
• Case: one input plus what good looks like. Its key (O01, bath-1) is its identity across versions. Never reuse a key for a different input.
• Run: one execution of a suite version with one model + prompt configuration. Baseline and candidates are separate runs that share a batch.
• Output: a case's result in a run (sample 0, 1, 2… for repeats), with files, cost, latency, error, and optional automatic scores.
• Judgments (made by people in the UI): which output is best per case (or tie / all bad) with a note, per-output ratings shaped by the suite's review settings, and case labels: active, irrelevant, needs_fixing (+ note).

Setup

export JUSTEVALS_API_KEY=je_...             # project key: Settings → API keys
export JUSTEVALS_URL=https://justevals.com  # default
curl -fsSL https://justevals.com/downloads/justevals.py -o justevals.py   # stdlib-only client
python justevals.py check

Workflow

1. Before changing a prompt or model, read the feedback on the latest runs:
   python justevals.py feedback --suite core-v2 [--batch evr_x3m54f] [--since 2026-10-01T00:00:00Z]
   • suite.cases[].status == "needs_fixing": fix the case (input, expectation) before trusting results on it. Read status_note.
   • status == "irrelevant": drop the case from the next upload.
   • comparisons[]: which run won per case and why (note). reviews[]: per-output ratings and notes.
2. Run your existing eval pipeline as usual.
3. Upload a manifest (see below): python justevals.py push manifest.json. It prints a link per run.
4. Ask a human to review the link (Compare view). Do not judge outputs yourself through the API unless asked; automatic judges must use "source": "auto".
5. Re-run the same suite version after a fix and compare again.

Manifest

{
  "suite": {
    "slug": "core-v2",
    "name": "Core v2",
    "review_config": "score_1_5",
    "cases": [
      {
        "key": "O01",
        "title": "Exterior, sunny",
        "group": "Outdoor",
        "instructions": "optimize, sunshine, declutter",
        "expected": "Brighter, sunnier, cleaner",
        "failure_conditions": "Any window, door or roof changes",
        "inputs": [{"path": "cases/O01.jpg", "role": "input"}, {"path": "cases/O01-mask.png", "role": "mask"}],
        "params": {"mode": "outdoor"},
        "tags": ["outdoor"]
      }
    ]
  },
  "runs": [
    {
      "name": "Sunburst (production)",
      "model": "fal-ai/gpt-image-2.5",
      "batch": "evr_x3m54f",
      "is_baseline": true,
      "external_id": "evr_x3m54f:baseline",
      "config": {"prompt_version": "v12", "code_revision": "abc123"},
      "outputs": [
        {"case": "O01", "sample": 0, "files": [{"path": "out/O01-0.jpg"}],
         "cost_usd": 0.09, "latency_ms": 41000, "scores": {"judge_score": 4}},
        {"case": "O02", "sample": 0, "status": "error", "error": "FAL timeout after 300s"}
      ]
    }
  ]
}

• File refs: {"path": ...} (relative to the manifest), {"url": ...} (downloaded and re-uploaded, so expiring provider URLs are safe; the client remembers uploaded URLs in ~/.cache/justevals), or {"file_id": ...}. Optional role (input, mask, reference, output, step, diagnostic) and label.
• The suite's cases is the full list. An upload identical to the latest version keeps it; one identical to an older version returns that older version (so re-uploading past runs creates no duplicates); anything else creates a new version. A version's cases never change once it exists, so a run always gets exactly the cases it was uploaded with.
• external_id makes re-uploads idempotent: the same run is updated (name, model, batch, baseline flag, config), not duplicated. Outputs are keyed by case + sample. A run never moves to another suite version.
• review_config: a preset (best, good_bad, score_1_5, video, prompt_harness) or a full config. Default: pick the best. Once someone changes a suite's review settings in the UI, uploads no longer overwrite them.
• scores are automatic only (judges, pixel metrics). Human labels never go there.

Importing past human labels

Add "judgments" to the manifest (or POST /api/v1/judgments):

{"comparisons": [{"outputs": [{"run": "evr_x3m54f:baseline", "case": "O01"}, {"run": "evr_x3m54f:nb2", "case": "O01"}],
                  "verdict": "winner", "winner": {"run": "evr_x3m54f:nb2", "case": "O01"},
                  "note": "keeps the window frames", "source": "import", "reviewer": "simon",
                  "created_at": "2026-09-12T10:00:00Z"}],
 "reviews": [{"output": {"run": "evr_x3m54f:nb2", "case": "O01"}, "values": {"score": 4},
              "source": "import", "reviewer": "simon"}]}

verdict is winner, tie or all_bad. Review values must match the suite's review settings. Re-importing an
unchanged judgment leaves it untouched (its updated_at stays). To remove imported judgments (e.g. a vote undone at
the source), DELETE /api/v1/judgments with {"comparisons": [{"outputs": [...], "source": "import", "reviewer": "simon"}], "reviews": [{"output": ..., "source": "import", "reviewer": "simon"}]}.

API (Bearer key, JSON, snake_case)

Method	Path	Purpose
GET	/api/v1/project	Check the key
POST	/api/v1/files	Declare a file by sha256, size, content_type; returns upload (PUT URL) unless already stored
POST	/api/v1/files/{id}/complete	Confirm an upload
PUT	/api/v1/suites/{slug}	Create/update a suite from its full case list
GET	/api/v1/suites/{slug}[?version=N]	Cases with labels
GET	/api/v1/runs[?suite=…][&batch=…]	List runs (newest first)
POST	/api/v1/runs	Create or update a run (suite, name, model, batch, is_baseline, config, external_id)
POST	/api/v1/runs/{id}/outputs	Add/replace up to 500 outputs
POST	/api/v1/runs/{id}/complete	Mark complete or failed
GET	/api/v1/runs/{id}	Run (with suite, suite_version) and its outputs including meta
GET	/api/v1/feedback?suite=…[&runs=…][&batch=…][&since=…]	Labels and notes (since filters on updated_at)
POST	/api/v1/judgments	Import past labels / automatic judges
DELETE	/api/v1/judgments	Remove imported / automatic judgments

{id} for runs also accepts the run's external_id.