retriEVAL

Author a custom metric from a plain-language description (+ optional golden examples). The judge converts your criteria into explicit evaluation steps, which run as a G-Eval-style scorer in run_eval / evaluate_case.

Compare several runs across all shared metrics. fmt="markdown" (default) returns a markdown matrix; fmt="text" returns plain ASCII; fmt="image" returns a PNG grouped-bar chart.

Score a single output inline, without loading a golden set. The result is saved as a one-case run (golden set "inline") so it has a dashboard link; pass save=False to score without recording it.

Current judge spend, the cap (RETRIEVAL_BUDGET_USD), and remaining headroom.

Fetch a web page and check OUTPUT for *consistency* with it: faithfulness (every claim supported by the page) and, if a question is given, answer relevancy. Use it to ask "does my answer agree with what this page says?". IMPORTANT — this is NOT a correctness check. There are no labels here: the page is an unverified source, not ground truth. A high score means the output matches the page, not that either is right. The returned `caveat` says so on every call; surface it to the user.

Show loaded golden sets and their sizes.

List built-in metrics and any custom metrics authored this session.

List saved runs (most recent last) with their per-metric mean scores.

Load a golden set from a file path, a JSON array string, or JSONL text. Field names are normalized (question->input, answer->actual_output, ground_truth->expected_output, contexts->context, passages->retrieval_context, ...).

One metric's mean score across recent runs, with a threshold reference. fmt="markdown" (default) returns a rich markdown report (tables, inline bars, case strip, distribution); fmt="text" returns plain ASCII; fmt="image" returns a PNG line chart (only useful where images display).

Every metric's mean score for a single run (default: latest). fmt="markdown" (default) returns a rich markdown report (tables, inline bars, case strip, distribution); fmt="text" returns plain ASCII; fmt="image" returns a PNG bar chart (only useful where images display).

Reset the cumulative spend ledger to $0 (the cap itself is unchanged).

Run metrics over a set of cases. Pass `cases` with the data inline (JSON, JSONL, CSV or TSV text, or a file path) to score in a single call — nothing is stored server-side and no prior load is needed. Or pass `golden_set` to reuse a set loaded earlier with load_golden_set. Inline `cases` is preferred: named sets live only in this server process, are shared with anyone else using it, and are lost when it restarts. All cases are scored, but only the `limit` lowest-scoring cases are returned by default (3) to keep replies short; `total_cases`/`shown` tell you how many more exist — call show_run_cases to page. Pass `generator_model` (which LLM produced the outputs) and `judge_model` (which LLM scored them) so the dashboard can compare across models; judge_model defaults to the configured judge. The run is saved to history (file or Supabase). IMPORTANT FOR CALLERS: the reply includes `view_url`, a link to the full visual report. ALWAYS include that link in your response to the user, even when you summarise everything else. `summary_md` is a ready-to-render markdown block (score table + link) that can be shown verbatim. Raises if the spend cap is hit mid-run; partial spend is still metered.

Page through the full per-case results of a saved run (the cases run_eval didn't show). Optionally sort by a single metric's score (worst first).