IA-QA Eval & RAG — 38 tools

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.

{"variant_a":{"name":"A","scores":[70,71,69]},"variant_b":{"name":"B","scores":[70,71,69]}}

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). For a 2-way head-to-head with structural JSON diff use compare_responses instead.

{"reference":"Paris is the capital of France.","responses":[{"label":"A","text":"Lyon is the capital."},{"label":"B","text":"Paris is the capital of France."}]}

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

{"documents":["shipping times","refund for damaged items","office hours"],"query":"refund damaged"}

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

{"chunks":[{"source":"a.md","text":"c1 text"}],"query":"q?"}

Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.

{"models":["gpt-6.1-sol","claude-sonnet-5-5"]}

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

{"response_a":"Paris is the capital.","response_b":"Paris is the capital."}

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.

{"responses":["The capital of France is Paris.","Paris is the capital of France."]}

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

{"messages":[{"content":"hi","role":"user"}],"model":"gpt-4o"}

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

{"messages":[{"content":"What is your refund policy?","role":"user"},{"content":"Refunds are available within 30 days.","role":"assistant"}]}

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

{"expected_conclusion":"7","reasoning":"Step 1: 12 apples minus 5 eaten.\nStep 2: 12 - 5 = 7.\nTherefore, the answer is 7."}

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.

{"input":"Hello world, this is a test."}

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

{"text_a":"the cat sat","text_b":"quantum field theory"}

Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.

{"input_tokens":1000000,"model":"gpt-4o","output_tokens":1000000}

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.

{"input":"Here:\n```json\n{\"a\":[1,2]}\n```\nthanks"}

Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, encoding, text, llm, qa, rag, dev, security, web.

{"query":"leaked api key"}

Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.

{}

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

{"description":"Summarize a customer support ticket in one sentence."}

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.

{"topic":"rag-pipeline"}

Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". For entailment use run_semantic_tests (NLI/embedding) or a calibrated judge.

{"answer":"Le remboursement prend plus de 7 jours ouvrés.","context":"Le remboursement prend plus de 7 jours ouvrés."}

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

{"provider":"groq"}

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

{"expected_format":"yaml","output":"a: 1\nb: [1, 2]"}

Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.

{"prompt":"Reply with the single word OK."}

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

{"output":"{\"name\":\"a\",\"age\":3}","schema":{"properties":{"age":{"minimum":0,"type":"integer"},"name":{"type":"string"},"role":{"enum":["admin","user"]}},"required":["name","age"],"type":"object"}}

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

{"expected_format":"json","must_include":"a","must_not_include":"TODO","output":"{\"a\":1}"}

Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.

{"base":{"output":"Paris is the capital and largest city of France."},"variants":[{"output":"Paris is the capital and largest city of France.","relation":"case"}]}

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

{"model":"gpt-4o"}

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

{"needle":"The code is 4242.","position":"middle","question":"What is the code?","tokens":2000}

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

{"system_prompt":"You are a helpful assistant.","user_prompt":"Say hi"}

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

{"chunks":["a b c","d e f"],"query":"quantum"}

Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

{"comment":"Formatted a 2 MB payload correctly.","score":4,"tool_name":"format_json"}

Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels. Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).

{"api_key":"\u003cyour-api-key\u003e","passages":[{"relevant":true,"text":"Paris is the capital of France."},{"relevant":false,"text":"Bananas are rich in potassium."}],"query":"capital of France"}

Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). For meaning against a reference answer, use run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.

{"question":"q","response":"anything"}

Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.

{"inline_contract":{"expectations":{"contains":["Paris"],"min_score":70},"metadata":{"name":"smoke","version":"1.0"},"scenarios":[{"ground_truth":"Paris","id":"capital","input":"Capital of France? One word."}]}}

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

{"cases":[{"actual":"Paris is the capital of France","expected":"The capital of France is Paris"}]}

Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, an overall PASS/WARNING/FAIL against threshold, and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.

{"api_key":"\u003cyour-api-key\u003e","image_url":"https://www.ia-qa.com/media/discover-report-poster.png","model":"gemini-2.5-flash","test_cases":[{"assertion_type":"contains","assertion_value":"yes","question":"Does the image contain text? Answer yes or no."}]}

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

{"hypothesis":"the cat sat on the mat","reference":"the cat sat on the mat","threshold":0.5}

Test a SKILL.md (Agent Skills format). (1) lint: the spec's frontmatter rules, no model — an error returns BLOCK; gate on lint.valid. (2) triggering: a model sees only the names + descriptions of this skill and its neighbours and picks which to load (temperature 0); trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK. Your own scenarios give a reproducible gate; generated ones are a signal. Unanswered requests make the run INCONCLUSIVE; so do scenarios left when a call reaches its 22 s budget (split long runs). Default model: Groq on the server key.

{"skill_md":"---\nname: json-formatter\ndescription: Pretty-print a JSON string the user pastes.\n---\nFormat the JSON with 2-space indentation."}