# IA-QA Eval & RAG — 38 tools (remote · www.ia-qa.com)

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

- Trust score: 79/100 (medium)
- Change this week: +6
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-11-05

## Components

- remote · `www.ia-qa.com`: 79/100 (this document), [markdown](https://verifymcp.io/servers/jcjamet-ia-qa-eval/mcp-eval.md), [page](https://verifymcp.io/servers/jcjamet-ia-qa-eval/mcp-eval)

## Channel facts

- Endpoint: `https://www.ia-qa.com/mcp/eval`
- Transports: `streamable-http`
- Auth: `none`
- Version: `1.0.1`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically against the live server, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-11-05.

- **Endpoint Security**: 83/100
  - The endpoint's TLS certificate is valid, in date, and uses a strong key.
  - No authorisation is required to call this server. Every tool declares its destructiveHint and none is destructive, so open access doesn't expose one.
  - HTTPS is enforced; there's no plaintext access path.
  - The HSTS (Strict-Transport-Security) header is present.
  - DNSSEC is configured correctly; the domain's records validate against the full chain to the root.
- **Transport & Reachability**: 100/100
  - Verified streamable-http transport via a live MCP handshake.
- **Schema Quality & AI Usability**: 81/100
  - AI-judged instruction clarity (excellent).
  - Context-footprint check failed: tool/resource definitions use about 7560 tokens (~198/item across 38 items; 38 tools + 0 resources), over budget; trim descriptions and params.
  - Tools include usage examples.
- **Stability & Change Management**: 33/100
  - Stability observed for 10 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 99/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 96% of tool parameters carry a description.
  - Structured output schemas are declared (100% of tools); any adoption earns full credit.
- **Tool Safety**: 75/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - 0 of 2 tool(s) whose name or description implies an irreversible operation declare an MCP destructiveHint annotation; "run_eval_contract" implies "eval" and declares readOnlyHint instead, contradicting what its own name says it does.
  - An AI judge read all 39 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 100/100
  - Implements a current MCP spec version (2026-07-28).

## Install

### How do I install the IA-QA Eval & RAG — 38 tools MCP server?

IA-QA Eval & RAG — 38 tools is a hosted endpoint at https://www.ia-qa.com/mcp/eval, so there is nothing to install locally. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add --transport http jcjamet-ia-qa-eval 'https://www.ia-qa.com/mcp/eval'
```

### Cursor

```json
{
  "mcpServers": {
    "jcjamet-ia-qa-eval": {
      "url": "https://www.ia-qa.com/mcp/eval"
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "jcjamet-ia-qa-eval": {
      "type": "http",
      "url": "https://www.ia-qa.com/mcp/eval"
    }
  }
}
```

### Codex

```toml
[mcp_servers.jcjamet-ia-qa-eval]
url = "https://www.ia-qa.com/mcp/eval"
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "jcjamet-ia-qa-eval": {
      "type": "remote",
      "url": "https://www.ia-qa.com/mcp/eval",
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add jcjamet-ia-qa-eval --url 'https://www.ia-qa.com/mcp/eval' --transport streamable-http
```

### Hermes

```yaml
mcp_servers:
  jcjamet-ia-qa-eval:
    url: "https://www.ia-qa.com/mcp/eval"
```

### Netclaw

```json
{
  "McpServers": {
    "jcjamet-ia-qa-eval": {
      "Transport": "http",
      "Url": "https://www.ia-qa.com/mcp/eval"
    }
  }
}
```

### Vellum

```bash
assistant mcp add jcjamet-ia-qa-eval -t streamable-http -u 'https://www.ia-qa.com/mcp/eval'
```

### Other

```json
{
  "mcpServers": {
    "jcjamet-ia-qa-eval": {
      "type": "http",
      "url": "https://www.ia-qa.com/mcp/eval"
    }
  }
}
```

The mcpServers block is a cross-client convention. Remote transports vary, so check your client's docs.

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-11-05 (score 79, +1)

- [security] The server rewrote its instructions, which are the text every model session reads
- [functional] MCP protocol: Implements a current MCP spec version (2026-07-28).
- [functional] MCP protocol version: 2025-11-25 → 2026-07-28

### 2026-11-04 (score 78, 0)

- [security] Tool “find_tool” rewrote its description, which is the text the model reads
- [security] Tool “rate_tool” rewrote its description, which is the text the model reads
- [security] Tool “run_eval_contract” rewrote its description, which is the text the model reads
- [cosmetic] “run_eval_contract” reworded the description of “inline_contract”

### 2026-11-03 (score 78, +1)

- [security] Tool “compare_models” rewrote its description, which is the text the model reads
- [security] Tool “estimate_llm_cost” rewrote its description, which is the text the model reads
- [security] Tool “llm_generate” rewrote its description, which is the text the model reads
- [security] Tool “token_budget_calculator” rewrote its description, which is the text the model reads
- [cosmetic] “compare_models” reworded the description of “models”
- [cosmetic] “context_window_check” reworded the description of “model”
- [cosmetic] “llm_generate” reworded the description of “model”
- [cosmetic] “token_budget_calculator” reworded the description of “model”
- [cosmetic] “compare_models” dropped the optional parameter “use_case”

### 2026-11-01 (score 77, +3)

- [security] The server rewrote its instructions, which are the text every model session reads
- [security] Tool “generate_ci_workflow” rewrote its description, which is the text the model reads
- [security] Tool “metamorphic_check” rewrote its description, which is the text the model reads
- [security] Tool “rerank_evaluate” rewrote its description, which is the text the model reads
- [security] Tool “run_eval_contract” rewrote its description, which is the text the model reads
- [security] Tool “run_vlm_test_suite” rewrote its description, which is the text the model reads
- [security] Tool “test_skill” rewrote its description, which is the text the model reads
- [functional improvement] Schema quality: 221 → 197
- [functional improvement] Schema quality: fail → pass
- [cosmetic] “hallucination_check” added an optional parameter “cases”
- [cosmetic] “generate_ci_workflow” reworded the description of “app_base_url”
- [cosmetic] “generate_ci_workflow” reworded the description of “app_start_command”
- [cosmetic] “generate_ci_workflow” reworded the description of “cli_tools”
- [cosmetic] “generate_ci_workflow” reworded the description of “contract_path”
- [cosmetic] “generate_ci_workflow” reworded the description of “cron”
- [cosmetic] “generate_ci_workflow” reworded the description of “fail_on_fail”
- [cosmetic] “generate_ci_workflow” reworded the description of “gate”
- [cosmetic] “generate_ci_workflow” reworded the description of “min_score”
- [cosmetic] “generate_ci_workflow” reworded the description of “node_version”
- [cosmetic] “generate_ci_workflow” reworded the description of “provider”
- [cosmetic] “generate_ci_workflow” reworded the description of “triggers”
- [cosmetic] “generate_ci_workflow” reworded the description of “workflow_name”
- [cosmetic] “hallucination_check” reworded the description of “answer”
- [cosmetic] “hallucination_check” reworded the description of “context”
- [cosmetic] “hallucination_check” reworded the description of “strict”
- [cosmetic] “metamorphic_check” reworded the description of “api_key”
- [cosmetic] “metamorphic_check” reworded the description of “base”
- [cosmetic] “metamorphic_check” reworded the description of “baseline_guard”
- [cosmetic] “metamorphic_check” reworded the description of “mode”
- [cosmetic] “metamorphic_check” reworded the description of “require_all”
- [cosmetic] “metamorphic_check” reworded the description of “thresholds”
- [cosmetic] “metamorphic_check” reworded the description of “variants”
- [cosmetic] “rerank_evaluate” reworded the description of “api_key”
- [cosmetic] “rerank_evaluate” reworded the description of “passages”
- [cosmetic] “rerank_evaluate” reworded the description of “query”
- [cosmetic] “rerank_evaluate” reworded the description of “threshold”
- [cosmetic] “rerank_evaluate” reworded the description of “top_k”
- [cosmetic] “response_quality_score” reworded the description of “expected_keywords”
- [cosmetic] “run_eval_contract” reworded the description of “api_keys”
- [cosmetic] “run_eval_contract” reworded the description of “contract_path”
- [cosmetic] “run_eval_contract” reworded the description of “inline_contract”
- [cosmetic] “run_vlm_test_suite” reworded the description of “api_key”
- [cosmetic] “run_vlm_test_suite” reworded the description of “image_base64”
- [cosmetic] “run_vlm_test_suite” reworded the description of “image_mime_type”
- [cosmetic] “run_vlm_test_suite” reworded the description of “image_url”
- [cosmetic] “run_vlm_test_suite” reworded the description of “model”
- [cosmetic] “run_vlm_test_suite” reworded the description of “system_prompt”
- [cosmetic] “run_vlm_test_suite” reworded the description of “test_cases”
- [cosmetic] “run_vlm_test_suite” reworded the description of “threshold”
- [cosmetic] “test_skill” reworded the description of “api_key”
- [cosmetic] “test_skill” reworded the description of “dir_name”
- [cosmetic] “test_skill” reworded the description of “lint_only”
- [cosmetic] “test_skill” reworded the description of “model”
- [cosmetic] “test_skill” reworded the description of “other_skills”
- [cosmetic] “test_skill” reworded the description of “scenario_count”
- [cosmetic] “test_skill” reworded the description of “scenarios”
- [cosmetic] “hallucination_check” made “answer” optional
- [cosmetic] “hallucination_check” made “context” optional

### 2026-10-31 (score 74, +1)

No change was recorded against any check on this day. Stability & Change Management went from 13 to 17. That category is still filling its 30-day observation window: 4 days of observed history at the previous scan, 5 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-29 (score 73, 0)

- [security] Tool “llm_generate” rewrote its description, which is the text the model reads
- [cosmetic] “llm_generate” reworded the description of “model”

### 2026-10-28 (score 73, +1)

- [security] Tool “ab_test_report” rewrote its description, which is the text the model reads
- [security] Tool “rate_tool” rewrote its description, which is the text the model reads

### 2026-10-27 (score 72, 0)

- [functional improvement] Stability: unverified → 0.03

## MCP tools (38)

### `get_testing_guidelines` (~227 tokens)

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.

Input parameters:

- `topic` (string): The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics.

Output parameters:

- `available_topics` (array)
- `keywords` (array)
- `start_here` (object)
- `tip` (string)
- `topic`
- `usage` (string)

### `find_tool` (~194 tokens)

Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, encoding, text, llm, qa, rag, dev, security, web.

Input parameters:

- `category` (string): Optional: filter by category — data | encoding | text | llm | qa | rag | dev | security | web
- `max_results` (number): Maximum tools to return (default 10, max 50). Results are ranked by IDF-weighted relevance, so common words like "test" do not inflate the list.
- `query` (string, required): Keyword(s) to search in tool name and description (e.g. "cors", "token", "vector", "json")
- `with_schema` (boolean): Set true to include inputSchema in results (default: false)

Output parameters:

- `category`
- `count` (number)
- `hint`
- `query`
- `score`
- `tool`
- `tools` (array)
- `total_matches` (number)
- `truncated` (boolean)

### `hallucination_check` (~247 tokens)

Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". For entailment use run_semantic_tests (NLI/embedding) or a calibrated judge.

Input parameters:

- `answer` (string): The LLM answer to verify (with context).
- `cases` (array): Batch instead of answer + context: one result per case, and the worst gate overall.
- `context` (string): The source text that should ground it.
- `strict` (boolean): Every sentence must be supported (default false).

Output parameters:

- `analysis`
- `contradictions`
- `detail` (string)
- `entities`
- `grounded` (boolean)
- `grounded_count` (number)
- `grounding_score`
- `matched_words` (number)
- `message` (string)
- `numbers`
- `overlap` (number)
- `sentence`
- `total_sentences` (number)
- `total_words` (number)
- `unbound_claims`
- `ungrounded_count` (number)
- `unsupported_claims`
- `verdict` (string)

### `run_eval_contract` (~244 tokens)

Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.

Input parameters:

- `api_keys` (object): Your keys per provider. Only Groq falls back to the server key; any other provider needs yours.
- `contract_path` (string): A .ia-eval.yaml under the server's working directory (local use).
- `inline_contract` (object): The contract as an object: "metadata" ({name, version, model?, provider?}), "expectations", "evaluators" and "scenarios" ([{id, input, expectations?}]). generate_eval_yaml scaffolds one.
- `overrides` (object): Override contract defaults
- `validate_only` (boolean): Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid.

Output parameters:

- `contract_path`
- `issues`
- `metadata` (object)
- `scenario_results`
- `summary` (object)
- `valid` (boolean)
- `warnings`

### `run_semantic_tests` (~216 tokens)

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

Input parameters:

- `api_key` (string): OpenAI API key — required only when mode is embeddings.
- `cases` (array, required): Array of (actual, expected) pairs to evaluate.
- `mode` (string): tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key.
- `require_all` (boolean): If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS.
- `thresholds` (object): Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).

Output parameters:

- `failed` (number)
- `mode` (string)
- `passed` (number)
- `results` (array)
- `total` (number)
- `verdict` (string)

### `generate_ci_workflow` (~380 tokens)

Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.

Input parameters:

- `app_base_url` (string): [selector_drift] Default http://127.0.0.1:3000; must match .ia-qa/config.json.
- `app_start_command` (string): [selector_drift] Boots the app, e.g. "npm run start:ci". Never guessed: omitted, the step is a TODO.
- `cli_tools` (array): Tools to gate on, e.g. ["secret_scan","prompt_injection_scan"].
- `contract_path` (string): Default evals/smoke.ia-eval.yaml.
- `cron` (string): For schedule (default '0 6 * * 1').
- `fail_on_fail` (boolean): Default true; false reports without gating.
- `gate` (string): Default eval_contract. both = CLI checks + eval; all = CLI checks, drift, eval.
- `min_score` (number): Override the contract min_score (0-100).
- `node_version` (string): Default "20".
- `provider` (string): Decides which repo secret is wired (default groq).
- `triggers` (array): Default push + pull_request.
- `workflow_name` (string)

Output parameters:

- `gate` (string)
- `next_steps` (array)
- `notes` (array)
- `path` (string)
- `secrets_required` (array)
- `yaml` (string)

### `needle_haystack_generate` (~171 tokens)

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

Input parameters:

- `needle` (string, required): The fact to hide (e.g. "The secret code is ALPHA-42")
- `position` (string): Where to insert the needle: "start", "middle", "end", "random" (default: "middle")
- `question` (string, required): The question to ask the LLM (e.g. "What is the secret code?")
- `tokens` (integer): Target haystack size in tokens (default: 5000, max: 100000)

Output parameters:

- `estimated_tokens`
- `haystack`
- `insert_block`
- `needle`
- `position`
- `question`
- `total_blocks` (number)

### `generate_eval_yaml` (~192 tokens)

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

Input parameters:

- `description` (string, required): Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints.
- `scenario_count` (number): Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial.
- `system_prompt` (string): Optional system prompt of the LLM under test. Helps generate more accurate test cases.
- `task_type` (string): Optional task type hint to guide evaluator selection.

Output parameters:

- `model_used` (string)
- `scenario_count` (number)
- `task_type` (string)
- `yaml` (string)

### `metamorphic_check` (~299 tokens)

Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.

Input parameters:

- `api_key` (string): OpenAI key, embeddings mode only.
- `base` (object, required): The original run.
- `baseline_guard` (object): Floor on the BASE answer; failing it returns INVALID.
- `mode` (string): tfidf (default, lexical: gate on case/typo, read paraphrase as a trend) or embeddings (OpenAI, needs api_key; required for translation).
- `require_all` (boolean): Default true: every gated variant must pass. Keep it for a gate — false lets one easy typo row hold the run at PASS while a paraphrase fails.
- `thresholds` (object): Per-relation overrides, 0-1. Calibrate on your own data before gating.
- `variants` (array, required): Answers to transformed versions of the question.

Output parameters:

- `baseline` (object)
- `mode` (string)
- `summary` (object)
- `thresholds` (object)
- `variants` (array)
- `verdict` (string)
- `warnings` (array)
- `weakest` (object)

### `test_skill` (~323 tokens)

Test a SKILL.md (Agent Skills format). (1) lint: the spec's frontmatter rules, no model — an error returns BLOCK; gate on lint.valid. (2) triggering: a model sees only the names + descriptions of this skill and its neighbours and picks which to load (temperature 0); trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK. Your own scenarios give a reproducible gate; generated ones are a signal. Unanswered requests make the run INCONCLUSIVE; so do scenarios left when a call reaches its 22 s budget (split long runs). Default model: Groq on the server key.

Input parameters:

- `api_key` (string): Key for the chosen model's provider; not needed for the default.
- `dir_name` (string): Folder holding the SKILL.md; name must match it.
- `lint_only` (boolean): Format check only: no model, no key. The CI gate for a SKILL.md.
- `model` (string): e.g. gpt-4o-mini. Default openai/gpt-oss-20b (Groq, server key).
- `other_skills` (array): The skills installed next to this one. Default: six generic built-ins.
- `scenario_count` (number): Requests to generate when scenarios is omitted (default 6).
- `scenarios` (array): Fixed requests, near misses included (should_trigger false). Max 20.
- `skill_md` (string, required): Full content of the SKILL.md file to test, YAML frontmatter included.

Output parameters:

- `lint` (object)
- `scenarios` (array)
- `scenarios_source` (string)
- `score` (number)
- `step_adherence` (number)
- `trigger_accuracy` (number)
- `verdict` (string)

### `llm_output_validator` (~240 tokens)

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

Input parameters:

- `check_json_schema` (string): If expected_format is JSON, provide required keys as comma-separated list to validate the structure
- `check_safety` (boolean): Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage
- `expected_format` (string): Expected output format
- `expected_language` (string): Expected language of the output (en, fr, es, de…). Checks for common words.
- `max_length` (number): Maximum character length for the output
- `min_length` (number): Minimum character length for the output
- `must_include` (string): Comma-separated strings that MUST appear in the output
- `must_not_include` (string): Comma-separated strings that must NOT appear (e.g. "TODO, FIXME, undefined, NaN")
- `output` (string, required): The LLM output text to validate

Output parameters:

- `checks`
- `failed`
- `passed`
- `total` (number)
- `verdict`

### `compare_responses` (~350 tokens)

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

Input parameters:

- `api_key` (string): Optional API key for the judge model (BYOK). Used only for the judge call; never stored.
- `check_json` (boolean): Try to parse as JSON and compare structurally (keys, types, values)
- `label_a` (string): Label for output A (e.g. "GPT-4o", "v1.0")
- `label_b` (string): Label for output B (e.g. "GPT-5-nano", "v1.1")
- `model` (string): Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner.
- `reference` (string): Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic).
- `response_a` (string, required): First output (e.g. model A's answer)
- `response_b` (string, required): Second output (e.g. model B's answer)
- `task` (string): The task/prompt both outputs were answering — used by the LLM judge for context

Output parameters:

- `judge`
- `labelA`
- `labelB`
- `metrics`
- `summary` (string)
- `verdict`

### `analyze_responses` (~227 tokens)

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). For a 2-way head-to-head with structural JSON diff use compare_responses instead.

Input parameters:

- `reference` (string): Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named.
- `responses` (array, required): The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { "label": "GPT-4o", "text": "..." }. At least 2 required.

Output parameters:

- `consensus`
- `count`
- `reference_ranking`
- `summary` (string)

### `prompt_test_suite` (~201 tokens)

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

Input parameters:

- `adversarial_prompts` (boolean): Auto-generate adversarial test variants (jailbreak, injection, edge cases)
- `check_safety` (boolean): Include safety/PII checks in the rubric
- `expected_behavior` (string): Description of what the LLM should do (free text)
- `expected_format` (string): Expected output format
- `max_tokens` (number): Max token budget for the test
- `must_include` (string): Required content (comma-separated)
- `must_not_include` (string): Forbidden content (comma-separated)
- `system_prompt` (string, required): The system prompt under test
- `temperature` (number): Temperature to use
- `user_prompt` (string, required): The user prompt to send

Output parameters:

- `categories` (array)
- `instructions` (string)
- `rubric`
- `test_suite_name` (string)
- `total_tests` (number)

### `llm_format_check` (~92 tokens)

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

Input parameters:

- `expected_format` (string, required): Expected format
- `output` (string, required): The LLM output to validate
- `regex_pattern` (string): Custom regex pattern (only when expected_format is "regex")

Output parameters:

- `checks`
- `expected_format`
- `failed`
- `passed`
- `total_checks` (number)
- `valid` (boolean)

### `consistency_check` (~143 tokens)

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.

Input parameters:

- `check_facts` (boolean): Check for contradictory numbers/facts across responses (default: true)
- `responses` (array, required): Array of 2+ LLM responses to compare (same prompt, different runs)

Output parameters:

- `avg_similarity`
- `fact_contradiction`
- `fact_drift`
- `length_variance_percent`
- `pairwise_scores`
- `response_count` (number)
- `verdict`

### `llm_json_schema_check` (~85 tokens)

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

Input parameters:

- `output` (string, required): The LLM JSON output (raw string, will be parsed)
- `schema` (object, required): JSON Schema (draft-07 subset) to validate against

Output parameters:

- `error_count` (number)
- `errors` (array)
- `parse_error`
- `parsed_type`
- `valid` (boolean)

### `response_quality_score` (~189 tokens)

Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). For meaning against a reference answer, use run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.

Input parameters:

- `expected_keywords` (array): Keywords a correct answer must contain, as whole words (case-insensitive; a plural in -s/-es counts). Scored as coverage.
- `max_length` (number): Maximum character length. Scored as a proportional penalty beyond it.
- `question` (string, required): The original question/prompt
- `response` (string, required): The LLM response to score

Output parameters:

- `breakdown` (object)
- `grade`
- `max_score` (number)
- `not_measured` (array)
- `reason` (string)
- `signals` (object)
- `stats` (object)
- `total_score`

### `conversation_analyze` (~53 tokens)

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

Input parameters:

- `messages` (array, required): Conversation messages in order

Output parameters:

- `assistant_messages` (number)
- `avg_response_length` (number)
- `context_retention`
- `has_system_prompt` (boolean)
- `repetition_detected` (boolean)
- `repetitions`
- `topic_drift`
- `turn_count` (number)
- `user_messages` (number)

### `cot_analyzer` (~104 tokens)

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

Input parameters:

- `expected_conclusion` (string): Expected final answer to check against (optional)
- `reasoning` (string, required): The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)

Output parameters:

- `backtracking_signals`
- `conclusion_matches_expected`
- `has_conclusion` (boolean)
- `markers`
- `reasoning_depth`
- `reasoning_depth_label`
- `step_count` (number)
- `total_chars` (number)
- `total_lines` (number)

### `ab_test_report` (~114 tokens)

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.

Input parameters:

- `variant_a` (object, required): First variant configuration with name and score array
- `variant_b` (object, required): Second variant configuration with name and score array

Output parameters:

- `count` (number)
- `improvement_percent`
- `max`
- `mean` (string)
- `median` (string)
- `min`
- `recommendation` (number)
- `std_dev` (string)
- `variant_a` (object)
- `variant_b` (object)
- `winner`

### `similarity_score` (~206 tokens)

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

Input parameters:

- `batch` (array): Batch mode: array of {reference, hypothesis} pairs.
- `hypothesis` (string): Hypothesis / actual text (LLM output)
- `metrics` (array): Metrics to compute (default: all). Options: "cosine_bow", "cosine_tfidf", "jaccard", "rouge1", "rouge2", "rougeL", "bleu"
- `reference` (string): Reference / expected text (ground truth)
- `threshold` (number): Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score.

Output parameters:

- `count` (number)
- `f1` (number)
- `mode` (string)
- `precision` (number)
- `recall` (number)
- `results`

### `run_vlm_test_suite` (~209 tokens)

Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, an overall PASS/WARNING/FAIL against threshold, and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.

Input parameters:

- `api_key` (string, required): Key for the model's provider.
- `image_base64` (string)
- `image_mime_type` (string): For image_base64 (default image/jpeg).
- `image_url` (string): Public image URL (or image_base64).
- `model` (string, required)
- `system_prompt` (string)
- `test_cases` (array, required)
- `threshold` (number): Pass rate for an overall PASS (default 80).

Output parameters:

- `failed` (number)
- `model` (string)
- `passed` (number)
- `results` (array)
- `total` (number)
- `verdict` (string)

### `embedding_similarity` (~186 tokens)

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

Input parameters:

- `batch` (array): Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided.
- `methods` (array): Algorithms to use (default: all three). Options: "bow", "tfidf", "ngram"
- `text_a` (string): First text to compare (single-pair mode)
- `text_b` (string): Second text to compare (single-pair mode)

Output parameters:

- `count` (number)
- `mode` (string)
- `results`
- `scores`
- `text_a`
- `text_b`

### `rag_relevance_rank` (~81 tokens)

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

Input parameters:

- `chunks` (array, required): Array of text chunks to rank
- `query` (string, required): The user query
- `top_k` (number): Return top K results (default: all)

Output parameters:

- `chunk_preview`
- `index`
- `keyword_overlap`
- `query`
- `rank` (number)
- `results`
- `returned` (number)
- `score` (string)
- `total_chunks` (number)

### `bm25_score` (~127 tokens)

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

Input parameters:

- `b` (number): Length normalization factor (default: 0.75)
- `documents` (array, required): Array of documents to rank
- `k1` (number): Term frequency saturation (default: 1.5)
- `query` (string, required): The search query
- `top_k` (number): Return top K results (default: all)

Output parameters:

- `avg_doc_length` (number)
- `b`
- `bm25_score` (number)
- `doc_length` (number)
- `doc_preview`
- `documents_count` (number)
- `index`
- `k1`
- `query`
- `results`

### `build_rag_prompt` (~163 tokens)

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

Input parameters:

- `chunks` (array, required): Retrieved context chunks with .text (required), .source (optional), .score (optional)
- `cite_sources` (boolean): Add [1], [2] citation numbers (default: true)
- `language` (string): Response language instruction (e.g. "French", "Spanish")
- `max_context_tokens` (number): Max tokens for context section (default: 2000)
- `query` (string, required): The user question to answer
- `system_instruction` (string): Custom system instruction (default: standard RAG grounding instruction)

Output parameters:

- `chunks_included` (number)
- `chunks_truncated` (number)
- `context_tokens_estimate` (number)
- `included_chunks`
- `prompt`
- `system_prompt`
- `total_tokens_estimate` (number)

### `rerank_evaluate` (~195 tokens)

Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels. Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).

Input parameters:

- `api_key` (string): NVIDIA key, used only when no passage has a score; never stored.
- `passages` (array, required): 2 to 20 passages.
- `query` (string, required)
- `threshold` (number): Minimum Precision@k to PASS (default 0.5).
- `top_k` (integer): Default 3.

Output parameters:

- `has_ground_truth` (boolean)
- `k` (number)
- `latency_ms` (number)
- `mode` (string)
- `model` (string)
- `precision_at_k` (number)
- `ranked` (array)
- `recall_at_k` (number)
- `threshold` (number)
- `total_passages` (number)
- `verdict` (string)

### `count_tokens` (~77 tokens)

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.

Input parameters:

- `input` (string, required): Text to count tokens for

Output parameters:

- `chars`
- `tokens_estimate`
- `words`

### `extract_json_from_text` (~88 tokens)

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.

Input parameters:

- `input` (string, required): Raw text (e.g., LLM output) that may contain a JSON object or array

Output parameters:

- `json`
- `source` (string)

### `estimate_llm_cost` (~120 tokens)

Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.

Input parameters:

- `input_tokens` (number, required): Number of input/prompt tokens
- `model` (string, required): Model name, e.g. "gpt-4o", "claude-3.5-sonnet", "deepseek-v3"
- `output_tokens` (number): Number of output/completion tokens (default: 0)

Output parameters:

- `input_cost_usd` (string)
- `input_tokens`
- `model`
- `output_cost_usd` (string)
- `output_tokens`
- `rates` (object)
- `total_cost_usd` (string)

### `token_budget_calculator` (~194 tokens)

Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).

Input parameters:

- `context` (string): Actual context text (will estimate tokens)
- `context_tokens` (number): Token count for RAG context / documents
- `expected_output_tokens` (number): Expected max output tokens
- `model` (string, required): Model id from list_llm_models (e.g. gpt-6.1-sol, claude-sonnet-5-5, gemini-3.8-flash)
- `system_prompt` (string): Actual system prompt text (will estimate tokens)
- `system_prompt_tokens` (number): Token count for system prompt
- `user_input` (string): Actual user input text (will estimate tokens)
- `user_input_tokens` (number): Token count for user message

Output parameters:

- `breakdown` (object)
- `context_window`
- `fits_in_window`
- `model`
- `remaining_tokens`
- `utilization_percent`
- `warnings`

### `context_window_check` (~112 tokens)

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

Input parameters:

- `max_output_tokens` (number): Reserved tokens for output (default: 4096)
- `messages` (array, required): Array of messages (system/user/assistant)
- `model` (string, required): Model id from list_llm_models (e.g. gpt-6.1-sol, claude-sonnet-5-5)

Output parameters:

- `breakdown` (object)
- `chars` (number)
- `context_window`
- `fits`
- `index`
- `message_count` (number)
- `model`
- `per_message`
- `reserved_output_tokens`
- `role`
- `tokens`
- `total_input_tokens`
- `total_tokens`
- `utilization_percent` (string)
- `warnings`

### `model_info` (~110 tokens)

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

Input parameters:

- `model` (string, required): Model name (e.g. "gpt-4o", "claude-3.5-sonnet", "gemini-3.1-pro-preview")

Output parameters:

- `model`
- `pricing_per_1k` (object)

### `compare_models` (~107 tokens)

Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.

Input parameters:

- `models` (array, required): Array of 2-5 model names (e.g. ["gpt-5","claude-sonnet-4-6","gemini-3.5-flash"])

Output parameters:

- `cheapest`
- `largest_context`
- `models_compared` (number)
- `prices_as_of`
- `rows`

### `list_llm_models` (~137 tokens)

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

Input parameters:

- `provider` (string): Filter by provider name (case-insensitive). E.g. "Groq", "HuggingFace", "OpenAI", "Anthropic", "Google", "DeepSeek", "xAI", "Ollama". Omit for full catalog.

Output parameters:

- `filter`
- `models`
- `providers`
- `total` (number)

### `llm_generate` (~275 tokens)

Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.

Input parameters:

- `max_tokens` (number): Maximum tokens to generate (default: 2048, max: 4096)
- `model` (string): Model ID (default: "openai/gpt-oss-20b"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b. Other ids from list_llm_models, HuggingFace ones included, are…
- `prompt` (string, required): The user prompt / instruction to send to the model
- `system` (string): Optional system prompt to set context or persona
- `temperature` (number): Sampling temperature 0.0–1.5 (default: 0.7)

Output parameters:

- `content`
- `latency_ms` (number)
- `model`
- `provider`
- `usage`

### `rate_tool` (~308 tokens)

Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

Input parameters:

- `comment` (string): Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating. Up…
- `score` (number, required): Rating from 1 (poor) to 5 (excellent)
- `tool_name` (string, required): Name of the MCP tool to rate (e.g. "format_json", "shield_analyze")

Output parameters:

- `comment`
- `message` (string)
- `ok` (boolean)
- `rated_at` (string)
- `score`
- `stored_chars` (number)
- `tool_name`
- `truncated` (boolean)

## Diagnostics

Captured diagnostic sections: TLS, DNSSEC, Authorisation, Transports. The full working is on the page: https://verifymcp.io/servers/jcjamet-ia-qa-eval/mcp-eval#diagnostics

## Score history

- 2026-11-05: 79
- 2026-11-04: 78
- 2026-11-03: 78
- 2026-11-02: 77
- 2026-11-01: 77
- 2026-10-31: 74
- 2026-10-30: 73
- 2026-10-29: 73
- 2026-10-28: 73
- 2026-10-27: 72
- 2026-10-26: 72

## Common questions

### What is the IA-QA Eval & RAG — 38 tools MCP server?

IA-QA Eval & RAG — 38 tools is an MCP server listed in the public MCP registry as io.github.JcJamet/ia-qa-eval. LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key. This page covers its hosted endpoint (https://www.ia-qa.com/mcp/eval).

### Is the IA-QA Eval & RAG — 38 tools MCP server safe to use?

IA-QA Eval & RAG — 38 tools scores 79 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the IA-QA Eval & RAG — 38 tools MCP server expose?

IA-QA Eval & RAG — 38 tools exposes 38 tools: get_testing_guidelines, find_tool, hallucination_check, run_eval_contract, run_semantic_tests, and 33 more. Their descriptions and schemas cost roughly 6,986 tokens of context every time the server is loaded.

### Does the IA-QA Eval & RAG — 38 tools MCP server require authentication?

No. We connected to IA-QA Eval & RAG — 38 tools without credentials and it answered, so anything it exposes is reachable by anyone who knows the address.

### Is the IA-QA Eval & RAG — 38 tools MCP server still maintained?

IA-QA Eval & RAG — 38 tools is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

## Links

- Remote endpoint: https://www.ia-qa.com/mcp/eval
- Repository: https://github.com/ia-qa-com/mcp
- Changelog RSS feed: https://verifymcp.io/servers/jcjamet-ia-qa-eval/mcp-eval.xml
- Changelog JSON feed: https://verifymcp.io/servers/jcjamet-ia-qa-eval/mcp-eval.json
- HTML version of this page: https://verifymcp.io/servers/jcjamet-ia-qa-eval/mcp-eval