IA-QA — 130+ QA & Dev Tools for AI Agents

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

L2-normalize a float vector (produce a unit vector with norm=1). Required by many vector DBs (Pinecone, Qdrant cosine). Supports batch normalization of up to 1000 vectors.

Normalize whitespace: trim trailing spaces, collapse blank lines, normalize line endings (LF/CRLF), convert tabs to spaces. Useful for cleaning code, configs, and text before processing.

Convert numbers between bases: decimal, binary, octal, hexadecimal, or any base 2–36. Auto-detects 0x, 0b, 0o prefixes.

Validate the structure of an OpenAPI 3.x specification (JSON or YAML). Checks required top-level fields (openapi, info.title, info.version, paths), validates each operation (responses, operationId uniqueness), detects undeclared $ref components, and flags missing 2xx responses. Returns a PASS/FAIL verdict, a 0–100 compliance score, and a list of errors and warnings with JSON-pointer locations. Use before publishing an API spec or generating SDK code.

Compress an LLM prompt by removing filler words, verbose phrases, duplicate sentences, and unnecessary whitespace. Returns optimized text with token savings breakdown. 100% deterministic, no API key needed.

Parse a CSV string into a JSON array of objects (or raw arrays). Handles RFC 4180 quoted fields, escaped quotes, and custom delimiters. Use when processing spreadsheet exports, data imports, or structured text pipelines where the source is CSV. Supports up to 200 KB.

Parse a raw HTTP headers block into a structured JSON object. Detects multi-value headers, masks Authorization values, and optionally audits for missing security headers (HSTS, CSP, X-Frame-Options, etc.).

Post the output of jira_to_test_suite as a formatted comment on the source Jira ticket. Converts Gherkin, E2E steps, API tests, and ambiguities into Atlassian Document Format (ADF). STATEFUL — creates a comment on the issue.

Compound quality gate for pull requests. Runs three sequential checks: (1) secret detection — scans diff for API keys, tokens, passwords matching 16 regex patterns; (2) bug analysis — heuristic scan for eval(), innerHTML, empty catch, console.log, TODO/FIXME; (3) commit message linting against Conventional Commits spec. Returns gate verdict (PASS/WARN/BLOCK), blockers, and actionable warnings. Use before merging any code change.

Scan user input or prompts for common prompt injection patterns. Detects system prompt overrides, jailbreak attempts, role manipulation, encoding tricks, delimiter attacks, template/interpolation injection ({{...}}, ${...}), and context-exfiltration attempts ("repeat everything above").

Fill a prompt template with variables. Supports {{variable}} syntax and {{#if key}}...{{/if}} conditional blocks. Returns the filled prompt and lists unfilled variables.

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Ratings are aggregated on a public dashboard at /devtools/mcp-ratings. Skip rating routine successes — we want signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

Automatically detect and redact Personally Identifiable Information (PII) from text. Replaces emails, phone numbers, SSNs, credit cards, IP addresses, and JWT tokens with [REDACTED_TYPE] placeholders. Safe to use before logging or sending to an LLM.

Test a regular expression pattern against an input string and return all matches with their index positions and named capture groups. Use for validating user inputs, extracting structured data from text, or debugging regex patterns. Supports flags g, i, m, s, u, y.

Evaluate RAG retrieval quality using the NVIDIA neural reranker (nv-rerankqa-mistral-4b-v3). Ranks passages by semantic relevance to a query and computes Precision@k and Recall@k. Optionally accepts ground-truth relevance labels to produce a PASS/FAIL CI/CD verdict.

Score an LLM response on multiple quality dimensions: relevance, completeness, clarity, conciseness, formatting. Returns a weighted 0-100 score with detailed breakdown.

Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.

Review triage for a pull request. Takes a unified git diff (`git diff HEAD`) and returns: diff-lint findings with the lines that produced them, regression impact areas, a risk score 0–100 with the factors that built it (churn, files touched, sensitive paths, whether any test file changed, lint severities), generated test cases, and a PASS / CONDITIONAL / BLOCK recommendation. Advisory: the score measures properties of the diff, not the correctness of the change — it does not read the code semantically and does not replace a reviewer or a static analyser. See notAnalysed in the response.

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL verdicts, a pass rate, an overall PASS/WARNING/FAIL verdict (customizable threshold), and latency stats. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains (TF-IDF cosine similarity ≥ 0.4). BYOK: requires your own API key for the target provider.

Compare multiple VLMs on the same test suite in parallel — send an image (URL or base64) + N test cases to all models simultaneously. Returns per-model PASS/FAIL verdicts, pass rates, latency stats, and a comparison table. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains. BYOK: requires API keys for each provider.

Analyze a webpage <head> HTML (or full HTML) for GEO (Generative Engine Optimization) signals. Returns a score /60 with per-check results and improvement tips. GEO = optimizing pages for AI-powered search engines (ChatGPT Search, Perplexity, etc.).

Search Jira using JQL (Jira Query Language). Returns matching issues with key fields. Ideal for finding open bugs, sprint tickets, or issues by label/assignee/component. BYOK — credentials transit in-memory only, never stored.

Scan text or code for leaked secrets: API keys (AWS, GCP, Azure, OpenAI, Anthropic, Stripe, GitHub, GitLab, Slack, Twilio, SendGrid, HuggingFace), private keys (RSA/EC/PGP), JWTs, database connection strings, Bearer tokens, and Basic auth headers. Returns a list of findings with type, severity, line number, and a redacted preview. Use before committing code, sharing logs, or sending text to an LLM. 100% regex-based, zero network calls.

Analyse the HTTP security headers of a public URL OR of raw response headers you paste in. Grades each header (A–F) for: Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, X-XSS-Protection, Cross-Origin-Opener-Policy, Cross-Origin-Resource-Policy, and Cross-Origin-Embedder-Policy. Returns an overall score (0–100), per-header grades, missing headers, and fix snippets for Express, Nginx, and Apache. For localhost/private targets the remote server cannot reach, pass the `headers` parameter instead of `url`.

Run a comprehensive AI guardrail analysis on an LLM response. Orchestrates 6 deterministic safety checks plus an optional LLM-powered deep analysis in parallel: hallucination detection (grounding score), prompt injection scan, toxicity scan, output validation (PII/safety), guardrail rules, response quality scoring, and AI verdict (via Qwen, Gemma, Llama, etc.). Returns a unified PASS/FIX/BLOCK verdict with a 0-100 safety score, per-check results, and actionable fix recommendations. Use this as a single-call safety gate before surfacing any LLM output to users.

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

Sort, deduplicate, reverse, or filter lines of text. Useful for cleaning import lists, dependencies, log files, and config entries.

Split text into chunks of at most N tokens (cl100k_base: ~4 chars/token) with optional overlap. Designed for RAG ingestion pipelines.

Analyse the SSL/TLS certificate of any HTTPS host. Returns certificate subject, issuer, validity dates, days until expiry, protocol version, cipher suite, key exchange info, and an overall grade (A+, A, B, C, F). Detects expired, self-signed, and weak certificates. Use this to audit TLS posture before production deployment or during security reviews.

Strip all Markdown formatting (headers, bold, italic, code fences, links, lists) from text and return clean plain text. Run this before injecting scraped documentation, README files, or user content into an LLM prompt to eliminate redundant markup tokens and reduce cost.

Build a structured system prompt from components: role, task, constraints, output format, tone, language, and examples. Generates a production-ready system prompt with token estimate.

Validate a SKILL.md definition (Cursor / GitHub Copilot / Windsurf) by auto-generating trigger-positive and trigger-negative scenarios, running each through the model with the skill injected as a system prompt, and scoring trigger accuracy + step adherence. Returns a PASS/FIX/BLOCK verdict with per-scenario breakdown. Uses Groq llama-3.3-70b by default (server key, no api_key needed). Pass api_key + model to use your own provider.

Compute comprehensive statistics for any text: character count (with and without spaces), word count, line count, sentence count, paragraph count, and estimated reading time in minutes. Use for validating form field lengths, evaluating LLM output verbosity, or content auditing.

Convert between Unix timestamps (seconds or milliseconds) and ISO-8601 / UTC date strings. Auto-detects epoch vs. millisecond format. Omit input to get the current time. Returns iso, unix_s, unix_ms, utc, date, and time fields.

Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.

Scan text for toxic language, bias indicators, profanity, and harmful content categories. Returns risk scores per category. Useful for LLM safety guardrail testing.

Transform a JSON array using common operations: pluck (extract specific fields), filter (by field value), sort_by (field), group_by (field), count_by (field), uniq_by (field). Useful for processing MCP tool results and LLM structured outputs.

Truncate text to at most N tokens (cl100k_base: ~4 chars/token) to avoid exceeding an LLM context window. Optionally keeps the end of the text instead of the start (useful for keeping recent conversation history). Reports whether truncation occurred and the estimated token count.