llmshift

Test edited prompts: finds the system prompts that changed since a git base (default origin/main) and runs only their test cases, with the edited text, on the model in production. Tells you whether a prompt edit changes how the model answers. Use it after editing a prompt in the code. Calls the model once per affected test case.

Run the project's test set (made by llmshift_snapshot) on a model and report exactly what changed versus the current model's recorded answers: broken formats (invalid JSON, code fences), wrong or missing facts and numbers, tone and length changes, tool-call differences and refusals. Returns the pass rate, whether it meets the threshold, the changed cases with their failing checks, and HTML/JSON report files. Use it after changing a model ID or a prompt to prove nothing broke. It calls the model once per test case (cached on re-runs).

Check the llmshift setup: version, the config file in use, which provider credential environment variables are set (names only, never values), whether the test set exists, and whether the configured baseline, candidate and judge models resolve. Use it when another llmshift tool fails with a configuration or credentials error.

Search for system-prompt changes that make a new model reproduce the current model's behaviour on the test set, validated by re-running the tests on the new model. Returns, per prompt template, the instructions to add or the full suggested prompt, where that prompt lives in the code (file:line), and the pass rate before and after. It never edits code: apply the prompts you agree with, then call llmshift_compare again.

Look up retirement dates and replacements in llmshift's model lifecycle registry (OpenAI, Anthropic, Google, Mistral, Cohere, xAI, DeepSeek). Pass `model` for one exact ID (Bedrock, Vertex, Azure and OpenRouter forms are understood) or `query` to search, optionally filtered by provider, status or days until retirement.

Find the cheapest model that still passes the project's test set: runs the tests on several candidate models at once and reports each one's pass rate (with a 95% range), cost per 1,000 requests and latency, then recommends the cheapest that meets the quality bar, with the saving against today's model (per month when `volume` is given). Calls every candidate once per test case (cached on re-runs).

Check whether a new model is riskier than the current one: refusals on the project's real requests, prompt-injection probes (hidden instructions in messages, quoted documents or tool results) and leaks of confidential data from the system prompt, all built from the project's own test cases and run on both models. Reports old vs new rates, the change, a paired significance test and examples. Calls both models.

Find every AI model ID the code in this project calls (Python, JS/TS, Go, Java, configs, .env, notebooks) and report each one's lifecycle: retired, retiring (date and days left), deprecated or active, the recommended replacement, and every file:line where it is used. Use it first when asked to migrate off a model, to find all the places to change, and again afterwards to confirm that no retired model remains.

Turn LLM usage logs (JSONL, JSON or CSV, optionally .gz: OpenAI chat/responses/batch, Anthropic, Gemini, Helicone, Langfuse/LangSmith exports, llmshift.record logs) into a regression test set: real requests, the current model's answers, and checks inferred from them. Run it once before switching models; llmshift_compare and llmshift_fix use it.

Plan (and optionally apply) the code changes that move this project off a retiring model: swaps the model ID exactly where llmshift_scan finds it, renames max_tokens to max_completion_tokens where OpenAI's newer models need it, and lists settings to check by hand. Returns a unified diff. Nothing is written unless apply is true. Use `auto` for every retiring model that has a known replacement.