io.github.houtini-ai/lm

Send a task to a local LLM — a sidekick running on the user's hardware or a configured OpenAI-compatible endpoint. It does not consume the user's Claude quota. Trades latency for tokens: local inference is typically 3-30× slower than frontier models, so delegation wins when the task is bounded and self-contained. Good fit: • Explain or summarise code/docs you already have in context • Generate boilerplate, test stubs, type definitions, mock data • Answer factual questions about languages, frameworks, APIs • Draft commit messages, PR descriptions, comments • Translate or reformat content (JSON↔YAML, snake_case↔camelCase) • Brainstorm approaches before committing to one Less good when: the task needs tool access, depends on multi-file context you have not captured, or is quick enough for you to answer directly before the round-trip completes. Prompt tips (local models take instructions literally): (1) Send COMPLETE context — the local LLM cannot read files. (2) Be explicit about output format ("respond as a JSON array", "return only the function"). (3) Specific system persona beats generic — "Senior TypeScript dev" not "helpful assistant". (4) State constraints — "no preamble", "reference line numbers", "max 5 bullets". (5) Leave max_tokens UNSET — the server sizes the budget from the model's real context window. Tiny caps like 256 waste the model: reasoning burns the budget before any visible output. Routing picks the best loaded model automatically. Call `discover` to see what is loaded and, after the first real call, its measured speed. The footer shows cumulative tokens kept in the user's quota.

Send a code-specific task to the local LLM, wrapped with an optimised code-review system prompt. Temperature is locked low (0.2 or the routed model's hint) for deterministic output. Good fit: • Explain what a function/class does • Find bugs or suggest improvements • Generate unit tests or type definitions for existing code • Add error handling, logging, or validation • Convert between languages or patterns For best results: • Provide COMPLETE source — the local LLM cannot read files. • Include imports and type definitions so the model has full context. • Be specific: "Write 3 Jest tests for the error paths in fetchUser" beats "Write tests". • Set the language field — it shapes the system prompt and improves accuracy. Verify generated code compiles, handles edge cases, and follows project conventions before committing.

Like code_task, but the local LLM reads files directly from disk — source never passes through the MCP client's context window. Use when reviewing multiple files or a single large file. How it works: • Provide absolute paths. Relative paths are rejected. • Files are read in parallel (Promise.allSettled) — one unreadable file does not sink the call. • Files are concatenated with `=== filename ===` headers and sent to the same code-review pipeline as code_task. • Read failures are surfaced inline with the reason so the LLM can still reason about the rest. • Pre-flight prefill estimate: if measured per-model data shows the input would exceed the MCP client's ~60s request timeout during prompt processing, the call is refused early with a diagnostic instead of hanging. Split or trim when this fires. Good fit: • Reviewing related files together (module + its tests, client + server pair) • Auditing a single large file too big to paste comfortably • Any code_task where keeping source out of the Claude context window matters Size guidance: on slow hardware (< 25 tok/s generation), keep total input under ~8,000 tokens (~32,000 chars) to stay safely under the client timeout. Faster hardware handles much more — the pre-flight estimator adapts once you've done a few calls and real per-model timings are in the SQLite cache. Same review discipline as code_task — verify the output before acting on it.

Structured analysis via the local LLM with explicit system/context/instruction separation. The 3-part format prevents context bleed in smaller models — the local LLM acknowledges the context in a fake assistant turn before receiving the instruction. Good fit when prompt structure matters: • Code review — paste full source, ask for bugs/improvements • Comparison — paste two implementations, ask which is better and why • Refactoring suggestions — paste code, ask for a cleaner version • Content analysis — paste text, ask for structure/tone/issues • Any task where separating context from instruction improves clarity Field guidance (each has a job — keep them focused): • system: persona + constraints, under 30 words. "Expert Python developer focused on performance and correctness." • context: COMPLETE data — full source, full logs, full text. Never truncate. • instruction: exactly what to produce, under 50 words. Specify format: "Return a JSON array of {line, issue, fix}." Review the output before acting on it — local model capability varies.

Check whether the local LLM is online and what model is loaded. Returns model name, context window size, response latency, and cumulative session stats (tokens offloaded so far). Call this if you are unsure whether the local LLM is available before delegating work. Fast — typically responds in under 1 second, or returns an offline status within 5 seconds if the host is unreachable.

Generate text embeddings via the local LLM server. Requires an embedding model to be loaded (e.g. Nomic Embed). Returns a vector representation of the input text for semantic search, similarity comparison, or RAG pipelines. Uses the OpenAI-compatible /v1/embeddings endpoint.

List all models on the local LLM server — both loaded (ready) and available (downloaded but not active). Shows rich metadata for each model: type (llm/vlm/embeddings), architecture, quantization, context window, and a capability profile describing what the model is best at. Use this to understand which models are available and suggest switching when a different model would suit the task better.

Show user stats: tokens offloaded, calls made, per-model performance — for the current session AND lifetime (persisted in SQLite at ~/.houtini-lm/model-cache.db). Unlike `discover` which includes the model catalog, `stats` returns just the numbers in a compact markdown table — cheap to call repeatedly to see the 💰 Claude-quota savings counter climb. Useful for quantifying how much work the local model is genuinely doing, and for noticing when a model's reasoning-token ratio is drifting.