Datris

List all columns and their data types for a specific PostgreSQL table. Use this to understand table structure before writing a query_postgres SQL query.

List all PostgreSQL databases available in the Datris platform. Use this as the first step when exploring what data has been ingested into PostgreSQL destinations.

List all schemas in a PostgreSQL database. Schemas organize tables within a database (e.g., 'public', 'analytics'). Use after list_postgres_databases to drill into a specific database.

List all tables in a PostgreSQL schema. Set vector_only=true to show only pgvector embedding tables, or false (default) to show regular data tables. Use after list_postgres_schemas.

List all collections in the Qdrant vector database. Use this to discover available collections before running search_qdrant.

List the version HISTORY of the script the pipeline runs for its AI data-quality rule or AI transformation (`kind`), newest first. Each entry has `version`, `createdAt`, `createdBy`, `changeNote`, `origin` (save, regenerate, run, restore-version, restore, repository), `model`, `commitSha` (repository-backed scripts), `fingerprintMatchesCurrent` (written for the pipeline's current instruction and source schema) and `isLive` (the script runs now use). Every change to the script is kept; the pipeline's definition version does not change when its script does. Read-only. See also `get_script_version`, `diff_script_versions`, `restore_script_version`.

List the names of tap secrets that already exist (secrets tagged _type=tap — both agent-created and human-owned ones surfaced via the UI's Tap Secrets section). ALWAYS call this before create_tap_secret: if a suitable secret already exists, prefer reusing it by passing its name as secret_name to create_tap. Only ask the user to provide credentials when no existing secret covers the need.

List the change HISTORY of a tap's definition, newest first. Each entry has `version`, `createdAt`, `createdBy`, and `changeNote`. The platform snapshots a tap's config + script every time it is created or updated. Read-only. IMPORTANT: an EMPTY result does NOT mean the tap has no version — it means the tap has not been edited since versioning was enabled, so no change snapshots exist yet. The tap's CURRENT version number is the `version` field on the tap itself (from `list_taps` or `get_tap`), which is at least 1 for every tap. Use this tool for 'what changed and when'; use `list_taps`/`get_tap` for 'what version is it on now'. See also `get_tap_version`, `diff_tap_versions`, `restore_tap_version`.

List all taps with their status, target pipeline, schedule, and last run info. CALL THIS FIRST (alongside list_pipelines) on any data-related user request — the user's existing taps often reveal what's already being pulled and where, which short-circuits the entire 'should I suggest an external API?' conversation.

List all classes in the Weaviate vector database. Use this to discover available classes before running search_weaviate.

Send data and use AI to generate a comprehensive data profile: summary statistics per column, data quality issues detected, and suggested validation rules. Use the suggested aiRule when building a pipeline's dataQuality section. The server may withhold row values from the model (DATRIS_AI_SAMPLE_VALUES=false): the profile is then built from column statistics or structure only, sampleValues are empty, and the result carries valuesWithheld: true. If the AI model declines the request (a provider safety decline), the result carries aiDeclined: true and holds the server's column statistics only, with empty qualityIssues and recommendations: read that as no AI review, not as a clean file.

Run a read-only query against the Databricks workspace a pipeline loads into. Pass the pipeline name; the server resolves the workspace, credentials, SQL warehouse, and catalog from the pipeline's config — same connection the loader uses, credentials never leave the server. Only works for pipelines whose destination is Databricks (database with useDatabricks=true). Allowed statements: SELECT (WITH/CTE), SHOW, and DESCRIBE — LIMIT is auto-appended to SELECTs. Omit `sql` to preview the pipeline's destination table (the 'did my load land?' check). Metadata discovery uses the same tool — there are no separate list_databricks_* tools: `SHOW CATALOGS`, `SHOW SCHEMAS IN <catalog>`, `SHOW TABLES IN <catalog>.<schema>`, `DESCRIBE TABLE <catalog>.<schema>.<table>`, or query <catalog>.information_schema.columns. Queries run on the customer's SQL warehouse (which costs them compute) — keep them targeted and let the default LIMIT stand unless the user asks for more. If the warehouse is stopped, the first query auto-starts it and may take longer.

Query a MongoDB collection with optional filter and projection. Use list_mongodb_databases and list_mongodb_collections first to discover available data. Returns matching documents as JSON.

Ask a question in natural language about data in a PostgreSQL table. The AI generates a SQL query from the question and table schema, executes it, and returns the results. Use this instead of writing SQL manually.

Read rows from a pipeline's objectStore destination (Parquet files, ORC files, or an Iceberg table in MinIO or AWS S3). Pass the pipeline name; the server resolves the bucket, prefix, format, and credentials from the pipeline config — same code path the writer uses, so MinIO and S3 destinations both work. Returns up to `limit` rows as JSON objects keyed by column name, plus the resolved s3a:// path and format for transparency. For an Iceberg table the response also carries snapshotId (a decimal string, not a number) and snapshotTimestamp (ISO-8601) of the snapshot read; both are null for other formats. Returns 0 rows (not an error) when the pipeline has no successful runs yet. This is the right tool when list_pipelines shows objectStore as the destination — query_postgres / query_mongodb / search_* will not work against objectStore data.

Execute a read-only SQL SELECT query against PostgreSQL. Use the metadata discovery tools (list_postgres_databases, list_postgres_schemas, list_postgres_tables, list_postgres_columns) first to explore available data before constructing queries. Only SELECT is allowed; LIMIT is auto-appended if missing. Queries are cancelled if they run too long, so avoid full-table scans. For approximate row counts — "how many rows / records does table X have", overviews, or summarizing several tables — do NOT run exact `SELECT COUNT(*)` (it scans the whole table and can take many seconds on a large one). Instead read the planner estimate, which is effectively instant regardless of table size: `SELECT reltuples::bigint AS estimate FROM pg_class WHERE relname = '<table>'`. Use exact `COUNT(*)` only when the user explicitly needs an exact count of a known-small or filtered result.

Run a read-only query against the Snowflake account a pipeline loads into. Pass the pipeline name; the server resolves the account, credentials, warehouse, and role from the pipeline's config — same connection the loader uses, credentials never leave the server. Only works for pipelines whose destination is Snowflake (database with useSnowflake=true). Allowed statements: SELECT (WITH/CTE), SHOW, and DESCRIBE — LIMIT is auto-appended to SELECTs. Omit `sql` to preview the pipeline's destination table (the 'did my load land?' check). Metadata discovery uses the same tool — there are no separate list_snowflake_* tools: `SHOW DATABASES`, `SHOW SCHEMAS IN DATABASE <db>`, `SHOW TABLES IN SCHEMA <db>.<schema>`, `DESCRIBE TABLE <db>.<schema>.<table>`, or query <db>.information_schema.columns. Queries run on the customer's configured warehouse (which costs them compute) — keep them targeted and let the default LIMIT stand unless the user asks for more.

Rename a catalog: every tap and pipeline labelled with `catalog` is relabelled to `new_name` in one server-side operation, and the empty-catalog placeholder moves with it. Call this ONLY when the user has explicitly asked to rename that catalog — never to emulate a rename by calling set_catalog on each item. `new_name` must not already exist: a rename never merges catalogs and is refused if the target exists. To combine two catalogs, move their items with set_catalog. Uncataloged cannot be renamed, and `new_name` must use letters, digits, '_' and '-' only. Items that could not be moved come back under `failed` (the rest still move); report them to the user. API keys whose capabilities are scoped to the old catalog name are listed under `affectedKeys` — they keep the old scope and no longer match the renamed items, so tell the user an administrator may need to update those keys. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Restore a catalog archive produced by export_catalog as catalog `catalog` (the same or a new name). Call this ONLY when the user has explicitly asked to restore and named the archive to use. A restore never merges: it is refused (with the clashing names) when the catalog already exists or any archived tap or pipeline name exists anywhere on this install. mode=overwrite replaces only items already in `catalog` and needs `confirm` equal to the catalog name. It never writes secrets: missing ones come back under `missingSecrets` for the user to create. Restored taps are left disabled; tell the user to test and enable them. Items that could not be restored come back under `failed`; report them. Archived AI scripts are restored as-is unless `regenerate_scripts` is set; `scriptsImported`, `scriptsRegenerated` and `scriptsKept` (a live script left in place; both with the reason) say which were which. The response carries `sourceEnvironment` (where the archive was exported) and `targetEnvironment` (this install), plus a warning when they differ; tell the user. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Roll a pipeline back (or forward) to a prior definition version. APPEND-ONLY and SIDE-EFFECTING: reads snapshot `version` and writes it as a NEW latest version, preserving full history. 'Rolling forward' is the same call with a higher version number. Only call this when the operator has explicitly asked to restore/roll back this specific pipeline to a specific version. Report the new version and stop. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Make an earlier version of a pipeline's AI data-quality rule or AI transformation script the one the pipeline runs. APPEND-ONLY and SIDE-EFFECTING: the restore writes the chosen text back as a NEW latest version (no model call); the pipeline's definition version does not change. If that text is already the live script, nothing is written and the response carries `noop: true`. Refused when the pipeline's instruction or source schema no longer matches the version (HTTP 422 with the reason; nothing is changed), and refused with HTTP 409 when a repository-backed script file was edited since its recorded commit. Only call this when the operator has explicitly asked to restore this specific script version. Report the new version and stop. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Roll a tap back (or forward) to a prior definition version. This is APPEND-ONLY and SIDE-EFFECTING: it reads snapshot `version` and writes it as a NEW latest version (config + that version's script), preserving the full history — nothing is overwritten or lost. 'Rolling forward' is the same call with a higher version number. Only call this when the operator has explicitly asked to restore/roll back this specific tap to a specific version. It does NOT run the tap — after restoring, report the new version and stop; do not chain into `run_tap` unless separately and explicitly asked. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Run the platform's operational self-check and return a report: Vault token expiry, AI slot secrets complete, embedding model actually loaded, disk usage, the payload staging area (writable, room for one run, no orphaned run directories), component version skew, and (opt-in) whether each AI model answers. Each non-ok check carries a remediation command for the operator; nothing is changed. Do NOT call run_doctor as part of the normal workflow — it is slow. Only use it for diagnostics when something fails or the user asks about the deployment's health. Host-level checks (Docker volumes, container env drift) need `datris doctor` on the machine running Docker and are not included here.

Manually trigger a tap. Executes the script; when a target pipeline is configured, hands records to the pipeline async. The response carries `recordCount`, `publisherToken`, `pipelineTokens`, `persisted`, `persistedReason`, and the script's `logs` — but NOT the records themselves. If you need to preview what the script produces, call `test_tap` instead. BEFORE the first run of a newly-created or newly-updated tap, you MUST have called `test_tap` and seen it succeed. See the VALIDATION RULE in the server instructions. Skipping the test on a fresh script pushes potentially-broken data into the destination — and `run_tap` doesn't return records, so you won't see the breakage from the response. PER-RUN PARAMS — pass a `params` object to drive this run with caller-supplied values (date range, id list, page cursor, etc.). Each key/value becomes an env var the script reads via `os.environ.get('DATRIS_TAP_PARAM_<key>')`. Keys must match `[A-Za-z_][A-Za-z0-9_]*` so they map cleanly onto env var names. Values are stringified; nested objects/arrays are JSON-encoded (script can `json.loads()` them back). Use this for anything that varies per-call — date windows, id lists, page cursors, batch sizes. Do NOT rewrite the tap secret to pass per-run params: secrets are for credentials (API keys, DB passwords); rewriting them on every call clobbers concurrent runs, pollutes audit history, and wastes Vault writes. If the tap script doesn't yet read a particular param, update the script by calling `create_tap` again with the same `name` and a revised `script` (create_tap upserts and replaces the existing script). That's the right shape for parameterized runs. REQUIRED next steps based on the response: • `persisted: true` → load is still running. Call `get_pipeline_status(publisher_token=response.publisherToken)` and poll until `rollup.allDone` is true. Then read `rollup.status` (`success`/`warning`/`error`) and `rollup.jobs[].lastError`. Do not report completion or query the desti…

Semantic search across a Chroma vector database collection. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Semantic search across a Milvus vector database collection. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Semantic search across a PostgreSQL pgvector table using cosine distance. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. Use list_postgres_tables with vector_only=true to discover available pgvector tables. For RAG: pass the returned text to ai_answer.

Semantic search across a Qdrant vector database collection. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Semantic search across a Weaviate vector database class. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Set or clear the catalog grouping label on an existing pipeline or tap. ONLY call this when the user has explicitly asked to organize work under a named catalog. Do NOT call it proactively — catalogs are a user-chosen organizational convention; assigning one for them puts the pipeline/tap into a taxonomy they didn't ask for. Pass exactly one of `pipeline` or `tap` to identify the target. Pass `catalog` to set the label, or omit it (or pass an empty string) to clear it. Survives subsequent re-ingests — `datris ingest` no longer rewrites an existing pipeline's config. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Overwrite or reset a tap's incremental-sync state. Pass `state` (a JSON object) to set the bookmark the next run receives via DATRIS_TAP_STATE — e.g. rewind a cursor so a window is re-fetched. Pass `reset: true` to delete the state entirely: the next run sees no DATRIS_TAP_STATE and does a full first-run fetch. The state shape is defined by the tap's own script (read its code via get_tap to see what keys it expects). Only use this on explicit request or to recover a broken cursor — normal runs manage state themselves, committing it only after a successful run. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Suggest per-field protection (hmac, mask with an optional preserve, redact, drop, or none, each with a one-line reason) for a pipeline's source fields, computed by the configured CodeGen model from field NAMES and TYPES ONLY: no row value is sent to the model. Stateless: nothing is saved or changed. Pass `pipeline` (an existing pipeline name; its stored source schema is used) or `fields` (an array of {name, type}, e.g. the schema you are about to create). A field that already carries `protect` shows it as `current` next to the suggestion. Show the suggestions to the user and ask which to accept before passing any of them as the `protect` map to `create_pipeline`; never apply them on your own. With `preset` (delimited sources only) the server's rule-based preset classifier is used instead of the model: each field gets an identifier class, a method and a reason, and the result lists the fields it did not recognise and review notes; show all of it, including the review notes, to the user.

Test-run a tap without pushing data to the pipeline. Executes the tap's script and returns results, record count, and any errors. Use this to validate a script before running it for real. By default a test pulls the first 20 records and returns; `recordCount` from a test is the preview size, not what `run_tap` will produce. If the error reports exit code -9 (or 137, or "killed") with no traceback, the script ran out of memory because fetch() built its whole result in memory: rewrite fetch() to yield records one at a time (read the source in chunks / pages) instead of returning a list, then test again. For a large columnar or tabular source, `yield` each batch (a pyarrow `RecordBatch`, a pandas `DataFrame`, or a list of dicts) instead of each row — the platform serialises a batch natively, several times faster — and read only the columns the pipeline needs (`columns=[...]`). Yielding one row at a time remains correct for small or API-paged sources. A test is bounded by `TAP_SCRIPT_TIMEOUT_SECONDS` (default 300 s), which is deliberately shorter than the ceiling a real run gets — a test that times out on a large source does not mean `run_tap` will. A run that timed out returns the script's logs, its `[wrapper] streamed N records` progress lines and a partial record count. Read them first: a script that reached the source and was still streaming records when the timeout hit is healthy and too long, not wrong — do not rewrite it; tell the user how far it got and offer a smaller window via `params` or the operator's timeout setting. After two consecutive failed tests, stop and report the exact error text to the user instead of iterating further; never probe the runner environment or read the wrapper to diagnose. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pa…

Update an AI provider secret in the Datris platform. Use this to configure your AI API keys so Datris can use AI features (data profiling, schema generation, AI transformations, RAG). Only AI-related secrets can be updated: anthropic, openai, azure, grok, ollama, embedding. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Update an existing tap's CONFIG without regenerating the script. Change the enabled state, CRON schedule, target pipeline, or description. USE THIS to set or adjust a tap's schedule (`cron_expression`) whenever the user describes a recurrence (nightly, daily, every morning, market open, etc.). The platform's scheduler runs the tap on the cadence you set — no external cron, Airflow DAG, or shell loop is needed (or wanted). See the SCHEDULING RULE in the server instructions. VALIDATION RULE: if you're enabling a `cron_expression` on a tap whose script has NEVER been validated, call `test_tap` FIRST and confirm it succeeds. The cadence-change path is safe for taps that have already run successfully; it is not safe to set a cron on a never-tested script. The platform enforces this: setting `cron_expression` on an untested script returns HTTP 409 with the remedy — save the tap without the cron, call `test_tap` until it succeeds, then call `update_tap` again with the cron. Cadence-only changes on a tap whose current script has passed `test_tap` are accepted. To change the SCRIPT itself, call `create_tap` again with the same `name` and the new `script` or `instruction` — create_tap upserts by name and replaces the existing script. There is no separate script-only update tool. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Upload a configuration file to the Datris platform. Supports 'validation-schema' (JSON Schema files used in pipeline dataQuality schema validation). Send the file content as base64. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Upload data to a registered pipeline for processing. Send the ENTIRE file content as a single base64-encoded string in ONE call — do not pre-split or chunk the content client-side. Vector destinations (pgvector, qdrant, weaviate, milvus, chroma) apply recursive chunking server-side using the pipeline's configured chunkSize/chunkOverlap; for those, one upload_data call yields many embedded chunks automatically. The pipeline's rules are applied: schema validation, data quality checks, transformations, then routing to the configured destination. A Live Read (`scratch`) pipeline validates or transforms the file and hands the rows back (status rollup `resultPreview`, then get_pipeline_result) instead of landing them. Returns a pipelineToken for tracking job status via get_job_status. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Sleep for a fixed number of seconds, then return. Use this to pace polling against long-running platform work (most often `get_pipeline_status` after `run_tap`, or `get_job_status` after `upload_data`) so you do not burn tool calls hammering an endpoint that is still in progress. ALWAYS poll once BEFORE the first wait — many runs finish in 1–5 seconds. Then use exponential backoff between polls: 5s, 10s, 20s, 30s, 60s, 60s, ... (cap at 60s normally, 120s only if the run is genuinely glacial). Reset to a short wait (5–15s) on the next cycle whenever a poll shows new jobs flipped to a terminal state — that means you're close to done. If 80%+ of jobs are terminal, wait ~15s; if only 1–2 jobs remain, wait ~10s. Hard upper bound is 120 seconds per call — for longer waits, call this tool in a loop interleaved with a status check.