Back to the index · Previous: ML Models
Step by step with screenshots: Demo Agents — a practical guide builds a space, a trusted answer, a benchmark, an agent that asks the space, versions through the test gate, a supervisor, answer quality, and the move to the next environment, on one Gold table.
How it works#
Agent Studio lets anyone build their own assistant for one job, such as a Sales analyst, a Data quality helper or a Policy assistant that answers from your documents. An agent is three things:
- a name and instructions: who it helps, what it answers, how;
- the tools it may use, ticked from DataSimple's own list (read a table, look up a metric, check data quality, explain a model, search documents, …);
- whether it searches documents.
Five rules hold for every agent:
- It runs as the person who asks. Their grants, row filters and masks apply. Sharing an agent never shares access to data.
- Only the tools you ticked. The agent is offered those tools and no others; a call to any other tool is refused.
- Changes wait for approval. A tool that changes something (run a pipeline, train a model, write a table) is queued in Agent → Approvals. Nothing changes until a person approves it.
- The platform rules come first. An agent never deletes data or changes table structure, whatever its instructions say.
- Every answer is traced. Each step (the model's turns, every tool call, its input, output and time) is kept, with the tokens used and the cost.
Build an agent#
- Left rail → ML & AI → Agent Studio → New agent.
- Start from a template (Sales analyst, Data quality helper, Policy assistant, ML helper) or from scratch.
- Write the instructions, and keep them specific: "Answer only about retail sales from Gold. Always say the table and the period. If a number is not in the data, say so."
- Tick the tools. All read tools is a quick start; tools marked needs approval propose changes.
- Tick Search the workspace documents to let it answer from files (see Documents below).
- Add a few starter questions and press Create agent.
A new agent starts as version 1, in Production. From then on, Build edits a draft (see Versions below): press Save draft, and say in one line what changed.
Chat#
Chat keeps each conversation. The owner chooses whether to talk to the Draft or to Production (the draft is the default while there is one); everyone else always talks to Production. Each answer shows its steps and cost; 👍 / 👎 records whether it helped, and the counts show on the agent card. Proposed changes appear under the answer as Waiting for approval.
Test it#
Tests holds lists of questions with what a good answer must contain. There are three ways to check an answer:
| Check | Passes when |
|---|---|
| contains | the answer contains every part (separate parts with ;, e.g. East; 75) |
| exact | the answer is exactly the expected text |
| judged by AI | a grader model decides whether the answer says what a good answer must say |
Run tests asks every question and shows the pass rate, the cost and each answer with its verdict. For the owner it asks the draft when there is one, else Production; each run shows the version it asked (for example v3). A question the agent could not answer at all (an error, an empty answer, or no verdict from the grader) is marked with a warning sign: it counts as an error, not just a fail. Test runs never queue changes: a tool that would propose one reports what it would have queued. Run the tests after every change to the instructions or tools; the latest pass rate shows on the agent card.
Versions#
Changing an agent no longer changes it for everyone at once.
- Draft. Every save on Build goes to the agent's one draft. The name and description change right away; instructions, tools, documents and starter questions wait in the draft.
- Production. People the agent is shared with, and every API call, always get the Production version. Each answer and test run records the version it used.
- Promote. On Versions, Promote asks every test question to the version first (the eval gate), then makes it Production. The version that was Production is archived. No version is ever deleted.
- Roll back. On an older version, Roll back to vN makes it Production again, after the same check.
- Check. Runs the gate without promoting.
The eval gate#
The gate passes when the version's pass rate, over every question of every test set, reaches the pass-rate bar:
- the bar set on Versions (for example 80%), or
- when no bar is set, the current Production's last pass rate on the same questions. A version must do at least as well as what people use today. If Production has never answered a test set, it is asked once to get that number.
The gate fails closed. It is blocked when:
- the agent has no test questions;
- any question errored, got no answer, or got no verdict from the grader, whatever the pass rate;
- the pass rate is below the bar.
When the gate blocks, the result shows each question and why it failed. The owner (or an administrator) can still Promote anyway, but only with a reason, which is kept with the decision. Right after a blocked check, promoting anyway uses that check and does not ask the questions again.
Gate history lists every check: when, which version, pass rate, bar, questions passed and errored, who, whether it came from the app or the API, and the reason for a forced promotion.
Check a draft from a build pipeline#
POST /api/agent-studio/agents/:id/versions/:v/gate runs the gate and never promotes. :v is a version number, draft or production. Call it with a personal API token (Settings → Personal API tokens) of the agent's owner:
curl -s -X POST -H "Authorization: Bearer $DATASIMPLE_TOKEN" https://<your DataSimple address>/api/agent-studio/agents/12/versions/draft/gate
The answer:
{ "success": true, "passed": false, "decision": "blocked", "passRate": 0.75, "threshold": 0.9,
"why": "Pass rate 75% is below the bar of 90%.",
"results": [ { "set": "Main questions", "question": "Orders in East last week?", "passed": true, "error": false, "reason": "contains every expected part" } ] }
Fail the build step when passed is false. The call waits until every question is answered. The check is stored in the gate history, marked API.
Scoring with an ML model#
Tick score_with_model to let an agent answer "what would the model say for this customer?". It scores one row with the Production version of an ML model (see ML Models), exactly like Try your own values there: the prediction, the probability for a class, and how much each column pushed it. Columns the row leaves out are listed as missing. It runs as the person who asks, so they need access to ML Models.
Answer quality#
Answer quality shows how well an agent answers real questions, what it costs, and where it goes wrong.
Live judging. After an answer has been sent, a share of live answers (default 5 %; set 0 to turn it off) is graded by an AI judge. The person asking never waits for it. The judge reads the question, what the agent's tools returned, and the answer, and scores each from 1 to 5:
| Score | Asks |
|---|---|
| Correct | Does the answer agree with what the tools returned? |
| Grounded | Is every number, name and table backed by tool output or a quoted document? |
| Relevant | Does it answer the question that was asked? |
The three make one score from 0 to 100. Test runs are not judged here; they have their own checks.
What the tab shows
- Score, last 7 days against the 7 days before, the thumbs up and down, and the cost per question.
- Charts by day: the judge score (a gap is a day with nothing judged), thumbs up and down, and dollars per question. Hover a day for its numbers: the three scores, how many answers were judged, tokens per question, the day's total, and what tests and judging cost that day.
- Worst answers: answers scored under 60 or given a thumbs down, worst first, with the judge's reason. Trace opens what the agent did: each tool call, its input and what it returned. The owner can press Judge this answer now for an answer that was not in the sample.
- Add to tests: one click on a bad answer, type what a good answer must say, pick the check (judged by AI, contains or exact) and the test set (by default a set called From live answers). The question goes in exactly as it was asked. Run the tests after you fix the instructions.
- Alerts: when the 7-day score falls more than the amount you allow (default 15 points) below the 7 days before, the agent gets an alert at the top of the tab until someone acknowledges it. At least 3 judged answers are needed in each week before an alert is raised.
Settings (owner or administrator): the share of answers to judge and the alert drop, under Judging and alerts.
Cost per agent. Every question's tokens come from the agent's run. Dollars use the token prices set for the server (by default $3 per million input tokens and $15 per million output tokens). When agents carry versions, the cost is also shown per version.
Suggested fixes#
The Suggested fixes tab turns bad answers into tested changes. When someone gives an answer a thumbs-down, or the judge scores it under 60, the assistant reads the question, the answer, what the agent's tools returned and the agent's instructions, and proposes one change with a one-line reason:
- change the instructions (shown as a diff against today's), or
- add a test question, when the instructions are not the cause.
When the right answer is known (a reviewer's note, or you type it in), it also brings the question as a test.
Each card shows the question, the answer that was given, the problem in plain words and the change. Click Test & apply or Reject:
- Test & apply never ships an untested change. It saves the change as the agent's draft version, adds the test question to the test set Suggested fixes, and runs the eval gate. Only if the gate passes is the draft promoted to Production. If it does not, Production stays as it was, the draft stays for you to look at, the test question is taken out again, and the card shows the gate's pass rate and the questions that failed.
- A suggestion that only adds a test question changes no answer, so it is added as it is.
- Reject marks it rejected; nothing changes.
You can also pick a worst answer and click Suggest yourself. Suggestions are made in the background and never slow an answer down. At most one is open per answer, at most 20 a day for one agent, and Suggest automatically switches them off. Only the agent's owner or an administrator sees the tab.
What it costs: one model call per suggestion, and on Test & apply one gate run (about one answer per test question).
Supervisor#
A supervisor is an agent whose tools are other agents. Use it as one front door: "Ask the Front desk" and it sends sales questions to the Sales analyst and policy questions to the Policy assistant.
- New agent → Kind: Supervisor: asks other agents.
- Tick the agents it may ask (up to 10): your own, and agents shared with you.
- Write instructions on how to route if it needs them ("Sales numbers go to Sales analyst; anything about refunds goes to Policy assistant"), and press Create agent.
How it works:
- It routes each question to the right agent (or several, for a question with several parts) and combines their answers, saying which agent each part came from. Under the answer, Answered by names them, and the trace shows each call.
- Each agent runs as the person who asks. A person only reaches the agents shared with them: an agent in the supervisor that is not shared with them is skipped. Their grants and approvals apply as usual: a change an agent proposes waits in Agent → Approvals.
- Two levels at most, no loops. A supervisor may ask another supervisor, whose agents must be plain agents. A supervisor never asks itself or an agent already answering higher up, and a save that would make a loop is refused.
- The cost of an answer includes the agents it asked. Each asked agent's own run also shows in that agent's Answer quality tab.
Answer receipt#
Under every answer in Chat, next to the thumbs, Receipt shows how the answer was made:
- Who asked and when, the question and the answer.
- Tables read, from the SQL the agent ran: when each table's data last changed and by whom or what, the tables it is built from one step up, whether a row filter or column mask applied to you, and its latest data-quality result.
- The SQL of every query, with its row count or error. For a supervisor this includes the queries of the agents it asked.
- Versions: the agent and the version that answered; for a supervisor, which agents answered and with which version.
- Changes proposed and their approvals: every change the agent asked to make (a write, a grant, a schedule…) with its state: waiting for approval, approved (by whom, when), rejected or failed.
- Cost: dollars, tokens, steps and time taken.
- Verified: chain intact. Receipts can only be added, never changed or removed; each carries a fingerprint of its content and of the receipt before it, so one changed afterwards shows Not verified.
Governed actions. An agent never changes anything by itself: every change it proposes waits for a person. Right after an answer that proposed changes, the chat lists them with their state, and people allowed to decide see Approve and Reject (the same rules as Agent → Approvals: nobody approves their own request unless they are an administrator, and table changes need the right privileges). The receipt shows the same list.
A receipt is shown to the person who asked, the agent's owner and administrators, inside the same workspace only. It adds no model call: a few small reads of the application database after the answer is returned, and the answer never waits for it.
Share it#
Share gives the agent to a user, a group, or everyone in the workspace. They chat with its Production version and can run its tests against Production; only the owner (or an administrator) can change, promote or roll it back. Agents are never visible to other workspaces.
Export and import as YAML#
An agent can be saved as a file and brought into another workspace or environment.
- Export YAML (on the agent) downloads
<agent name>.yaml: its instructions, tools, documents on/off, starter questions, shares and test sets. No ids, owner, conversations, answers or secrets. - Import YAML (top of the page): paste a file or Choose file, then Preview changes. The preview says whether it creates a new agent or changes an existing one with the same name, and lists each field that changes, now and after. Only the agent's owner or an administrator can change an existing agent. Create agent / Apply changes saves it; when Agent Studio keeps versions it is Save as draft: a new draft version, Production untouched.
- A file that does not read, has no tools, or carries an id or something that looks like a password or key is refused with the reason; nothing is saved. Tools this workspace does not have are left out and named.
- With Git integration switched on, every agent is also a file on the environment's branch (
agents/<name>.yaml) and a Git badge marks it in the list. Agents also move with the Deployment screen. See Git integration and Deployment.
Documents#
Documents holds the files agents can search: PDF (with a text layer), text, Markdown, CSV, JSON or HTML, up to 20 MB each.
- Each file is split into passages. Search ranks them by keywords, and also by meaning when the server has an embeddings key.
- Agents that search documents quote the passages and cite file and page.
- A file can be visible to everyone in the workspace or only me. Removing a file removes its passages.
- Try a search shows exactly what an agent would find.
What it costs#
Agent answers and test runs are billed by the language model provider per token; Agent Studio shows the estimated cost of every answer and test run. A gate check costs about one answer per test question (twice that the first time, if Production has never answered the test sets); promoting anyway right after a blocked check costs nothing more. Live judging adds one model call per judged answer (about 2,000–5,000 tokens, under 3 US cents at the default prices); at the default 5 %, an agent asked 1,000 questions a month costs about 50 judge calls, under $1 a month. A supervisor answer costs its own call plus each agent it asks (each answers with its Production version). Document search runs in the application. Adding documents costs embeddings only when an embeddings key is set (about one US cent per 100 pages). No extra servers are started.
API#
| Method and path | What it does |
|---|---|
GET /api/agent-studio/tools |
the tools an agent may be given |
GET /api/agent-studio/agents / POST /api/agent-studio/agents |
list the agents you can use / create one |
PATCH /api/agent-studio/agents/:id |
name, description, gateThreshold (0 to 1, or null); instructions, tools, useDocuments, starterQuestions, model, versionNote go to the draft |
POST /api/agent-studio/agents/:id/ask |
{question, threadId?, version?}; API calls always get Production |
POST /api/agent-studio/agents/:id/evals |
save a test set |
POST /api/agent-studio/agents/:id/evals/:setId/run |
run a test set {version?} |
GET /api/agent-studio/agents/:id/versions |
versions, the Production version, the pass-rate bar and gate history |
POST /api/agent-studio/agents/:id/versions/:v/gate |
run the gate only: {passed, passRate, threshold, results} |
POST /api/agent-studio/agents/:id/versions/:v/promote |
gate, then make :v Production; {force: true, reason} to promote anyway |
POST /api/agent-studio/agents/:id/shares |
{add: [{kind, principal}], remove: [...]} |
POST /api/agent-studio/runs/:runId/feedback |
{rating: "up" | "down"} |
GET /api/agent-studio/agents/:id/quality?days=30 |
daily scores and thumbs, worst answers, alerts, settings |
POST /api/agent-studio/agents/:id/quality/settings |
{samplePct: 0-100, alertDrop: 0-1} |
GET /api/agent-studio/agents/:id/quality/runs/:runId |
one answer's trace and verdict |
POST /api/agent-studio/agents/:id/quality/to-test |
{runId, expected, checkKind?, setId?} |
POST /api/agent-studio/agents/:id/quality/alerts/:alertId/ack |
acknowledge an alert |
GET /api/agent-studio/agents/:id/cost?days=30 |
tokens and dollars per question by day |
POST /api/agent-studio/agents with {kind: "supervisor", memberAgents: [ids]} |
create a supervisor |
GET /api/agent-studio/documents / POST /api/agent-studio/documents |
list files / add one {name, mime, data: base64} |
POST /api/agent-studio/documents/search |
{query} |
GET /api/agent-studio/agents/:id/spec |
the agent as YAML {fileName, yaml} |
POST /api/agent-studio/specs/preview |
{yaml} → create or change, and the fields that change |
POST /api/agent-studio/specs/import |
{yaml} → created / updated (a draft when versions exist) |
GET /api/agent-studio/specs/git |
which agents have a file on this environment's Git branch |