Everything is one page where a business person runs DataSimple by talking, typing, pointing, drawing or dropping. It covers the whole platform, from the first file that lands to production. Sarah is the voice you talk to. Behind her, two helpers do the work: Sarah answers business questions from Gold, and the agent runs the platform. Every change waits for a person with the right role to press Approve.
This chapter puts it all in one place:
- how a spoken request is handled;
- what keeps it safe;
- everything you can ask for;
- a full day with Sarah, from the morning brief to a deployment.
The screen itself (voice ring, held answers, Pen, drag-and-drop) is described in chapter 7, Everything.
How a request is handled#
you speak ─▶ Sarah hears the sentence (live voice, or the browser's voice)
│
├─ small talk ─────────────▶ Sarah answers in conversation
│
├─ a business question ────▶ Sarah (Gold only, your grants)
│ KPI tiles, charts, proof SQL, one spoken sentence
│
└─ a platform request ─────▶ the agent: reads catalog, loads, runs, quality, access
│
├─ reading: answers straight away
└─ a change: a "Needs your approval" card
└─ Approve (a click, by someone allowed)
└─ it runs as the approver, with their access
every step ─▶ Activity (who asked, how, who answered, what was approved or refused)
- Who answers. Plain keyword scoring decides, so the choice is instant and explainable. Business words (sales, revenue, top 10, by region) go to Sarah. Platform words (pipeline, schedule, grant, failed, star schema, metric, quality rule, certify, deploy) go to the agent.
@sarahor@agentat the start forces the choice, and each answer offers Ask the other one. - Conversation or data. With
TYPESAFE_API_KEYset, an independent check confirms whether a sentence is conversation or a request for data. Small talk never starts a lookup, and a data question never gets a made-up answer. - Held answers. While you talk, a finished answer waits off screen until you say show me.
- Three at once. Up to three questions run at the same time; a fourth waits for one to finish.
- Spoken answers. The answer is spoken in one short sentence, with the numbers it was built from.
What keeps it safe#
| Rule | What it means |
|---|---|
| The AI never deletes or restructures. | Sarah and the agent never drop, delete, truncate, alter, overwrite or replace, for anyone. Every SQL statement the AI writes is checked before it runs and again before it is saved. A person can still do it in Notebooks if their role allows (AI guardrail). |
| The AI builds new things only. | A pipeline build, star schema or OBT from Everything must use new table names. An existing table is never replaced. |
| Every change waits for a click. | A change appears as a Needs your approval card and nothing happens until someone presses Approve. You approve on screen, never by voice. |
| You cannot approve your own request. | Someone else approves it. An admin may approve their own, so a lone admin is not stuck; AGENT_ADMIN_SELF_APPROVE=false makes admins need another admin too. |
| Some approvals need a role. | See the next table. |
| It runs as the approver. | An approved action runs with the approver's session, so their grants, row filters and column masks apply. |
| Proof for every number. | Proof lists every query and its SQL; rerun it and you get the same figure. |
| Everything is recorded. | Activity, the audit log, agent_runs / agent_actions, and for objects the definition log (History). |
Who may approve what
| Card | May approve |
|---|---|
| Change data with SQL | Someone with MODIFY (or CREATE TABLE for a new table) on every table it writes, or an admin |
| Grant, revoke, row filter, column mask | The owner of the object (MANAGE) or an admin |
| Approve a business term or metric; certify a table; a quality rule enforced on write | A data steward, compliance officer, DPO or admin (NB_DQ_CERT_ISSUER_ROLES adds roles) |
| Deploy to QA / Prod | An admin |
| Create a group or change its members | An admin |
| Make someone the owner of a table | The table's current owner (MANAGE) or an admin |
| Add a user, change a role, lock or unlock; roll a table back; bring back a dropped table | An admin (a roll back or undrop can also only be asked for by an admin or super admin) |
| Back up a table | Someone with MODIFY on the table, or an admin |
| Everything else (pipelines, schedules, loads, draft definitions, quality rules, star schemas, reruns) | Anyone other than the person who asked, or an admin |
What you can ask for#
| Area | Say, for example | What happens |
|---|---|---|
| Business answers | Tell me today's sales; revenue by region this quarter vs last | Sarah answers from Gold: one sentence, then KPI tiles and charts chosen from the shape of the data, with What stands out and Proof. She uses approved metrics first, then an OBT if one has the columns, then a star schema; she says which answered. |
| Charts | show the labels; make it a bar chart; show it as a table | Controls the chart you named or clicked, else the latest one. |
| What-if | what if West drops 10%; solve for 40,000; save this as peak season | The scenario panel, ELS Analyst's engine. It is arithmetic on stored values, never a forecast. |
| A table's summary | drag a table onto the page | One sentence from the data, plus freshness, quality, keys, lineage and size. |
| Point at data | Pen on a preview: days between this and this as delivery_days | The code, with Open in notebook. |
| Meaning | what does gold.fact_sales mean?; draft the meaning of every Gold table | Business meaning, written by AI from sample rows. Every new table, in any layer, gets one automatically and it is approved by the workspace's standing decision (MEANING_AUTOAPPROVE=false to require a person). |
| Catalog and history | how was gold.obt_sales built?; what feeds gold.sales? | The definition log (every CREATE, ALTER, WRITE with its code, who and when) and lineage. |
| Access | give finance read access to gold.sales; mask email for everyone but HR | A grant, row filter or mask card for the owner to approve. |
| Loads | load s3://…/orders.csv into Bronze; the New file alert's Load | A DataBridge or file load card. |
| Pipelines and workflows | load orders and customers from our SQL Server into Silver and Gold every day at 6 | A saved Workflow (and schedule) or Pipeline. A summary rebuilt each run becomes a materialized view (gold.mv_*). |
| Star schema and OBT | build a star schema for silver.shop_orders with a customer dimension that keeps history, and a One Big Table | The plan drawn on the card (fact, dimensions, keys, OBT); saves a DS Tables notebook and a pipeline (chapter 15). |
| Metrics and terms | define a metric Net order value = total amount excluding negatives from gold.orders, approve it; define Active customer; retire the old Revenue metric; what is net order value by region? | Draft or approved definitions; the card shows the metric's value today; metric_value runs the agreed SQL. Details below. |
| Quality rules | amount must not be negative on silver.orders, make it an Error; run the checks on silver.orders; can we certify gold.sales?; certify it | Rules tried on the real table before saving, sent for a second person's review; checks run now; certification readiness and the certificate. Details below. |
| Groups | create a group finance with Ann and Raj; add Bob to finance; give finance read on gold.sales; make finance the owner of gold.sales | Groups are named like people in grants, row filters, masks and ownership; a member holds what the group holds. Creating a group or changing members: an admin approves. Also Governance → Groups. |
| Meaning, owner, audit | the net column of gold.sales means sales after returns, in USD; what changed today and who approved it? | Sets the business meaning of a table or column (card); reads what changed (objects, approvals with who, AI refusals, grants and groups). |
| Usage | what used the most compute this week?; what's slow?; which tables take the most storage? | Spark time, runs, rows and storage by user, table, job or day; the slowest runs against the usual for the same job. No money amounts (see below). |
| Users and roles | add Priya as a data steward; make Raj an analyst; lock John's sign-in | A card only an administrator can approve. Users are never deleted from Everything, and the AI never sees or sets a password. |
| Backup and roll back | show the history of gold.sales; back up gold.sales; roll gold.sales back to version 12; bring back the dropped orders table | History: every version with when and what. Backup: a full copy as a new table. Roll back and undrop: administrators only. Details below. |
| Watch and fix | what failed last night?; brief me; rerun the sales pipeline | The Morning brief card and its Rerun / Why?; a rerun card by voice. |
| Deploy | what is ready to go to QA?; deploy the sales pipeline to QA | New and changed items; a dry run on an admin-only card. |
| What happened | Activity | One timeline of questions, runs, approvals and refusals. |
Metrics and terms#
A term is an agreed definition of a business word, such as Active customer: bought in the last 90 days. It has synonyms and the tables and columns that represent it. A term is documentation and runs no SQL.
A metric is a named measure with the SQL that computes it. The SQL is either an aggregate expression over a table (SUM(CASE WHEN amount > 0 THEN amount ELSE 0 END) on silver.orders) or a full SELECT. A metric can also have a grain, a unit and default dimensions.
- Defining. Define a metric … shows a card with the name, definition, SQL and the value the metric gives today, computed before anyone approves it. The SQL must be read-only: one statement, no writes.
- Draft or approved. Without approve it the definition is saved as a draft. With it, the card needs a data steward or an admin. Approve the Net order value metric or retire the old Revenue metric changes only the status. Nothing is deleted, so old notebooks stay explainable.
- Using it. Once approved, Sarah uses the metric's SQL whenever the question needs it and says so under the answer (approved metric). The agent runs it by name with
metric_value, split by dimensions or filtered: net order value by region. - Where else. Semantics in the Lakehouse (chapter 9) shows and edits the same terms and metrics.
Quality rules and certification#
A rule is the condition every good row meets, in SQL: amount >= 0, customer_id IS NOT NULL, status IN ('open', 'closed').
- Creating. Amount must not be negative on silver.orders, make it an Error shows a card. Once approved, the rule is tried on the real table first: a misspelled column is refused with Spark's reason, and nothing is saved. Then it is saved and sent for review, and someone other than the approver approves it on the Data Quality page (four eyes).
- Levels. A Warning rule only records a failure. An Error rule stops the pipeline that runs it.
- Enforced on write. Keep bad rows out on every write quarantines failing rows on every write to the table, optionally failing the write above a share (
max_reject_ratio). Only a data steward or an admin can approve that. - Running. Run the checks on silver.orders runs the table's rules now. It reports what passed and what failed, and records the results on the Data Quality page.
- Certifying. Can we certify gold.sales? lists the checks, their coverage, anything blocking, and whether you may issue. Certify gold.sales shows a card that a steward, compliance officer or admin approves. The certificate records the table version, the checks that passed and the coverage. It is superseded when the data changes, and a table with a failing rule cannot be certified.
Usage, users, backup and roll back#
- Usage. Everything reads the usage record of every run: Spark seconds, duration, rows read and written, by user, table, job, kind or day. It also finds the slowest runs and compares each with the usual time for the same job. Money amounts are not shown. When they are added they will be visible to administrators only, priced from the cloud provider (AWS, Azure or GCP).
- Users and roles. Add Priya as a data steward, make Raj an analyst, lock or unlock a sign-in. Only an administrator can approve the card. A new account is created without a password anyone knows: an administrator sets it with Reset password on the Users screen. Everything never deletes a user.
- History. Show the history of gold.sales lists the table's versions, newest first, with when, the operation (write, merge, restore) and rows written. Use it to pick the version to go back to.
- Backup. Back up gold.sales copies the whole table into a new table,
gold.sales__backup_<yyyymmdd_hhmm>(UTC), noting the version it was taken at. Someone with MODIFY on the table, or an administrator, approves it. - Roll back. Roll gold.sales back to version 12 (or to 2026-10-25 18:00, UTC) is for administrators and super admins only. Only they may ask for it and approve it. A full backup is taken first, so a roll back can itself be undone. It works for Delta and Iceberg tables. On our dev system, rolling a demo fact back from 5 rows to its first version (3 rows) and then forward again worked, with a backup each time.
- Bring back a dropped table. Bring back the dropped orders table restores a table dropped within the retention window, with its data and history. Administrators only.
- Never. The AI never drops a table, deletes data or deletes a user, for anyone. Rolling back exists only as the card above, never as SQL. People delete or drop by hand, in Notebooks, only if their role allows it.
A day with Sarah, by voice#
This walkthrough follows one person, Maya, through a working day. Every step uses features described above. Where we ran the step on our dev system, the figures are the real ones (marked real). The rest show the shape of the answer.
Maya opens Everything and presses Let's talk.
1 · Morning brief. The page opens with the brief.
Maya: Morning, Sarah. What failed last night? Sarah: One thing failed: the ship mode workflow. DataBridge answered "tenant required". The other four runs finished fine.
The brief lists it with Rerun and Why? (real; the cause was fixed in this release).
Maya: Why did it fail? Sarah: The workflow called DataBridge without saying which tenant it was for. That's fixed in the new version; rerun it once it's deployed.
Maya presses Rerun failed on the card. It runs as Maya, with her permissions.
2 · A new file lands. A line appears under the box: New orders file arrived · Load it into Bronze?
Maya: Yes, load it.
A Load into Bronze card appears. Her colleague Raj approves it, because Maya cannot approve her own request.
3 · Shape it every day.
Maya: Load orders and customers from Bronze into Silver and Gold every day at 6.
The card describes a Workflow (Bronze → Silver → Gold with keys, checks and masks) and a 06:00 schedule. Once approved, it is saved, not run, and it shows on the Workflows screen.
4 · A star schema and a One Big Table.
Maya: Build a star schema for the orders in silver.dsdemo2_orders, with a customer dimension that keeps history, and a One Big Table. Sarah: I've drafted it: a customer dimension that keeps history on updated_at, an orders fact at one row per order, and an event-time One Big Table. All names are new. It's waiting for approval.
The card draws the star: star_demo_fact_orders in the middle, star_demo_dim_customer (history) on the customer_id line, and star_demo_obt_orders underneath (real: the agent's plan). After approval, a notebook Star schema · Star Demo Orders and a pipeline are saved. Built on Spark, the OBT showed customer C3 as West for a September order and East for an October one (real).
5 · Agree a metric.
Maya: Define a metric called Net order value: the total order amount, excluding negative amounts, from silver.dsdemo2_orders. Approve it. Sarah: I've queued Net order value: the sum of positive amounts. It gives 445 today; one order is negative, minus 5. Approving it needs a data steward.
The card shows the SQL SUM(CASE WHEN amount > 0 THEN amount ELSE 0 END) on silver.dsdemo2_orders (real). Priya, a data steward, approves it.
6 · Agree a term.
Maya: Define the business term Active customer: a customer who ordered in the last 90 days.
This produces a draft term card (real). Sarah adds that the table only holds three days of orders, so the window will match everyone for now.
7 · A quality rule.
Maya: On silver.dsdemo2_orders, amount must not be negative. Make it an Error. Sarah: Queued: amount ≥ 0, level Error. Today order 3 has minus 5, so the rule will fail until that row is fixed.
That is the real answer. After approval, the rule is tried on the table, saved, and sent to someone else to review on the Data Quality page.
Maya: Run the checks on silver.dsdemo2_orders. Sarah: One rule ran: amount ≥ 0 failed on 1 of 4 rows. It's on the Data Quality page.
8 · Certify a table.
Maya: Can we certify gold.fact_sales? Sarah: Not yet: it has no checks, so coverage is zero. Add rules first.
The rule exists for a reason: a green stamp over thin coverage is worse than none. Once rules pass, certify it produces a card for a steward.
9 · Ask the business questions.
Maya: What's net order value by customer? Sarah: By the approved Net order value: C3 200, C1 150, C2 95.
The agent ran the agreed metric split by customer_id, with no SQL written on the spot. The figures are computed from the dev data.
Maya: Show me order amount by region, as a bar chart. Sarah: East 295 and West 145, from gold.star_demo_obt_orders.
This answer comes from Sarah, who reads Gold only. She used the One Big Table because it already carries each order's region. Order 3's minus 5 counts against West, because this is order amount, not the net metric. The bars show both regions, and Proof lists the SQL.
Maya: Show the labels.
The labels appear on the bars.
10 · What if.
Maya: Let's do a scenario. What if West drops 10%? Sarah: With West down 10%, the total goes from 440 to 425.5, minus 3.3%.
The scenario panel moves West's slider. The figures are worked out from the bars above.
Maya: Save this as cautious plan.
11 · Share it safely.
Maya: Give the finance group read access to gold.star_demo_obt_orders.
A grant card appears for the table's owner to approve.
12 · Ship it.
Maya: What's ready to go to QA? Sarah: Four new items: the workflow, its schedule, the star schema pipeline and its notebook.
Maya: Deploy the star schema pipeline to QA.
A card shows the dry run, including anything missing on QA. Only an admin can approve it. New schedules arrive disabled on QA.
13 · Look back. Maya opens Activity. It shows every question she asked by voice, who answered, the cards she created, who approved each one, and the one request the guardrail refused (she had asked Sarah to clear out the test orders).
The same day typed instead of spoken works identically: Show chat opens the box.
Behind the scenes#
Agent tools added for Everything (all in server/lib/agent-tools.js; read answers at once, write makes a card):
| Tool | Tier | Does |
|---|---|---|
daily_brief, brief_rerun |
read, write | the morning brief; rerun a listed failure |
notebooks_find, workflow_build, pipeline_create |
read, write, write | find notebooks; save a Workflow; save a Pipeline |
deploy_status, deploy_promote |
read, write | what changed vs QA / Prod; deploy with a dry run |
star_schema_suggest, star_schema_create |
read, write | draft a star from the catalog; save its DS Tables notebook and pipeline |
audit_changes, groups_list |
read | what changed and who approved; the groups and members |
meaning_set, table_owner_set, group_create, group_members |
write | set a meaning; set an owner (MANAGE); create a group; add / take out members (admin) |
usage_summary, slow_runs, storage_by_table, table_history, users_list |
read | usage, slowest runs, storage, a table's versions, users and roles (admins) |
user_add, user_role_set, user_access |
write (admin) | add a user, change a role, lock / unlock |
table_backup, table_restore, table_undrop |
write | back up as a new table; roll back (admin, backup first); bring back a dropped table (admin) |
metric_value, term_define, metric_define, definition_status |
read, write, write, write | run an agreed metric; define terms and metrics; approve / retire |
quality_rules, quality_checks_run, certify_readiness |
read | a table's rules; run them now; can it be certified |
quality_rule_create, table_certify |
write | add a rule (tried first, then sent for review); certify |
object_history |
read | how an object was built and changed |
They sit beside the earlier tools for the catalog, lineage, quality, access, schedules, loads and pipelines.
APIs it uses. These are the same ones the screens use, so behaviour and audit are identical:
/api/everything/*: brief, rerun, activity, scenarios, table summary./api/notebooks/semantic/*: terms, metrics, preview, run, meaning./api/notebooks/dq/*: checks, validate-rule, results, certificates./api/pipelines,/api/workflows,/api/deploy/*,/api/notebooks/spark/*.
Settings an administrator sets
| Setting | Effect |
|---|---|
OPENAI_API_KEY |
live voice that hears and speaks at once (else the browser's voice in Chrome / Edge) |
TYPESAFE_API_KEY |
the independent conversation-or-data check |
FILE_ALERTS_ENABLE, FILE_ALERT_PREFIXES, FILE_ALERT_INTERVAL_MS |
new-file alerts |
MEANING_AUTODRAFT, MEANING_AUTODRAFT_SCHEMAS, MEANING_AUTOAPPROVE |
AI business meaning for every new table (every schema by default), approved automatically unless MEANING_AUTOAPPROVE=false |
NB_DQ_CERT_ISSUER_ROLES, NB_DQ_CERT_MIN_COVERAGE |
extra roles that may approve definitions and certify; minimum coverage to certify |
Good to know#
- Approve and Reject are clicks. Pointing with the Pen and dragging a table are on screen too. Everything else can be said.
- Who reviews a rule. A quality rule approved from Everything is still reviewed on the Data Quality page by someone other than the approver. That is the four-eyes rule of the Data Quality screen.
- The Semantics screen. Saving a term or metric directly on the Semantics screen (chapter 9) does not check roles, so anyone signed in can mark a definition approved there. Only Everything's cards are limited to stewards.
- Missing tables are not created from chat. Workflows deploy like other objects; creating missing lakehouse tables and views on the target, and copying their grants, is offered on the Deployment screen only (chapter 12).
- What is still to come. TODO.md lists what is still to come: money amounts for administrators, priced from the cloud provider; the agent's own Silver step doing de-duplication, type fixes and history.
Previous: 15. DS Tables · Next: 17. Data Vault · Back to the index