Glossary#
| Term | Meaning in DataSimple |
|---|---|
| Tenant | A customer workspace, identified by a three-digit id (001). Every row the platform writes carries it, and every query is scoped to it. |
| Catalog name | datasimple.<schema>.<table>. SQL cells also accept <schema>.<table>. |
| Layer | source (copy of a source), bronze (raw history), silver (clean state and history), gold (business-ready). |
| Source copy | DataBridge's copy of a source table, one row per key, stored as Parquet part files, Delta or Iceberg. The Parquet copy is named <schema>_<table>, the Delta copy <schema>_<table>_DELTA and the Iceberg copy <schema>_<table>_ICEBERG. |
| Change log | Bronze history of a source table: one event per insert (I), update (U) or delete (D), numbered by _cdc_seq. Stored as append-only part files. |
| Watermark | A column (usually a timestamp) used to read only rows changed since the last load. |
| Look-back | A short window before the stored watermark that is read again, so late commits are not missed. |
| Detect deletes | Comparing source keys with Bronze to record deletes a watermark load cannot see. It has a safety limit (max_delete_ratio, default 20 %). |
| SCD Type 1 / Type 2 | Current state only, or one row per version with __start_at, __end_at, __is_current. |
| Declarative table | A table defined with @ds.table (or @ds.dimension, @ds.fact, @ds.obt) and built by ds.run() in dependency order. |
| Quarantine | Rows that fail a quality rule, written to a separate table instead of the target. |
| Spark session | The Spark engine instance that runs a user's cells. Depending on platform setup, a user's notebooks share one session or each has its own, up to a per-user limit. Notebooks run by Dataflow get their own session. |
| SQL guard | The check every statement goes through in the Spark session: tenant stamping, protected columns, refusal of unsafe statements, and the tenant boundary (only your tenant's tables and lake paths can be read or written). |
| Policy snapshot | The grants, row filters and column masks sent with a cell so the Spark session can enforce them. |
| Object log | The history of table definitions (CREATE, ALTER, DROP, RENAME). |
| Run log | One entry per cell, pipeline step or DataBridge load, with status, time and steps. |
| Lineage edge | "Table A feeds table B", recorded by cells, pipelines, DataBridge, dbt and declarative tables. |
| Share / recipient | A set of tables offered through the Delta Sharing protocol, and an outside party holding a bearer token for it. |
| Environment | A complete DataSimple instance (development, QA, production) that work is promoted between. |
| Deploy key | The stable identity of an object across environments, used to remap ids on promotion. |
API route map#
All routes are relative to https://<your-datasimple-host>. Call them with a personal API token (Authorization: Bearer <PERSONAL_TOKEN>). Your role must include the matching screen. A personal token cannot create or revoke tokens, sign in or out, change or reset passwords, manage users, roles or role templates, or create sharing recipients or dbt client keys. Do those in the application while signed in.
| Prefix | Area | Chapter |
|---|---|---|
/api/auth/me |
Who you are (works with a personal token) | 19 |
/api/auth/tokens |
Personal API tokens | 18 |
/api/notebooks |
Notebooks, folders, schedules, external connections, saved joins, run log | 02, 03, 10 |
/api/notebooks/spark |
Run cells (/run, /sql), stop, status, pre-warm, Trash, join builder |
02, 23 |
/api/notebooks/unity-catalog, /api/2.1/unity-catalog |
Catalog, governance, Unity Catalog-compatible REST | 07 |
/api/notebooks/governance |
Governance actions | 07 |
/api/notebooks/semantic |
Glossary terms, metrics, business meaning | 13 |
/api/notebooks/dq |
Quality rules, results, alerts, sweeps | 12 |
/api/notebooks/sharing |
Shares, recipients, access log. A share is managed by its creator, anyone with MANAGE on it, or an administrator |
14 |
/delta-sharing |
Delta Sharing protocol (bearer token, no session) | 14 |
/api/pipelines (also /api/notebooks/pipelines) |
Pipelines, runs, stop, re-run failed | 08 |
/api/workflows |
Workflows, runs, schedule as pipeline | 09 |
/api/notebooks/airflow |
Dataflow (DAGs, runs, tasks, logs) | 11 |
/api/notebooks/dbt, /api/notebooks/dbt/clients, /dbt-sql |
dbt projects, clients, SQL gateway for dbt | 22 |
/api/databridge |
DataBridge loads, jobs, schedules, file ingest | 04 |
/api/bronze-replication |
Bronze loads screen | 05 |
/api/cloud-connections |
Lake destinations | 03 |
/api/notebooks/query-history |
Query history | 15 |
/api/usage |
Usage and cost | 15 |
/api/deploy |
Environments, inventory, compare, promote, accept, history, roll back | 16 |
/api/sarah, /api/agents, /api/everything |
Assistants | 17 |
/api/admin |
Users, roles, role templates, tab access, tenants, activity and error logs | 18 |
/api/kinetic |
Kinetic Pen (turn pointers and words into code, saved recordings, save as a business formula) | 27 |
/api/notebooks/git |
Git integration for notebooks | 02 |
/api/activity |
History activity (History tab) | 15 |
/api/file-alerts |
New-file alerts | 04 |
/api/notebooks/flink |
Streaming jobs (Pipelines tab) | 08 |
/help/handbook |
This Developer Handbook (older help links forward here) | — |
Run states#
| Object | States |
|---|---|
| Notebook cell | ok, error (run log) |
| Notebook schedule run | running, success, partial (a cell failed), error |
| Pipeline run | running, success, partial, error, cancelled; step results carry kept_from_run after a re-run |
| Workflow run | running, ok, error, cancelled; boxes ok, error, skipped |
| DataBridge job | queued, running, done, failed, cancelled; only a file copy can also end partial (some files copied, some failed) |
| Delete check | ok, in_sync, refused (with a reason such as above_threshold), skipped (no key, or no Bronze yet), error (nothing recorded) |
| File ingest run | ok, no_new_files, empty (new files had no readable rows), busy (another run holds the lock), cancelled, error |
| Quality alert | open, acknowledged, resolved |
Common errors#
| Message or code | Where | Meaning and what to do |
|---|---|---|
401 Login required |
Any /api route |
No session, or it was closed because the same user signed in elsewhere (one session per user). Sign in again. |
403 with a tab name |
Any /api route |
Your role lacks that tab. Ask an administrator. |
403 TOKEN_NOT_ALLOWED |
Personal token | A token cannot create or revoke tokens, sign in or out, change passwords, manage users, roles or role templates, or create sharing recipients or dbt client keys. Sign in to the application to do this. |
401 TOKEN_INVALID |
Personal token | Revoked or expired token. Create a new one. |
409 Pipeline "…" is already running (run #N) |
Pipelines | A run is in progress; the answer names it. Wait for it to finish, or stop it. |
409 This schedule is already running |
Schedules (Run now) | Wait for the current run to finish, then run it again. A clock-triggered run that comes while one is running is skipped. |
409 Stop or wait for the running workflow first. |
Workflows | The workflow is running. Wait or stop it. |
All Spark sessions are busy; try again in a minute. |
Notebook cells | Every Spark session is busy and the cap is reached. |
All your Spark sessions are busy (limit N per user) |
Notebook cells | All of your own sessions are running. Try again in a minute, or stop a running cell. |
Spark session restarted |
Notebook cells | Your session was closed to free capacity for others, to make room for another of your notebooks, or after it sat idle. Variables are gone; re-run setup cells. |
REQUIRES_SINGLE_PART_NAMESPACE |
Python spark.sql("… datasimple.gold.t") |
Known limit: in Python, use gold.t or read_table("datasimple.gold.t"). SQL cells accept the full name. |
TABLE_OR_VIEW_NOT_FOUND |
Cells | Check the name; a table loaded by DataBridge may need one SQL cell to attach it before Python spark.sql can see it. |
Table or view … was not found in your catalog. |
Cells | The table is not in your tenant's catalog. Check the name. |
The lake path … is not in your catalog. |
Cells | The path belongs to another tenant or is outside your catalog. Use a table name from your catalog. |
The table name … cannot be used: choose another name. |
Cells | The name is already taken outside your tenant. Pick another name. |
Direct file-path queries bypass Unity Catalog governance |
SQL on parquet.\s3a://…`` |
Query the table by its catalog name. |
… are missing from the source, above the 20% limit; no deletes recorded |
DataBridge delete check | Check the source. If the deletes are real, raise max_delete_ratio for that table. |
Another file-ingest run is already loading … |
File ingest | Another run holds the table lock. Nothing was written; run again when it finishes. |
You submitted this rule, so you cannot approve it |
Data quality approval | Four-eyes rule. Another person must approve. |
a metric must read, not write |
Semantics | Metric SQL must be a SELECT or an aggregate expression. |
RESOURCE_DOES_NOT_EXIST / UNAUTHENTICATED / PERMISSION_DENIED |
Delta Sharing | Share not granted, token revoked or expired, or table has filters or masks. |
Naming and data rules#
| Rule | Detail |
|---|---|
| Schema and table names | Letters, digits and underscore; start with a letter or underscore |
| DataBridge source copies | datasimple.source.<schema>_<table> for the Parquet copy (for example public_orders), <schema>_<table>_DELTA for the Delta copy and <schema>_<table>_ICEBERG for the Iceberg copy |
| Audit columns | Bronze and change logs carry _op_type, _cdc_seq, _load_ts, _batch_id, _row_hash and related columns. See Bronze. |
| Tenant column | _tenantid, stamped on insert |
| SCD Type 2 columns | __start_at, __end_at, __is_current |
| Star schema | Dimensions carry a surrogate key (<name>_key) and an Unknown member with key -1 |
| VACUUM | Retention of at least 168 hours |
| Dropped tables | Go to Trash; restore with the Trash panel or UNDROP TABLE |
Related content#
- Architecture
- End-to-end walkthrough
- Spark SQL reference: https://spark.apache.org/docs/latest/sql-ref.html
- Delta Lake utility commands (VACUUM, DESCRIBE HISTORY, RESTORE): https://docs.delta.io/latest/delta-utility.html