DataSimpleDeveloper HandbookLog inOpen DataSimple
DataSimple
Chapter 02Bring data in

Notebooks

Write, run and schedule SQL, Python and Scala code against the governed lakehouse, with the platform helpers already loaded.

Overview#

The Notebooks screen is the main place where developers and data engineers work with data in DataSimple. A notebook is a document of cells. Each cell is code (SQL, Python, Scala, R or shell), text (Markdown), a chart, or a parameter block. Code cells run in a Spark session that DataSimple starts for you. That session has helper functions preloaded, so reading and writing lake tables by catalog name is one line of code.

Who uses it:

  • Data engineers build Bronze, Silver and Gold tables, apply changes and declare tables with ds.* helpers.
  • Analysts query Gold tables in SQL cells and chart the results.
  • Developers test helpers, inspect run traces and automate notebooks through the API or a schedule.

Where it is: left rail → Notebooks. The same editor opens when you click Open in notebook from Everything, or Open in a new notebook from the Kinetic page.

How it fits the platform: every cell you run is checked against your permissions, runs on the Spark engine, and is recorded in the run log, the audit history, lineage and usage. Tables that a cell creates are registered in the catalog automatically. See Architecture for the full picture.

Key concepts#

Term Meaning
Cell type sql, python, spark (Spark / Delta), scala, r, shell, markdown, chart, params. Code cells are sql, python, spark, scala, r, shell.
Spark session Your personal Spark session. Depending on how your platform is set up, you have one Spark session for all your notebooks (variables are shared) or one per notebook, up to a per-user limit. It starts at sign-in when possible, or when you open a notebook. Capacity is shared fairly between users. Variables live in it until it is restarted.
Helpers Python functions available in every session without an import: read_table, save_table, merge_delta, check_table, ds.* and more. They resolve catalog names and apply grants, row filters and column masks.
Catalog name datasimple.<layer>.<table>, <layer>.<table> or a bare <table>. Layers are source, bronze, silver, gold.
Parameters Values declared in a Params cell. Python uses the variable; SQL uses {{name}}. A schedule or pipeline can pass other values.
Preview The rows a cell returns to the screen. Default 200 rows. The table itself is not limited.
Compute profile The size of the Spark session. Today only Serverless (use all available capacity) is offered.
Run log One entry per cell run, shown on the Log screen.
Fast path A plain read of registered lake tables can be answered by an always-on SQL engine instead of your session, which is faster. The run trace says warm SQL engine when this happens.

How it works#

Notebooks: your notebook, what happens each time a cell runs, and the lake layers it reads and writes
Notebooks: your notebook, what happens each time a cell runs, and the lake layers it reads and writes

What happens when you run a cell:

  1. DataSimple checks your sign-in, your tenant's rate limit and any storage-path restrictions, and for SQL substitutes {{parameters}}.
  2. A SQL cell that is a plain read of registered lake tables may be answered by the always-on SQL engine. Everything else runs in your Spark session.
  3. The code runs with the helpers loaded. Grants, row filters and column masks apply on every read by name.
  4. New tables are registered in the catalog, lineage and usage are recorded, and PII columns are masked in the preview for viewers without access.
  5. The screen shows the result grid, and a run-log entry with the run trace is written.

Running cells#

Cell type What happens
SQL Parameters are substituted. Statements are split on ; and run one by one. The last statement's result is the preview. Catalog names are resolved, grants and masks applied, new tables registered and lineage recorded.
Python, Spark The code runs in your Spark session with the helpers loaded. The value of the last expression, if it is a DataFrame, becomes the preview.
Scala, R, shell The code runs in a session for that language on the Spark engine.
Markdown, Chart, Params Rendered in the browser. A chart redraws from the result of the cell above it.

Request limits: a cell source is at most 200 KB. Tables dragged into a Python cell as pandas data can be up to 25 MB.

First-line options: %sql, %python, %md#

Type an option on the first line of a cell and the cell switches to that type. The option line is removed, so it never reaches Spark. Upper or lower case both work, and %% works the same as %.

First line Cell becomes Also accepted
%sql SQL %%sql
%python Python %py, %%python
%pyspark Python on Spark %spark
%md Markdown %markdown
%sh Shell %shell
%scala Scala
%r R

Only the first line counts. %run, %pip and %fs are not available: use a pipeline to run notebooks in order, ask your administrator for packages, and read files as tables.

Markdown cells#

Markdown cells render headings (#, ##, ###), bold, italic, `code`, bullet lists, quotes (>), links, images and pipe tables (:---: centres a column, ---: right-aligns it). Numbered lists, fenced code blocks and HTML are not rendered. The toolbar inserts each of them; Show preview shows the result.

Keyboard shortcuts#

Keys Does
Ctrl+Enter (⌘+Enter on a Mac) Run the current cell
Shift+Enter New line inside the cell
Ctrl+S Save the notebook (and keep a version)
Ctrl+P Command palette

Every SQL command and Python helper, with examples, is in SQL, Python and streaming reference.

Python helpers#

Every session has the helpers without an import, plus spark, ds, F (pyspark.sql.functions), Window and T (pyspark.sql.types). In the editor, autocomplete shows each helper's signature and description; the same list is available from GET /api/notebooks/spark/helpers.

Helper Purpose
read_table(name) Governed read. Resolves Parquet, Delta or Iceberg from the catalog, checks SELECT, applies row filters and column masks.
read_delta(name_or_path), read_parquet(...), read_iceberg(...) Same as read_table for a catalog name. A raw lake path that belongs to a registered table is resolved to that table and read the governed way.
read_delta_at(name, version=None, timestamp=None), list_delta_history(name) Delta time travel and history.
save_table(df, name, mode="overwrite", partition_by=None, merge_schema=False, fmt=None) Writes a Delta table (or Iceberg with fmt="iceberg"), applies enforce-on-write quality rules, registers the table and records lineage.
merge_delta(target, source_df, on, ...), write_delta(...) Delta MERGE and append/overwrite writes.
check_table(name_or_df, checks, ...), profile_table(name), table_freshness(name), suggest_checks(name) Data quality. Results appear on the Data Quality screen.
grant(...), revoke(...) Catalog grants. Administrators or MANAGE holders only.
optimize_table(name), drop_table(name, purge=True) Compaction and drop. Files are purged only for managed tables.
ds.apply_changes(...), ds.apply_changes_from_snapshot(...), @ds.table, ds.run(), @ds.dimension, @ds.fact, @ds.obt Change handling (Type 1 and Type 2) and declared tables. See Declarative tables and change handling.
ds.load_control() The DataBridge load-control rows for your tenant (read only). See DataBridge.

A plain df.write.save(...) to a path works, but the table is not registered and no lineage is recorded. Use the helpers.

SQL cells and catalog names#

In a SQL cell:

  1. datasimple.<layer>.<table> works for the four layers, whatever the table's format (Parquet, Delta or Iceberg).
  2. Grants, row filters and column masks apply per statement.
  3. GRANT and REVOKE are recorded in the catalog.
  4. CREATE TABLE ... USING DELTA with a column list, and CREATE TABLE ... AS SELECT, create a managed table and register it.
  5. DROP TABLE is a soft drop (files kept, UNDROP TABLE restores). DROP TABLE ... PURGE deletes files.
  6. Path queries such as delta.`s3a://...` are refused unless you are a catalog administrator for your tenant.
  7. APPLY CHANGES [FROM SNAPSHOT] INTO ... and MERGE CHANGES are supported. See Declarative tables and change handling.

Scheduled and pipeline runs apply rules 1 and 2 the same way, with the grants, row filters and column masks of the notebook owner (pipeline steps fall back to the pipeline creator).

Where SQL cells run. The notebook header shows Lakehouse and a Query a source system switch. Off, SQL cells run on the Lakehouse (POST /api/notebooks/spark/sql). On, you pick one Data Source from a grouped, searchable list (Databases, Cloud warehouses, Business apps, Streams, Files & storage); SQL cells then run on it read-only (POST /api/notebooks/external-database/query), and the cell badge shows its name. Kafka sources, file drops and storage targets are listed greyed: read Kafka with STREAM KAFKA('<Data Source>', '<topic>') in a Lakehouse cell, and load files with Databridge. The app's own database is never a notebook source; an old notebook saved with it opens on the Lakehouse.

Combine sources. A SQL cell can read several slots, each the Lakehouse or one Data Source, and join them with a combine query over the slot aliases (POST /api/notebooks/cross-query; Python cells get each slot as a pandas DataFrame through POST /api/notebooks/cross-fetch). Slot SQL must be read-only (SELECT / WITH). A slot saved with the old local source, or with no source, is read from the Lakehouse.

Known limit: three-part names in Python spark.sql. spark.sql("SELECT * FROM datasimple.gold.sales_by_region") in a Python cell fails with REQUIRES_SINGLE_PART_NAMESPACE. A Python spark.sql on a table loaded by DataBridge can also fail with "table not found" until a SQL cell has used it. Use read_table("gold.sales_by_region") in Python, or put the query in a SQL cell.

Parameters#

Add a Params cell and declare values such as region_id = 3. Python cells use the variable directly. SQL cells write {{region_id}}; a cell with a parameter that has no value is refused with Parameter {{region_id}} has no value. Schedules and pipeline steps pass their own values with the same substitution, so one notebook serves many regions or dates.

Previews#

The preview is a LIMIT on the last result. Writes, counts and aggregations always run on the full data. See Limits and known issues for the numbers.

Compute profiles#

Only Serverless is offered: the session uses all capacity the Spark engine allows. Notebooks and schedules that saved an older size (small, medium, large) run as Serverless, so a saved value never breaks a run.

Run log, versions and history#

  • Each cell run is recorded with cell type, status, duration, rows, error, the first 4,000 characters of code and the run trace. The trace shows steps such as warm SQL engine, the tables read and written, and the DataBridge origin for loads. Errors and the run trace hide storage locations, internal addresses and tokens. Cell code is shown as written, so keep passwords out of code.
  • A save that changes the cells is kept as a version. The latest 100 versions of each notebook are kept. Notebook menu → Version history… lists them and restores one.
  • Notebook menu → Audit log… shows one notebook's timeline: edits, versions, scheduled runs and queries.

REST API#

Call these from your own scripts with Authorization: Bearer <PERSONAL_TOKEN>.

Method Path Purpose
GET, POST /api/notebooks List notebooks; create one.
GET, PUT, DELETE /api/notebooks/:id Read, save (keeps a version), delete.
POST /api/notebooks/spark/sql Run a SQL cell.
POST /api/notebooks/spark/run Run a Python or Spark cell.
POST /api/notebooks/spark/run-kernel Run a Scala, R or shell cell (language, code).
POST /api/notebooks/spark/run-notebook Run a whole notebook, stopping on the first error.
POST /api/notebooks/spark/cancel Stop a running cell.
GET /api/notebooks/spark/status Your session state: cold, starting, ready or busy.
POST /api/notebooks/spark/prewarm Start your session in the background.
GET /api/notebooks/spark/helpers Helper names, signatures and descriptions.
GET /api/notebooks/run-log?notebookId=&status=error&mine=1&limit= Run-log entries, as on the Log screen.
GET /api/notebooks/:id/versions Version history.
POST /api/notebooks/:id/versions/:vid/restore Restore a version.
GET /api/notebooks/:id/timeline One notebook's timeline.
GET, POST /api/notebooks/:id/schedules Schedules for this notebook. See Schedules and scheduler.

Errors from the run endpoints:

Status Meaning
400 code is required, sql is required, or a parameter with no value (missingParams lists them).
403 A storage path in the code is outside the locations allowed for your tenant; a path query was used by someone who is not a catalog administrator for your tenant; the AI guardrail blocked SQL sent by Sarah or the Agent (code: "AI_GUARDRAIL"); DROP TABLE or UNDROP TABLE without ownership or MANAGE; or RESTORE by someone who is not an administrator.
413 The cell source or the dragged data is too large.
429 Your tenant's rate limit is reached (code: "RATE_LIMITED").
503 The Spark engine is not available, or every Spark session is busy (All Spark sessions are busy; try again in a minute).

Developer examples#

1. A SQL cell that builds a Gold table#

SQL
CREATE TABLE IF NOT EXISTS datasimple.gold.sales_by_region AS
SELECT o.region, SUM(o.amount) AS revenue, COUNT(*) AS orders
FROM datasimple.silver.orders o
WHERE o.order_date >= {{from_date}}
GROUP BY o.region;

SELECT * FROM datasimple.gold.sales_by_region ORDER BY revenue DESC;

The first statement creates and registers a managed table. The second returns the preview. from_date comes from the Params cell or the schedule. Text and date values are quoted for you; do not add quotes around a parameter.

2. A Python cell with governed reads and writes#

Python
orders = read_table("silver.orders")                      # grants, filters and masks apply
live = orders.filter("NOT _is_deleted")
by_region = (live.groupBy("region")
                 .agg(F.sum("amount").alias("revenue"),
                      F.count("*").alias("orders")))

save_table(by_region, "gold.sales_by_region")             # Delta, registered, lineage recorded
check_table("gold.sales_by_region",
            lambda c: c.isComplete("region").isUnique("region"))
read_table("gold.sales_by_region")                        # last expression -> preview

3. Run a SQL cell through the API#

JavaScript
const res = await fetch("https://<your-datasimple-host>/api/notebooks/spark/sql", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": "Bearer <PERSONAL_TOKEN>",
  },
  body: JSON.stringify({
    notebookId: 12,
    cellId: "c-3",
    sql: "SELECT region, revenue FROM datasimple.gold.sales_by_region WHERE revenue > {{min_rev}}",
    params: { min_rev: 1000 },
    previewRows: 50,
  }),
});
const out = await res.json();
// {
//   "success": true,
//   "df": { "columns": ["region", "revenue"], "rows": [{ "region": "West", "revenue": 48210.5 }] },
//   "rowCount": 1,
//   "stdout": "", "stderr": "", "error": null,
//   "durationMs": 912,
//   "trace": [ ... ]
// }

A write statement (INSERT, MERGE, UPDATE, DELETE) also returns affectedRows. Send notebookId and cellId so the run is tied to that notebook. A SQL run from a script is not added to the run log by itself (the editor writes its own entry); to record one, call POST /api/notebooks/run-log. A Python run from a script (example 4) is recorded when the request includes "source": "script".

4. Run a Python cell from a script#

Shell
curl -s https://<your-datasimple-host>/api/notebooks/spark/run \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <PERSONAL_TOKEN>" \
  -d '{ "notebook_id": 12, "cell_id": "c-4",
        "code": "df = read_table(\"gold.sales_by_region\")\ndf.orderBy(F.desc(\"revenue\"))" }'

Both notebookId and notebook_id are accepted. The response has success, stdout, stderr, df, error, durationMs and trace.

Security and permissions#

  • Sign-in and access. Every call needs a session or a personal token, and your role must include the Notebooks screen.
  • Tenant isolation. Your session and catalog names are resolved for your tenant. Every table a cell writes carries _tenantid; cells may not drop, rename or retype it.
  • Data access. Reads by name check SELECT and apply row filters and column masks. Writes need CREATE_TABLE on the schema or MODIFY on an existing table. PII columns are masked in the preview for viewers without access.
  • Storage locations. Your administrator can restrict which storage locations notebook code may use. When restrictions are set, every s3://, s3a://, gs:// or abfs(s):// path in a cell must start with an allowed prefix, or the cell is refused with HTTP 403.
  • AI guardrail. SQL sent by Sarah or the Agent can never drop, delete, truncate, alter, overwrite or replace. A person running the same SQL is governed by their own role.
  • Audit. Cell runs and whole-notebook runs appear in History. Definitions (CREATE, ALTER, DROP) go to the definition log.

Settings you can change#

Setting Where Effect
Parameters Params cell, schedule, pipeline step, or params in the API Values substituted into {{name}} and set as Python variables.
Preview rows previewRows in the API Rows returned for this call, up to 5,000. A read answered by the always-on SQL engine returns up to that many; a result from your Spark session is capped at 200 rows.
Compute profile Notebook header Only Serverless today.

Your administrator can lower the default number of preview rows, change the cell time limit, and enable CREATE TABLE ... USING ICEBERG. Spark session capacity and fair sharing are covered in Spark engine.

Monitoring and troubleshooting#

  • Log screen: one line per cell run with status, duration, rows, error and trace. Filter by notebook, errors only and origin.
  • History screen: actions such as created, run, scheduled, deleted.
  • Notebook menu → Audit log…: one notebook's full timeline.
  • Spark status chip in the notebook header: starting... 12 s, ready, or nothing when no session runs.
Symptom Likely cause
REQUIRES_SINGLE_PART_NAMESPACE in a Python cell Three-part name in spark.sql. Use read_table or a SQL cell.
Parameter {{x}} has no value Add x to the Params cell or pass it from the schedule.
Spark session restarted with a reason Your idle session was closed to free capacity. Re-run earlier cells or Run all.
All Spark sessions are busy (503) Every session is running a cell. Try again in a minute.
HTTP 403 for a storage path Ask an administrator to allow the location for your tenant, or read by catalog name.
The first cell is slow The session is starting. It starts at sign-in when possible, or when you open a notebook.

Limits and known issues#

Limit Default
Preview rows per cell 200 (up to 5,000 from the always-on SQL engine when previewRows asks for more; the table itself is not limited)
Cell time limit 5 minutes
Cell source size 200 KB
Data dragged into a Python cell 25 MB
  • Three-part names do not work in Python spark.sql yet (see above).
  • Statements in a SQL cell are split on ;. A semicolon inside a string literal is not handled; run such a statement on its own.
  • Only the Serverless compute profile exists.
  • The first cell of a new session can take up to about a minute while the session starts.
  • Lineage from notebook code does not see table names built dynamically (for example f-strings); the lineage view reports how many such cells it could not attribute.