DataSimplePractical GuideLog inOpen DataSimple
DataSimple
Chapter 07aTransform and model

Spark: the engine behind your notebooks

Back to the index · Previous: Notebooks · Next: 7b. SQL reference

What Spark is#

Apache Spark is the open-source engine that reads and writes your lakehouse tables. It splits a query into many small tasks and runs them side by side, so a scan of millions of rows or a join of two large tables finishes in seconds or minutes. You do not install or size anything: DataSimple runs Spark for you, in your own cloud, next to your data.

You talk to Spark in two languages:

  • SQL in a SQL cell: Spark SQL, the same dialect as the Apache Spark documentation, with the Delta Lake and Apache Iceberg commands (7b).
  • Python in a Python cell: PySpark DataFrames, plus the DataSimple helpers (7c).

Both read the same tables, under the same access rules.

Two ways a SQL cell is answered#

Engine Answers Speed
SQL engine (always on) Plain reads of lake tables: SELECT, joins of catalog tables, DESCRIBE TABLE, SHOW TABLES About a second, even right after you sign in
Your Spark session Everything else: Python cells, writes, MERGE, time travel, temporary views, file paths, database connections Starts in up to about a minute; then fast

You do not choose: DataSimple sends each cell to the right one. The run log shows warm SQL engine when the SQL engine answered, or warm SQL engine skipped with the reason. Your grants, row filters and masks apply the same way in both.

Your Spark session#

  1. Start. Your first Python cell, or a SQL cell the SQL engine cannot answer, starts your session. The notebook header shows Spark: starting… N s, then Spark: ready.
  2. Work. All your Python cells share the session, so variables stay from cell to cell. The first read of a table takes a few seconds longer; later reads are faster.
  3. Idle. Without an open notebook, an idle session closes after about 12 minutes (Scala and R after about 10). The next cell starts a new session and shows Spark session restarted: run the earlier cells again to set your variables.
  4. Busy. The platform runs a limited number of sessions at once. When every session is busy, your run waits about 20 seconds, then fails with All Spark sessions are busy; try again in a minute.

Spark status in the notebook toolbar shows your sessions. An administrator can keep sessions warm during business hours.

How results come back#

  • Previews show 200 rows from your session. Reads answered by the SQL engine can return up to 5,000 rows when Settings allows it.
  • For more rows, write the result to a table (CREATE TABLE … AS SELECT, or save_table in Python), or page through it with LIMIT / OFFSET or a WHERE clause.
  • Writes report the rows inserted, updated or deleted, read from the Delta commit.
  • Errors come back short, with a hint (for example column count mismatch, unresolved column, concurrent write, divide by zero), and the full text below.

What DataSimple checks on every statement#

Check What happens
Grants Reads, DESCRIBE and DESCRIBE HISTORY need SELECT. Writes, TRUNCATE, OPTIMIZE, VACUUM and RESTORE need MODIFY. Creating a new table or view needs CREATE_TABLE on the schema
Row filters and column masks Applied to every read, also through a view
Tenant Every lake row keeps the tenant that wrote it (_tenantid). Inserts and CREATE TABLE … AS SELECT add it; dropping or changing it is refused. Tables of other tenants do not exist for you
Drops DROP TABLE is a soft drop (the files are kept, UNDROP TABLE brings it back); PURGE deletes the files
Older versions RESTORE TABLE is for administrators only
Storage paths Queries on a path (delta.`s3://…` ) are refused for non-administrators: read by table name
Declared keys A primary key declared on a table is enforced on writes: a duplicate is rejected and the write rolled back

Scheduled runs, pipelines and the API apply the same checks.

Performance tips#

  1. Read by catalog name (silver.orders), never by storage path: faster, and your access rules apply.
  2. Filter and aggregate in Spark, then collect() or toPandas(): pandas loads everything into memory.
  3. Write big results to a table instead of showing them.
  4. Use SQL cells for quick reads: plain SELECTs are often answered by the SQL engine without waiting for your session.
  5. Compact tables with many small files: OPTIMIZE silver.orders in SQL, or optimize_table("silver.orders") in Python.
  6. Partition only large tables, by a column you filter on (a date): save_table(df, "silver.events", partition_by=["event_date"]).
  7. Keep a notebook open while you work, so your session stays warm.

Things to know#

  • Python and three-part names. In Python, spark.sql("SELECT … FROM gold.t") uses schema.table; datasimple.gold.t fails there. SQL cells accept both.
  • A table just loaded by DataBridge may read as "not found" in Python until it has been read once from a SQL cell.
  • Python, Scala and R run in separate sessions; only Python shares variables with the helpers.
  • Memory. A cell that collects too much data to the session stops it: The Spark session stopped while the cell ran. Filter or aggregate earlier.

Read more#

  • Apache Spark SQL reference and the PySpark API (the open-source documentation).
  • Developer Handbook, chapter Spark engine: the REST API, administrator settings and messages.

Back to the index · Next: 7b. SQL reference