DataSimpleDeveloper HandbookLog inOpen DataSimple
DataSimple
Chapter 20Shape and govern data

Lake storage

How DataSimple lays out data on object storage, which table format it uses where, and how tables are registered, versioned, compacted and stamped with the tenant.

Overview#

All table data lives on object storage (Amazon S3, Azure Data Lake Storage Gen2 or Google Cloud Storage), in one bucket or container per environment. An administrator saves it once as a target Data Source. DataBridge, notebooks and the Workflow write under a fixed folder layout inside it, and every table is registered in the catalog, so you read by name (datasimple.silver.orders) and never by path (Lakehouse and governance).

This chapter is for developers who debug a load or need to know what a folder holds. You see the layout in Data Lake (raw folders) and the registered tables in Lakehouse. For the platform map, see Architecture.

Key concepts#

Term Meaning
Medallion layers source (raw copies from DataBridge), bronze (one row per key with delete markers, or the change log), silver (cleaned, current state and history), gold (business tables, star schemas, summaries).
Lake prefix An optional prefix under the bucket. With no prefix, layers sit at the root: s3://<lake-bucket>/source/….
Source folder The saved Data Source name typed in the DataBridge wizard. It is the first folder under the layer and names the source in lineage.
Format copy One source table can be copied as Parquet, Delta and Iceberg side by side. Each copy is its own catalog entry.
Managed root <lake root>/__uc_managed/<layer>/<table>: where tables created in notebooks (save_table, CREATE TABLE … USING DELTA without a location) are written.
Part files A large incremental Parquet copy kept as many files in a generation folder instead of one object.
Change log The Bronze change (CDC) event log of a table: append-only part files under cdc/, plus the latest state per key.
_tenantid Bookkeeping column on lake rows: the tenant (three digits) that wrote the row. Protected.

How it works#

Delta Lake, Apache Iceberg and Apache Parquet
Delta Lake, Apache Iceberg and Apache Parquet

Delta Lake is a trademark of LF Projects, LLC. Apache, Apache Iceberg, Iceberg, Apache Parquet and Parquet are trademarks of The Apache Software Foundation. The logos identify the open formats DataSimple reads and writes.

Lake storage: the folder layout per layer, which format where, the change log, part-file copies, registration and the tenant column
Lake storage: the folder layout per layer, which format where, the change log, part-file copies, registration and the tenant column

DataBridge writes Parquet, Delta and Iceberg directly; notebooks write through the Spark engine. Both register what they wrote in the catalog. Readers read every Parquet file in a table folder and skip file names that start with _.

Folder layout#

Text
s3://<lake-bucket>/<prefix>/
  source/<Source>/<database>/<schema>/<table>/
      parquet/<TABLE>.parquet                    single-file Parquet copy
      parquet_parts/<generation>/…               part-file Parquet copy (large incremental tables)
      delta/                                     Delta copy (_delta_log/ + data files)
      iceberg/                                   Iceberg copy (data/ + metadata/)
  bronze/<Source>/<database>/<schema>/<table>/
      parquet/<TABLE>.parquet | delta/ | iceberg/   Bronze table in the run's format
      cdc/                                       change log (Parquet Bronze with CDC)
  silver/<Source>/<database>/<schema>/<table>/delta            DataBridge Silver, Type 1
  silver/<Source>/<database>/<schema>/<table>_history/delta    DataBridge Silver, Type 2
  __uc_managed/<layer>/<table>/                  tables created in notebooks (Silver, Gold, your own)
  <schema>/_volumes/<tenant>/<name>/             volumes (default location)
  • Database and schema folders appear only when the source has them; a schema equal to the database is not repeated.
  • Path segments keep case (Snowflake and Oracle names are upper-case) and replace unsafe characters with _.
  • A Silver or Gold table built in a notebook or by ds.* goes to the managed root unless you name a path (target="lake", path=…).

Change log folder (cdc/)#

On S3 targets, a Bronze change log is append-only:

Text
<table>/cdc/part-<first sequence number, 20 digits>.parquet    one per run that had events
<table>/cdc/_common_metadata                                   schema of the log
<table>/cdc/_state/…                                           latest state per key
  • Events are never rewritten. Each run's events are one new file. Two runs can never commit the same events twice.
  • The part file is the commit point. If a run stops after its part file is written, the next run completes it automatically. The per-key state is derived and is rebuilt from the part files if needed.
  • Cost per run does not grow with history: a run reads only what it needs and writes one new part.
  • ADLS targets keep a single change-log file, cdc/cdc_bronze.parquet.

Part-file Parquet copies#

A table loaded incrementally (watermark, change log or file drop) and large enough to be streamed keeps its Parquet copy as part files:

Text
<table>/parquet_parts/<generation>/bucket-00000.parquet …   with a key: rows spread by a hash of the key
<table>/parquet_parts/<generation>/part-….parquet           no key: one file per batch, appended
<table>/parquet_parts/<generation>/_common_metadata         table schema
  • The generation folder is the table's location in the catalog.
  • A full load writes a new generation and switches the catalog to it only after every file is written. Readers never see a half-written copy.
  • An incremental run with a key rewrites only the files its keys belong to. A key always lives in one file, so a reader sees each key once.
  • A schema change (new column, new type, other key) rewrites the generation once.
  • One previous generation is kept for readers that started before the switch.

Delta, Iceberg and Parquet#

Format Used for Registered as Keep in mind
Parquet Fastest plain copy; any tool reads it datasimple.source.<schema>_<table> No history. A single-file copy is rewritten on every incremental run unless it uses part files. Parquet Bronze appends.
Delta Default for Bronze when ticked; DataBridge Silver; notebook-managed tables …_DELTA for source copies; plain name in other layers History, MERGE, time travel, RESTORE, OPTIMIZE, VACUUM. Columns are written nullable.
Iceberg Snapshots, Iceberg engines and catalogs …_ICEBERG for source copies; Spark name iceberg.<layer>.<table> Must be enabled for notebooks by your administrator; renames are not supported; a soft drop only removes the catalog entry until the table is purged.

Iceberg tables. An Iceberg table is a folder with data/ and metadata/ on object storage. Tables written by DataBridge are immediately readable in notebooks by their Spark name; snapshots and time travel work in SQL. In SQL cells you may use the catalog name and DataSimple routes it to the Iceberg table automatically.

Bronze format. Bronze follows the formats of the run: Delta if Delta was ticked, else Iceberg, else Parquet.

Registration#

DataBridge registers each copy it writes with its format, layer, type (EXTERNAL) and storage location. Notebook writers register managed tables. Plain Spark writes in Python (saveAsTable, insertInto, writeTo) are registered too. A SQL CREATE TABLE may use the datasimple. prefix and PRIMARY KEY / FOREIGN KEY clauses (stored in the catalog); Iceberg tables are routed to iceberg.<schema>.<table>, and a table without a location goes to the managed root.

Time travel and history#

Format Look at history Read an old version Roll back
Delta DESCRIBE HISTORY <table>; Lakehouse → History; list_delta_history(name) SELECT … FROM <table> VERSION AS OF n; read_delta_at(name, version=n) or timestamp=… RESTORE TABLE (administrators only); Agent restore table (administrators, backup first)
Iceberg SELECT * FROM <table>.snapshots; Lakehouse → History Iceberg snapshot reads Agent restore table (administrators)
Parquet None None None

DESCRIBE HISTORY needs SELECT on the table; RESTORE needs MODIFY and an administrator role.

OPTIMIZE and VACUUM#

Path What it does Rules
Workflow / Pipelines Optimize step OPTIMIZE (optional ZORDER BY) then VACUUM … RETAIN n HOURS on the Delta tables of the chosen layers; Iceberg and Parquet layers are skipped Vacuum retention defaults to 168 hours. A value between 1 and 167 is refused: VACUUM must keep at least 168 hours (7 days) of history, or set it to 0 to skip VACUUM.
optimize_table(name, min_files=8) / optimize_tables([...]) in a Python cell Compacts a Delta table only when it has at least min_files files (unless force=True) Prints files before and after
SQL OPTIMIZE / VACUUM in a cell Runs in Spark Needs MODIFY on the table. Sarah and the Agent may never run VACUUM.
Iceberg maintenance Rewrites data files and manifests, expires old snapshots and removes orphan files on every Iceberg table Turned on by your administrator; keeps at least 10 snapshots, expires snapshots older than 7 days, removes orphan files older than 72 hours

Why 168 hours: a VACUUM with a short retention deletes the files that time travel, RESTORE and readers that started earlier still need.

Tenant column _tenantid#

Every lake row carries the tenant that wrote it:

  • DataBridge adds _tenantid when it prepares Bronze rows, from the job's tenant; a row that already has it keeps its value.
  • Notebook writers (save_table, write_delta, create_managed_table, merge_delta, ds.apply_changes*, @ds.table) add it when the DataFrame lacks it. A column dropped in Python comes back on write.
  • SQL in a notebook: CREATE TABLE … AS SELECT gains the tenant column when the query does not produce it; INSERT INTO / INSERT OVERWRITE into a table that has the column gets the tenant added. An INSERT OVERWRITE that cannot be stamped is refused.
  • Protected: ALTER TABLE … DROP / RENAME / ALTER COLUMN … TYPE / CHANGE COLUMN on _tenantid, and a REPLACE COLUMNS list without it, are refused.
  • A table written before the column existed gains it on its next append.
  • Reads: read_table on a table with _tenantid returns only your tenant's rows (rows with NULL, written before the column existed, still show).

See Spark engine for the other statements the SQL guard refuses.

REST API#

Base URL https://<your-datasimple-host>, header Authorization: Bearer <PERSONAL_TOKEN>.

Method Path Purpose
POST /api/notebooks/unity-catalog/register Register a table you wrote yourself
POST /api/notebooks/spark/history Delta history of a table ({ path, limit })
GET /api/notebooks/spark/s3-list?path= List the data and log files of one Delta table. path is the full path of the table in the lake browse location (as on the Data Lake screen). A path outside the lake browse storage answers 403; a table not in your catalog answers 403 This table is not in your catalog.; a malformed path answers 400 invalid s3 path. When the lake browse location is off, the answer has enabled: false and no files.

Developer examples#

1. Compare the format copies of one source table#

SQL
SELECT COUNT(*) FROM datasimple.source.dbo_orders;            -- Parquet copy
SELECT COUNT(*) FROM datasimple.source.dbo_orders_DELTA;      -- Delta copy
DESCRIBE HISTORY datasimple.source.dbo_orders_DELTA;          -- one version per load
SELECT snapshot_id, committed_at FROM datasimple.source.dbo_orders_ICEBERG.snapshots;

2. Read an older Delta version and compact#

Python
before = read_delta_at("silver.orders", version=3)
now = read_table("silver.orders")
print(now.count() - before.count(), "rows added since version 3")

optimize_table("silver.orders", min_files=8)      # skipped when there are fewer than 8 files

3. Register a table written to your own path#

Shell
curl -s -X POST "https://<your-datasimple-host>/api/notebooks/unity-catalog/register" \
  -H "Authorization: Bearer <PERSONAL_TOKEN>" -H "Content-Type: application/json" \
  -d '{"name":"orders_archive","layer":"silver","format":"DELTA","tableType":"EXTERNAL",
       "storageLocation":"s3://<lake-bucket>/<prefix>/silver/archive/orders/delta",
       "columns":[{"name":"order_id","type":"bigint"},{"name":"amount","type":"decimal(18,2)"}],
       "primaryKey":["order_id"]}'

4. The tenant column is kept#

SQL
CREATE TABLE gold.sales_by_region AS
SELECT region, order_date, SUM(amount) AS revenue FROM silver.orders GROUP BY region, order_date;
-- the table gets _tenantid automatically

ALTER TABLE gold.sales_by_region DROP COLUMN _tenantid;
-- refused: _tenantid is a protected column: every lake table keeps the tenant that wrote it

Security and permissions#

  • Read by name, not by path. Grants, row filters and column masks apply to reads by catalog name (Lakehouse and governance). Always use the catalog name.
  • The source layer is read-only for users; write to bronze, silver, gold or your own schemas.
  • Tenant column on every row, protected in SQL and re-stamped by every writer.
  • Storage names stay private. The Lakehouse tooltip shows the Data Source name; error messages never name storage.
  • Audit. DataBridge loads and notebook writes are recorded in the definition log and lineage; drops and undrops in the catalog audit.

Settings your administrator controls#

Behaviour Default
Part-file Parquet copies for large incremental tables Automatic
Append-only change log Automatic
Iceberg in notebooks Off
Iceberg maintenance Off (keeps 10 snapshots, 7 days, orphans 72 hours when on)
Automatic purge of soft-dropped tables Off. When your administrator turns it on, tables dropped more than 7 days ago are purged.

Monitoring and troubleshooting#

  • Log tab / job report: DataBridge records each step's time (read, copy, change log, catalog) per job.
  • History: Lakehouse → table → History for Delta versions and Iceberg snapshots.
  • Data Lake screen: check the folders and files of a table.
Symptom Cause and fix
Iceberg is off: this statement uses an Iceberg table… Iceberg is not enabled for notebooks. Read the _DELTA copy, or ask an administrator to enable Iceberg (the first Iceberg cell after that restarts your session once).
The location holds no Delta table Wrong path or table never written. Check the name.
Iceberg tables in this lake cannot be renamed Renames are not supported. CREATE TABLE <new> USING ICEBERG AS SELECT * FROM <old>, then drop the old one.
Another job wrote this table at the same moment Concurrent Delta commit. Run again.
VACUUM must keep at least 168 hours… Optimize step retention too short. Use 168 or more, or 0 to skip.
TRUNCATE on a Delta table Run as DELETE FROM (keeps time travel); Iceberg keeps TRUNCATE.

Limits#

  • Single-file Parquet copies of tables that are not streamed are rewritten whole on each incremental run.
  • ADLS targets keep a single-file change log.
  • Iceberg in notebooks is off by default because it adds session start-up time.
  • Iceberg renames are not supported.
  • Parquet has no history, so there is no undo of a Parquet overwrite.
  • Bookkeeping columns on Bronze tables can differ slightly between load types (Parquet, Delta/Iceberg merge, change log, file stream).