DataSimplePractical GuideLog inOpen DataSimple
DataSimple
Chapter 18Reference

What is tested

Every feature this guide and the developer handbook describe, with whether it has been proven working by a real run, when, where, and where to see the proof in the app. Anything not proven is marked so: a feature is only called tested when a run with a result is on record.

Where to see the proof. Sign in to the workspace (tenant 001) and open:

  • Notebooks → folder Doc checks: one notebook per document, holding that document's code examples as cells (table names prefixed zz_docs_, so no real table is touched). Each notebook has a schedule named Doc check; its runs list every cell's result.
  • Log (Scheduled filter): one line per cell run, with the code, status, time and error.
  • Lakehouse → object history on any zz_docs_* table: what each example created or changed.
  • The practical guides (Manufacturing, Retail, Finance) for the end-to-end builds, with screenshots, numbers and times.

Environments. dev is the shared development server (Spark as a remote service, the standalone catalog server). local is a developer machine using the same lake. A result on one is not automatically a result on the other; the table says which.

Code examples in the documentation#

Run on dev on 2026-11-04 as scheduled notebook runs. 50 notebooks, 46 documents, 791 cells (442 of them the documents' own examples, the rest setup and result checks). 330 of 442 examples ran as written.

Document Examples run Worked Did not Notebook (folder Doc checks)
DELTA-LOADS 1 1 0 Doc check - docs delta loads (run 51)
PHASE-A-AWS-DEPLOY 3 2 1 Doc check - docs phase a aws deploy (run 52)
UNITY-CATALOG-E2E 4 3 1 Doc check - docs unity catalog e2e (run 53)
UNITY-CATALOG-FULL-PARITY 12 10 2 Doc check - docs unity catalog full parity (run 54)
demo-manufacturing / PRACTICAL-GUIDE 18 12 6 Doc check - demo manufacturing practical guide (run 55)
demo-retail / PRACTICAL-GUIDE 11 9 2 Doc check - demo retail practical guide (run 56)
developer-guide / 02-lakehouse-storage 6 1 5 Doc check - dev guide 02 lakehouse storage (run 57)
developer-guide / 05-notebook-helpers 1 0 1 Doc check - dev guide 05 notebook helpers (run 58)
developer-guide / 06-end-to-end 4 2 2 Doc check - dev guide 06 end to end (run 59)
developer-guide / 08-notebook-cells 3 3 0 Doc check - dev guide 08 notebook cells (run 60)
developer-guide / 09-git-integration 2 2 0 Doc check - dev guide 09 git integration (run 61)
developer-guide / 10-dbt 1 1 0 Doc check - dev guide 10 dbt (run 62)
developer-guide / 11-dbt-from-your-own-dbt 1 1 0 Doc check - dev guide 11 dbt from your own dbt (run 63)
developer-handbook / 02-notebooks 2 1 1 Doc check - handbook 02 notebooks (run 64)
developer-handbook / 04-databridge 2 2 0 Doc check - handbook 04 databridge (run 65)
developer-handbook / 05-bronze-loads 2 2 0 Doc check - handbook 05 bronze loads (run 66)
developer-handbook / 07-lakehouse-and-governance 4 2 2 Doc check - handbook 07 lakehouse and governance (run 67)
developer-handbook / 08-pipelines 1 1 0 Doc check - handbook 08 pipelines (run 68)
developer-handbook / 10-schedules-and-scheduler 1 1 0 Doc check - handbook 10 schedules (run 69)
developer-handbook / 12-data-quality 3 0 3 Doc check - handbook 12 data quality (run 70)
developer-handbook / 15-observability 1 0 1 Doc check - handbook 15 observability (run 71)
developer-handbook / 20-lake-storage 4 3 1 Doc check - handbook 20 lake storage (run 72)
developer-handbook / 21-ds-tables-and-scd 9 5 4 Doc check - handbook 21 ds tables and scd (run 73)
developer-handbook / 23-spark-engine 2 1 1 Doc check - handbook 23 spark engine (run 74)
developer-handbook / 24-end-to-end-walkthrough 7 5 2 Doc check - handbook 24 end to end walkthrough (run 75)
developer-handbook / 26-unity-catalog-step-by-step 21 10 11 Doc check - handbook 26 unity catalog step by step (run 76)
developer-handbook / 27-kinetic-pen 5 0 5 Doc check - handbook 27 kinetic pen (run 77)
developer-handbook / 30-sql-python-reference 46 30 16 Doc check - handbook 30 sql python reference (part 1) (run 79); Doc check - handbook 30 sql python reference (part 2) (run 80)
guide / 03-lake-targets-and-formats 4 3 1 Doc check - guide 03 lake targets (run 81)
guide / 05-bronze 4 4 0 Doc check - guide 05 bronze (run 82)
guide / 06-silver-apply-changes 7 6 1 Doc check - guide 06 silver apply changes (run 83)
guide / 07-notebooks 13 11 2 Doc check - guide 07 notebooks (run 84)
guide / 07b-sql-reference 114 93 21 Doc check - guide 07b sql reference (run 85); Doc check - guide 07b sql reference part 2 (run 86); Doc check - guide 07b sql reference part 3 (run 87); Doc check - Delta and Iceberg commands on dev (run 101)
guide / 07c-python-reference 5 3 2 Doc check - guide 07c python reference (run 88)
guide / 07d-streaming-and-incremental 10 6 4 Doc check - guide 07d streaming and incremental (run 89)
guide / 08b-dbt-end-to-end-tested 1 1 0 Doc check - guide 08b dbt end to end tested (run 90)
guide / 09-lakehouse-governance 35 30 5 Doc check - guide 09 lakehouse governance (run 91); Doc check - guide 09 lakehouse governance (part 2) (run 92)
guide / 10-audit-trail-columns 8 8 0 Doc check - guide 10 audit trail columns (run 93)
guide / 11-monitoring-logs-recovery 7 7 0 Doc check - guide 11 monitoring logs recovery (run 94)
guide / 13-reference 19 16 3 Doc check - guide 13 reference (run 95)
guide / 15-ds-tables 5 5 0 Doc check - guide 15 ds tables (run 96)
user-guide / 01-connect-and-load 1 1 0 Doc check - user guide 01 connect and load (run 97)
user-guide / 02-notebooks 22 19 3 Doc check - user guide 02 notebooks (run 98)
user-guide / 04-lakehouse 3 2 1 Doc check - user guide 04 lakehouse (run 99)
user-guide / 06-apply-changes 7 5 2 Doc check - user guide 06 apply changes (run 100)

Examples that did not run as written, and why:

Document Line Example Error
PHASE-A-AWS-DEPLOY 171 CREATE TABLE datasimple.bronze.zz_docs_pha_ice_test (id BIGINT, region t 'UnresolvedRelation [bronze, zz_docs_pha_ice_test], [required_write_privileges=INSERT], false, false, false, false / +- LocalRelation [col1#376, col2#377]
UNITY-CATALOG-E2E 130 save_table(df, "bronze.zz_docs_uce_orders", mode="overwrite") If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches.
UNITY-CATALOG-FULL-PARITY 196 SELECT * FROM system.access.audit; SELECT * FROM system.access.table_l cross-database references are not implemented: "system.access.audit"
UNITY-CATALOG-FULL-PARITY 239 GRANT SELECT ON TABLE datasimple.bronze.zz_docs_ucp_orders TO alice; VALID_ARGUMENT","details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.system.access.table_lineage"}
demo-manufacturing / PRACTICAL-GUIDE 555 check_table("gold.fact_zz_docs_gmfg_sales", level="Error", checks=lamb = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
demo-manufacturing / PRACTICAL-GUIDE 598 -- MFG Inspection fail rate (by plant) SELECT p.plant_code, COUNT(*) A 8302L,plant_id#178303,plant_code#178304,plant_name#178305,country#178306,city#178307,manager_name#178308,capacity_units_per_day#178309,_tenantid#178310] parquet
demo-manufacturing / PRACTICAL-GUIDE 610 -- MFG Scrap rate (by plant and line) SELECT p.plant_code, l.line_code +- Relation datasimple.gold.dim_zz_docs_gmfg_line[line_key#178334L,line_id#178335,plant_id#178336,line_code#178337,line_name#178338,_tenantid#178339] parquet
demo-manufacturing / PRACTICAL-GUIDE 1209 ds.reset() @ds.dimension("gold.dim_zz_docs_gmfg_ice_supplier", keys=[ h ds.dimension)" % (_dn, dsk)) / ValueError: ds.fact: gold.dim_zz_docs_gmfg_product has no surrogate key zz_docs_gmfg_product_key (declare it with ds.dimension)
demo-manufacturing / PRACTICAL-GUIDE 1340 REFRESH MATERIALIZED VIEW gold.mv_zz_docs_gmfg_fulfilment_monthly; SE information_schema takes one statement — remove the ";"
demo-manufacturing / PRACTICAL-GUIDE 1465 df = read_table("gold.fact_zz_docs_gmfg_quality") df.show(10) nd catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; / 'UnresolvedRelation [gold, fact_zz_docs_gmfg_quality], [], false
demo-retail / PRACTICAL-GUIDE 382 ds.run() ds.checks() #302825,city#302826,state_code#302827,loyalty_tier#302828,home_store_id#302829,updated_at#302830,_tenantid#302831,__sequence#302832,__updated_at#302833] parquet
demo-retail / PRACTICAL-GUIDE 491 check_table("gold.fact_zz_docs_grtl_sales", level="Error", checks=lamb = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-guide / 02-lakehouse-storage 120 SELECT COUNT(*) FROM datasimple.source.zz_docs_d02_tpch_sf1_nation; SE operationMetrics#327321, userMetadata#327322, engineInfo#327323] / +- 'UnresolvedTable [source, zz_docs_d02_tpch_sf1_nation_DELTA], DESCRIBE HISTORY
developer-guide / 02-lakehouse-storage 123 SELECT COUNT(*) FROM datasimple.source.zz_docs_d02_tpch_sf1_nation_ICE :[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.source.zz_docs_d02_tpch_sf1_nation_ICEBERG.snapshots"}
developer-guide / 02-lakehouse-storage 126 CREATE TABLE datasimple.bronze.zz_docs_d02_orders (id BIGINT, amount D rs` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects.
developer-guide / 02-lakehouse-storage 217 df = read_table("datasimple.source.zz_docs_d02_tpch_sf1_nation_DELTA") = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-guide / 02-lakehouse-storage 353 CREATE MATERIALIZED VIEW gold.mv_zz_docs_d02_sales_revenue_by_region_d IALIZED VIEW gold.mv_zz_docs_d02_sales_revenue_by_region_daily AS SELECT region, order_date, SUM(amount) AS revenue FROM silver.zz_docs_d02_orders GROUP BY 1, 2
developer-guide / 05-notebook-helpers 121 src = read_table("datasimple.source.zz_docs_d05_tpch_sf1_nation_DELTA" = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-guide / 06-end-to-end 34 SELECT COUNT(*) AS n FROM datasimple.source.zz_docs_e2e_tpch_sf1_natio :[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.source.zz_docs_e2e_tpch_sf1_nation_ICEBERG.snapshots"}
developer-guide / 06-end-to-end 100 DROP TABLE datasimple.bronze.zz_docs_e2e_nation PURGE ssage) from None / pyspark.errors.exceptions.connect.UnsupportedOperationException: [UNSUPPORTED_FEATURE.PURGE_TABLE] The feature is not supported: Purge table.
developer-handbook / 02-notebooks 190 orders = read_table("silver.zz_docs_hb02_orders") = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-handbook / 07-lakehouse-and-governance 161 GRANT SELECT ON TABLE silver.zz_docs_hb07_orders TO analyst; information_schema takes one statement — remove the ";"
developer-handbook / 07-lakehouse-and-governance 186 COMMENT ON TABLE datasimple.gold.zz_docs_hb07_sales_by_region IS 'One information_schema takes one statement — remove the ";"
developer-handbook / 12-data-quality 106 check_table("gold.zz_docs_hb12_sales_by_region", level="Error", checks = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-handbook / 12-data-quality 228 check_table("gold.zz_docs_hb12_sales_by_region", level="Error", checks = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-handbook / 12-data-quality 269 good, bad = quarantine_write(df, "silver.zz_docs_hb12_orders", rules=[ ror( / ValueError: replay_errors('silver.zz_docs_hb12_orders') has no rules to re-test with. Either register them for the table, or pass rules=[...] explicitly.
developer-handbook / 15-observability 160 -- What feeds gold.sales_by_region, and from which job? SELECT source_ information_schema takes one statement — remove the ";"
developer-handbook / 20-lake-storage 186 ALTER TABLE gold.zz_docs_hb20_sales_by_region DROP COLUMN _tenantid; _tenantid is a protected column: every lake table keeps the tenant that wrote it
developer-handbook / 21-ds-tables-and-scd 124 MERGE CHANGES INTO silver.zz_docs_h21_mcustomers FROM bronze.zz_docs_h ON ds_t.customer_id = ds_s.customer_id / AND ds_s.updated_at > ds_t.updated_at / WHEN MATCHED THEN DELETE
developer-handbook / 21-ds-tables-and-scd 149 CREATE MATERIALIZED VIEW gold.mv_zz_docs_h21_sales_revenue_daily SCH silver.zz_docs_h21_orders o / JOIN silver.zz_docs_h21_customers c ON c.customer_id = o.customer_id AND c.__is_current / GROUP BY c.region, o.order_date
developer-handbook / 21-ds-tables-and-scd 159 DROP MATERIALIZED VIEW gold.mv_zz_docs_h21_sales_revenue_daily rrent_schema() output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.
developer-handbook / 21-ds-tables-and-scd 225 ds.reset() @ds.dimension("gold.dim_zz_docs_h21_customer", keys=["cust If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches.
developer-handbook / 23-spark-engine 150 spark.sql("""CREATE TABLE gold.zz_docs_h23_region_totals AS ciated location ('s3a://datasimple-the Spark service-serverless/delta-table-emrtest/gold.db/zz_docs_h23_region_totals') is not empty and also not a Delta table.
developer-handbook / 24-end-to-end-walkthrough 149 ds.reset() @ds.dimension("gold.dim_zz_docs_h24_customer", keys=["cust gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
developer-handbook / 24-end-to-end-walkthrough 196 check_table("gold.fact_zz_docs_h24_orders", level="Warning", checks=la = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-handbook / 26-unity-catalog-step-by-step 228 -- Managed table, declared columns and primary key CREATE TABLE gold.z gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
developer-handbook / 26-unity-catalog-step-by-step 246 # Python: write a DataFrame and register it df = read_table("silver.zz gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
developer-handbook / 26-unity-catalog-step-by-step 298 COMMENT ON TABLE gold.zz_docs_g26_sales IS 'One row per order, net of Table gold.zz_docs_g26_sales is not in the catalog; nothing to describe.
developer-handbook / 26-unity-catalog-step-by-step 328 -- gold.sales already has its primary key (order_id, declared in Step process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / >
developer-handbook / 26-unity-catalog-step-by-step 337 ALTER TABLE gold.zz_docs_g26_sales ADD COLUMNS (customer_id BIGINT); process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / >
developer-handbook / 26-unity-catalog-step-by-step 338 ALTER TABLE gold.zz_docs_g26_sales ADD CONSTRAINT fk_sales_customer process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / >
developer-handbook / 26-unity-catalog-step-by-step 721 ALTER TABLE gold.zz_docs_g26_sales RENAME TO gold.zz_docs_g26_sales_or S.; line 1 pos 12; / 'RenameTable [gold, zz_docs_g26_sales_orders], false / +- 'UnresolvedTableOrView [gold, zz_docs_g26_sales], ALTER TABLE ... RENAME TO, true
developer-handbook / 26-unity-catalog-step-by-step 729 DROP TABLE gold.zz_docs_g26_sales_orders; -- soft drop: hidden rrent_schema() output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.
developer-handbook / 26-unity-catalog-step-by-step 738 UNDROP TABLE gold.zz_docs_g26_sales_orders; Nothing to UNDROP: no dropped table 'gold.zz_docs_g26_sales_orders' within the retention window.
developer-handbook / 26-unity-catalog-step-by-step 745 DESCRIBE HISTORY gold.zz_docs_g26_sales_orders; nd#505404, operationMetrics#505405, userMetadata#505406, engineInfo#505407] / +- 'UnresolvedTable [gold, zz_docs_g26_sales_orders], DESCRIBE HISTORY
developer-handbook / 26-unity-catalog-step-by-step 746 RESTORE TABLE gold.zz_docs_g26_sales_orders TO VERSION AS OF 3; output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; line 1 pos 0
developer-handbook / 27-kinetic-pen 129 SELECT region, sum(amount) AS total_amount FROM datasimple.silv gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
developer-handbook / 27-kinetic-pen 136 from pyspark.sql import functions as F df = read_table("silver.zz_doc gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
developer-handbook / 27-kinetic-pen 148 SELECT order_id, region, customer_id, first_name, gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
developer-handbook / 27-kinetic-pen 162 SELECT * FROM datasimple.silver.zz_docs_k27_orders WHERE amount < 0 gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
developer-handbook / 27-kinetic-pen 168 SELECT * FROM datasimple.silver.zz_docs_k27_orders WHERE region = 'Eas process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / >
developer-handbook / 30-sql-python-reference 123 SHOW SCHEMAS; -- source, bronze, sil e queried on its own; join its result in Python with read_table("information_schema."). (references outside information_schema: silver.zz_docs_r30_orders)
developer-handbook / 30-sql-python-reference 172 -- A view: a saved query, read through its tables' access rules CREATE ders in a way DataSimple cannot check against the tables' access rules. Name it as . after FROM or JOIN (for example FROM gold.retail_sales_view).
developer-handbook / 30-sql-python-reference 245 CREATE MATERIALIZED VIEW gold.mv_zz_docs_r30_sales_revenue_by_region_d ---------^^^ / AS SELECT region, order_date, SUM(net_amount) AS revenue / FROM silver.zz_docs_r30_sales GROUP BY region, order_date
developer-handbook / 30-sql-python-reference 255 SELECT * FROM datasimple.information_schema.materialized_views; syntax error at or near ";"
developer-handbook / 30-sql-python-reference 262 -- Append: every new source row is added once CREATE STREAMING TABLE s datasimple.silver.zz_docs_r30_orders_log already exists. Use CREATE OR REFRESH STREAMING TABLE to change it, or REFRESH STREAMING TABLE to load the new rows.
developer-handbook / 30-sql-python-reference 279 REFRESH STREAMING TABLE silver.zz_docs_r30_orders_live; -- on == SQL == / REFRESH STREAMING TABLE silver.zz_docs_r30_orders_live / ^^^
developer-handbook / 30-sql-python-reference 372 -- Columns ALTER TABLE silver.zz_docs_r30b_customers ADD COLUMNS (emai ualifiedColType(None,visits,IntegerType,true,None,None,None), QualifiedColType(None,fax,StringType,true,None,None,None), QualifiedColType(None,telex,StringType,
developer-handbook / 30-sql-python-reference 395 -- Rename the table: from here on it is silver.customer_master ALTER T xception(info, status.message) from None / pyspark.errors.exceptions.connect.SparkConnectGrpcException: (scala.NotImplementedError) an implementation is missing
developer-handbook / 30-sql-python-reference 398 -- Views: change one by creating it again CREATE OR REPLACE VIEW gold. ders in a way DataSimple cannot check against the tables' access rules. Name it as . after FROM or JOIN (for example FROM gold.retail_sales_view).
developer-handbook / 30-sql-python-reference 402 -- Drop and bring back DROP TABLE silver.zz_docs_r30b_customer_master; Nothing to UNDROP: no dropped table 'silver.zz_docs_r30b_customer_master' within the retention window.
developer-handbook / 30-sql-python-reference 422 SELECT * FROM silver.zz_docs_r30b_orders TIMESTAMP AS OF '2026-10-16 1 tamp (2026-10-16 18:00:00.0) is before the earliest version available to this / table (2026-11-04 17:25:36.0). Please use a timestamp after 2026-11-04 17:25:36.
developer-handbook / 30-sql-python-reference 668 check_table("silver.zz_docs_r30b_orders_clean", lambda c: c.isComplete = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
developer-handbook / 30-sql-python-reference 749 REFRESH STREAMING TABLE silver.zz_docs_r30b_orders_current; REFRESH MA == SQL == / REFRESH STREAMING TABLE silver.zz_docs_r30b_orders_current / ^^^
developer-handbook / 30-sql-python-reference 759 @ds.table("silver.zz_docs_r30b_orders_current_ds", mode="merge", keys= If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches.
developer-handbook / 30-sql-python-reference 776 CREATE STREAMING TABLE silver.zz_docs_r30b_orders_log AS SELECT * FROM CREATE STREAMING TABLE silver.zz_docs_r30b_orders_log / -------^^^ / AS SELECT * FROM STREAM bronze.zz_docs_r30b_dbo_orders
developer-handbook / 30-sql-python-reference 792 CREATE MATERIALIZED VIEW gold.mv_zz_docs_r30b_sales_revenue_short SCHE e queried on its own; join its result in Python with read_table("information_schema."). (references outside information_schema: silver.zz_docs_r30b_sales)
guide / 03-lake-targets-and-formats 64 SELECT snapshot_id, committed_at FROM datasimple.bronze.zz_docs_g03_or "details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.bronze.zz_docs_g03_orders_iceberg.snapshots"}
guide / 06-silver-apply-changes 126 @ds.table("silver.zz_docs_g06_orders_clean", mode="merge", keys=["Orde If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches.
guide / 07-notebooks 160 SELECT * FROM datasimple.bronze.zz_docs_n07_nation_eu TIMESTAMP AS OF tamp (2026-10-16 18:58:10.0) is before the earliest version available to this / table (2026-11-04 16:20:30.0). Please use a timestamp after 2026-11-04 16:20:30.
guide / 07-notebooks 161 SELECT snapshot_id, committed_at FROM datasimple.bronze.zz_docs_n07_or "details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.bronze.zz_docs_n07_orders_iceberg.snapshots"}
guide / 07b-sql-reference 162 CREATE TABLE silver.zz_docs_g07b_events (event_id BIGINT, event_type S ts` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects.
guide / 07b-sql-reference 182 INSERT INTO silver.zz_docs_g07b_customers VALUES (1001, 'Ana Lopez', ' b)) / ValueError: Constraint violation on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1001,) x3. The write was rolled back.
guide / 07b-sql-reference 184 INSERT INTO silver.zz_docs_g07b_customers SELECT customer_id, customer on on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1002,) x2; (1003,) x2; (1007,) x2; (1001,) x3. The write was rolled back.
guide / 07b-sql-reference 190 UPDATE silver.zz_docs_g07b_customers SET segment = 'Premier' WHERE cus on on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1002,) x2; (1003,) x2; (1007,) x2; (1001,) x3. The write was rolled back.
guide / 07b-sql-reference 197 MERGE INTO silver.zz_docs_g07b_customers t USING (SELECT * FROM bronze aint violation on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1002,) x2; (1003,) x2; (1001,) x3. The write was rolled back.
guide / 07b-sql-reference 260 CREATE STREAMING TABLE silver.zz_docs_g07b_orders_log COMMENT 'Every o datasimple.silver.zz_docs_g07b_orders_log already exists. Use CREATE OR REFRESH STREAMING TABLE to change it, or REFRESH STREAMING TABLE to load the new rows.
guide / 07b-sql-reference 282 UNDROP STREAMING TABLE silver.zz_docs_g07b_orders_live; -- b ve` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects.
guide / 07b-sql-reference 371 ALTER TABLE silver.zz_docs_g07b_customers RENAME COLUMN phone TO phone umnMapping.mode' = 'name') / Note, if your table is not on the required protocol version it will be upgraded. / Column mapping requires at least protocol (2, 5)
guide / 07b-sql-reference 372 ALTER TABLE silver.zz_docs_g07b_customers DROP COLUMN email; umnMapping.mode' = 'name') / Note, if your table is not on the required protocol version it will be upgraded. / Column mapping requires at least protocol (2, 5)
guide / 07b-sql-reference 373 ALTER TABLE silver.zz_docs_g07b_customers DROP COLUMNS (fax, telex); umnMapping.mode' = 'name') / Note, if your table is not on the required protocol version it will be upgraded. / Column mapping requires at least protocol (2, 5)
guide / 07b-sql-reference 375 ALTER TABLE silver.zz_docs_g07b_customers ALTER COLUMN visits TYPE BIG onnect.AnalysisException: [DELTA_UNSUPPORTED_ALTER_TABLE_CHANGE_COL_OP] ALTER TABLE CHANGE COLUMN is not supported for changing column visits from INT to BIGINT
guide / 07b-sql-reference 380 ALTER TABLE silver.zz_docs_g07b_customers ADD CONSTRAINT valid_segment e-serverless/delta-table-emrtest/__uc_managed/silver/zz_docs_g07b_customers` violate the new CHECK constraint (segment IN ( 'Retail' , 'Business' , 'Premier' ))
guide / 07b-sql-reference 381 ALTER TABLE silver.zz_docs_g07b_customers DROP CONSTRAINT valid_segmen ror, provide the parameter IF EXISTS or set the SQL session configuration spark.other lakehouse platforms.delta.constraints.assumesDropIfExists.enabled to true.
guide / 07b-sql-reference 392 ALTER TABLE silver.zz_docs_g07b_customers RENAME TO silver.zz_docs_g07 xception(info, status.message) from None / pyspark.errors.exceptions.connect.SparkConnectGrpcException: (scala.NotImplementedError) an implementation is missing
guide / 07b-sql-reference 399 DROP TABLE silver.zz_docs_g07b_customer_master; -- rrent_schema() output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.
guide / 07b-sql-reference 400 UNDROP TABLE silver.zz_docs_g07b_customer_master; -- Nothing to UNDROP: no dropped table 'silver.zz_docs_g07b_customer_master' within the retention window.
guide / 07b-sql-reference 401 DROP TABLE zz_docs_g07b_sandbox.zz_docs_g07b_scratch PURGE; ssage) from None / pyspark.errors.exceptions.connect.UnsupportedOperationException: [UNSUPPORTED_FEATURE.PURGE_TABLE] The feature is not supported: Purge table.
guide / 07b-sql-reference 416 SELECT * FROM silver.zz_docs_g07b_orders_hist TIMESTAMP AS OF '2026-10 tamp (2026-10-16 18:00:00.0) is before the earliest version available to this / table (2026-11-04 16:47:13.0). Please use a timestamp after 2026-11-04 16:47:13.
guide / 07c-python-reference 141 @ds.table("silver.zz_docs_g07c_orders_clean", mode="merge", keys=["Ord If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches.
guide / 07c-python-reference 168 check_table("silver.zz_docs_g07c_orders_clean", lambda c: c.isComplete = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
guide / 07d-streaming-and-incremental 65 @ds.table("silver.zz_docs_g07d_py_orders_current", mode="merge", keys= If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches.
guide / 07d-streaming-and-incremental 82 CREATE STREAMING TABLE silver.zz_docs_g07d_orders_log AS SELECT * FROM CREATE STREAMING TABLE silver.zz_docs_g07d_orders_log / -------^^^ / AS SELECT * FROM STREAM bronze.zz_docs_g07d_dbo_orders
guide / 07d-streaming-and-incremental 99 CREATE MATERIALIZED VIEW gold.mv_zz_docs_g07d_sales_revenue_by_region_ z_docs_g07d_sales_revenue_by_region_daily already exists. Use CREATE OR REPLACE MATERIALIZED VIEW to change it, or REFRESH MATERIALIZED VIEW to update its data.
guide / 07d-streaming-and-incremental 109 SELECT * FROM datasimple.information_schema.materialized_views; -- syntax error at or near ";"
guide / 09-lakehouse-governance 151 SELECT kind, count(*) AS runs, sum(spark_seconds) AS spark_seconds FRO syntax error at or near ";"
guide / 09-lakehouse-governance 219 CREATE MATERIALIZED VIEW gold.mv_zz_docs_g09_sales_revenue_by_region_d zz_docs_g09_sales_revenue_by_region_daily already exists. Use CREATE OR REPLACE MATERIALIZED VIEW to change it, or REFRESH MATERIALIZED VIEW to update its data.
guide / 09-lakehouse-governance 264 ALTER TABLE datasimple.gold.zz_docs_g09_orders ADD CONSTRAINT fk_zz_do ---------------------------------------------------------------------------^^^ / FOREIGN KEY (customer_id) REFERENCES gold.zz_docs_g09_customers (customer_id)
guide / 09-lakehouse-governance 318 profile_table("silver.zz_docs_g09_orders_scored") check_table("silver. = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
guide / 09-lakehouse-governance 230 SELECT * FROM gold.mv_zz_docs_g09_sales_revenue_by_region_daily; +- 'LocalLimit 200 / +- 'Project [*] / +- 'UnresolvedRelation [gold, mv_zz_docs_g09_sales_revenue_by_region_daily], [], false
guide / 13-reference 55 REFRESH STREAMING TABLE silver.zz_docs_g13_orders_live; -- FU == SQL == / REFRESH STREAMING TABLE silver.zz_docs_g13_orders_live / ^^^
guide / 13-reference 129 @ds.table("silver.zz_docs_g13_py_orders_clean", mode="merge", keys=["O If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches.
guide / 13-reference 135 grant("SELECT", "silver.zz_docs_g13_py_orders_live", "analyst") profil = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
user-guide / 02-notebooks 79 SELECT snapshot_id, committed_at FROM datasimple.source.zz_docs_u02_tp :[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.source.zz_docs_u02_tpch_sf1_nation_iceberg.snapshots"}
user-guide / 02-notebooks 148 check_table("silver.zz_docs_u02_nation_priority", lambda c: c.isComple = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
user-guide / 02-notebooks 163 SELECT * FROM datasimple.source.zz_docs_u02_tpch_sf1_nation_delta WHER FROM source.zz_docs_u02_tpch_sf1_nation_delta WHERE N_REGIONKEY = {{region_id}} / ---------------------------------------------------------------------------^^^
user-guide / 04-lakehouse 46 check_table("bronze.zz_docs_u04_nation_eu", lambda c: c.isComplete("na = importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ'
user-guide / 06-apply-changes 95 APPLY CHANGES INTO silver.zz_docs_a06_orders_current FROM bronze.zz_do gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
user-guide / 06-apply-changes 186 @ds.table("silver.zz_docs_a06_orders_clean", mode="merge", keys=["Orde Invalid column names: sum(Amount). / Please use other characters and try again. / Alternatively, enable Column Mapping to keep using these characters.
guide / 07b-sql-reference UNDROP TABLE silver.zz_docs_cmd_customers rs` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects.
guide / 07b-sql-reference INSERT INTO silver.zz_docs_cmd_ice VALUES (1, 'us'), (2, 'eu') nt 'UnresolvedRelation [silver, zz_docs_cmd_ice], [required_write_privileges=INSERT], false, false, false, false / +- LocalRelation [col1#13413, col2#13414]
guide / 07b-sql-reference SELECT snapshot_id, committed_at FROM silver.zz_docs_cmd_ice.snapshots _ARGUMENT","details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.silver.zz_docs_cmd_ice.snapshots"}

Commands, by how often their documented examples ran#

Every documented example that uses the command, on dev on 2026-11-04. A failure can be the command or the example around it; the table below lists each one.

Command Examples Worked
APPLY CHANGES (SQL) 38 37
ds.apply_changes (Python) 19 19
MERGE CHANGES / MERGE SNAPSHOT 1 0
Streaming tables 27 19
Materialized views 27 17
ALTER TABLE (columns, constraints, properties) 34 15
COMMENT ON 8 6
Time travel (VERSION / TIMESTAMP AS OF) 18 14
RESTORE TABLE 6 5
OPTIMIZE / VACUUM 4 4
DROP / UNDROP 12 4
Views (CREATE VIEW) 6 4
GRANT / REVOKE 23 19
information_schema 66 57
Iceberg tables (USING ICEBERG) 9 8
save_table 16 12
merge_delta 5 5
DS Tables (@ds.table, ds.run) 10 2
Star schema (@ds.dimension / fact / obt) 9 5
check_table (data quality checks) 13 0

Known gaps on dev, being fixed (see the list of failed examples above): check_table needs a library that dev's remote Spark cannot use (a built-in replacement is planned); ALTER TABLE ... RENAME TO and DROP ... PURGE with the standalone catalog server; a cell that mixes a streaming-table or materialized-view command with other statements in a scheduled run; DS Tables re-runs with the standalone catalog server.

Features#

92 features: 44 tested, 27 partly tested, 21 not tested.

Connect sources#

Feature Where Status Proven (date, environment) Proof Still to prove
Add / test a Data Source (DS-n) Data Source tab; DataBridge step 1 Tested 2026-10-29, both DS-22 MFG_SQLServer, DS-23 MFG_SensorEvents; DS-24, DS-25; local Postgres, saved connection (deleted); DS-18 CdcDemo-SQL; DS-20 Kafka-Test; DS-16 (Snowflake), DS-15 Target_AWS Only SQL Server, Postgres, Snowflake, Kafka, S3 and file drop were exercised live. Oracle, MySQL, Db2 (live, not as backup), SAP HANA, OData, Salesforce, MongoDB, Azure: tested with fakes only (phase-3 'No live source').
Cloud connect (S3, GCS, Azure Blob, ADLS) Data Lake tab Partly tested 2026-10-29, local gold/mfg_*, silver/mfg_*; 22.3 Data Lake Only the S3 lake (Target_AWS DS-15) was browsed. GCS, Azure Blob and ADLS connections never opened (phase-3: no live Azure). Adding a cloud connection through the dialog is not shown.
Database drivers Server side (blocked from the browser) Not tested No document reports installing a dynamic driver or an engine appearing in DataBridge. Browser access is blocked by design (phase-0 test of the 403 on /api/install-python-driver is a unit test plus a local read-only request).
Git for a notebook Notebook > Git dialog Not tested Browser click-through: Git No run of the per-notebook Git dialog (commit, pull, branch, PR) is reported; needs a personal access token.

Ingest#

Feature Where Status Proven (date, environment) Proof Still to prove
Bronze apply (automatic after a load, or by hand) Bronze loads tab Tested 2026-10-29, local job 141; bronze.mfg_*; bronze.retail_*; _batch_id 168; job 183; job 74 Manual 'Apply to bronze' for Delta/Iceberg Bronze merge and the retry-Bronze button are not shown with results.
Hash CDC Bronze log Implicit (Parquet Bronze with CDC or watermark loads) Tested 2026-10-28, local public.zz_r3_big3; cdc/part-*.parquet in dev S3; zz_r3_big2, zz_r3_big3 ADLS keeps the single-file log and was never run live; catalog registration of the log, compaction of old parts and the 403-on-new-table case are open.
Detect deletes (delete capture) DataBridge step 4; Load History > Schedules Tested 2026-10-27, local zz_r3_dc on connection CdcDemo-PG, bucket Target_AWS; job 163 Daily cadence only tested with fakes; Salesforce deleted records, and the rule that built-in 'Apply changes to Silver' does not receive these deletes, not run live.
File ingest Workflow 'files' step; autoIngest on a file connection Tested 2026-10-27, local bronze.zz_r3_files; landing/zz_r3_files/; file_stream.py Workflow 'files' step and Everything 'Load into Bronze' card not run; autoIngest scheduling see the scheduled file ingest row.
Kafka topic read directly in SQL (CREATE STREAMING TABLE ... STREAM KAFKA) SQL cell Tested 2026-11-04 STREAM KAFKA('DS-20','orders') Environment not stated; schedule (SCHEDULE CRON) firing not reported.
Streamed first load and part-file Parquet copies for large tables DataBridge (automatic over a row threshold) Tested 2026-10-27, local zz_r3_big* in dev S3; job 21 S3 only; ADLS keeps single-file merge and was not run. Stale Spark read after a full reload is a known limit.
Audit trail (bookkeeping) columns on every layer (_op_type, _batch_id, _load_ts, ...) Notebook result grid > Audit columns Tested 2026-10-30, local notebook MFG 08 Bronze change types; RTL 07 Protected-column rules and Silver Type 2 audit columns for Iceberg are not separately shown.
DataBridge load (five-step wizard): full, watermark, change-log CDC, file drop, backup drop, Kafka, Camel Databridge tab Partly tested 2026-10-30, both DS-22; jobs 141, 145, 148, 158, 163; DS-24; jobs 165-171; DS-18; job 74; jobs 27, 28, 29; FIN_CoreBanking; jobs 183, 186; jobs 40-49 (also 39); jobs 50-61; DS-20 Kafka-Test; job 107; DS-16; jobs 15-21 (EC2 box and laptop Camel connectors never run. Backup drop for Postgres/MySQL is 'recipe ready, run on request'; Oracle and SAP HANA 'recipe only, untested flag'. No live Oracle, MySQL, Salesforce, SAP, ADLS source or target. Postgres logical-decoding CDC has a documented the container test environment but no result table.
Reconcile No screen; API /api/databridge/api/reconcile/* Partly tested 2026-10-30, local batch reconcile-2026-10-30; Tests The reconcile endpoint itself (tombstone mode and event_log mode) is not shown run end to end; tombstone mode untested.
Load control (reset, lookback, disable) Bronze loads tab and the DataBridge page Partly tested 2026-10-30, local jobs 148, 161 Only viewing positions is shown; reset, lookback change and disable never exercised.
Bulk load DataBridge wizard checkbox Not tested Section 1 inventory (Bulk bulk.py + bulk_spark.py) No run with 'Bulk load into Source tables' ticked (Snowflake COPY INTO or Spark JDBC) is reported. Bulk-refused-for-Bronze rule only in code.
File copy No React screen (API) Not tested Section 14: Tenant scope for files/copy and reconcile No real run of /files/copy (raw file copy, optional table registration) reported.
Source -> Delta Notebook wizard Not tested No document reports using the notebook Source-to-Delta wizard (100k default rows, overwrite/append).
Kafka streaming through Flink Workflow, ingestion mode 'streaming' Not tested No run of the Flink job path; Flink appears only in docs and unit tests.
dlt engine None (deprecated shim) Not tested Deprecated shim; no run reported.
New-file alerts Everything > File alerts Not tested A day with Sarah, by voice, step 2 The file watcher is off on the development server (FILE_ALERTS_ENABLE is not set); it needs that setting in the deploy configuration before an alert can be shown.
Load to ADLS (Azure) targets DataBridge destination Not tested Not tested here No live ADLS load, incremental load or file drop on ADLS.
Stop a DataBridge job / table, memory guard and streamed retry Load History, Ingestion Not tested Whole database again, streamed code Stop job, per-table stop and the memory-guard queue/retry are described but no run is reported.

Transform#

Feature Where Status Proven (date, environment) Proof Still to prove
SQL and Python cells Notebooks Tested 2026-10-29, both notebooks MFG 01-11 (NB-82 = MFG 01); gold.zz_lt_pipe_a, gold.e2e_orders; silver.zz_p1_tenant_check Interactive cell limit is 5 minutes (Gold build hit it); Python cell three-part-name bug found in first-run-research is an open finding.
DataBridge "Apply changes to Silver (MERGE)" DataBridge step 4 Tested 2026-10-16, local silver.dbo_Orders, silver.dbo_Orders_HISTORY; jobs 75, 77; Worked example, end to end Type 2 needs the Spark box and was not run without it; ADLS not run. The demos set this option to 'Not now'.
ds.apply_changes / ds.apply_changes_from_snapshot Python cell Tested 2026-10-27, both silver.orders_current, orders_history, orders_snapshot; silver.zz_p2_scd2_dim; silver.zz_p5_scd2_seq, ..._old Iceberg targets and the legacy fallback path not run live.
APPLY CHANGES [FROM SNAPSHOT] INTO SQL cell Tested 2026-10-30, both NB-82 MFG 01; silver.mfg_*; RTL 01, RTL 02; zz_decl_* tables The FROM SNAPSHOT SQL spelling was verified only on 2026-10-18 for the Python form; SQL snapshot form not shown with a result.
DS Tables (ds.table, ds.run()) Python cell Tested 2026-10-30, both MFG 04, MFG 09, MFG 10; gold.mfg_*; RTL 04; gold.retail_*; zz_decl_*; ds.dimension section
Kinetic pen Kinetic tab Tested 2026-10-30, local silver.mfg_sales_orders, mfg_work_orders; Kinetic tab; silver.retail_store_inventory, retail_sales_transactions Voice input and AI-fallback accuracy not tested; Semantics-metric export from Kinetic not shown.
dbt run / test Pipelines (dbt step); Workflow step Tested 2026-11-01 pipeline 'RETAIL dbt Silver and Gold'; dbt/retail_dbt; silver.dbt_retail_*, gold.dbt_retail_*; silver.ext_*, silver.spk_*; Phase 1: A real dbt run for lineage dbt as a Workflow step not run; dbt with a warehouse other than this workspace not run.
Data Vault page: hubs, links, satellites, PIT, KPI views Data Vault page Tested 2026-11-04 notebook #133 'Data Vault . finance', pipeline #35, runs 130/133; hub_fin_*, gold.pit_fin_*; gold.dv_retail_sales_by_region_month; Which scenarios are tested Unit-tested only: composite key proposed by scoring, empty multi-active set, parent changing after events. Not built: same-as and hierarchical links. Not run with a second real user.
MERGE CHANGES / MERGE SNAPSHOT INTO SQL cell Partly tested 2026-10-27, local zz_decl_* tables MERGE SNAPSHOT INTO never run; MERGE CHANGES only on a small Delta test table, not on Iceberg or a demo project.
Workflow medallion (Bronze MERGE, Silver, SCD2 checkbox, Gold) Workflow tab Partly tested 2026-10-30, local workflow 'MFG quality inspections (Workflow)'; mfg_wf_quality_inspections; workflow id 11; retail_wf_returns; zz_lt_workflow, gold.zz_lt_wf_summary SCD2 checkbox (workflow-scd2) and chat-built workflows (Sarah canvas / Everything workflow_build) not shown run; API weakness: unknown 'after' silently replaced.
Materialized views (... SCHEDULE CRON) SQL cell Partly tested 2026-10-30, local MFG 11; mv_mfg_*; RTL 08; zz_decl_* No report of a materialized view refreshed by its own cron timer or of the failed-refresh alert firing (alert is unit-tested).
Notebook parameters ({{param}} / Params cell) Notebook cell; pipeline step parameters Partly tested 2026-10-21, local notebook E2E 5 Parameter Params cell and parameter use inside an interactive notebook not shown; Chart cell not shown.
Stop a running cell; fair Spark sessions; 'session restarted' notice Notebooks; Spark status Partly tested 2026-10-27, local Phase 2: Stop on the real Spark; 3. Fair Spark sessions: live check on the running app; Not checked live Several users competing for sessions on the dev Spark never run live.
Flink SQL Pipelines (Flink step); Workflow step Not tested No run of a Flink SQL step.
Chart cell and result charts in a notebook Notebook Chart cell Not tested No run reported (Sarah's charts are a different feature).

Orchestrate#

Feature Where Status Proven (date, environment) Proof Still to prove
Notebook schedule Notebook > Schedule; Schedules tab Tested 2026-10-27, local zz_lt_pipe_a schedule; schedule 16; Phase 0: real restart Catch-up of a missed nightly run after a restart tested with a fake clock only; retry_count and email notification open.
Pipeline Pipelines tab Tested 2026-10-30, both pipelines 'E2E orders pipeline', 'E2E failure handling'; pipeline id 20; runs 93, 96, 99; pipeline id 24; runs 107, 109; zz_lt_pipeline_dag, zz_lt_pipeline_stop; run 57, 76 Email/webhook notification delivery open; QA-side runs not reported.
Workflow, "Schedule as pipeline" Workflow tab Tested 2026-10-27, local zz_lt_workflow and its schedule pipeline; 22.1 The one-step pipeline's run history for a demo workflow is not shown.
Dataflow Dataflow tab Tested 2026-10-30, both DAG e2e_orders_dataflow; zz_lt_flow_notebook, zz_lt_flow_pipeline; DAG mfg_daily_lakehouse_build; DAG retail_daily_lakehouse_build Daily cron of a DAG never fired on its own (all created paused and run manually); notebook/cell DAG running as principal 'dataflow' with grants only described.
DataBridge schedule DataBridge page > Load History > Schedules Partly tested 2026-10-18, local job 74; job 107 Only 'Run now' was used; no report of a schedule firing on its own interval, pause/resume, or Detect-deletes daily cadence.
Materialized view refresh SQL SCHEDULE CRON Partly tested 2026-10-30, local gold.mv_mfg_fulfilment_monthly No timer-driven refresh observed; failed-refresh alert unit-tested only.
Scheduled file ingest No screen (connection setting autoIngest) Not tested No run of autoIngest on a timer; only manual/API file ingest was run.
DQ sweep No screen (API /api/notebooks/dq/sweeps) Not tested 12. Data-quality sweep reports failed tables No sweep created or fired on a real cron.
Run alerts: email and webhook notify (every run / only on failure) Pipeline editor 'Notify when'; Schedules Not tested 2026-10-27, local Browser click-through; Phase 2: Email alert delivery No email or webhook delivery reported.

Quality#

Feature Where Status Proven (date, environment) Proof Still to prove
Data Quality dashboard Data Quality tab Tested 2026-10-29, both Data Quality filter 'mfg_'; 3 rules on gold.zz_lt_pipe_a; 21.6 PII tag setting not shown; governance finding: hand-made rule starts 'approved' (decision open).
DS Tables rules Python cell Tested 2026-10-30, both MFG 09, MFG 10; zz_decl_orders
Checks in notebooks (deequ_check, check_table, profile_table, table_freshness) Notebook cells Partly tested 2026-10-29, local notebook MFG 06; RTL 06; gold.zz_lt_pipe_a profile_table and table_freshness not shown run.
Quality rules and certification by voice Everything / Agent Partly tested silver.dsdemo2_orders; 22.1 No certificate actually issued and approved; voice not run; approval-card flow for rule creation only shown as walkthrough text.
Data Quality: freshness, profiles, quarantine, coverage, certificates Data Quality tab (Coverage, Suggestions, Anomalies, Alerts) Partly tested 2026-10-27, local Data Quality; Declarative pipelines Freshness, anomalies, suggestions and certificates tabs not shown with results.

Semantics and BI#

Feature Where Status Proven (date, environment) Proof Still to prove
Glossary terms, metrics, table and column meaning Semantics tab Tested 2026-10-29, both metrics MFG Net revenue, MFG Inspection fail rate, MFG Scrap rate; Step 12; term 'zz LT Customer Total', metric 'zz LT total amount', join zz_lt_orders_x_totals Grand-total metric with saved dimensions not possible (known limit).
Sarah: ask for a report Sarah tab; Everything Tested 2026-10-30, both Sarah report on gold.mfg_*; gold.retail_sales_view, gold.retail_store_day_view, shared_store_sales_by_region_test; gold.mfg_*_view Live voice (OPENAI key) and a second real user's access rules (store manager sees only 'department') shown by screenshot only.
dbt SQL gateway External (bearer token per dbt client) Tested 2026-11-02 dbt connection e2e_flow, owner client:001:e2e_flow; silver.e2e_stores, gold.e2e_region_stores
Meaning auto-draft Background, every 6 hours Partly tested 2026-10-27, local gold.zz_lt_pipe_a The 6-hourly sweep itself and the approval of auto-drafted meanings are not shown.
Star schema / One Big Table; what-if scenarios Everything Partly tested star_demo_* tables; ds.dimension section Scenario engine and saving a scenario ('cautious plan') not shown run; approval/pipeline creation of the star not shown with ids.

Govern#

Feature Where Status Proven (date, environment) Proof Still to prove
Register and browse tables Lakehouse tab; every load and write Tested 2026-10-29, both Lakehouse, datasimple catalog; Update 2026-10-15 / tenant isolation 2026-10-16; silver.zz_p1_tenant_check External Unity Catalog server mode (when configured) not covered in these docs; local DB has 110 ownerless tables never backfilled.
Lineage Lineage tab Tested 2026-10-30, both gold.mfg_fact_sales; Q-1227; Pipelines / Other findings; Step 4 Impact mode not shown; SQL CTAS lineage in scheduled runs was a gap on 2026-10-21 (later recorded per live test).
Definition log and information_schema Lakehouse > object history; SQL Tested 2026-10-30, local 21.9; Pipelines; silver.zz_p1_tenant_check Redaction of definitions without SELECT (phase 5 round 1) is unit-tested only ('Checked live: -').
Trash and UNDROP Lakehouse (Trash); SQL UNDROP TABLE Tested 2026-10-27, local gold.zz_tenant_probe; UNDROP section
Tenant isolation (tenant id on every object and write) Sign-in; every layer Tested 2026-10-27, both Tenant isolation 2026-10-16; silver.zz_p1_tenant_check; Data Share
Unity Catalog REST (open-source subset) /api/2.1/unity-catalog Partly tested 2026-10-27, local Tests: unity-catalog-owner.test.js; 8. Verified working flows Spark or an external tool actually speaking the protocol to the app is not reported.
Grants, row filters, column masks, volumes, credentials, locations, UC audit Lakehouse; Administration > Governance Partly tested 2026-11-04, both rules: greenville_plant_only; gold.mfg_obt_sales, silver.mfg_customers; chicago_store_only x2; Grants section; bronze.zz_doc21_public_customers (mask on name, filter zz_west_only) Volumes, credentials, locations and the UC audit view never shown run.
Time travel, restore a version, Iceberg snapshots and rollback SQL cell; Lakehouse history Partly tested 2026-10-26, local NATION tables; Phase 1 RESTORE TABLE (admin only) and Iceberg rollback never run.
PII tags and display masking Data Quality > table detail Not tested No run of setting a column PII tag or seeing a masked display. (Column masks, a different feature, are tested.)
Retention sweeper Background Not tested 3. Catalog-screen delete goes to Trash Never run.
Iceberg maintenance Background Not tested No run of snapshot expiry or compaction reported.
Volumes, functions and models in the catalog Lakehouse Not tested No run reported.

Share#

Feature Where Status Proven (date, environment) Proof Still to prove
DataShare (Delta Sharing): shares, recipients, profile file, revoke DataShare tab Tested 2026-10-27, local share zz_lt_share (recipient revoked); mfg_fulfilment_share; retail_supplier_share Real client (Spark, pandas, Power BI) not used; profile file download and 15-minute link not shown; creating a share needs only the Sharing tab (decision open).
dbt clients Pipelines > dbt client admin Tested 2026-11-02 e2e_flow; Tested on 2026-11-02 with dbt-spark 1.10.3
Personal API tokens Settings Tested 2026-10-27, local dsp_TElQ... (revoked)

Deploy and promote#

Feature Where Status Proven (date, environment) Proof Still to prove
Git integration: commit a business model, pull request, Update from Git Deployment > Git panel; Administration > Git integration Tested 2026-11-01 repo datasimple/ds_git_deploy branches Dev, QA, Prod; pipelines 'RETAIL Daily Lakehouse Build'; Verified 2026-10-31 Three tenants of one install used as three environments; Roll back not run.
Deployment screen: register QA/Prod, compare, deploy notebooks, pipelines, workflows, lake objects (created empty) and grants; history; restore plan and restore Deployment tab Partly tested 2026-11-01, both notebooks MFG 01, 02, 04; metrics MFG Net revenue, mfg_net_unit_price, mfg_scrap_rate_pct; schema zz_r3_deploy; zz_dim_customer, v_dim_current; Browser click-through: Deployment; 22.2 Deployment Never run between two real instances (QA/Prod not reachable); Roll back and restore plan, prod approval request and per-environment values not run live; Phase 0-4 deploy tests use stubs.
Code deploy Outside the app (the Deploy workflow) Partly tested 2026-10-16, dev containers datasimple-qa :5002, datasimple-dev :5001; Checked on the running app No document records a Deploy workflow run result; QA is on a 2026-09-15 build (165 commits behind on 2026-10-16).
Notebook versions -> restore Notebook menu Not tested No run of restoring a notebook version.

Monitor and admin#

Feature Where Status Proven (date, environment) Proof Still to prove
History History tab Tested 2026-10-29, both 22.7 History — who did what; 22.7
Log Log tab Tested 2026-10-29, both Q-726; NB-82; Browser click-through: Log > Run log > Errors
Load History DataBridge page Tested 2026-10-30, both jobs 74, 75, 77; jobs 148, 158 Schedules panel (Pause, Delete) not shown beyond Run now.
Ingestion Ingestion tab (administrators) Tested 2026-10-30, both jobs 141-163; jobs 165, 168; Step 6 Cancel and 'retry Bronze' buttons not shown run.
Usage Usage tab Tested 2026-10-30 22.10 Usage — what it costs; needtofix H75 Export CSV and ledger accuracy not verified; H75 open.
Notebook Query history dialog Notebook > Query history Tested 2026-10-27, local Browser click-through: Notebook > Query history
Users, roles, tab permissions, governance, Activity Logs, Error Logs Administration Partly tested 2026-11-04, both Browser click-through: Admin > Roles; 22.12 Administration; Step 10; Administration > Users / Roles; roles 8-17 Editing a role's permissions on dev waits for the deploy of 12f77c2; the access check only reports (ACCESS_GUARD_MODE=log), and signing in as a role-limited user is not done yet; SSO/OIDC/SAML sign-in and Activity/Error Logs content not shown.
Settings Settings tab Partly tested 2026-11-04, both 22.11 Settings — your defaults; 2. Personal API tokens; Settings tab Spark warm window and per-user notebook defaults effect not shown.
Sign-in with tenant id, roles, and single sign-on Login page Partly tested 2026-10-27, local Other findings; 2. Personal API tokens Google/OIDC/SAML SSO and email-domain checks (admin page) never run.
ELS_DB browser ELS_DB tab Partly tested 2026-10-30 22.5 ELS_DB; H49 SQL/preview not available (H49).
App activity feed /api/activity (no screen calls it) Not tested Unused by design (no writers); nothing to run.

Assistants#

Feature Where Status Proven (date, environment) Proof Still to prove
Sarah Sarah tab Tested 2026-10-30, both 6. Sarah; Step 19 Live voice not tested.
Agent Agent tab Tested 2026-10-30, both Agent screen; 22.6; Custom groups section Approval-queue approve/reject on a demo write card not shown with a result in the demos.
Kinetic Kinetic tab Tested 2026-10-30, local Step 20; Step 20 Voice input not tested.
Sarah: business views, forecasts, chart filters, Open in notebook, Share with Sarah Sarah tab Tested 2026-10-30 gold.shared_store_sales_by_region_test, ...view; gold.mfg*_view
Everything Header toggle Partly tested silver.dsdemo2_orders; star_demo_* tables Voice never run; file alerts, deploy card, grant card, scenario save, table backup/restore cards not shown run; no dates or ids.
RoleAgent Administration sub-tab Partly tested 2026-10-30 22.12 Administration Toggling access and its effect on a user not shown; scope still 'to confirm' in the catalogue.

Not testable on dev yet#

These need a system that is not connected to the development server. They are marked not tested above until they are run:

  • Google Cloud Storage, Azure Blob and ADLS lakes
  • Oracle, MySQL production, SAP, Salesforce and HANA sources; Camel connectors; Flink
  • Google, OIDC and SAML sign-in
  • Deployment between two real installations (QA, Prod)
  • Email delivery of alerts
  • Speaking a request to Everything (a typed request is tested)