Every feature this guide and the developer handbook describe, with whether it has been proven working by a real run, when, where, and where to see the proof in the app. Anything not proven is marked so: a feature is only called tested when a run with a result is on record.
Where to see the proof. Sign in to the workspace (tenant 001) and open:
- Notebooks → folder Doc checks: one notebook per document, holding that document's code examples as cells (table names prefixed
zz_docs_, so no real table is touched). Each notebook has a schedule named Doc check; its runs list every cell's result. - Log (Scheduled filter): one line per cell run, with the code, status, time and error.
- Lakehouse → object history on any
zz_docs_*table: what each example created or changed. - The practical guides (Manufacturing, Retail, Finance) for the end-to-end builds, with screenshots, numbers and times.
Environments. dev is the shared development server (Spark as a remote service, the standalone catalog server). local is a developer machine using the same lake. A result on one is not automatically a result on the other; the table says which.
Code examples in the documentation#
Run on dev on 2026-11-04 as scheduled notebook runs. 50 notebooks, 46 documents, 791 cells (442 of them the documents' own examples, the rest setup and result checks). 330 of 442 examples ran as written.
| Document | Examples run | Worked | Did not | Notebook (folder Doc checks) |
|---|---|---|---|---|
| DELTA-LOADS | 1 | 1 | 0 | Doc check - docs delta loads (run 51) |
| PHASE-A-AWS-DEPLOY | 3 | 2 | 1 | Doc check - docs phase a aws deploy (run 52) |
| UNITY-CATALOG-E2E | 4 | 3 | 1 | Doc check - docs unity catalog e2e (run 53) |
| UNITY-CATALOG-FULL-PARITY | 12 | 10 | 2 | Doc check - docs unity catalog full parity (run 54) |
| demo-manufacturing / PRACTICAL-GUIDE | 18 | 12 | 6 | Doc check - demo manufacturing practical guide (run 55) |
| demo-retail / PRACTICAL-GUIDE | 11 | 9 | 2 | Doc check - demo retail practical guide (run 56) |
| developer-guide / 02-lakehouse-storage | 6 | 1 | 5 | Doc check - dev guide 02 lakehouse storage (run 57) |
| developer-guide / 05-notebook-helpers | 1 | 0 | 1 | Doc check - dev guide 05 notebook helpers (run 58) |
| developer-guide / 06-end-to-end | 4 | 2 | 2 | Doc check - dev guide 06 end to end (run 59) |
| developer-guide / 08-notebook-cells | 3 | 3 | 0 | Doc check - dev guide 08 notebook cells (run 60) |
| developer-guide / 09-git-integration | 2 | 2 | 0 | Doc check - dev guide 09 git integration (run 61) |
| developer-guide / 10-dbt | 1 | 1 | 0 | Doc check - dev guide 10 dbt (run 62) |
| developer-guide / 11-dbt-from-your-own-dbt | 1 | 1 | 0 | Doc check - dev guide 11 dbt from your own dbt (run 63) |
| developer-handbook / 02-notebooks | 2 | 1 | 1 | Doc check - handbook 02 notebooks (run 64) |
| developer-handbook / 04-databridge | 2 | 2 | 0 | Doc check - handbook 04 databridge (run 65) |
| developer-handbook / 05-bronze-loads | 2 | 2 | 0 | Doc check - handbook 05 bronze loads (run 66) |
| developer-handbook / 07-lakehouse-and-governance | 4 | 2 | 2 | Doc check - handbook 07 lakehouse and governance (run 67) |
| developer-handbook / 08-pipelines | 1 | 1 | 0 | Doc check - handbook 08 pipelines (run 68) |
| developer-handbook / 10-schedules-and-scheduler | 1 | 1 | 0 | Doc check - handbook 10 schedules (run 69) |
| developer-handbook / 12-data-quality | 3 | 0 | 3 | Doc check - handbook 12 data quality (run 70) |
| developer-handbook / 15-observability | 1 | 0 | 1 | Doc check - handbook 15 observability (run 71) |
| developer-handbook / 20-lake-storage | 4 | 3 | 1 | Doc check - handbook 20 lake storage (run 72) |
| developer-handbook / 21-ds-tables-and-scd | 9 | 5 | 4 | Doc check - handbook 21 ds tables and scd (run 73) |
| developer-handbook / 23-spark-engine | 2 | 1 | 1 | Doc check - handbook 23 spark engine (run 74) |
| developer-handbook / 24-end-to-end-walkthrough | 7 | 5 | 2 | Doc check - handbook 24 end to end walkthrough (run 75) |
| developer-handbook / 26-unity-catalog-step-by-step | 21 | 10 | 11 | Doc check - handbook 26 unity catalog step by step (run 76) |
| developer-handbook / 27-kinetic-pen | 5 | 0 | 5 | Doc check - handbook 27 kinetic pen (run 77) |
| developer-handbook / 30-sql-python-reference | 46 | 30 | 16 | Doc check - handbook 30 sql python reference (part 1) (run 79); Doc check - handbook 30 sql python reference (part 2) (run 80) |
| guide / 03-lake-targets-and-formats | 4 | 3 | 1 | Doc check - guide 03 lake targets (run 81) |
| guide / 05-bronze | 4 | 4 | 0 | Doc check - guide 05 bronze (run 82) |
| guide / 06-silver-apply-changes | 7 | 6 | 1 | Doc check - guide 06 silver apply changes (run 83) |
| guide / 07-notebooks | 13 | 11 | 2 | Doc check - guide 07 notebooks (run 84) |
| guide / 07b-sql-reference | 114 | 93 | 21 | Doc check - guide 07b sql reference (run 85); Doc check - guide 07b sql reference part 2 (run 86); Doc check - guide 07b sql reference part 3 (run 87); Doc check - Delta and Iceberg commands on dev (run 101) |
| guide / 07c-python-reference | 5 | 3 | 2 | Doc check - guide 07c python reference (run 88) |
| guide / 07d-streaming-and-incremental | 10 | 6 | 4 | Doc check - guide 07d streaming and incremental (run 89) |
| guide / 08b-dbt-end-to-end-tested | 1 | 1 | 0 | Doc check - guide 08b dbt end to end tested (run 90) |
| guide / 09-lakehouse-governance | 35 | 30 | 5 | Doc check - guide 09 lakehouse governance (run 91); Doc check - guide 09 lakehouse governance (part 2) (run 92) |
| guide / 10-audit-trail-columns | 8 | 8 | 0 | Doc check - guide 10 audit trail columns (run 93) |
| guide / 11-monitoring-logs-recovery | 7 | 7 | 0 | Doc check - guide 11 monitoring logs recovery (run 94) |
| guide / 13-reference | 19 | 16 | 3 | Doc check - guide 13 reference (run 95) |
| guide / 15-ds-tables | 5 | 5 | 0 | Doc check - guide 15 ds tables (run 96) |
| user-guide / 01-connect-and-load | 1 | 1 | 0 | Doc check - user guide 01 connect and load (run 97) |
| user-guide / 02-notebooks | 22 | 19 | 3 | Doc check - user guide 02 notebooks (run 98) |
| user-guide / 04-lakehouse | 3 | 2 | 1 | Doc check - user guide 04 lakehouse (run 99) |
| user-guide / 06-apply-changes | 7 | 5 | 2 | Doc check - user guide 06 apply changes (run 100) |
Examples that did not run as written, and why:
| Document | Line | Example | Error |
|---|---|---|---|
| PHASE-A-AWS-DEPLOY | 171 | CREATE TABLE datasimple.bronze.zz_docs_pha_ice_test (id BIGINT, region |
t 'UnresolvedRelation [bronze, zz_docs_pha_ice_test], [required_write_privileges=INSERT], false, false, false, false / +- LocalRelation [col1#376, col2#377] |
| UNITY-CATALOG-E2E | 130 | save_table(df, "bronze.zz_docs_uce_orders", mode="overwrite") |
If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches. |
| UNITY-CATALOG-FULL-PARITY | 196 | SELECT * FROM system.access.audit; SELECT * FROM system.access.table_l |
cross-database references are not implemented: "system.access.audit" |
| UNITY-CATALOG-FULL-PARITY | 239 | GRANT SELECT ON TABLE datasimple.bronze.zz_docs_ucp_orders TO alice; |
VALID_ARGUMENT","details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.system.access.table_lineage"} |
| demo-manufacturing / PRACTICAL-GUIDE | 555 | check_table("gold.fact_zz_docs_gmfg_sales", level="Error", checks=lamb |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| demo-manufacturing / PRACTICAL-GUIDE | 598 | -- MFG Inspection fail rate (by plant) SELECT p.plant_code, COUNT(*) A |
8302L,plant_id#178303,plant_code#178304,plant_name#178305,country#178306,city#178307,manager_name#178308,capacity_units_per_day#178309,_tenantid#178310] parquet |
| demo-manufacturing / PRACTICAL-GUIDE | 610 | -- MFG Scrap rate (by plant and line) SELECT p.plant_code, l.line_code |
+- Relation datasimple.gold.dim_zz_docs_gmfg_line[line_key#178334L,line_id#178335,plant_id#178336,line_code#178337,line_name#178338,_tenantid#178339] parquet |
| demo-manufacturing / PRACTICAL-GUIDE | 1209 | ds.reset() @ds.dimension("gold.dim_zz_docs_gmfg_ice_supplier", keys=[ |
h ds.dimension)" % (_dn, dsk)) / ValueError: ds.fact: gold.dim_zz_docs_gmfg_product has no surrogate key zz_docs_gmfg_product_key (declare it with ds.dimension) |
| demo-manufacturing / PRACTICAL-GUIDE | 1340 | REFRESH MATERIALIZED VIEW gold.mv_zz_docs_gmfg_fulfilment_monthly; SE |
information_schema takes one statement — remove the ";" |
| demo-manufacturing / PRACTICAL-GUIDE | 1465 | df = read_table("gold.fact_zz_docs_gmfg_quality") df.show(10) |
nd catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; / 'UnresolvedRelation [gold, fact_zz_docs_gmfg_quality], [], false |
| demo-retail / PRACTICAL-GUIDE | 382 | ds.run() ds.checks() |
#302825,city#302826,state_code#302827,loyalty_tier#302828,home_store_id#302829,updated_at#302830,_tenantid#302831,__sequence#302832,__updated_at#302833] parquet |
| demo-retail / PRACTICAL-GUIDE | 491 | check_table("gold.fact_zz_docs_grtl_sales", level="Error", checks=lamb |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-guide / 02-lakehouse-storage | 120 | SELECT COUNT(*) FROM datasimple.source.zz_docs_d02_tpch_sf1_nation; SE |
operationMetrics#327321, userMetadata#327322, engineInfo#327323] / +- 'UnresolvedTable [source, zz_docs_d02_tpch_sf1_nation_DELTA], DESCRIBE HISTORY |
| developer-guide / 02-lakehouse-storage | 123 | SELECT COUNT(*) FROM datasimple.source.zz_docs_d02_tpch_sf1_nation_ICE |
:[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.source.zz_docs_d02_tpch_sf1_nation_ICEBERG.snapshots"} |
| developer-guide / 02-lakehouse-storage | 126 | CREATE TABLE datasimple.bronze.zz_docs_d02_orders (id BIGINT, amount D |
rs` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects. |
| developer-guide / 02-lakehouse-storage | 217 | df = read_table("datasimple.source.zz_docs_d02_tpch_sf1_nation_DELTA") |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-guide / 02-lakehouse-storage | 353 | CREATE MATERIALIZED VIEW gold.mv_zz_docs_d02_sales_revenue_by_region_d |
IALIZED VIEW gold.mv_zz_docs_d02_sales_revenue_by_region_daily AS SELECT region, order_date, SUM(amount) AS revenue FROM silver.zz_docs_d02_orders GROUP BY 1, 2 |
| developer-guide / 05-notebook-helpers | 121 | src = read_table("datasimple.source.zz_docs_d05_tpch_sf1_nation_DELTA" |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-guide / 06-end-to-end | 34 | SELECT COUNT(*) AS n FROM datasimple.source.zz_docs_e2e_tpch_sf1_natio |
:[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.source.zz_docs_e2e_tpch_sf1_nation_ICEBERG.snapshots"} |
| developer-guide / 06-end-to-end | 100 | DROP TABLE datasimple.bronze.zz_docs_e2e_nation PURGE |
ssage) from None / pyspark.errors.exceptions.connect.UnsupportedOperationException: [UNSUPPORTED_FEATURE.PURGE_TABLE] The feature is not supported: Purge table. |
| developer-handbook / 02-notebooks | 190 | orders = read_table("silver.zz_docs_hb02_orders") |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-handbook / 07-lakehouse-and-governance | 161 | GRANT SELECT ON TABLE silver.zz_docs_hb07_orders TO analyst; |
information_schema takes one statement — remove the ";" |
| developer-handbook / 07-lakehouse-and-governance | 186 | COMMENT ON TABLE datasimple.gold.zz_docs_hb07_sales_by_region IS 'One |
information_schema takes one statement — remove the ";" |
| developer-handbook / 12-data-quality | 106 | check_table("gold.zz_docs_hb12_sales_by_region", level="Error", checks |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-handbook / 12-data-quality | 228 | check_table("gold.zz_docs_hb12_sales_by_region", level="Error", checks |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-handbook / 12-data-quality | 269 | good, bad = quarantine_write(df, "silver.zz_docs_hb12_orders", rules=[ |
ror( / ValueError: replay_errors('silver.zz_docs_hb12_orders') has no rules to re-test with. Either register them for the table, or pass rules=[...] explicitly. |
| developer-handbook / 15-observability | 160 | -- What feeds gold.sales_by_region, and from which job? SELECT source_ |
information_schema takes one statement — remove the ";" |
| developer-handbook / 20-lake-storage | 186 | ALTER TABLE gold.zz_docs_hb20_sales_by_region DROP COLUMN _tenantid; |
_tenantid is a protected column: every lake table keeps the tenant that wrote it |
| developer-handbook / 21-ds-tables-and-scd | 124 | MERGE CHANGES INTO silver.zz_docs_h21_mcustomers FROM bronze.zz_docs_h |
ON ds_t.customer_id = ds_s.customer_id / AND ds_s.updated_at > ds_t.updated_at / WHEN MATCHED THEN DELETE |
| developer-handbook / 21-ds-tables-and-scd | 149 | CREATE MATERIALIZED VIEW gold.mv_zz_docs_h21_sales_revenue_daily SCH |
silver.zz_docs_h21_orders o / JOIN silver.zz_docs_h21_customers c ON c.customer_id = o.customer_id AND c.__is_current / GROUP BY c.region, o.order_date |
| developer-handbook / 21-ds-tables-and-scd | 159 | DROP MATERIALIZED VIEW gold.mv_zz_docs_h21_sales_revenue_daily |
rrent_schema() output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. |
| developer-handbook / 21-ds-tables-and-scd | 225 | ds.reset() @ds.dimension("gold.dim_zz_docs_h21_customer", keys=["cust |
If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches. |
| developer-handbook / 23-spark-engine | 150 | spark.sql("""CREATE TABLE gold.zz_docs_h23_region_totals AS |
ciated location ('s3a://datasimple-the Spark service-serverless/delta-table-emrtest/gold.db/zz_docs_h23_region_totals') is not empty and also not a Delta table. |
| developer-handbook / 24-end-to-end-walkthrough | 149 | ds.reset() @ds.dimension("gold.dim_zz_docs_h24_customer", keys=["cust |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| developer-handbook / 24-end-to-end-walkthrough | 196 | check_table("gold.fact_zz_docs_h24_orders", level="Warning", checks=la |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-handbook / 26-unity-catalog-step-by-step | 228 | -- Managed table, declared columns and primary key CREATE TABLE gold.z |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| developer-handbook / 26-unity-catalog-step-by-step | 246 | # Python: write a DataFrame and register it df = read_table("silver.zz |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| developer-handbook / 26-unity-catalog-step-by-step | 298 | COMMENT ON TABLE gold.zz_docs_g26_sales IS 'One row per order, net of |
Table gold.zz_docs_g26_sales is not in the catalog; nothing to describe. |
| developer-handbook / 26-unity-catalog-step-by-step | 328 | -- gold.sales already has its primary key (order_id, declared in Step |
process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / > |
| developer-handbook / 26-unity-catalog-step-by-step | 337 | ALTER TABLE gold.zz_docs_g26_sales ADD COLUMNS (customer_id BIGINT); |
process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / > |
| developer-handbook / 26-unity-catalog-step-by-step | 338 | ALTER TABLE gold.zz_docs_g26_sales ADD CONSTRAINT fk_sales_customer |
process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / > |
| developer-handbook / 26-unity-catalog-step-by-step | 721 | ALTER TABLE gold.zz_docs_g26_sales RENAME TO gold.zz_docs_g26_sales_or |
S.; line 1 pos 12; / 'RenameTable [gold, zz_docs_g26_sales_orders], false / +- 'UnresolvedTableOrView [gold, zz_docs_g26_sales], ALTER TABLE ... RENAME TO, true |
| developer-handbook / 26-unity-catalog-step-by-step | 729 | DROP TABLE gold.zz_docs_g26_sales_orders; -- soft drop: hidden |
rrent_schema() output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. |
| developer-handbook / 26-unity-catalog-step-by-step | 738 | UNDROP TABLE gold.zz_docs_g26_sales_orders; |
Nothing to UNDROP: no dropped table 'gold.zz_docs_g26_sales_orders' within the retention window. |
| developer-handbook / 26-unity-catalog-step-by-step | 745 | DESCRIBE HISTORY gold.zz_docs_g26_sales_orders; |
nd#505404, operationMetrics#505405, userMetadata#505406, engineInfo#505407] / +- 'UnresolvedTable [gold, zz_docs_g26_sales_orders], DESCRIBE HISTORY |
| developer-handbook / 26-unity-catalog-step-by-step | 746 | RESTORE TABLE gold.zz_docs_g26_sales_orders TO VERSION AS OF 3; |
output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; line 1 pos 0 |
| developer-handbook / 27-kinetic-pen | 129 | SELECT region, sum(amount) AS total_amount FROM datasimple.silv |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| developer-handbook / 27-kinetic-pen | 136 | from pyspark.sql import functions as F df = read_table("silver.zz_doc |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| developer-handbook / 27-kinetic-pen | 148 | SELECT order_id, region, customer_id, first_name, |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| developer-handbook / 27-kinetic-pen | 162 | SELECT * FROM datasimple.silver.zz_docs_k27_orders WHERE amount < 0 |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| developer-handbook / 27-kinetic-pen | 168 | SELECT * FROM datasimple.silver.zz_docs_k27_orders WHERE region = 'Eas |
process/resource limits reached" / debug_error_string = "UNKNOWN:unable to create native thread: possibly out of memory or process/resource limits reached" / > |
| developer-handbook / 30-sql-python-reference | 123 | SHOW SCHEMAS; -- source, bronze, sil |
e queried on its own; join its result in Python with read_table("information_schema. |
| developer-handbook / 30-sql-python-reference | 172 | -- A view: a saved query, read through its tables' access rules CREATE |
ders in a way DataSimple cannot check against the tables' access rules. Name it as |
| developer-handbook / 30-sql-python-reference | 245 | CREATE MATERIALIZED VIEW gold.mv_zz_docs_r30_sales_revenue_by_region_d |
---------^^^ / AS SELECT region, order_date, SUM(net_amount) AS revenue / FROM silver.zz_docs_r30_sales GROUP BY region, order_date |
| developer-handbook / 30-sql-python-reference | 255 | SELECT * FROM datasimple.information_schema.materialized_views; |
syntax error at or near ";" |
| developer-handbook / 30-sql-python-reference | 262 | -- Append: every new source row is added once CREATE STREAMING TABLE s |
datasimple.silver.zz_docs_r30_orders_log already exists. Use CREATE OR REFRESH STREAMING TABLE to change it, or REFRESH STREAMING TABLE to load the new rows. |
| developer-handbook / 30-sql-python-reference | 279 | REFRESH STREAMING TABLE silver.zz_docs_r30_orders_live; -- on |
== SQL == / REFRESH STREAMING TABLE silver.zz_docs_r30_orders_live / ^^^ |
| developer-handbook / 30-sql-python-reference | 372 | -- Columns ALTER TABLE silver.zz_docs_r30b_customers ADD COLUMNS (emai |
ualifiedColType(None,visits,IntegerType,true,None,None,None), QualifiedColType(None,fax,StringType,true,None,None,None), QualifiedColType(None,telex,StringType, |
| developer-handbook / 30-sql-python-reference | 395 | -- Rename the table: from here on it is silver.customer_master ALTER T |
xception(info, status.message) from None / pyspark.errors.exceptions.connect.SparkConnectGrpcException: (scala.NotImplementedError) an implementation is missing |
| developer-handbook / 30-sql-python-reference | 398 | -- Views: change one by creating it again CREATE OR REPLACE VIEW gold. |
ders in a way DataSimple cannot check against the tables' access rules. Name it as |
| developer-handbook / 30-sql-python-reference | 402 | -- Drop and bring back DROP TABLE silver.zz_docs_r30b_customer_master; |
Nothing to UNDROP: no dropped table 'silver.zz_docs_r30b_customer_master' within the retention window. |
| developer-handbook / 30-sql-python-reference | 422 | SELECT * FROM silver.zz_docs_r30b_orders TIMESTAMP AS OF '2026-10-16 1 |
tamp (2026-10-16 18:00:00.0) is before the earliest version available to this / table (2026-11-04 17:25:36.0). Please use a timestamp after 2026-11-04 17:25:36. |
| developer-handbook / 30-sql-python-reference | 668 | check_table("silver.zz_docs_r30b_orders_clean", lambda c: c.isComplete |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| developer-handbook / 30-sql-python-reference | 749 | REFRESH STREAMING TABLE silver.zz_docs_r30b_orders_current; REFRESH MA |
== SQL == / REFRESH STREAMING TABLE silver.zz_docs_r30b_orders_current / ^^^ |
| developer-handbook / 30-sql-python-reference | 759 | @ds.table("silver.zz_docs_r30b_orders_current_ds", mode="merge", keys= |
If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches. |
| developer-handbook / 30-sql-python-reference | 776 | CREATE STREAMING TABLE silver.zz_docs_r30b_orders_log AS SELECT * FROM |
CREATE STREAMING TABLE silver.zz_docs_r30b_orders_log / -------^^^ / AS SELECT * FROM STREAM bronze.zz_docs_r30b_dbo_orders |
| developer-handbook / 30-sql-python-reference | 792 | CREATE MATERIALIZED VIEW gold.mv_zz_docs_r30b_sales_revenue_short SCHE |
e queried on its own; join its result in Python with read_table("information_schema. |
| guide / 03-lake-targets-and-formats | 64 | SELECT snapshot_id, committed_at FROM datasimple.bronze.zz_docs_g03_or |
"details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.bronze.zz_docs_g03_orders_iceberg.snapshots"} |
| guide / 06-silver-apply-changes | 126 | @ds.table("silver.zz_docs_g06_orders_clean", mode="merge", keys=["Orde |
If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches. |
| guide / 07-notebooks | 160 | SELECT * FROM datasimple.bronze.zz_docs_n07_nation_eu TIMESTAMP AS OF |
tamp (2026-10-16 18:58:10.0) is before the earliest version available to this / table (2026-11-04 16:20:30.0). Please use a timestamp after 2026-11-04 16:20:30. |
| guide / 07-notebooks | 161 | SELECT snapshot_id, committed_at FROM datasimple.bronze.zz_docs_n07_or |
"details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.bronze.zz_docs_n07_orders_iceberg.snapshots"} |
| guide / 07b-sql-reference | 162 | CREATE TABLE silver.zz_docs_g07b_events (event_id BIGINT, event_type S |
ts` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects. |
| guide / 07b-sql-reference | 182 | INSERT INTO silver.zz_docs_g07b_customers VALUES (1001, 'Ana Lopez', ' |
b)) / ValueError: Constraint violation on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1001,) x3. The write was rolled back. |
| guide / 07b-sql-reference | 184 | INSERT INTO silver.zz_docs_g07b_customers SELECT customer_id, customer |
on on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1002,) x2; (1003,) x2; (1007,) x2; (1001,) x3. The write was rolled back. |
| guide / 07b-sql-reference | 190 | UPDATE silver.zz_docs_g07b_customers SET segment = 'Premier' WHERE cus |
on on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1002,) x2; (1003,) x2; (1007,) x2; (1001,) x3. The write was rolled back. |
| guide / 07b-sql-reference | 197 | MERGE INTO silver.zz_docs_g07b_customers t USING (SELECT * FROM bronze |
aint violation on datasimple.silver.zz_docs_g07b_customers: PRIMARY KEY (customer_id) duplicated: (1002,) x2; (1003,) x2; (1001,) x3. The write was rolled back. |
| guide / 07b-sql-reference | 260 | CREATE STREAMING TABLE silver.zz_docs_g07b_orders_log COMMENT 'Every o |
datasimple.silver.zz_docs_g07b_orders_log already exists. Use CREATE OR REFRESH STREAMING TABLE to change it, or REFRESH STREAMING TABLE to load the new rows. |
| guide / 07b-sql-reference | 282 | UNDROP STREAMING TABLE silver.zz_docs_g07b_orders_live; -- b |
ve` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects. |
| guide / 07b-sql-reference | 371 | ALTER TABLE silver.zz_docs_g07b_customers RENAME COLUMN phone TO phone |
umnMapping.mode' = 'name') / Note, if your table is not on the required protocol version it will be upgraded. / Column mapping requires at least protocol (2, 5) |
| guide / 07b-sql-reference | 372 | ALTER TABLE silver.zz_docs_g07b_customers DROP COLUMN email; |
umnMapping.mode' = 'name') / Note, if your table is not on the required protocol version it will be upgraded. / Column mapping requires at least protocol (2, 5) |
| guide / 07b-sql-reference | 373 | ALTER TABLE silver.zz_docs_g07b_customers DROP COLUMNS (fax, telex); |
umnMapping.mode' = 'name') / Note, if your table is not on the required protocol version it will be upgraded. / Column mapping requires at least protocol (2, 5) |
| guide / 07b-sql-reference | 375 | ALTER TABLE silver.zz_docs_g07b_customers ALTER COLUMN visits TYPE BIG |
onnect.AnalysisException: [DELTA_UNSUPPORTED_ALTER_TABLE_CHANGE_COL_OP] ALTER TABLE CHANGE COLUMN is not supported for changing column visits from INT to BIGINT |
| guide / 07b-sql-reference | 380 | ALTER TABLE silver.zz_docs_g07b_customers ADD CONSTRAINT valid_segment |
e-serverless/delta-table-emrtest/__uc_managed/silver/zz_docs_g07b_customers` violate the new CHECK constraint (segment IN ( 'Retail' , 'Business' , 'Premier' )) |
| guide / 07b-sql-reference | 381 | ALTER TABLE silver.zz_docs_g07b_customers DROP CONSTRAINT valid_segmen |
ror, provide the parameter IF EXISTS or set the SQL session configuration spark.other lakehouse platforms.delta.constraints.assumesDropIfExists.enabled to true. |
| guide / 07b-sql-reference | 392 | ALTER TABLE silver.zz_docs_g07b_customers RENAME TO silver.zz_docs_g07 |
xception(info, status.message) from None / pyspark.errors.exceptions.connect.SparkConnectGrpcException: (scala.NotImplementedError) an implementation is missing |
| guide / 07b-sql-reference | 399 | DROP TABLE silver.zz_docs_g07b_customer_master; -- |
rrent_schema() output, or qualify the name with the correct schema and catalog. / To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. |
| guide / 07b-sql-reference | 400 | UNDROP TABLE silver.zz_docs_g07b_customer_master; -- |
Nothing to UNDROP: no dropped table 'silver.zz_docs_g07b_customer_master' within the retention window. |
| guide / 07b-sql-reference | 401 | DROP TABLE zz_docs_g07b_sandbox.zz_docs_g07b_scratch PURGE; |
ssage) from None / pyspark.errors.exceptions.connect.UnsupportedOperationException: [UNSUPPORTED_FEATURE.PURGE_TABLE] The feature is not supported: Purge table. |
| guide / 07b-sql-reference | 416 | SELECT * FROM silver.zz_docs_g07b_orders_hist TIMESTAMP AS OF '2026-10 |
tamp (2026-10-16 18:00:00.0) is before the earliest version available to this / table (2026-11-04 16:47:13.0). Please use a timestamp after 2026-11-04 16:47:13. |
| guide / 07c-python-reference | 141 | @ds.table("silver.zz_docs_g07c_orders_clean", mode="merge", keys=["Ord |
If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches. |
| guide / 07c-python-reference | 168 | check_table("silver.zz_docs_g07c_orders_clean", lambda c: c.isComplete |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| guide / 07d-streaming-and-incremental | 65 | @ds.table("silver.zz_docs_g07d_py_orders_current", mode="merge", keys= |
If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches. |
| guide / 07d-streaming-and-incremental | 82 | CREATE STREAMING TABLE silver.zz_docs_g07d_orders_log AS SELECT * FROM |
CREATE STREAMING TABLE silver.zz_docs_g07d_orders_log / -------^^^ / AS SELECT * FROM STREAM bronze.zz_docs_g07d_dbo_orders |
| guide / 07d-streaming-and-incremental | 99 | CREATE MATERIALIZED VIEW gold.mv_zz_docs_g07d_sales_revenue_by_region_ |
z_docs_g07d_sales_revenue_by_region_daily already exists. Use CREATE OR REPLACE MATERIALIZED VIEW to change it, or REFRESH MATERIALIZED VIEW to update its data. |
| guide / 07d-streaming-and-incremental | 109 | SELECT * FROM datasimple.information_schema.materialized_views; -- |
syntax error at or near ";" |
| guide / 09-lakehouse-governance | 151 | SELECT kind, count(*) AS runs, sum(spark_seconds) AS spark_seconds FRO |
syntax error at or near ";" |
| guide / 09-lakehouse-governance | 219 | CREATE MATERIALIZED VIEW gold.mv_zz_docs_g09_sales_revenue_by_region_d |
zz_docs_g09_sales_revenue_by_region_daily already exists. Use CREATE OR REPLACE MATERIALIZED VIEW to change it, or REFRESH MATERIALIZED VIEW to update its data. |
| guide / 09-lakehouse-governance | 264 | ALTER TABLE datasimple.gold.zz_docs_g09_orders ADD CONSTRAINT fk_zz_do |
---------------------------------------------------------------------------^^^ / FOREIGN KEY (customer_id) REFERENCES gold.zz_docs_g09_customers (customer_id) |
| guide / 09-lakehouse-governance | 318 | profile_table("silver.zz_docs_g09_orders_scored") check_table("silver. |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| guide / 09-lakehouse-governance | 230 | SELECT * FROM gold.mv_zz_docs_g09_sales_revenue_by_region_daily; |
+- 'LocalLimit 200 / +- 'Project [*] / +- 'UnresolvedRelation [gold, mv_zz_docs_g09_sales_revenue_by_region_daily], [], false |
| guide / 13-reference | 55 | REFRESH STREAMING TABLE silver.zz_docs_g13_orders_live; -- FU |
== SQL == / REFRESH STREAMING TABLE silver.zz_docs_g13_orders_live / ^^^ |
| guide / 13-reference | 129 | @ds.table("silver.zz_docs_g13_py_orders_clean", mode="merge", keys=["O |
If your intention is to keep the existing schema, you can omit the / schema from the create table command. Otherwise please ensure that / the schema matches. |
| guide / 13-reference | 135 | grant("SELECT", "silver.zz_docs_g13_py_orders_live", "analyst") profil |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| user-guide / 02-notebooks | 79 | SELECT snapshot_id, committed_at FROM datasimple.source.zz_docs_u02_tp |
:[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.source.zz_docs_u02_tpch_sf1_nation_iceberg.snapshots"} |
| user-guide / 02-notebooks | 148 | check_table("silver.zz_docs_u02_nation_priority", lambda c: c.isComple |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| user-guide / 02-notebooks | 163 | SELECT * FROM datasimple.source.zz_docs_u02_tpch_sf1_nation_delta WHER |
FROM source.zz_docs_u02_tpch_sf1_nation_delta WHERE N_REGIONKEY = {{region_id}} / ---------------------------------------------------------------------------^^^ |
| user-guide / 04-lakehouse | 46 | check_table("bronze.zz_docs_u04_nation_eu", lambda c: c.isComplete("na |
= importlib.import_module("pydeequ.configs") / return _bootstrap._gcd_import(name[level:], package, level) / ModuleNotFoundError: No module named 'pydeequ' |
| user-guide / 06-apply-changes | 95 | APPLY CHANGES INTO silver.zz_docs_a06_orders_current FROM bronze.zz_do |
gle.common.util.concurrent.ExecutionError) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached |
| user-guide / 06-apply-changes | 186 | @ds.table("silver.zz_docs_a06_orders_clean", mode="merge", keys=["Orde |
Invalid column names: sum(Amount). / Please use other characters and try again. / Alternatively, enable Column Mapping to keep using these characters. |
| guide / 07b-sql-reference | UNDROP TABLE silver.zz_docs_cmd_customers |
rs` because it already exists. / Choose a different name, drop or replace the existing object, or add the IF NOT EXISTS clause to tolerate pre-existing objects. | |
| guide / 07b-sql-reference | INSERT INTO silver.zz_docs_cmd_ice VALUES (1, 'us'), (2, 'eu') |
nt 'UnresolvedRelation [silver, zz_docs_cmd_ice], [required_write_privileges=INSERT], false, false, false, false / +- LocalRelation [col1#13413, col2#13414] | |
| guide / 07b-sql-reference | SELECT snapshot_id, committed_at FROM silver.zz_docs_cmd_ice.snapshots |
_ARGUMENT","details":[{"reason":"INVALID_ARGUMENT","@type":"google.rpc.ErrorInfo"}],"message":"Invalid table name: datasimple.silver.zz_docs_cmd_ice.snapshots"} |
Commands, by how often their documented examples ran#
Every documented example that uses the command, on dev on 2026-11-04. A failure can be the command or the example around it; the table below lists each one.
| Command | Examples | Worked |
|---|---|---|
| APPLY CHANGES (SQL) | 38 | 37 |
| ds.apply_changes (Python) | 19 | 19 |
| MERGE CHANGES / MERGE SNAPSHOT | 1 | 0 |
| Streaming tables | 27 | 19 |
| Materialized views | 27 | 17 |
| ALTER TABLE (columns, constraints, properties) | 34 | 15 |
| COMMENT ON | 8 | 6 |
| Time travel (VERSION / TIMESTAMP AS OF) | 18 | 14 |
| RESTORE TABLE | 6 | 5 |
| OPTIMIZE / VACUUM | 4 | 4 |
| DROP / UNDROP | 12 | 4 |
| Views (CREATE VIEW) | 6 | 4 |
| GRANT / REVOKE | 23 | 19 |
| information_schema | 66 | 57 |
| Iceberg tables (USING ICEBERG) | 9 | 8 |
| save_table | 16 | 12 |
| merge_delta | 5 | 5 |
| DS Tables (@ds.table, ds.run) | 10 | 2 |
| Star schema (@ds.dimension / fact / obt) | 9 | 5 |
| check_table (data quality checks) | 13 | 0 |
Known gaps on dev, being fixed (see the list of failed examples above): check_table needs a library that dev's remote Spark cannot use (a built-in replacement is planned); ALTER TABLE ... RENAME TO and DROP ... PURGE with the standalone catalog server; a cell that mixes a streaming-table or materialized-view command with other statements in a scheduled run; DS Tables re-runs with the standalone catalog server.
Features#
92 features: 44 tested, 27 partly tested, 21 not tested.
Connect sources#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
Add / test a Data Source (DS-n) |
Data Source tab; DataBridge step 1 | Tested | 2026-10-29, both | DS-22 MFG_SQLServer, DS-23 MFG_SensorEvents; DS-24, DS-25; local Postgres, saved connection (deleted); DS-18 CdcDemo-SQL; DS-20 Kafka-Test; DS-16 (Snowflake), DS-15 Target_AWS | Only SQL Server, Postgres, Snowflake, Kafka, S3 and file drop were exercised live. Oracle, MySQL, Db2 (live, not as backup), SAP HANA, OData, Salesforce, MongoDB, Azure: tested with fakes only (phase-3 'No live source'). |
| Cloud connect (S3, GCS, Azure Blob, ADLS) | Data Lake tab | Partly tested | 2026-10-29, local | gold/mfg_*, silver/mfg_*; 22.3 Data Lake | Only the S3 lake (Target_AWS DS-15) was browsed. GCS, Azure Blob and ADLS connections never opened (phase-3: no live Azure). Adding a cloud connection through the dialog is not shown. |
| Database drivers | Server side (blocked from the browser) | Not tested | No document reports installing a dynamic driver or an engine appearing in DataBridge. Browser access is blocked by design (phase-0 test of the 403 on /api/install-python-driver is a unit test plus a local read-only request). | ||
| Git for a notebook | Notebook > Git dialog | Not tested | Browser click-through: Git | No run of the per-notebook Git dialog (commit, pull, branch, PR) is reported; needs a personal access token. |
Ingest#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| Bronze apply (automatic after a load, or by hand) | Bronze loads tab | Tested | 2026-10-29, local | job 141; bronze.mfg_*; bronze.retail_*; _batch_id 168; job 183; job 74 | Manual 'Apply to bronze' for Delta/Iceberg Bronze merge and the retry-Bronze button are not shown with results. |
| Hash CDC Bronze log | Implicit (Parquet Bronze with CDC or watermark loads) | Tested | 2026-10-28, local | public.zz_r3_big3; cdc/part-*.parquet in dev S3; zz_r3_big2, zz_r3_big3 | ADLS keeps the single-file log and was never run live; catalog registration of the log, compaction of old parts and the 403-on-new-table case are open. |
| Detect deletes (delete capture) | DataBridge step 4; Load History > Schedules | Tested | 2026-10-27, local | zz_r3_dc on connection CdcDemo-PG, bucket Target_AWS; job 163 | Daily cadence only tested with fakes; Salesforce deleted records, and the rule that built-in 'Apply changes to Silver' does not receive these deletes, not run live. |
| File ingest | Workflow 'files' step; autoIngest on a file connection | Tested | 2026-10-27, local | bronze.zz_r3_files; landing/zz_r3_files/; file_stream.py | Workflow 'files' step and Everything 'Load into Bronze' card not run; autoIngest scheduling see the scheduled file ingest row. |
| Kafka topic read directly in SQL (CREATE STREAMING TABLE ... STREAM KAFKA) | SQL cell | Tested | 2026-11-04 | STREAM KAFKA('DS-20','orders') | Environment not stated; schedule (SCHEDULE CRON) firing not reported. |
| Streamed first load and part-file Parquet copies for large tables | DataBridge (automatic over a row threshold) | Tested | 2026-10-27, local | zz_r3_big* in dev S3; job 21 | S3 only; ADLS keeps single-file merge and was not run. Stale Spark read after a full reload is a known limit. |
| Audit trail (bookkeeping) columns on every layer (_op_type, _batch_id, _load_ts, ...) | Notebook result grid > Audit columns | Tested | 2026-10-30, local | notebook MFG 08 Bronze change types; RTL 07 | Protected-column rules and Silver Type 2 audit columns for Iceberg are not separately shown. |
| DataBridge load (five-step wizard): full, watermark, change-log CDC, file drop, backup drop, Kafka, Camel | Databridge tab | Partly tested | 2026-10-30, both | DS-22; jobs 141, 145, 148, 158, 163; DS-24; jobs 165-171; DS-18; job 74; jobs 27, 28, 29; FIN_CoreBanking; jobs 183, 186; jobs 40-49 (also 39); jobs 50-61; DS-20 Kafka-Test; job 107; DS-16; jobs 15-21 (EC2 box and laptop | Camel connectors never run. Backup drop for Postgres/MySQL is 'recipe ready, run on request'; Oracle and SAP HANA 'recipe only, untested flag'. No live Oracle, MySQL, Salesforce, SAP, ADLS source or target. Postgres logical-decoding CDC has a documented the container test environment but no result table. |
| Reconcile | No screen; API /api/databridge/api/reconcile/* | Partly tested | 2026-10-30, local | batch reconcile-2026-10-30; Tests | The reconcile endpoint itself (tombstone mode and event_log mode) is not shown run end to end; tombstone mode untested. |
| Load control (reset, lookback, disable) | Bronze loads tab and the DataBridge page | Partly tested | 2026-10-30, local | jobs 148, 161 | Only viewing positions is shown; reset, lookback change and disable never exercised. |
| Bulk load | DataBridge wizard checkbox | Not tested | Section 1 inventory (Bulk bulk.py + bulk_spark.py) | No run with 'Bulk load into Source tables' ticked (Snowflake COPY INTO or Spark JDBC) is reported. Bulk-refused-for-Bronze rule only in code. | |
| File copy | No React screen (API) | Not tested | Section 14: Tenant scope for files/copy and reconcile | No real run of /files/copy (raw file copy, optional table registration) reported. | |
| Source -> Delta | Notebook wizard | Not tested | No document reports using the notebook Source-to-Delta wizard (100k default rows, overwrite/append). | ||
| Kafka streaming through Flink | Workflow, ingestion mode 'streaming' | Not tested | No run of the Flink job path; Flink appears only in docs and unit tests. | ||
| dlt engine | None (deprecated shim) | Not tested | Deprecated shim; no run reported. | ||
| New-file alerts | Everything > File alerts | Not tested | A day with Sarah, by voice, step 2 | The file watcher is off on the development server (FILE_ALERTS_ENABLE is not set); it needs that setting in the deploy configuration before an alert can be shown. | |
| Load to ADLS (Azure) targets | DataBridge destination | Not tested | Not tested here | No live ADLS load, incremental load or file drop on ADLS. | |
| Stop a DataBridge job / table, memory guard and streamed retry | Load History, Ingestion | Not tested | Whole database again, streamed code | Stop job, per-table stop and the memory-guard queue/retry are described but no run is reported. |
Transform#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| SQL and Python cells | Notebooks | Tested | 2026-10-29, both | notebooks MFG 01-11 (NB-82 = MFG 01); gold.zz_lt_pipe_a, gold.e2e_orders; silver.zz_p1_tenant_check | Interactive cell limit is 5 minutes (Gold build hit it); Python cell three-part-name bug found in first-run-research is an open finding. |
| DataBridge "Apply changes to Silver (MERGE)" | DataBridge step 4 | Tested | 2026-10-16, local | silver.dbo_Orders, silver.dbo_Orders_HISTORY; jobs 75, 77; Worked example, end to end | Type 2 needs the Spark box and was not run without it; ADLS not run. The demos set this option to 'Not now'. |
ds.apply_changes / ds.apply_changes_from_snapshot |
Python cell | Tested | 2026-10-27, both | silver.orders_current, orders_history, orders_snapshot; silver.zz_p2_scd2_dim; silver.zz_p5_scd2_seq, ..._old | Iceberg targets and the legacy fallback path not run live. |
APPLY CHANGES [FROM SNAPSHOT] INTO |
SQL cell | Tested | 2026-10-30, both | NB-82 MFG 01; silver.mfg_*; RTL 01, RTL 02; zz_decl_* tables | The FROM SNAPSHOT SQL spelling was verified only on 2026-10-18 for the Python form; SQL snapshot form not shown with a result. |
DS Tables (ds.table, ds.run()) |
Python cell | Tested | 2026-10-30, both | MFG 04, MFG 09, MFG 10; gold.mfg_*; RTL 04; gold.retail_*; zz_decl_*; ds.dimension section | |
| Kinetic pen | Kinetic tab | Tested | 2026-10-30, local | silver.mfg_sales_orders, mfg_work_orders; Kinetic tab; silver.retail_store_inventory, retail_sales_transactions | Voice input and AI-fallback accuracy not tested; Semantics-metric export from Kinetic not shown. |
| dbt run / test | Pipelines (dbt step); Workflow step | Tested | 2026-11-01 | pipeline 'RETAIL dbt Silver and Gold'; dbt/retail_dbt; silver.dbt_retail_*, gold.dbt_retail_*; silver.ext_*, silver.spk_*; Phase 1: A real dbt run for lineage | dbt as a Workflow step not run; dbt with a warehouse other than this workspace not run. |
| Data Vault page: hubs, links, satellites, PIT, KPI views | Data Vault page | Tested | 2026-11-04 | notebook #133 'Data Vault . finance', pipeline #35, runs 130/133; hub_fin_*, gold.pit_fin_*; gold.dv_retail_sales_by_region_month; Which scenarios are tested | Unit-tested only: composite key proposed by scoring, empty multi-active set, parent changing after events. Not built: same-as and hierarchical links. Not run with a second real user. |
MERGE CHANGES / MERGE SNAPSHOT INTO |
SQL cell | Partly tested | 2026-10-27, local | zz_decl_* tables | MERGE SNAPSHOT INTO never run; MERGE CHANGES only on a small Delta test table, not on Iceberg or a demo project. |
| Workflow medallion (Bronze MERGE, Silver, SCD2 checkbox, Gold) | Workflow tab | Partly tested | 2026-10-30, local | workflow 'MFG quality inspections (Workflow)'; mfg_wf_quality_inspections; workflow id 11; retail_wf_returns; zz_lt_workflow, gold.zz_lt_wf_summary | SCD2 checkbox (workflow-scd2) and chat-built workflows (Sarah canvas / Everything workflow_build) not shown run; API weakness: unknown 'after' silently replaced. |
Materialized views (... SCHEDULE CRON) |
SQL cell | Partly tested | 2026-10-30, local | MFG 11; mv_mfg_*; RTL 08; zz_decl_* | No report of a materialized view refreshed by its own cron timer or of the failed-refresh alert firing (alert is unit-tested). |
| Notebook parameters ({{param}} / Params cell) | Notebook cell; pipeline step parameters | Partly tested | 2026-10-21, local | notebook E2E 5 Parameter | Params cell and parameter use inside an interactive notebook not shown; Chart cell not shown. |
| Stop a running cell; fair Spark sessions; 'session restarted' notice | Notebooks; Spark status | Partly tested | 2026-10-27, local | Phase 2: Stop on the real Spark; 3. Fair Spark sessions: live check on the running app; Not checked live | Several users competing for sessions on the dev Spark never run live. |
| Flink SQL | Pipelines (Flink step); Workflow step | Not tested | No run of a Flink SQL step. | ||
| Chart cell and result charts in a notebook | Notebook Chart cell | Not tested | No run reported (Sarah's charts are a different feature). |
Orchestrate#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| Notebook schedule | Notebook > Schedule; Schedules tab | Tested | 2026-10-27, local | zz_lt_pipe_a schedule; schedule 16; Phase 0: real restart | Catch-up of a missed nightly run after a restart tested with a fake clock only; retry_count and email notification open. |
| Pipeline | Pipelines tab | Tested | 2026-10-30, both | pipelines 'E2E orders pipeline', 'E2E failure handling'; pipeline id 20; runs 93, 96, 99; pipeline id 24; runs 107, 109; zz_lt_pipeline_dag, zz_lt_pipeline_stop; run 57, 76 | Email/webhook notification delivery open; QA-side runs not reported. |
| Workflow, "Schedule as pipeline" | Workflow tab | Tested | 2026-10-27, local | zz_lt_workflow and its schedule pipeline; 22.1 | The one-step pipeline's run history for a demo workflow is not shown. |
| Dataflow | Dataflow tab | Tested | 2026-10-30, both | DAG e2e_orders_dataflow; zz_lt_flow_notebook, zz_lt_flow_pipeline; DAG mfg_daily_lakehouse_build; DAG retail_daily_lakehouse_build | Daily cron of a DAG never fired on its own (all created paused and run manually); notebook/cell DAG running as principal 'dataflow' with grants only described. |
| DataBridge schedule | DataBridge page > Load History > Schedules | Partly tested | 2026-10-18, local | job 74; job 107 | Only 'Run now' was used; no report of a schedule firing on its own interval, pause/resume, or Detect-deletes daily cadence. |
| Materialized view refresh | SQL SCHEDULE CRON |
Partly tested | 2026-10-30, local | gold.mv_mfg_fulfilment_monthly | No timer-driven refresh observed; failed-refresh alert unit-tested only. |
| Scheduled file ingest | No screen (connection setting autoIngest) |
Not tested | No run of autoIngest on a timer; only manual/API file ingest was run. | ||
| DQ sweep | No screen (API /api/notebooks/dq/sweeps) | Not tested | 12. Data-quality sweep reports failed tables | No sweep created or fired on a real cron. | |
| Run alerts: email and webhook notify (every run / only on failure) | Pipeline editor 'Notify when'; Schedules | Not tested | 2026-10-27, local | Browser click-through; Phase 2: Email alert delivery | No email or webhook delivery reported. |
Quality#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| Data Quality dashboard | Data Quality tab | Tested | 2026-10-29, both | Data Quality filter 'mfg_'; 3 rules on gold.zz_lt_pipe_a; 21.6 | PII tag setting not shown; governance finding: hand-made rule starts 'approved' (decision open). |
| DS Tables rules | Python cell | Tested | 2026-10-30, both | MFG 09, MFG 10; zz_decl_orders | |
Checks in notebooks (deequ_check, check_table, profile_table, table_freshness) |
Notebook cells | Partly tested | 2026-10-29, local | notebook MFG 06; RTL 06; gold.zz_lt_pipe_a | profile_table and table_freshness not shown run. |
| Quality rules and certification by voice | Everything / Agent | Partly tested | silver.dsdemo2_orders; 22.1 | No certificate actually issued and approved; voice not run; approval-card flow for rule creation only shown as walkthrough text. | |
| Data Quality: freshness, profiles, quarantine, coverage, certificates | Data Quality tab (Coverage, Suggestions, Anomalies, Alerts) | Partly tested | 2026-10-27, local | Data Quality; Declarative pipelines | Freshness, anomalies, suggestions and certificates tabs not shown with results. |
Semantics and BI#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| Glossary terms, metrics, table and column meaning | Semantics tab | Tested | 2026-10-29, both | metrics MFG Net revenue, MFG Inspection fail rate, MFG Scrap rate; Step 12; term 'zz LT Customer Total', metric 'zz LT total amount', join zz_lt_orders_x_totals | Grand-total metric with saved dimensions not possible (known limit). |
| Sarah: ask for a report | Sarah tab; Everything | Tested | 2026-10-30, both | Sarah report on gold.mfg_*; gold.retail_sales_view, gold.retail_store_day_view, shared_store_sales_by_region_test; gold.mfg_*_view | Live voice (OPENAI key) and a second real user's access rules (store manager sees only 'department') shown by screenshot only. |
| dbt SQL gateway | External (bearer token per dbt client) | Tested | 2026-11-02 | dbt connection e2e_flow, owner client:001:e2e_flow; silver.e2e_stores, gold.e2e_region_stores | |
| Meaning auto-draft | Background, every 6 hours | Partly tested | 2026-10-27, local | gold.zz_lt_pipe_a | The 6-hourly sweep itself and the approval of auto-drafted meanings are not shown. |
| Star schema / One Big Table; what-if scenarios | Everything | Partly tested | star_demo_* tables; ds.dimension section | Scenario engine and saving a scenario ('cautious plan') not shown run; approval/pipeline creation of the star not shown with ids. |
Govern#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| Register and browse tables | Lakehouse tab; every load and write | Tested | 2026-10-29, both | Lakehouse, datasimple catalog; Update 2026-10-15 / tenant isolation 2026-10-16; silver.zz_p1_tenant_check | External Unity Catalog server mode (when configured) not covered in these docs; local DB has 110 ownerless tables never backfilled. |
| Lineage | Lineage tab | Tested | 2026-10-30, both | gold.mfg_fact_sales; Q-1227; Pipelines / Other findings; Step 4 | Impact mode not shown; SQL CTAS lineage in scheduled runs was a gap on 2026-10-21 (later recorded per live test). |
Definition log and information_schema |
Lakehouse > object history; SQL | Tested | 2026-10-30, local | 21.9; Pipelines; silver.zz_p1_tenant_check | Redaction of definitions without SELECT (phase 5 round 1) is unit-tested only ('Checked live: -'). |
| Trash and UNDROP | Lakehouse (Trash); SQL UNDROP TABLE |
Tested | 2026-10-27, local | gold.zz_tenant_probe; UNDROP section | |
| Tenant isolation (tenant id on every object and write) | Sign-in; every layer | Tested | 2026-10-27, both | Tenant isolation 2026-10-16; silver.zz_p1_tenant_check; Data Share | |
| Unity Catalog REST (open-source subset) | /api/2.1/unity-catalog | Partly tested | 2026-10-27, local | Tests: unity-catalog-owner.test.js; 8. Verified working flows | Spark or an external tool actually speaking the protocol to the app is not reported. |
| Grants, row filters, column masks, volumes, credentials, locations, UC audit | Lakehouse; Administration > Governance | Partly tested | 2026-11-04, both | rules: greenville_plant_only; gold.mfg_obt_sales, silver.mfg_customers; chicago_store_only x2; Grants section; bronze.zz_doc21_public_customers (mask on name, filter zz_west_only) | Volumes, credentials, locations and the UC audit view never shown run. |
| Time travel, restore a version, Iceberg snapshots and rollback | SQL cell; Lakehouse history | Partly tested | 2026-10-26, local | NATION tables; Phase 1 | RESTORE TABLE (admin only) and Iceberg rollback never run. |
| PII tags and display masking | Data Quality > table detail | Not tested | No run of setting a column PII tag or seeing a masked display. (Column masks, a different feature, are tested.) | ||
| Retention sweeper | Background | Not tested | 3. Catalog-screen delete goes to Trash | Never run. | |
| Iceberg maintenance | Background | Not tested | No run of snapshot expiry or compaction reported. | ||
| Volumes, functions and models in the catalog | Lakehouse | Not tested | No run reported. |
Share#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| DataShare (Delta Sharing): shares, recipients, profile file, revoke | DataShare tab | Tested | 2026-10-27, local | share zz_lt_share (recipient revoked); mfg_fulfilment_share; retail_supplier_share | Real client (Spark, pandas, Power BI) not used; profile file download and 15-minute link not shown; creating a share needs only the Sharing tab (decision open). |
| dbt clients | Pipelines > dbt client admin | Tested | 2026-11-02 | e2e_flow; Tested on 2026-11-02 with dbt-spark 1.10.3 | |
| Personal API tokens | Settings | Tested | 2026-10-27, local | dsp_TElQ... (revoked) |
Deploy and promote#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| Git integration: commit a business model, pull request, Update from Git | Deployment > Git panel; Administration > Git integration | Tested | 2026-11-01 | repo datasimple/ds_git_deploy branches Dev, QA, Prod; pipelines 'RETAIL Daily Lakehouse Build'; Verified 2026-10-31 | Three tenants of one install used as three environments; Roll back not run. |
| Deployment screen: register QA/Prod, compare, deploy notebooks, pipelines, workflows, lake objects (created empty) and grants; history; restore plan and restore | Deployment tab | Partly tested | 2026-11-01, both | notebooks MFG 01, 02, 04; metrics MFG Net revenue, mfg_net_unit_price, mfg_scrap_rate_pct; schema zz_r3_deploy; zz_dim_customer, v_dim_current; Browser click-through: Deployment; 22.2 Deployment | Never run between two real instances (QA/Prod not reachable); Roll back and restore plan, prod approval request and per-environment values not run live; Phase 0-4 deploy tests use stubs. |
| Code deploy | Outside the app (the Deploy workflow) | Partly tested | 2026-10-16, dev | containers datasimple-qa :5002, datasimple-dev :5001; Checked on the running app | No document records a Deploy workflow run result; QA is on a 2026-09-15 build (165 commits behind on 2026-10-16). |
| Notebook versions -> restore | Notebook menu | Not tested | No run of restoring a notebook version. |
Monitor and admin#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| History | History tab | Tested | 2026-10-29, both | 22.7 History — who did what; 22.7 | |
| Log | Log tab | Tested | 2026-10-29, both | Q-726; NB-82; Browser click-through: Log > Run log > Errors | |
| Load History | DataBridge page | Tested | 2026-10-30, both | jobs 74, 75, 77; jobs 148, 158 | Schedules panel (Pause, Delete) not shown beyond Run now. |
| Ingestion | Ingestion tab (administrators) | Tested | 2026-10-30, both | jobs 141-163; jobs 165, 168; Step 6 | Cancel and 'retry Bronze' buttons not shown run. |
| Usage | Usage tab | Tested | 2026-10-30 | 22.10 Usage — what it costs; needtofix H75 | Export CSV and ledger accuracy not verified; H75 open. |
| Notebook Query history dialog | Notebook > Query history | Tested | 2026-10-27, local | Browser click-through: Notebook > Query history | |
| Users, roles, tab permissions, governance, Activity Logs, Error Logs | Administration | Partly tested | 2026-11-04, both | Browser click-through: Admin > Roles; 22.12 Administration; Step 10; Administration > Users / Roles; roles 8-17 | Editing a role's permissions on dev waits for the deploy of 12f77c2; the access check only reports (ACCESS_GUARD_MODE=log), and signing in as a role-limited user is not done yet; SSO/OIDC/SAML sign-in and Activity/Error Logs content not shown. |
| Settings | Settings tab | Partly tested | 2026-11-04, both | 22.11 Settings — your defaults; 2. Personal API tokens; Settings tab | Spark warm window and per-user notebook defaults effect not shown. |
| Sign-in with tenant id, roles, and single sign-on | Login page | Partly tested | 2026-10-27, local | Other findings; 2. Personal API tokens | Google/OIDC/SAML SSO and email-domain checks (admin page) never run. |
| ELS_DB browser | ELS_DB tab | Partly tested | 2026-10-30 | 22.5 ELS_DB; H49 | SQL/preview not available (H49). |
| App activity feed | /api/activity (no screen calls it) | Not tested | Unused by design (no writers); nothing to run. |
Assistants#
| Feature | Where | Status | Proven (date, environment) | Proof | Still to prove |
|---|---|---|---|---|---|
| Sarah | Sarah tab | Tested | 2026-10-30, both | 6. Sarah; Step 19 | Live voice not tested. |
| Agent | Agent tab | Tested | 2026-10-30, both | Agent screen; 22.6; Custom groups section | Approval-queue approve/reject on a demo write card not shown with a result in the demos. |
| Kinetic | Kinetic tab | Tested | 2026-10-30, local | Step 20; Step 20 | Voice input not tested. |
| Sarah: business views, forecasts, chart filters, Open in notebook, Share with Sarah | Sarah tab | Tested | 2026-10-30 | gold.shared_store_sales_by_region_test, ...view; gold.mfg*_view | |
| Everything | Header toggle | Partly tested | silver.dsdemo2_orders; star_demo_* tables | Voice never run; file alerts, deploy card, grant card, scenario save, table backup/restore cards not shown run; no dates or ids. | |
| RoleAgent | Administration sub-tab | Partly tested | 2026-10-30 | 22.12 Administration | Toggling access and its effect on a user not shown; scope still 'to confirm' in the catalogue. |
Not testable on dev yet#
These need a system that is not connected to the development server. They are marked not tested above until they are run:
- Google Cloud Storage, Azure Blob and ADLS lakes
- Oracle, MySQL production, SAP, Salesforce and HANA sources; Camel connectors; Flink
- Google, OIDC and SAML sign-in
- Deployment between two real installations (QA, Prod)
- Email delivery of alerts
- Speaking a request to Everything (a typed request is tested)