DataSimplePractical GuideLog inOpen DataSimple
DataSimple
Chapter 08aOrchestrate

Pipelines and Dataflow: a tested walkthrough

Back to the index · Previous: Pipelines and workflows

Everything on this page was run end to end through the application on 2026-10-21 and passed. Repeat it on your own environment to confirm the setup before relying on it.

The two features in one line each#

  • A Pipeline is the application's own orchestration: steps with dependencies, parallel branches, a failure policy per step, retries and a cron, run by the application's scheduler.
  • Dataflow gives the calendar to an external scheduler service: the application writes the definition, the service triggers it with its own retries, and the service calls the application, which runs the steps as a pipeline. Same engine for the steps; a different owner of the clock.

Notebooks used#

Five notebooks with one SQL cell each.

Notebook Cell
E2E 1 Load orders CREATE OR REPLACE TABLE datasimple.gold.e2e_orders AS SELECT order_id, customer, amount, status FROM silver.orders
E2E 2 Totals by status CREATE OR REPLACE TABLE datasimple.gold.e2e_totals AS SELECT status, count(*) AS orders, sum(amount) AS amount FROM gold.e2e_orders GROUP BY status
E2E 3 Top customers CREATE OR REPLACE TABLE datasimple.gold.e2e_top AS SELECT customer, sum(amount) AS amount FROM gold.e2e_orders GROUP BY customer ORDER BY amount DESC LIMIT 5
E2E 4 Broken step SELECT * FROM gold.table_that_does_not_exist_e2e (fails on purpose)
E2E 5 Parameter SELECT count(*) AS n FROM gold.e2e_orders WHERE status = {{status}}

Pipeline 1: dependencies, parallel branches, a parameter#

Left rail → Pipelines → + Create Pipeline. Steps:

Step Notebook Depends on On failure Parameters
load E2E 1 stop
totals E2E 2 load continue
top E2E 3 load continue
param E2E 5 totals, top continue status = paid
Steps with their dependencies and failure policy
Steps with their dependencies and failure policy

Run on the row, then open the pipeline. What was checked:

Check Result
Run status Success, every step Success pass
load finished before totals and top started; param started after both finished pass
{{status}} replaced by 'paid' (6 rows) pass
Tables written: gold.e2e_orders 20 rows, gold.e2e_totals 5, gold.e2e_top 5 pass
Cron 0 */6 * * * (UTC) saved and enabled; the list shows the next run pass
Toggle off pass
The run log: load, then totals and top, then the parameter step
The run log: load, then totals and top, then the parameter step

Pipeline 2: a failing step#

Step Notebook Depends on On failure
load E2E 1 stop
broken E2E 4 load skip dependents
after_broken E2E 2 broken continue
independent E2E 3 load continue
Check Result
broken fails with the table-not-found message as the headline pass
after_broken recorded as Skipped pass
independent still runs to Success pass
Run status Partial pass
A failed step, its skipped dependent and the branch that still ran
A failed step, its skipped dependent and the branch that still ran
Pipelines list after the runs: status, steps and schedule per pipeline
Pipelines list after the runs: status, steps and schedule per pipeline

Dataflow: the same steps on the external scheduler#

Left rail → Dataflow → + New DAG → granularity pipeline → the three notebooks with totals and top depending on load → one retry, no schedule → Create. Then Run on the row.

Check Result
Scheduler service health: scheduler and metadata healthy pass
Definition created from the application and listed by the service pass
Run triggered from the application pass
Run finished Success; its task Success pass
The same run appears under Pipelines as Schedule: e2e_orders_dataflow, status Success, triggered externally pass
Dataflow list: the definition, its schedule, state and last run
Dataflow list: the definition, its schedule, state and last run
Run history of the definition
Run history of the definition
The task graph: one task that calls the application
The task graph: one task that calls the application

Totals#

Area Checks Passed
Pipelines 11 11
Dataflow 5 5

Before you repeat it#

  • Pipelines need nothing beyond a signed-in user with the Pipelines tab.
  • Dataflow needs the scheduler service configured by an administrator (its address and credentials, the folder it reads definitions from, and the connection it uses to call the application). An administrator's checklist is in chapter 12.

Next: 9. Lakehouse governance