DataSimplePractical GuideLog inOpen DataSimple
DataSimple
Chapter 08Orchestrate

Pipelines and workflows

Back to the index · Previous: 7d. Streaming, materialized views and incremental tables

How it works#

A notebook is the unit of work; three things run notebooks for you. Schedules run one notebook on a cron (chapter 7). Pipelines run a graph of steps with dependencies: notebook, dbt project, Flink SQL statement or Workflow; steps with no dependency between them run at the same time, and a failed step stops its dependents. Dataflow is the orchestration view for teams that already run a scheduler service: the application writes the definition files and proxies the service's API, so the service owns the calendar and the retries while the application still runs the steps. A Workflow is different in kind: a canvas built by chatting with Sarah that creates a source → bronze → silver → gold → governance flow over one Data Source, with a notebook behind every box, which you can test on a sample, execute, and save as a pipeline.

Pipelines#

Pipelines
Pipelines
  1. Left rail → Pipelines → + Create Pipeline.
  2. General: name, description, cron and timezone (leave the cron empty to run by hand), enabled, retry count, email recipients, webhook, compute profile.
  3. Steps: type (Notebook, dbt, Flink SQL, Workflow), a short step key (bronze, silver, gold), depends on, on-failure policy (stop, continue, skip_dependents), parameters, timeout. dbt steps take the project folder, the command (run / test / compile) and the warehouse; Flink steps the SQL, mode (batch or streaming) and parallelism; Workflow steps the workflow.
  4. Save. Run on the row starts it; open the row for the graph, each step's status and logs, the run history, Stop, Toggle, AI lineage and Fix error.

Run can take parameters; they override the notebook steps' own parameters for that run and are kept on the run record. Example: step bronze runs the notebook from chapter 7; silver depends on bronze and builds silver.orders_scored; gold depends on silver. If bronze fails, the others do not start. The scheduler ticks every minute; a pipeline cron, a notebook schedule and the retry queue all run from it.

HTTP
POST /api/pipelines
{ "name": "orders_daily", "cron_expression": "0 6 * * *", "timezone": "UTC",
  "steps": [ { "step_key": "bronze", "notebook_id": 12 },
             { "step_key": "silver", "notebook_id": 13, "depends_on": ["bronze"] },
             { "step_key": "dbt_marts", "step_kind": "dbt", "depends_on": ["silver"],
               "parameters": { "_dbt": { "projectDir": "dbt/orders", "command": "run" } } } ] }
POST /api/pipelines/:id/run        { "parameters": { "region_id": 3 } }
GET  /api/pipelines/:id/runs
GET  /api/pipelines/:id/graph                   -- table lineage DAG plus latest run
PUT  /api/pipelines/:id/toggle     { "enabled": false }
POST /api/pipelines/:id/stop

An invalid step is rejected with 400 and a message naming the step. Pipelines promoted by Deployment land disabled on the target.

A worked run#

Four steps: load builds gold.e2e_orders; totals and top both depend on load, so they start together once it finishes; param depends on both and receives status = paid.

Steps with their dependencies and failure policy
Steps with their dependencies and failure policy
The run log: load, then totals and top, then the parameter step
The run log: load, then totals and top, then the parameter step

When a step fails under the skip dependents policy, its dependents are recorded as skipped, independent branches still run, and the run ends Partial with the failing message as the headline:

A failed step, its skipped dependent and the branch that still ran
A failed step, its skipped dependent and the branch that still ran
Pipelines list: status, step count and schedule per pipeline
Pipelines list: status, step count and schedule per pipeline

Dataflow#

Left rail → Dataflow → + New DAG → DAG id, a notebook (or pipeline steps), granularity (one task per notebook, per cell, or a pipeline), schedule, retries → Create. Rows offer pause, resume, trigger, backfill, clear, mark state, logs and a code editor. Cell runs land in Log tagged Dataflow. When no Dataflow scheduler service is configured the screen says Dataflow isn't reachable and offers Retry; Schedules and Pipelines do not need the orchestration service.

Dataflow list: the definition, its schedule, state and last run
Dataflow list: the definition, its schedule, state and last run
Run history of one definition
Run history of one definition

With pipeline granularity the definition has one task that calls back into the application, which runs the steps exactly as a Pipeline would; the run therefore also appears under Pipelines as Schedule: <dag id>.

The task graph of a pipeline-granularity definition
The task graph of a pipeline-granularity definition

Pipeline or Dataflow?#

Pipeline Dataflow
Who owns the calendar the application's own scheduler (cron on the pipeline) the external scheduler service
Who runs the steps the application the application (the service calls it)
Step types notebook, dbt, Flink SQL, Workflow notebooks, cells or a pipeline
Dependencies and parallel branches depends_on, waves the same, saved as a pipeline
Failure handling per step: stop, continue, skip dependents; retry count service-level retries and backoff on top
Needs anything installed no the scheduler service
Where you see runs Pipelines, Log, History Dataflow (runs, grid, graph, logs) and Pipelines

Rule of thumb: start with a Pipeline; move the calendar to Dataflow only when an operations team already runs the scheduler service and wants every job in one place.

HTTP
POST /api/notebooks/airflow/dags   { "dagId": "orders_daily", "notebookId": 12, "schedule": "0 6 * * *", "granularity": "notebook" }
POST /api/notebooks/airflow/dags/orders_daily/trigger   { "conf": {} }
GET  /api/notebooks/airflow/dags/orders_daily/runs

Workflows built by chat (Sarah)#

Workflow canvas
Workflow canvas
  1. Left rail → Workflow → + New workflow (ask Sarah). Say what you want: "Load Orders and Customers from DS-16 into bronze, silver and gold, mask the email column."
  2. Sarah builds the boxes: connection, source (the raw copy into the lake through DataBridge), bronze, silver, gold, governance (masks, grants, certificates), plus optional steps (data quality, dbt, Flink, files, SQL, optimize, webhook) and AND/OR gates. Per table you can set the primary key, watermark, SCD2, the silver mapping and dedupe, the gold group-by and measures, and masks.
  3. Click a box to open its notebook. Test runs the box against a <box>_test target with sampled rows; Execute step runs one box for real; Execute workflow runs the whole graph in waves of three.
  4. Runs shows the history; Lineage the tables the run produced; Re-run failed and Stop act on the current run.
  5. Schedule as pipeline saves a nightly Pipeline (cron 0 2 * * * UTC) with one workflow step.

Inside a box notebook: read_staged(connection_id, table) reads the raw copy, workflow_table("orders") names the box's target, and save_table(df, workflow_table("orders")) writes it. Runs register bronze, silver and gold tables in Unity Catalog and record lineage.

HTTP
POST /api/workflows
{ "name": "Orders flow", "connectionId": 16, "sourceSchema": "dbo",
  "tables": [{ "name": "Orders", "primaryKey": "OrderID" }],
  "options": { "nodes": ["connection", "source", "bronze", "silver", "gold"] } }
POST /api/workflows/:id/run        {}  or  { "tables": ["Orders"] }
GET  /api/workflows/:id/runs
GET  /api/workflows/:id/lineage
POST /api/workflows/:id/pipeline   {}        -- nightly pipeline

Choosing between them#

You want Use
one notebook, nightly a Schedule
several notebooks in order, dbt or Flink in the chain, run-time parameters a Pipeline
a whole medallion flow over one source without writing it yourself a Workflow, then Schedule as pipeline
your existing orchestration service to own the schedule Dataflow

Read more#

Next: 9. Lakehouse governance