DataSimpleDeveloper HandbookLog inOpen DataSimple
DataSimple
Chapter 04Bring data in

DataBridge

Connect many source systems, extract them three ways (full, incremental, change data capture), and land source copies and the Bronze change log in the lake. DataBridge's job ends at the lake.

Overview#

DataBridge is the ingestion service of DataSimple. It reads a whole database, selected tables, a folder of exported files, a restored backup or a set of Kafka topics, and writes each table into the lake's source layer in Parquet, Delta and Iceberg. It registers every copy in the catalog as datasimple.source.<schema>_<table>[_DELTA|_ICEBERG]. The first run of a table is a full load. Later runs bring only what changed. After the copies are written, DataBridge records every insert, update and delete in the Bronze change log. Silver and Gold are built afterwards, in notebooks and pipelines. (Optionally, a load can hand its batch straight to Silver apply; this is off by default.)

Who uses it:

  • Data engineers run the load wizard, pick load methods, set keys and watermarks, and schedule loads.
  • Administrators watch jobs on the Ingestion screen, retry Bronze, and steer watermarks in load control.
  • Developers start and follow loads from scripts through the REST API.

Where it is: left rail → Databridge. Load History (top right of the DataBridge page) lists jobs, schedules and load control. Administrators also see jobs on Ingestion.

How it fits the platform: every DataBridge call goes through DataSimple, which checks your sign-in, supplies the saved connection settings and adds your tenant. See Architecture and Data sources.

Key concepts#

Term Meaning
Job One load over one or more tables. Has an id (job_id), per-table results and a log. The id is also the _batch_id on every row it wrote.
Load method How a later run finds changes: full load, watermark column, database change log, file drop, backup snapshot, Kafka offsets. Automatic picks the change log when the table is tracked by the database, else the watermark, else a full load.
Watermark A change column (updated_at, change_seq). A later run reads rows past the last committed value.
Look-back For timestamp watermarks, the next run starts this many seconds below the committed value, to catch late rows. New keyed tables start with 300 seconds.
Load control The registry of every loaded table: load type, watermark, committed position, last window, counters, plus the run ledger.
Change log The Bronze event log per table (_op_type I/U/D, _cdc_seq). Written only when Bronze is Parquet. See The CDC change log.
Detect deletes An opt-in key comparison after a watermark run that records delete events for keys gone from the source, with a safety limit. Needs the change log, so Bronze must be Parquet.
Target The lake destination: a Target data source or a saved Data Lake cloud connection. Either way, its keys stay on the server. Every job writes to one.
Schedule A saved load plus an interval in minutes.

How it works#

DataBridge: source systems, the five steps of a load, and the lake target
DataBridge: source systems, the five steps of a load, and the lake target
Step Purpose
1. Connect Open a saved connection. Passwords and keys stay on the server.
2. Extract Read the table one of three ways: full load (the whole table), incremental (rows past the watermark), or change data capture (the database's own change log). Files, backups and Kafka keep their own place.
3. Land Write the table into the lake's Source layer as Parquet, Delta and/or Iceberg.
4. Record changes When Bronze is Parquet, append insert, update and delete events to the Bronze change log.
5. Register and commit Register the copies in the catalog. Commit the watermark only after the copies and the change log are written.

What happens in a load:

  1. You start a load from the wizard or POST /api/databridge/api/convert with connectionId and targetConnectionId. DataSimple checks your sign-in, supplies the connection settings and adds your tenant.
  2. DataBridge creates a job and starts the tables, up to the chosen parallelism. Very large tables are held back so the service stays within its memory budget.
  3. For each table it reads the full table, the watermark window, the change-log range, the new files or the Kafka offsets. Rows get the bookkeeping columns (_batch_id, _load_ts, _row_hash and more; see Bronze loads).
  4. The table is written in the chosen formats, and the copies are registered in the catalog. Silver apply runs next if it is switched on.
  5. When Bronze is Parquet, for watermark and change-log runs, the batch is compared with the change log and the insert, update and delete events are recorded.
  6. The position (watermark, log position or offset) is committed after the copies and the change log are written. A failure before that point leaves the position unchanged, so the next run re-reads the same window.
  7. Detect deletes and the automatic Bronze apply run after the commit. Detect deletes runs per table; the automatic Bronze apply runs at the end of the job.
  8. The job appears on the Log screen, in History and in lineage.
Delta Lake, Apache Iceberg and Apache Parquet
Delta Lake, Apache Iceberg and Apache Parquet

Delta Lake is a trademark of LF Projects, LLC. Apache, Apache Iceberg, Iceberg, Apache Parquet and Parquet are trademarks of The Apache Software Foundation. The logos identify the open formats DataSimple reads and writes.

Load options in the wizard#

Option What it does
Process with DataBridge engine (parallel reads, streams big tables, Bronze auto-apply available) or Spark box (bulk) for very large tables (writes Source staging only).
Source layer formats Parquet, Delta and/or Iceberg. Bronze is Delta when Delta and Parquet are both ticked, Iceberg when Iceberg and Parquet are ticked, otherwise Parquet. Delta and Iceberg Bronze need an S3 target.
Parallel threads Tables read at once (default 16). Higher values load faster but use more memory.
Enable watermark + CDC Turns on incremental loads and, when Bronze is Parquet, the change log into Bronze.
Change detection on later runs Automatic, Watermark column, Database change log, or Full load every run.
Watermark column The change column for watermark loads.
Also read rows whose watermark is empty See Rows with an empty watermark.
Detect deletes on later runs Off, After every run, Once a day. See Detect deletes.
Primary key column(s) Leave blank to use the source's declared key. Without a key, changes can only be appended. Check keys & watermark shows what DataBridge found per table.
Apply changes to Silver (MERGE) Not now, Type 1 (current state, deletes applied), or Type 1 + Type 2 history table. Needs a key.
Auto-apply to Bronze CDC after extract On by default. Writes Bronze right after the load.

Load methods#

Method What a later run reads Deletes Position kept
Full load every run The whole table. Copies replaced, or merged by key when a key exists. By snapshot comparison none
Watermark column Rows past the committed watermark, minus the look-back for timestamps. Only with Detect deletes; Salesforce also via IsDeleted last watermark value
Database change log SQL Server CDC or PostgreSQL logical decoding, from the stored position. The first run is a full snapshot, with the position taken before the read. The table must be CDC-enabled on the source. Yes log position
File drop Files not seen before under <folder>/<TABLE>/, tracked by object name and version. A file changed in the last two minutes waits for the next run, unless its folder has a _SUCCESS marker. Modes: merged by key, added, snapshot. Delete-marker column or snapshot mode files already loaded
Backup drop The newest backup file, restored into a temporary database, then read like a live connection. Keys missing from the new backup none
Kafka Messages from the stored offsets up to the end of each partition, as a micro-batch. By default a run reads at most 1,000,000 messages or for 120 seconds; you can change both per load. From the operation field topic offsets
Bulk (Spark box) A very large table read and written in parallel on Spark. Writes Source staging only; Bronze auto-apply is not available. n/a as above

The watermark window#

  • A run reads col > from AND col <= to. from is the committed watermark; for timestamps it is lowered by the look-back. For sequences, log positions and offsets it is exactly the committed value. to is the source's current maximum when the source can report it, else the largest value read.
  • The look-back applies only to keyed tables. An unkeyed table is appended, so re-read rows would be stored twice. Rows re-read by the look-back are merged by key and row hash, so they do not create duplicates.
  • A run with zero rows still moves the window. A failed run does not.
  • Two runs never load the same table at once.

Rows with an empty watermark

A row whose watermark is NULL fits no window. With Also read rows whose watermark is empty (include_null_watermark: true, keyed tables only), each later run also reads rows where the watermark is NULL and merges them by key. This works for SQL databases, BigQuery, Snowflake, HANA, MongoDB, Salesforce and SAP OData. REST API and Camel sources cannot ask for empty values; the option is ignored and the table result says so.

Large loads#

  • Streamed first load. When a table is too big to hold in memory, its full load (including the first load of a watermark or change-log table) is read in batches and written as it comes, so memory use is bounded by the batch, not the table. The watermark is stored only after every file is written. Your administrator can set a row count above which every full load is streamed. Streaming applies to S3 targets.
  • Part-file Parquet copies. A large incremental table keeps its Parquet copy as a folder of part files. With a key, rows are spread over bucket files by a hash of the key, and an incremental run rewrites only the buckets its keys fall into. Without a key, each batch adds one new part file. A full load or a schema change writes a complete new set of files and switches to it only when every file is uploaded, so readers never see a half-written copy. Small tables keep a single file.
  • Retries. A table that fails with a transient error before anything was written is tried again automatically (two attempts by default).

The CDC change log#

The change log is written only when Bronze is Parquet. For watermark and change-log runs, DataBridge then compares each batch with the table's change log and appends insert, update and delete events with a row hash and an increasing _cdc_seq. With Delta or Iceberg Bronze there is no change log; the automatic Bronze apply merges the batch instead.

On S3 targets the change log is a folder under the Bronze table:

Text
<table>/cdc/part-<first _cdc_seq>.parquet   one file per run that had events
<table>/cdc/_common_metadata                 the log's schema
<table>/cdc/_state/...                       latest state per key (bookkeeping)
  • Events are never rewritten or removed. Each run's events are one new file.
  • A run that stopped after writing its events is completed by the next run.
  • A run reads only the part of the per-key state its keys fall into, so the change-log step stays fast as the table's history grows.
  • When reading the folder with Spark, read every Parquet file in it; files and folders whose names start with _ are bookkeeping, not events.
  • ADLS targets keep the change log as a single file.

Detect deletes#

A watermark run never sees a deleted row. With Detect deletes set to After every run (detect_deletes: "every_run") or Once a day ("daily"), after a successful incremental run of a keyed table with Parquet Bronze, DataBridge reads the source's current key set, compares it with the keys live in Bronze, and appends one delete event (_op_type = 'D', _is_deleted = true, with the last known values) per key gone from the source. Nothing in Bronze is removed.

Nothing is recorded, and the table result says why, when:

  • the table has no primary key;
  • the table has no change log (Bronze is not Parquet, or watermark + CDC is off);
  • the key read failed;
  • the source returned no keys while Bronze has live rows;
  • more than the delete limit of the live keys would be deleted. The default limit is 20% (max_delete_ratio: 0.2); exactly the limit is allowed. Set max_delete_ratio per load or per schedule; your administrator can change the default.

A refused check never fails the load.

Salesforce deleted records. With Detect deletes on, an incremental Salesforce read also returns recycle-bin records (IsDeleted = true); each becomes a delete event. Use SystemModstamp as the watermark and Id as the key.

By API. POST /api/reconcile/preview shows what would be marked; POST /api/reconcile/run either stamps _deleted_at on the Bronze Parquet copy or appends delete events (event_log: true).

File ingest#

POST /api/files/run loads every file under a landing prefix that has not been loaded in its current version.

  • A new file is loaded; an unchanged file is skipped; a rewritten file is loaded again.
  • Each run writes one new Bronze part file under the table folder. Files count as loaded only after that part is written; a retried run replaces its own part.
  • Only one run per source and table at a time. A second run is refused with status busy. A run that died is taken over automatically after six hours.
  • Landing files are never modified or moved.

POST /api/files/preview shows what the next run would pick up, and GET /api/files/status shows recent runs. POST /api/files/copy copies raw files into the lake without reading rows.

Backup drop#

A backup drop hands DataBridge a database backup file. DataBridge restores it into a temporary database and reads it like a live connection. From the API:

  1. GET /api/backup-drop/health must answer ok: true.
  2. POST /api/backup-drop/prepare starts the restore and returns at once.
  3. Poll GET /api/backup-drop/status?id= until the phase is ready.
  4. Run the load, then DELETE /api/backup-drop/release?id= to free the temporary database.

Schedules#

A schedule is a saved load plus an interval in minutes. On the last step of the wizard, after a load has run, pick an interval (every 5 minutes up to every day) and click Repeat; each run picks up only what is new. Manage schedules under Load History → Schedules.

  • Due schedules are checked every 30 seconds and each starts a normal job, shown with triggered_by = schedule:<id> <name>.
  • A schedule whose previous job is still running is skipped and tried again one interval later, so runs never pile up.
  • A schedule built from saved Data Sources stores only their numbers and its owner, never the credentials. Each run looks the connections up again, so a changed password is picked up. If a connection was deleted, each run is skipped and the reason is shown on the schedule. The schedule stays on until you pause or delete it.
  • Pause or resume, run now, delete, or change Detect deletes per schedule.

Step timings#

Each table result carries the time of each step in milliseconds: _ms_read (read), _ms_copy (the Parquet copy write), _ms_cdc (comparing with and appending to the change log) and _ms_register (catalog registration). The Log screen shows them as read, copy, change log and catalog, for example read 1.2 s, copy 0.8 s, change log 3.1 s, catalog 0.4 s. Load History shows each table's read, write, upload, Delta and Iceberg times. Use them to see where a slow run spent its time.

REST API#

All paths are reached through DataSimple as https://<your-datasimple-host>/api/databridge/<path>, for example /api/databridge/api/convert, with Authorization: Bearer <PERSONAL_TOKEN>.

Method Path Purpose
GET /api/engines Source engines and whether each is available.
POST /api/test-connection, /api/test-target /api/test-connection tests a Kafka source with the settings you send. Test a database source with /api/databases or the Data Source test. /api/test-target tests a destination.
POST /api/databases, /api/tables, /api/tables-for-databases Discovery.
POST /api/key-check, /api/loaded-tables Keys and watermarks per table; tables already loaded.
POST /api/convert Start a job. Returns { job_id, options }.
GET /api/jobs, /api/jobs/{id}, /api/jobs/{id}/tables, /api/jobs/{id}/log, /api/jobs/{id}/report Job list, status, per-table results, log, report.
DELETE /api/jobs/{id}/cancel, /api/jobs/{id}/tables/cancel?table= Stop a job or one table.
POST /api/jobs/{id}/rerun, /api/jobs/{id}/bronze/retry Re-run a job; retry its Bronze apply.
POST, GET /api/schedules Create or list schedules.
POST /api/schedules/{id}/toggle?enabled=, /api/schedules/{id}/run Pause or resume; run now.
DELETE /api/schedules/{id} Delete a schedule.
POST /api/schedules/{id}/delete-capture Set Detect deletes and the delete limit for a schedule.
GET /api/delete-capture?schedule_id= Your tenant's delete checks, newest first.
POST /api/reconcile/preview, /api/reconcile/run Delete reconciliation by API.
GET, POST /api/files/status, /api/files/preview, /api/files/run File ingest.
POST, GET /api/files/copy, /api/files/copy/runs Copy raw files without reading rows.
GET, POST, DELETE /api/backup-drop/health, /prepare, /status, /release Backup restore.
GET, POST /api/load-control, /api/load-control/{connection_key}/{table}/reset, /lookback, /enabled Load control and its actions. See Bronze loads.
GET /api/load-runs The run ledger.

Developer examples#

1. Start a watermark load with Detect deletes#

JavaScript
const res = await fetch("https://<your-datasimple-host>/api/databridge/api/convert", {
  method: "POST",
  headers: { "Content-Type": "application/json", "Authorization": "Bearer <PERSONAL_TOKEN>" },
  body: JSON.stringify({
    source:  { connectionId: 18, database: "Sales" },
    target:  { targetConnectionId: 15, parallelism: 4 },
    tables:  ["dbo.Orders", "dbo.Customers"],
    formats: ["parquet"],                 // Detect deletes needs Parquet Bronze
    whole_database: false, connectionId: 18,
    enable_watermark: true, watermark_column: "updated_at",
    change_capture: "watermark",          // "auto", "watermark", "log" or "full"
    primary_keys: { "dbo.Orders": ["OrderID"], "dbo.Customers": ["CustomerID"] },
    auto_bronze_apply: true,
    detect_deletes: "daily", max_delete_ratio: 0.1,
    include_null_watermark: true,
  }),
});
const { job_id } = await res.json();   // e.g. { "job_id": 74, "options": { ... } }

2. Follow the job and read step timings#

Shell
H="Authorization: Bearer <PERSONAL_TOKEN>"
curl -s https://<your-datasimple-host>/api/databridge/api/jobs/74 -H "$H"
curl -s https://<your-datasimple-host>/api/databridge/api/jobs/74/tables -H "$H"
curl -s https://<your-datasimple-host>/api/databridge/api/jobs/74/log -H "$H"

The table results carry rows, formats, cdc (I/U/D counts), delete_capture (status, deletes, live_keys, source_keys, delete_ratio, message) and the step times.

3. Schedule the load every hour#

Shell
curl -s -X POST https://<your-datasimple-host>/api/databridge/api/schedules \
  -H "Content-Type: application/json" -H "Authorization: Bearer <PERSONAL_TOKEN>" \
  -d '{ "source": { "connectionId": 18, "database": "Sales" },
        "target": { "targetConnectionId": 15 },
        "tables": ["dbo.Orders"], "formats": ["parquet", "delta"],
        "connectionId": 18, "enable_watermark": true, "watermark_column": "updated_at",
        "interval_minutes": 60, "name": "Orders hourly" }'

4. Read load control and a batch from a notebook#

Python
ds.load_control()                 # your tenant's load-control rows, read only
                                  # pass a table name to get one table's row
SQL
SELECT OrderID, _op_type, _batch_id, _load_ts
FROM datasimple.source.dbo_Orders_DELTA
WHERE _batch_id = 74
ORDER BY OrderID;

Security and permissions#

  • Sign-in. Every DataBridge call needs a session or a personal token, and your role must include the Databridge screen.
  • Tenant. Every call carries your tenant. Jobs, schedules and load control are filtered by it, and everything a load writes is stamped with it (lake rows carry _tenantid, catalog registrations belong to your tenant).
  • Secrets. Saved connections are resolved inside DataSimple. Schedules store connection numbers, not credentials. Job reports leave out storage locations, hosts and similar details.
  • Audit and lineage. Each job appears on the Log screen with per-table steps, in History, and in lineage from the source table to the copies, Bronze and Silver.

Settings you can change#

Setting Where Effect
Load method, watermark, keys, formats, parallelism Wizard or /api/convert See Load options in the wizard.
Detect deletes and delete limit Wizard, /api/convert, or per schedule When to check and how many deletes are allowed (default 20%).
Look-back Load History → Load control → Lookback, or /api/load-control/.../lookback Seconds re-read before the committed watermark. 0 turns it off.
Reset, Disable Load History → Load control Start a table again from a full load, or skip it on later runs.
Schedule interval Repeat on the wizard's last step, or interval_minutes in /api/schedules Minutes between runs.

Your administrator can change service-wide defaults such as the default look-back (300 s), the default delete limit (20%), the File drop settle time (2 minutes), the row count above which full loads are streamed, the number of jobs that run at once (8) and the number of attempts per table (2).

Monitoring and troubleshooting#

  • Load History on the DataBridge page: jobs, per-table phases (reading, preparing rows, uploading Parquet, writing Delta, writing Iceberg, applying changes), Match / Merged / Mismatch, schedules and load control.
  • Log screen: one line per job with per-table formats, Bronze rows, Silver counts and step timings.
  • Ingestion (administrators): the job report, log, rerun, cancel and Retry Bronze.
  • History: jobs under the DataBridge filter.
Symptom Meaning
Login required (401) No session. Sign in or send a personal token.
Databridge service unavailable (502) DataBridge is not reachable right now. It restarts automatically; try again shortly, or contact your administrator.
Deletes: not recorded (...) Detect deletes refused: no key, failed key read, empty source, or above the delete limit.
File ingest returns busy Another run of the same source and table is in progress.
Mismatch Row counts differ between formats or from the source. Run again.
A table says run a full load The change-log position fell outside the source's retention. Reset the table in load control.

Limits and known issues#

  • Streamed full loads, part-file copies and the change-log folder apply to S3 targets. ADLS targets load in memory and keep a single-file change log.
  • Oracle (LogMiner) and MySQL (binlog) change-log capture are not available.
  • Apply changes to Silver merges the run's batch only; it does not receive Detect deletes events (it does receive Salesforce IsDeleted records).
  • Updating the Parquet copy of a keyed table can take several seconds even for a handful of changed rows.
  • Change-log files are not compacted yet.
  • Bulk (Spark box) loads write Source staging only; Bronze apply is not available for them.