how it works
How does metadata-driven orchestration work?
ETLMaestro stores your entire orchestration design as metadata: six linked concepts in a database (configurations, platforms, process types, processes, schedules, phases). The engine reads that metadata, runs your existing ADF or Fabric pipelines in governed phases, and reports every state change to a status report anyone can read. If you know the control-table pattern, you already know the shape of this. The difference is that here the framework is the product, with the debugging surface built in, and it doesn't retire when its author does.
What is the data model?
Six concepts, each a table, each linked to the next. Together they describe everything the master pipeline used to hide in one engineer's head.
- 01
configuration
Environment-level settings and parameter constants in one place, instead of scattered across fifty pipelines.
- 02
platform
Where work runs: Azure Data Factory or Microsoft Fabric. Supported side by side.
- 03
process_type
Reusable patterns: file ingest, table-to-ODS copy, dataset refresh. Define once, instantiate many times.
- 04
process
A concrete unit of work: this source, this target, these parameters. A row, not a rebuild.
- 05
schedule
When processes run and what they wait for. Dependencies are data instead of drawn arrows.
- 06
phase
The checkpointed steps inside a run. Phases are what make failure recovery surgical instead of total.
The numbering is the dependency order: each concept builds on the one before it.
What happens during a run?
The engine walks the schedule, launches eligible processes up to the configured concurrency ceiling (3 of 6 slots in use is a normal mid-night picture), and records every phase transition. A failure stops the process at its current phase. The retry policy re-launches from that phase: once, from checkpoint, with succeeded phases left alone. Compare that to rerun-from-failed-activity in the Monitor view, done by hand, per run, per failure.
- Running means visibly running: which process, which phase, since when.
- Failed means failed at a specific phase, with the retry state shown next to it.
- Succeeded means every phase checkpointed through to the end.
The status report is a Power BI report over the same metadata. A manager can read it without translation, and nobody needs contributor rights on the factory to see what ran last night.
- sales_ods_nightly running succeeded
- erp_gl_extract succeeded
- claims_file_ingest failed · phase 4 retrying · checkpoint succeeded
- fabric_lake_merge running
- pbi_dataset_refresh queued running
the live status report, with running, succeeded, and failed states, is shown in every demo
What ETLMaestro is, and what it isn't
| ETLMaestro is | ETLMaestro is not |
|---|---|
| An orchestration and management layer on top of ADF and Fabric | An ETL tool. It never moves data itself |
| A metadata database that describes your whole pipeline estate | A replacement for the Azure platform you already pay for |
| A status report your whole team can read | A premium capacity add-on priced like a second platform |
| Phased execution with checkpointed, automated retry | A re-run-everything button |
Scope, stated plainly: if your team doesn't consolidate data into a central repository on ADF or Fabric, ETLMaestro won't help you yet. For systems outside Microsoft, the Azure Function integration lets ETLMaestro orchestrate them alongside your ADF and Fabric work.
Weighing this against Airflow, Dagster, or Prefect? The honest comparison for Azure teams covers where each one genuinely fits.
plain answers
The questions every evaluation asks
What is metadata-driven orchestration?
An approach where the orchestration design lives as data instead of drawn pipelines: processes, schedules, dependencies, and phases are rows in a database. An engine reads those rows to run your pipelines, and the status report reads the same rows to show what happened. Change a row, change the behavior; no canvas to redraw.
Do we have to rebuild our pipelines to use ETLMaestro?
No. Your ADF pipelines and Fabric notebooks stay exactly where they are and keep doing the work. What moves into ETLMaestro is the coordination: what runs, in what order, on what schedule, and what happens when something fails.
What happens when a run fails?
The process stops at its current phase and the failure is visible immediately: which process, which phase, since when. The retry policy relaunches once from that checkpointed phase, leaving succeeded phases alone, and a notification goes out if the retry fails too. Nobody re-runs the whole night by hand.
Who can read the status report?
Anyone on the team. It's a Power BI report over the orchestration metadata, so a manager can check last night's run without contributor rights on the factory and without asking an engineer to translate.
Watch it run against real metadata
The demo walks the full loop: insert a metadata row, watch the process appear, fail it on purpose, watch it resume from checkpoint.