A metadata-driven ingestion and transformation layer that turns a blank Fabric workspace into a running, governed data platform. Sources, load patterns and dependencies are rows in a control table, not one hand-built pipeline per table. Adding a source becomes a configuration change instead of a development task.
Someone opens an empty workspace and starts building by hand: one copy activity per source table, one notebook per transformation, a naming convention that holds for the first two weeks, and a deployment story that is really just clicking through the portal. Months later the platform works, but the engineering effort went into the plumbing rather than into the data.
Adding a table means a new pipeline, a new notebook and a new set of parameters. Onboarding cost stays flat instead of dropping as the platform matures, and incremental logic ends up differing per source.
When a load fails at 03:00 there is no single place that says which table, which batch, which watermark, and what the last good state was. Recovery is manual, slow and easy to get wrong.
Pipeline definitions, connection strings and workspace IDs get edited in place rather than promoted. A pipeline that works in dev breaks the moment it lands in production, and nobody can say why.
Sources, target tables, load patterns and dependencies are registered as metadata. A small set of generic pipelines and Spark notebooks reads that metadata at runtime and executes the right pattern per entity. The medallion layout, audit columns, naming convention and partitioning strategy are baked in, so they are applied the same way everywhere by construction.
Source systems, entities, load patterns, watermark columns and dependencies live in control tables. One generic pipeline resolves them at runtime rather than one artifact per table.
Bronze landing, silver conformed and gold serving layers created with consistent schemas, audit columns and Delta table properties, including OPTIMIZE and VACUUM maintenance jobs.
Full refresh, watermark delta, and merge or upsert on business key. The pattern is selected per entity through configuration rather than written again as per-entity code.
Every execution writes batch, entity, row count, duration and error detail to a logging schema. A run can be traced end to end and restarted from the last successful batch instead of from zero.
Workspace and lakehouse references, connections and paths resolve through Variable Libraries and deployment pipelines, so dev, test and production run identical definitions with different values.
Scripted creation of the workspace layout, lakehouse and warehouse objects, control tables and baseline pipelines, so a new engagement starts from a working skeleton rather than an empty page.
Data Factory pipelines orchestrate and copy, PySpark notebooks transform, Lakehouse and Delta hold bronze and silver, and the Warehouse or SQL endpoint serves gold. Variable Libraries, deployment pipelines and Git handle promotion. Source connectivity depends on the engagement: SQL Server and Azure SQL, files on ADLS or SharePoint, REST APIs, and SAP through a dedicated extraction tool where SAP is in scope.
If you are about to start a Fabric platform, or you are three sprints in and still writing plumbing, we will show the control tables, run one source through the framework end to end, and be honest about where it fits your landscape and where it does not. This is an internal asset a consultant deploys and adapts with you, not software you install unattended.