Selected Work

Case 01 / 04 · 2026

AWS DLT Pipeline

A metadata-driven AWS lakehouse: Finland's open grid API, ingested incrementally and refined by Databricks Delta Live Tables into a governed star schema — provisioned by Terraform, shipped through CI/CD.

View the code

01 · The problem

Fingrid — the operator of Finland's national power grid — streams electricity consumption and wind- and solar-generation data through an open REST API. It is high-frequency, time-series, and paginated: useful only once it is ingested reliably, modeled cleanly, and governed.

I built a cloud-native lakehouse on AWS that does exactly that — and does it declaratively. Adding a new dataset is a row of configuration, not a new pipeline; every environment is provisioned and deployed the same way, from code.

02 · Architecture

Fig. 01 The data flow. A control table drives incremental ingestion from the Fingrid API into an S3 landing zone; a Delta Live Tables medallion refines bronze → silver → gold with Auto Loader and Auto CDC into a governed star schema for Power BI.
Fig. 02 The delivery. A push or merged PR triggers GitHub Actions, which builds a Databricks Asset Bundle and promotes it across three isolated environments — Terraform provisions each one’s S3 and Unity Catalog, the bundle deploys the job and DLT pipeline, and a manual approval gates production.

03 · How it works

  1. Metadata-driven ingestion

    A control table holds each source — its id, watermark, and batch size. One job reads the active rows, fans out with a for-each, and pulls only each dataset’s new window from the API (paginated, with rate-limit retries), landing raw JSON on S3.

  2. A medallion that builds itself

    The Delta Live Tables pipeline generates its tables from config: a bronze table per dataset via Auto Loader, then a silver layer that routes each dataset to its own cleansing logic. Add a dataset and the graph grows — no new code.

  3. Auto CDC into a star schema

    Intermediate DLT views join the cleaned streams to conformed date and time dimensions, then Auto CDC flows apply SCD-1 upserts into the gold facts and dimensions — fact_consumption and fact_generation_forecast, ready for BI.

  4. Governed, multi-env, from code

    Terraform provisions the S3 lake; Databricks Asset Bundles package the job and pipeline; GitHub Actions promote them across dev, test, and prod — each an isolated Unity Catalog, so governance and lineage are centralized, not bolted on.

04 · Proof

Databricks job graph: a configuration task, a lookup task, a for-each loop running source-to-landing, then the ETL pipeline.
Fig. 03 The orchestration job. A configuration step seeds the control table, a lookup reads the active datasets, a for-each fans the API pulls out across them, and the final task triggers the DLT pipeline.
Databricks Delta Live Tables lineage graph: silver streaming tables and dimension views feeding pre-fact views into gold fact and dimension tables.
Fig. 04 The DLT lineage graph. Per-dataset silver tables and conformed dimensions flow through pre-fact views into the gold fact and dimension tables — the medallion, materialized.

05 · Outcome

The result is a hands-off, config-driven lakehouse: a new Fingrid dataset is one row in a control table, and it flows — ingested incrementally, refined through bronze, silver, and gold, and modeled into the star schema — all the way to Power BI, governed end to end and reproducible across every environment.

View the code Ask the assistant about this