Selected Work

Case 02 / 04 · 2025–2026

Fingrid Data Platform

One grid API, two cloud implementations: Finland's national power-grid data, ingested incrementally and refined into a governed star schema — built first on Azure Data Factory, then rebuilt declaratively on AWS with Terraform and Delta Live Tables.

  • Databricks
  • Delta Lake
  • Medallion
  • Unity Catalog
  • Power BI

01 · The problem

Fingrid — the operator of Finland's national power grid — publishes electricity consumption and wind- and solar-generation data through an open REST API. It is high-frequency, paginated, and time-series by nature: a long way from anything an analyst can trust, and useful only once it is ingested reliably, modelled cleanly, and governed.

I built that platform twice. First on Azure: Data Factory driving a control-table pipeline into a Databricks medallion on Delta Lake. Then on AWS, rebuilt declaratively — the same medallion expressed as Delta Live Tables, with the lake and workspace provisioned by Terraform and shipped through Asset Bundles.

Building it twice is the point. The architecture is identical in both — metadata-driven ingestion, incremental by watermark, bronze → silver → gold on Delta, governed centrally and deployed from code. What differs is only which primitives each cloud gives you to express it. The toggle below switches between the two implementations.

02 · Architecture

Same architecture · two clouds

The declarative rebuild: Delta Live Tables generates the medallion from config, Terraform provisions the S3 lake underneath it, and Asset Bundles ship the whole thing as one versioned unit.

  • AWS S3
  • Delta Live Tables
  • Auto Loader
  • Terraform
  • Asset Bundles
  • GitHub Actions
Fig. 01 The data flow on AWS. A control table drives incremental ingestion from the Fingrid API into an S3 landing zone; a Delta Live Tables medallion refines bronze → silver → gold with Auto Loader and Auto CDC into a governed star schema for Power BI.
Fig. 02 The delivery on AWS. A dispatched GitHub Actions run takes the environment as an input, builds the Databricks Asset Bundle, and deploys the job and DLT pipeline into it. Two environments were configured and used — dev and test — each an isolated workspace over its own S3 buckets. Terraform provisions those buckets and was run from my machine; it was never wired into this workflow, and there is no approval gate. Naming that boundary matters more than implying a promotion chain I did not build.
View the AWS code

03 · How it works

  1. Metadata-driven ingestion

    A single control table holds every source — its endpoint, watermark, and batch size — so one parameterized pipeline serves all of them instead of a brittle pipeline per source. On Azure a Lookup feeds a ForEach; on AWS a lookup task fans out with a for-each. Adding a source is a row of configuration, not a deployment.

  2. Incremental by watermark

    Each run reads the last-loaded window from the control table, copies only the new slice, then writes the refreshed timestamp back — an Until loop on Azure, a paginated and rate-limit-aware pull on AWS. Loads stay cheap and idempotent, and a re-run costs nothing.

  3. A medallion on Delta

    Bronze (raw, as landed) → silver (cleansed, flattened, typed) → gold (a conformed star schema), with ACID transactions and schema enforcement throughout. On AWS the medallion is declarative: Delta Live Tables generates its graph from config and Auto CDC applies SCD-1 upserts into the gold facts, so the pipeline grows when the config does.

  4. Governed and reproducible from code

    Unity Catalog governs access and lineage across every layer, and the infrastructure underneath is code on both clouds — Terraform builds the AWS lake, ARM templates the Azure side. The pipeline is packaged once, as an Asset Bundle or an ADF release artifact, then promoted between environments as the same build rather than rebuilt for each one.

04 · Proof

AWS

Databricks job graph: a configuration task, a lookup task, a for-each loop running source-to-landing, then the ETL pipeline.
Fig. 03 The orchestration job. A configuration step seeds the control table, a lookup reads the active datasets, a for-each fans the API pulls out across them, and the final task triggers the DLT pipeline.
Databricks Delta Live Tables lineage graph: silver streaming tables and dimension views feeding pre-fact views into gold fact and dimension tables.
Fig. 04 The DLT lineage graph. Per-dataset silver tables and conformed dimensions flow through pre-fact views into the gold fact and dimension tables — the medallion, materialized.

05 · Outcome

The result is a hands-off, config-driven platform: a new Fingrid dataset is one row in a control table, and it flows — ingested incrementally, refined through bronze, silver, and gold, modelled into the star schema — all the way to Power BI, governed end to end and reproducible across every environment.

And because it exists twice, the design is demonstrably portable. The same medallion, the same governance posture, the same promotion discipline — expressed in whichever primitives the platform hands you. That portability is worth more than either implementation on its own.

Ask the assistant about this