Selected Work

Case 02 / 04 · 2025

Fingrid Data Platform

An end-to-end Azure lakehouse that turns Finland's national power-grid API into governed, analytics-ready data — built declaratively, shipped through CI/CD.

View the code

01 · The problem

Fingrid — the operator of Finland’s national power grid — publishes live grid and energy-market data through an open API. Raw, high-frequency, and time-series by nature, it is a long way from something an analyst can trust.

I built a cloud-native platform that ingests that API, governs it, and models it into a single source of truth for reporting — with no manual steps and full reproducibility across environments.

02 · Architecture

Fig. 01 The data flow. A control table in Azure Data Factory drives incremental, configurable-batch ingestion from the Fingrid API into an ADLS Gen2 landing zone; a Databricks medallion on Delta Lake refines bronze → silver → gold into a star schema for Power BI.
Fig. 02 The delivery. Feature branches merge to main in GitHub; an Azure DevOps pipeline builds the Data Factory release once, and the release pipeline promotes that same artifact across three isolated environments — engineers author in Dev, QA validates in Test, business users consume from Production.

03 · How it works

  1. Metadata-driven ingestion

    A single control table holds the state and parameters for every source, so one parameterized pipeline ingests every endpoint at a configurable batch size — instead of a brittle pipeline per source.

  2. Incremental by watermark

    Each run reads the last-loaded window from the control table, copies only the new slice in an Until loop, then writes the refreshed timestamp back — so loads stay cheap and idempotent.

  3. Medallion lakehouse on Delta

    Bronze (raw) → silver (validated) → gold (business-ready) on Delta Lake, with ACID transactions and schema enforcement, ending in a gold star schema as the single source of truth.

  4. Governed and reproducible

    Unity Catalog governs access and lineage, and the whole platform deploys declaratively, promoted through CI/CD across Dev, Test, and Production.

04 · Proof

Azure Data Factory pipeline: a Lookup activity feeds a ForEach that fans out the copy.
Fig. 03 Metadata-driven orchestration: a Lookup reads the load parameters, then a ForEach fans the copy out across every configured endpoint.
Azure Data Factory incremental-load pipeline with set-variable activities, an Until loop, and a notebook step.
Fig. 04 Incremental load: variables set the time window from the Fingrid API, an Until loop copies data incrementally, and a notebook updates the last-refresh timestamp.

05 · Outcome

The result is a hands-off platform: new grid data lands, is validated, and reaches Power BI without anyone touching it — governed end to end and reproducible across every environment.

View the code Ask the assistant about this