← eldouma.dev data project · Databricks lakehouse

every detector, every 15 minutes

Melbourne
Traffic Pulse

Victoria publishes a vehicle count for every loop detector at every traffic signal, every 15 minutes. This pipeline takes those files into a Databricks lakehouse each day, checks and reconciles them, and publishes what the city's traffic is doing. It runs like a production service: scheduled, monitored, with a freshness target, reconciliation checks and an incident record.

Pipeline…
Data freshness…
Checks…
30-day success…
Open incidents…

The pulse map

Every signal site in metropolitan Melbourne, placed by its coordinates and sized by its average weekday traffic over the last 28 days. No map underneath: the roads appear from the data. The glow follows the city's day, hour by hour.

The city's day

Average vehicles per hour across all signal sites, last 28 days.

Weekdays Weekends

Busiest intersections

Average weekday vehicles, last 28 days.

    Every day, against its baseline

    Vehicles counted per day. The line is the median of the same weekday over the previous four weeks; days more than 15% away from it are marked.

    How it runs

    One Databricks job, seven steps, every morning at 06:30 Melbourne time. Code lives in Git and deploys to Databricks automatically when it changes.

      Reconciliation and quality checks

      Detector health, newest day

      Network-wide gaps

      Intervals where every detector reported a fault at once: a system event (a clock change or a central outage), not a broken detector. Kept out of detector health and listed here.

        Performance tuning

        Incident record

        Incidents open automatically when a run fails, the publisher's data is late, or a quality check warns, and close when the condition clears. Each one gets a root-cause note.