Living in York for the majority of my life you kind of get used to flooding, it happens frequently. With the number of floods increasing and their chances of it being a real nasty one also increasing, it might be a good idea to start peeking at past data and seeing if there’s anything jumping out at us.

In NI We Don’t Need Open Weather Data….

There’s Cecelia who tells it like it is. It’s usually raining….*

Getting Historical Met Office Data

While Cecelia can with 98% accuracy tell the biblical level of weather for tomorrow and the day after that, historical data is another matter. That’s where open data comes in rather handy. And those Met Office folks opened up a lot of data in 2011 on the data.gov.uk portal.

Historical data comes by way of a form which goes and scuttles off to the data archive and sends back the data as a CSV file.

Fine if you want one day but rather a pain if you want to get data between two dates. Now you could do some handy unix scripting to pull the data in but there is an issue with that. When you fire the search form the data is pulled and then forwarded to another url with a unique id. So unless you’re great at handling redirects within unix it’s going to be a hard slog to get the data out…… well, it was, now I have good news for you.

A Clojure Alternative?

MastodonC created an open source project called kixi.hecuba.weather, the primary purpose is to pull historical weather data and send it up to the Hecuba platform. It’s fully open source so even if you’re not using Hecuba you can still use some of the component parts of k.h.w. to pull the Met Office data for your needs.

The project is hosted on github and anyone can use this project for grabbing the Met Office data.

Clone The Project

First of all you need to clone the project. From the command line run the following git command from your terminal or command prompt.

git clone git@github.com:MastodonC/kixi.hecuba.weather.git

You’ll see the repository download to your machine.

dev:mctemp jasonbell$ git clone git@github.com:MastodonC/kixi.hecuba.weather.git
Cloning into 'kixi.hecuba.weather'...
remote: Counting objects: 209, done.
remote: Total 209 (delta 0), reused 0 (delta 0), pack-reused 209
Receiving objects: 100% (209/209), 39.05 KiB | 0 bytes/s, done.
Resolving deltas: 100% (68/68), done.
Checking connectivity... done.
dev:mctemp jasonbell$

With that done you can now start the REPL and start running functions to pull down the data.

Starting The REPL

I’m going to first compile the k.h.d. project and the start the REPL from the command line:

lein compile
lein repl

Give it a few moments then you will see the output as the REPL starts and then the prompt, once you have the prompt you can start working.

nREPL server started on port 55453 on host 127.0.0.1 - nrepl://127.0.0.1:55453
REPL-y 0.3.7, nREPL 0.2.10
Clojure 1.7.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27
 Docs: (doc function-name-here)
 (find-doc "part-of-name-here")
 Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
 Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

kixi.hecuba.weather.core=>

The namespace where the Met Office functions are is called “kixi.hecuba.weather.metoffice-api“, so you will need to change namespace first.

kixi.hecuba.weather.core=> (ns kixi.hecuba.weather.metoffice-api)

Now we can turn our attention to retrieving the Met Office data.

Retrieving The Data

The run-data-pull function takes three parameters, a start date, end date and a path where to save the files to. For example if I’m wanting to pull historical data from the 1st of January 2013 to the 1st of February 2013 I would run the following. The dates are entered as a dd/mm/yyyy format, so for example:

kixi.hecuba.weather.metoffice-api=> (run-data-pull "01/01/2013" "01/02/2013" "/Users/jasonbell/Documents/work/projects/mctemp/")

The function will call the API and save the data for each hourly observation and then save it to the directory specified.

-rw-r--r-- 1 jasonbell staff 21427 7 Dec 13:18 01-01-2013-0000.csv
-rw-r--r-- 1 jasonbell staff 20759 7 Dec 13:18 01-01-2013-0100.csv
-rw-r--r-- 1 jasonbell staff 20690 7 Dec 13:18 01-01-2013-0200.csv
-rw-r--r-- 1 jasonbell staff 20755 7 Dec 13:18 01-01-2013-0300.csv
-rw-r--r-- 1 jasonbell staff 20734 7 Dec 13:18 01-01-2013-0400.csv
-rw-r--r-- 1 jasonbell staff 20809 7 Dec 13:18 01-01-2013-0500.csv
-rw-r--r-- 1 jasonbell staff 21328 7 Dec 13:18 01-01-2013-0600.csv
-rw-r--r-- 1 jasonbell staff 21234 7 Dec 13:18 01-01-2013-0700.csv
-rw-r--r-- 1 jasonbell staff 21312 7 Dec 13:18 01-01-2013-0800.csv
-rw-r--r-- 1 jasonbell staff 20789 7 Dec 13:18 01-01-2013-0900.csv
-rw-r--r-- 1 jasonbell staff 20673 7 Dec 13:18 01-01-2013-1000.csv
-rw-r--r-- 1 jasonbell staff 20825 7 Dec 13:18 01-01-2013-1100.csv
-rw-r--r-- 1 jasonbell staff 20812 7 Dec 13:18 01-01-2013-1200.csv
-rw-r--r-- 1 jasonbell staff 21003 7 Dec 13:18 01-01-2013-1300.csv
-rw-r--r-- 1 jasonbell staff 20888 7 Dec 13:18 01-01-2013-1400.csv

The make up of the CSV files is documented by the Met Office, here’s a quick look at one of the files for your reference.

Site Code,Site Name,Latitude,Longitude,Region,Observation Time,Observation Date,Wind Direction,Wind Speed,Wind Gust,Visibility,Screen Temperature,Pressure,Pressure Tendency, Significant Weather
"3005","LERWICK (S. SCREEN) (3005)","60.1390","-1.1830","Orkney & Shetland","12:00","2013-01-01","WNW","10","","14000","3.80","989","R","Light rain shower (Day)",
"3031","LOCH GLACARNOCH SAWS (3031)","57.7250","-4.8960","Highland & Eilean Siar","12:00","2013-01-01","WNW","16","29","18000","3.70","997","R","Heavy Rain",
"3041","AONACH MOR (3041)","56.8200","-4.9700","Highland & Eilean Siar","12:00","2013-01-01","W","17","32","","-1.20","","#","N/A",
"3063","AVIEMORE (3063)","57.2060","-3.8270","Highland & Eilean Siar","12:00","2013-01-01","WSW","3","","16000","3.50","997","R","Light rain shower (Day)",
"3066","KINLOSS (3066)","57.6494","-3.5606","Grampian","12:00","2013-01-01","WSW","17","","40000","5.20","996","R","(Black) Low-level cloud",