Big data, small Data, it’s all data.

When we think of Hadoop we think of huge clusters of machines, in a data centre somewhere dimming the lights of the cities they reside in, meanwhile data is being consumed, mined with the hope one day that sense will come out of it.

Take a look at your desk….

Now take a look at your desk, now I can’t vouch for everyone but I’m going to take a rough guess, there’s a phone or two. Perhaps a tablet, a Kindle maybes even a Raspberry Pi.

Everything’s a Node

Everything that’s connected is a node. Every device has some form of identification therefore it can be registered against a service of some form to know that it’s state is either idle or doing something. I’m thinking about all that time the Nexus S is sat on my desk doing nothing when it could be doing something useful, processing data.

Android being a good candidate for downtime processing as it can handle background tasks pretty well.

What if I said there IS a cure for……?

We’ve just not found it yet. Harking back to screensaver programs of old that did computations during the idle times (yes it has been done before), why not have the idle device work in the background pulling, processing and returning results back to a central server? Registered devices get paid for processing the data.

Think about it, a medical research firm could leave data with the main system and in turn farms out small segments of data for processing on connected and idle devices. Makes perfect sense to me considering that devices sit on our desks a lot of time on charge, while you’re asleep or working at your desk. Moore’s Law gives us the knowledge and foresight to know the power that these devices will have in the future.

Hadoop in your pocket is not far off (hopefully someone will port to iOS at some point, it has to be done).

It doesn’t really matter whether the data is big or small, while marketers will push the Big Data paradigm and how many billions it will potentially generate by 2017 or whatever there’s still a small matter of what can be done now.

Map/Reduce does have a significant role to play in processing health data in my opinion. But I also think there’s a plethora of devices not used to their full potential that, with the right code, just be able to do the job and send the results back.

Everyone can benefit.