If you like this then you might also like….

We’re into the second wave of personalisation, the big data version where colliding data and social influences could shape and influence our decisions. 

For the Amazons, Targets and Tesco’s of this world well they’ve been on it for years. Analysing scraps of data from all directions, Target’s system still blows me away and not for the pregnancy prediction models, just the directions they monitor from (ie all of them).  Even they don’t get it right. 

The newer kids on the block like Facebook who basically depend on targeted advert placement still haven’t got it right and they have more data than we’d be comfortable with I’m sure.  

It’s fair to say that the new battle ground is like the old battle ground, but bigger. It’s about the right ads in front of the right people at the right time. 

Recommendation for the rest of e-civilisation

For the smaller e-commerce store you might be working at an advantage but with a small pay off.

Any user recommendation system needs a user id, an item id and a score. You don’t want to give recommendations with a low rating score. The more transactions you have the better confidence you have of giving a decent prediction to the customer.

The real problem is scale. If you consider an online shop with 500 customers, 10,000 stock items and half a million transactions. Not impossible in this day and age. The basic classifer and recommender can be done but the time to calculate is the issue:

RecommendedItem[item:191, value:4.7008195]

RecommendedItem[item:9370, value:4.5856185]

RecommendedItem[item:8791, value:4.4668913]

Took: 57763 millis

So my recommendation engine mined 500,000 transactions and recommended three products to a user, one user. It took just under a minute. All 500 customer predictions would clock in at just under 8 hours on a single machine. For a startup we’re looking at doing this on a seperate service and perhaps within a cluster of machines.  This is not the sort of thing you want to be attempting in real time. 

For Tesco’s “Four Christmas’s a Year”, well now you know why. Data, data and more data to process. It takes time and practice. 

The incline to 2013

A weekly mailing list to customers isn’t going to be enough. The retail price isn’t going to be enough. And half baked recommendations aren’t going to make good conversions. If you can refine the recommendation process and get the customer to purchase even 1 more purchase at £12.99 then think about the long term customer value against the other 499 doing the same. Another £6,495 in the till, I thank you. 

The key to this is taking land inch by inch and running the predictions again and again. Once is never enough, it’s an ongoing process.

UPDATE 5th January 2013

I open sourced the code form the recommendation engine, you can either use the database version which I mentioned above or the much quicker file based recommender (using comma seperated files instead).

Have a look at this blog post or the Digital Circle article for more information.