Here’s what we do know, there’s a lot of data out there.  As 2011/12 appears to be the season of BigData, Hadoop and other lovely tools to play with. 

The question is do you really need the answers now? 

In terms of social media monitoring (Klout scores don’t count, they’re pretty meaningless to me) then there’s a good reason to attempt to process the firehose of data in realtime (or as close as you can get it), then react and respond.  Even then it’s questionable what represents a good response time. Most businesses are interested in their business and not constantly keeping an eye on online feedback. 

There’s a pretty good thing to remember in terms of any form of monitoring. If you can’t respond to it immediately then there’s no point measuring it immediately. 

Take Tesco for example, though times may of changed from point of sale to data warehouse (the Clubcard is a specialise subject of mine), you only get your points every three months it used the be called the “four Christmases a year”.  Even then to start off with only 10% of the data was mined. Once POS technology and connectivity came up to speed I’m sure a lot of more basket data is mined and processed a lot quicker. A perfect use for cloud computing elastic map reduce if you ask me, something I actually said in 2008.

Even with my own startup uVoucher the processing is batched though it can be turned on in to realtime if the retailer wants so.  With pure data volume of most small retailers a daily churn of customer data is fine.  If the stock is of a time critical nature then realtime customer notification could be the difference between a sale or stock that needs disposing.

I would suggest that 80% of the response could happen within 8-12 hours and there’s no real problem.  The customer (assuming they are a customer or a good chance of being a prospect) won’t mind some time between a response.  

Mining links, long term financial data rarely happen in realtime. The answers from that mining is then brought back into the decision making algorithms.  It all boils down to what answers you want, when you need them and the decisions your customers could make with them.