What Happens If We Can’t Get Big Data Skills?

Big Data is growing – and some people see it as the future of enterprise computing. But what if it takes off too fast?

It’s now nearing the top of the Gartner hype cycle – the iconic graph from the analyst firm, which tracks new technologies up to a peak when expectations are at their highest, and then down the other side when people get disillusioned with them, before climbing the steady incline of real exploitation.

Hype or transformation?

Accusations of hype are easy to make, especially when a technology appears in the product plans of every large company, but in a mere handful of case studies.

Hadoop creator Doug Cutting told TechWeekEurope last week that, whether or not Big Data is being hyped, it is not a bubble: the technology provides real benefits in exactly the areas where  IT can help business most.

Automation is the way to increase profits in any field, and Big Data promises quick responses working with the masses of data which automation provides, Cutting told us.

But if the excitement really spreads, and everyone decides they have to do Big Data, what then? Where will the experts come from?

That was the most interesting strand of discussion at an IBM event last week. A report from Oxford’s Said Business School said organisations are keen to do Big Data, but have little understanding of what that means.

“There will be a skills gap, because the market cannot wait,” said independent analyst Martha Bennett. In that situation, we are likely to see a host of hasty Hadoop projects set up – but if they don’t have enough people on them who know what they are doing, there must be a real danger of a rash of Big Data disasters.

Could we see a backlash against Big Data, if that happens? If the first projects fail to deliver, then the concept might suffer reputational damage, and everyone might start to look the other way.

Certainly, earlier business analytics approaches, like the corporate dashboard, the data warehouse and the decision support system were laughably hyped and under-delivered massively on their first iteration.

Why Big Data might get it right

Big Data could be different, for a few reasons.

Firstly, the risks and losses in any failed Big Data project could actually be quite small. Underlying technology such as Hadoop is open source, and the concept runs on low-cost hardware – most likely operating in the cloud. This means that the actual resources committed to Big Data projects can be quite low, and the concept can get a thorough testing before anyone signs an enterprise-grade contract for it.

In previous generations of analytics, the big firms actually had lots of money to throw at the problem, and proceeded to throw it, at whatever sounded best, with predictably mixed results.

This time around, there’s less money sloshing about, so people are actually going to be careful.

Secondly, the industry is aware of the issue, and taking steps. People with SQL skills can retrain, and there’s plenty of people who can see the market opportunity in that.

Cutting’s firm Cloudera trains about 1500 people a month in the use of Hadoop and related tools. The Said Business School is adding Big Data to its MBA programmes. It’s pretty clear that some Big Data training could be a good investment – and there seem to be places to get it.

There may well be people who think Big Data is just a case of downloading a free package and filling it with some data, but most people will find out pretty quickly there is more to it than that. You do need to understand at least some of the science behind it.

But while Big Data may produce some failures in its earlier stages, the dynamic of the market is such that they will not be spectacular, and will produce learning rather than losing millions.

I may be wrong of course, and would be interested to be proven wrong. So do let me know your Big Data experiences, both positive and negative.

TechWeekEurope is still backing Tech Success in our awards scheme and elsewhere, but big failures can be instructive.

Are you a patent expert? Try our quiz!