Is Your Big Data Project Clever Enough To Be A Tech Success?

It may be big, but is it clever? That’s the thought I often have when I hear about Big Data.

Vendors and service providers are very keen to promote their Big Data expertise, but often what they say seems to miss the vital task of explaining what Big Data is, why it is important and why it is happening now.

Pointers to Big Data success

That’s why I’m looking forward to riffling through the deck of entrants for the Big Data category of our Tech Success Awards.

Vendors might slip into hype-mode when promoting products, but these entrants will be actual end user projects, where IT leadership has put a Big Data project in place to gain real benefits.

What am I expecting to find? Well, in some senses, this will be business as usual – but business super-charged by some big trends that are affecting the tech industry ar the moment.

I see three trends that will drive Big Data into widespread use:

• the new capabilities of the cloud
• the new generation of analytics-for-the masses, and
• the new flood of data from an increasingly instrumented world

“It is not about Big Data,” says Ian Osborne of Intellect UK, the UK high tech trade association which is sponsoring our Awards.”It is about data. Sensors will give orders of magnitude more data, and Big Data allows you to look at that data.”

“Big data means never having to sample,” he memorably tweeted from London’s Big Data Week in April.

https://twitter.com/IanFOsborne/status/194848731529625600″ data-datetime=”2012-04-24T18:02:12+00:00

The clever part of Big Data is the analytics, that pulls new patterns out of big collections of data. The new part is the fact that complex analytics can now be applied quicker and cheaper to a bigger – and even a real-time – collection of data. Big Data is analytics for the hip kids.

What is Big Data doing for you?

But what are the hip kids – or I should say, serious-minded but far-sighted IT professionals – doing with their Big Data? The first thing I want to see from a Big Data project is an obvious benefit. Some previously unexplored aspect of your organisation’s data which helps you move in a new direction.

That’s a given. I also want to see an awareness of privacy.

I can imagine some big data projects – say seismic data, or astronomy – that don’t have any privacy implications because they are handling information from inanimate objects (and it would be cool to have a winner from this line of work).

But most Big Data projects will involve data that touches on people’s lives. When organisations realise the kind of data they can gather, they can just start saving everything, in case. And that creates an aggregated pile of data which might  produce privacy breaches without people realising it.

Remember Google’s Wi-Spy event? Street View cars slurped up all the data they could from Wi-Fi networks as they went past, just because Google techies thought they might have a use for it.

There is an apocryphal Big Data story where a mother finds her teenage daughter is pregnant because the family PC starts serving ads for baby-care products, targeted at the family because of her daughter’s messaging and purchasing patterns.

I would hope any Big Data contenders we see have considered this side of the story. In fact, I’d like to see explicit acknowledgements of the trades users make when their data is captured. I want to see Big Data projects which address the question of who owns the data.

I also suspect we will see projects addressing new kinds of data, or using new means to transfer it. A lot of data is widely distributed when it is collected – so how do you aggregate it without clogging up networks?

I’m looking forward to some surprises. If you have a Big Data project, implemented in the UK since May 2011, that you want to shout about, send us some details.  We have extended the deadline until 19 October.

UPDATE: We’ve made it easier to enter. Fill in a form here!

Are you an open source guru? Try our quiz!