Ingres: Why Vectorising Data Analysis Could Change Everything

Open-source database vendor Ingres today launched a plan to deliver radically-cheaper data analysis on mainstream hardware.

The VectorWise project is rewriting the database, so standard Intel chips can process multiple instructions at once – a “vector-processing” approach which has so far only been available on special purpose hardware. And putting it on standard hardware could change a lot of things, according to Ingres vice president of marketing, Ketan Karia.

“To use a specialised system for data analysis, from a company like Teradata, you have to put in place a seven figure piece of kit, which is incompatible to everyone else,” said Karia. “Only select organisations can afford to do that, and only select people within those organisations can use it.” Those with permission, have to ask the IT department for permission to analyse their figures, he said.

The new Ingres system would allow a normal laptop to handle a 600 milion row spreadsheet – and answer queries in less than a second, said Karia. By comparison existing software on a single core can process a hundred rows of a database simultaneously, and leaves the processor idling for 95 percent of the time.

This would allow “personal data marts”, he says, giving more people the chance to analyse their company’s raw data, without the intervention of the IT department – something that could also have security implications of course. “It opens up something that is today limited to a select few because of lack of resources.”

Analysis or transactions?

The breakthrough applies to data analysis, not transactions, but it should speed up all operations, he says.

“Most transactions include a query: say a telco wants to touch ten million customer records and mark them as to whether they are on a new tariff or not.

That is a transaction, but it involves sending a query.” Previously the database would have to chug along for a long while deciding which records to update: “Now the query part will happen in sub-seconds instead of minutes.”

The new module – which should boost performance by ten times, and continue to speed applications up beyond that, will be shown on Intel’s processors at the company’s IDF event in September, and will be delivered as a commercial product in 2010.

“It could be done on other chips but we chose to work with Intel first, so we can reach eighty percent of the business world,” said Karia. “We’re not talking to AMD but we cando – Intel is a benevolent partner.”

How it’s sold

Fast analysis might be seen as pulling the rug from under SAP and others, but Karia says it gives them new opportunities: “We are talking to SAP – they have a business warehouse, that suffers from latency. ” said Karia. SAP users are very loyal, but the software is cumbersome, he said, and companies like SAP could do a lot more business creating faster products allowing ad hoc queries.

The business plan is to offer this new feature to ISV third parties to build into software for their customers. Ingres does not expect to displace other databases like Oracle or Sybase or SQL Server in this way, but hopes to see its database built increasingly into the data analysis part of the solution vendors such as SAP or SAS sell.

“ISVs can port their analysis applications to Ingres, and get performance benefits,” he said. This adds ingres as another database platform.” This would be done under Ingres’ New Economics of IT programme, he said.

Ingres gets to do this kind of thing quicker because it’s open source, said Karia – and Ingres CEO Roger Burkhardt said similar things to eWEEK recently. “It had to be a complete rewrite of the database kernel,” he said. “The barrier to entry is quite high for all other database players. ”

The open source nature of Ingres was also a draw for Vectorwise‘s founders, he said . The group, founded by Peter Boncz and Marcin Zukowski, could have gone to Google – which saw a lecture about a prototype called MonetDB/X100 in a Google Tech Talk . “They chose to work with Ingre, to fully exploit the power of the research, unhindered by proprietary technologies,” said Karia.