IBM Scientists Demo Speedy GPFS Storage System

IBM has revealed an impressive demonstration of its new storage architecture design, dubbed the General Parallel File System (GPFS).

IBM had initially revealed GPFS back in November 2010 at the Supercomputing 2010 conference in New Orleans.

At that time IBM said that its boffins had a system that could convert terabytes of pure information into actionable insights twice as fast as previously possible.

Analytics record

And now IBM researchers demonstrated its GPFS storage management technology, by successfully scanning 10 billion files on a single system in just 43 minutes. This speed shatters the previous record of a data management system by a factor of 37.

To date, the fastest time that a system was able scan a billion files was three hours. That record was also set by IBM researchers, at the Supercomputing 2007 conference in Reno, Nevada.

IBM is calling this technology the storage performance breakthrough needed for organisations hoping to move to “Big Data Applications.”

This is because most organisations today are often swamped by the sheer volume of data being generated, and are not easily able to turn this data using analytic software into actionable insights.

Instead organisations are constantly fire fighting by trying to figure out how to manage and store it all, which is in turn fuelling the ongoing storage boom.

“Growing at unprecedented scales, this advance unifies data environments on a single platform, instead of being distributed across several systems that must be separately managed. It also dramatically reduces and simplifies data management tasks, allowing more information to be stored in the same technology, rather than continuing to buy more and more storage,” said IBM.

Long time coming

Although IBM had officially revealed GPFS last November, in actual fact the technology goes back over ten years.

Indeed it was back in 1998, when IBM researchers first unveiled a highly scalable, clustered parallel file system called General Parallel File System (GPFS), which IBM then fine tuned over the next ten years to make this breakthrough possible.

According to IBM, GPFS represents a major advance of scaling for storage performance and capacity, while keeping management costs flat.

This innovation, IBM believes, could help organisations cope with the exploding growth of data, transactions and digitally-aware sensors and other devices that comprise Smarter Planet systems.

Actionable data

IBM feels it is ideally suited for applications requiring high-speed access to large volumes of data such as data mining to determine customer buying behaviours across massive data sets, seismic data processing, risk management and financial analysis, weather modelling and scientific research.

IBM revealed that it managed to achieve the 43 minute breakthrough by running GPFS on a cluster of 10 eight core systems and solid state storage.

“Today’s demonstration of GPFS scalability will pave the way for new products that address the challenges of a rapidly growing, multi-zettabyte world,” said Doug Balog, VP, storage platforms, IBM. “This has the potential to enable much larger data environments to be unified on a single platform and dramatically reduce and simplify data management tasks such as data placement, aging, backup and migration of individual files.”

“Businesses in every industry are looking to the future of storage and data management as we face a problem springing from the very core of our success – managing the massive amounts of data we create on a daily basis,” said Bruce Hillsberg, director of storage systems, IBM Research – Almaden. “From banking systems to MRIs and traffic sensors, our day-to-day lives are engulfed in data. But, it can only be useful if it is effectively stored, analysed and applied, and businesses and governments have relied on smarter technology systems as the means to manage and leverage the constant influx of data and turn it into valuable insights.”