Data Grows Apace As We Look At 2010 Storage Trends

Chris Preimesberger,

The most significant trend continues unabated: the mounting deluge of newly created and duplicated files

As we take a break at the end of 2010 to brace ourselves for the onslaught of new products and services to come for 2011, here are eWEEK’s observations of key trends in storage. Some are new, and some are continuations of trends we saw previously but all are based upon daily conversations with storage vendors, analysts, data centre managers, CIOs and CTOs.

Data Stores Doubling Every Two Years

The most significant up-and-to-the-right trend continues unabated: the mounting enterprise and consumer deluge of newly created or duplicated data files. One prediction is consistent across a number of analyst reports: The universe of data continues to grow at an estimated 40 to 60 percent per year, and it is not slowing down or even levelling off.

Gartner Research reported November 1 that about half (47 percent) of 1,000 data centre managers surveyed ranked data growth as the No. 1 biggest daily challenge they face. The other problems are the perennials: system performance and scalability (37 percent) and network congestion and connectivity architecture (36 percent).

Big Trend No. 2: Storage media, including spinning disk hard drives, solid-state NAND and NOR Flash, and digital tape, continues to become more efficient to use as engineers and manufacturers improve these products at regular intervals. Vastly improved networking and processing speeds are also instrumental in these upgrades. Yet the industry is not catching up with the amount of data being created — by humans and by computers.

Big Trend No. 3: Pricing. Storage, whether as a product or a service, continues to fall in total cost, thanks to increased marketplace competition and improvements in manufacturing techniques.

The Six Most Important Sub-trends

1. Data retirement: As most of us know, data that is supposedly wiped from a hard drive is not always deleted permanently. In fact, when a file is dragged into a “trash bin”, you have simply changed the folder in which it resides.

When you subsequently “empty” the trash bin, most of the time you still have not deleted anything; you have simply designated those data blocks for reassignment. Thus, those blocks may not be reassigned to other files for days or weeks. As a result, the previously “deleted” file can still live in the system for an undetermined length of time.

In the data recovery field, knowing the details of this is what keeps companies such as Kroll Ontrack, DriveSavers, i365 EVault, and numerous others in business.

A data retirement project is one that examines all archived and backed-up data in a system to make sure it gets completely deleted from all storage nodes when its expiration date comes up. Projects of this nature can take weeks or months, depending upon the size and scope of the storage system. An increasing number of enterprises, concerned about long-term storage usage and power consumption, are looking into these initiatives.

In large IT systems, substantial amounts of data often do not get deleted on a regular schedule, despite company policies. This is most often due to multiple backup copies and mirrored systems that get passed over at cleanup time.

All those extra copies take up valuable space on digital disks or tape that can cost the enterprise dearly over time. The term “storage system constipation” is becoming a painful reality for many storage administrators.