Data volume explosion poses storage problem

By Alan Harten

Clearpace, a database archiving solutions company, commissioned Forrester Research to prepare a report into companies’ archiving habits.

The results seem to suggest that many companies are exposing themselves to considerable operational risk with inadequate archiving.

The research showed that there is a 50% annual growth in online transactional data and the repositories to house all that information.

Over a quarter of the companies in the survey have over 50 terabyte production databases.

Over 60% were still storing a big proportion of their data on tier 1 and a fifth of all data centre infrastructure spending was being put into data growth.

Clearpace deduce from these figures that company IT bosses are not focusing on the possible long term consequences of inadequate data storing or email and document retrieval.

The report indicates the IT managers are more concerned with other aspects of their workload and often pay a limited amount of attention to storage, concentrating on other critical areas such as security.

Clearpace contend that having old data at a company’s finger tips is essential for all kinds of potential compliance and other legal issues.

These include possible legal action, basic areas such as customer service and investigations of security issues.

They also consider that companies are not keeping pace with legal requirements on data storage but instead focus on lifting and shifting inactive data from production databases to improve application performance, without considering how to optimise the way archived data is stored.

They are also of the opinion that if a company has a good storage strategy and the archiving is up to date they will reduce storage costs and guarantee, simple, future-proofed access.

They also point out that inadequate retrieval or storage has heavy legal ramifications such as big fines and worst case scenario a prison term.

The report, with the catchy title “Why Database Archiving Should Be Part of Your DBMS Strategy”, emphasises that IT execs should be concerned about ensuring data integrity, security and availability rather than just the ability to select and move data.

The report collected 150 senior IT executives under bright lights and with no access to double latte coffee, half of them in the US the other half in the UK.

It discovered that enterprise data is growing in volume at an enormous rate, at around 50% per annum, due to new applications coming online and a trend towards retaining more data for analysis.

Three quarters stated that they were managing at least 10 databases with a minimum 1 terabyte capacity.

All of this activity is causing headaches for IT managers as they struggle with storage problems.

One of the frustrations is that up to 85% of production data is comprised of inactive records.

Another dilemma is that the perceived “value” of old data is weighed against the costs involved in rebuilding systems and data structures especially as large amounts of information is stored on slow retrieving tapes.

Moving data to an archive RDBMS adds complexity and requires additional resources similar to production databases for ongoing support - hence it rarely fully achieves the desired cost savings.

Even if all this data has “no value” there are still legal requirements that often demand viable storage, will all manor of government regulations making compliance a necessity and storage of huge amounts of information very often for many years.

In the end it is often a troublesome balancing act for IT managers trying to comply to government rules, make the data easily obtainable, keep it safe and secure and do this while spending as little money as possible.

Most data systems are currently aimed at requirements such as retrieving payment information so that customers can have quick and easy transactions rather than holding information on a client that had one transaction with the company six years ago.

The report highlights the need for more development of ever more economical and effective storage of mushrooming amounts of data.

About Clearpace
Clearpace Software Ltd is a UK-based software company that provides database archiving solutions for the long-term retention of structured data within the enterprise.

Clearpace has become a pioneer in the database archiving market by providing archive stores that are the optimal destination for inactive data that has been removed from production systems.

The Clearpace NParchive software enables organisations with large and growing data estates to massively reduce the cost and complexity of storing historical information while making archived data easily accessible for regulatory, legal and business purposes.

Using NParchive, companies are able to reduce the storage footprint of their historical data by more than 95%.

Clearpace serves a variety of industries, including Financial Services, Telecommunications, Energy, Utilities, Pharmaceuticals, Retail and the Public Sector.