The CxO's Ultimate Guide to Data Management Through Storage Consolidation
by Jaap van Duijvenbode on December 7, 2016
The amount of data enterprises are collecting and storing continues to increase at a staggering rate. According to one estimate, 90 percent of all the world's data has been created in just the past two years. And enterprises are storing 80 percent of that new data.
The traditional approach to meeting growing requirements for data storage has been to simply add more hardware. But that is becoming a less and less satisfactory solution due to several factors.
First, CIOs are quickly discovering that with ever-increasing storage requirements, trying to keep up by throwing more disk arrays and servers at the problem is simply too costly and too complex. One issue is the shear amount of additional devices that must be purchased not only to meet current needs, but also to be ready for future increases in demand. As Gartner analyst Bob Passmore observes, "Unless you are going to go out and buy new storage every week, you have to over-provision to allow for growth." Yet, a report from 451 Research notes that the IT budgets of most enterprises are already failing to keep pace with increases in the demand for storage. So, for most IT managers, continually adding storage hardware to existing data centers is not a viable strategy.
Another factor putting strain on data storage resources is that modern enterprises are typically geographically dispersed. Employees in offices around the globe must collaborate using the same datasets, often in real time. Providing local access to that data, while keeping it securely backed up, and ensuring that changes are propagated to all locations where the information may be stored, is proving to be a substantial storage management challenge. In fact, a recent survey reveals that managing their company's data storage infrastructure is the #1 concern voiced by IT professionals today.
Storage Consolidation Comes To The Rescue
Storage consolidation is a relatively new paradigm that is making the management of large-scale storage resources much less complex and costly. It involves centralizing a company's storage infrastructure by replacing a multitude of widely dispersed and heterogeneous hardware devices with a unified set of storage resources that can be managed as a single entity. For example, a company that has hundreds of individual storage servers in branch offices all over the world could consolidate all that storage into a single on-premises, cloud-based, or hybrid data center.
Key words that describe the storage consolidation concept include storage virtualization, software defined storage (SDS), and distributed data storage (DDS). In a consolidated storage system, sophisticated software allows administrators to provision and manage a large number of inexpensive commodity servers through a single control panel. The actual storage devices may be located in the company's own datacenter, in the cloud, or across some combination of the two (a hybrid approach). Yet both users and administrators interact with the storage system as if it was housed locally.
Advantages Of The Storage Consolidation Approach
Enterprises that have adopted storage consolidation are realizing a number of significant benefits. These range from lower overall costs to greater operational efficiency. Let's look at some of these in more detail.
Lower Storage Infrastructure and Operational Costs****
Large enterprises have historically had storage servers installed in numerous locations. Each major branch office, for example, might have its own storage infrastructure, including both hardware and some level of local IT support staff. With storage consolidation, many of these resources can be eliminated or redirected to more strategic uses. Having fewer storage devices in perhaps dozens of different locations translates into lowered expenses for server and disk array hardware, and for the electric power and cooling required to run them. And with most of the detailed storage management task shifted to the consolidated data center, remote sites no longer have to devote local employees to the care and feeding of their on-site storage infrastructure.
Many enterprises can realize even greater savings by leveraging the cloud. Although on-premises or hybrid implementations may be the optimal solutions for some companies, a cloud-based solution offers significant budget benefits. It relieves a business of the necessity of allocating CapEx expenditures to purchase and maintain storage hardware installed in the company's own data center. Instead, the exact amount of storage needed at a particular time can be purchased on a subscription basis through OpEx funding. Because the savings associated with cloud storage have proven to be substantial, business cloud storage has become the option of choice for many enterprises.
According to TechTarget, a recent study by Microsoft reveals that consolidating storage can save an enterprise more than $2 million per year.
Easier Data Management
With storage consolidation there are fewer devices to be managed. On-site administrators can provision and manage their local storage requirements through a "single pane of glass" portal into the consolidated data center. They no longer have to acquire, track, and maintain storage hardware for their sites. Details of servers, disk arrays, and the sophisticated software that controls the storage and retrieval of data need not be visible to local IT managers or their end users.
Scaling of storage capacity, as well as detection and replacement of failing devices, can occur without local sites even being aware that these processes are taking place. The result is a reduction in administrative complexity that leads to less time being devoted to managing storage, and a significant reduction in down time due to human errors in the management process.
Easier User Access and File Sharing
In a consolidated storage system, all authorized users are given concurrent access to the same centrally located datasets. Although that information may physically reside in the cloud, or on storage units that are widely dispersed across a network, all users, wherever they may be located, interact with that data as if it was stored on the same local server.
One of the most important issues that must be addressed when multiple users access the same datasets or files is ensuring that data integrity is maintained even when different users attempt to make changes simultaneously. When data is stored in on-site servers in each branch office, changes made in one location can easily clash with, or fail to be incorporated with changes made at other sites. But the sophisticated management software employed in consolidated storage implementations is designed to ensure that data integrity is maintained even when multiple users attempt to make changes at the same time.
For example, Talon's FAST™ solution uses a technology called Intelligent File Caching that combines data caching and file locking to protect against clashing changes to shared datasets. Users accessing a file are actually working with a local cached copy of the relevant data. However, when the file is opened, FAST™ automatically locks it from write access by other users, even if those users are located half a world away from one another.
Improved Data Security
Contrary to what might seem at first sight to be the case, storage consolidation can actually improve the security of a company's data. One reason is that there are simply fewer points of attack for intruders to access. Many locations where enterprise data was previously stored can be eliminated altogether, making the task of restricting physical access to storage locations easier. In addition, because the central data repository is under unified management, best-of-class security procedures can be expertly applied to prevent unauthorized access to the information, either while it is resident on storage devices or when it is in transit to and from the cloud or across a network.
Easier Backup and Disaster Recovery
When data is stored and backed up at local facilities, the amount of storage required, as well as the chances that differing backup procedures will lead to the loss or corruption of some data, expand. But in a consolidated storage system, end users work with cached copies of information that has been abstracted from a single data store, and there is no need to back up data locally. The fact that backup and recovery functions need be applied only to a single definitive version of the data, and not to a multitude of copies residing in branch offices around the world, greatly simplifies the task of assuring recovery of data that was corrupted or lost due to equipment failure or to man-made or natural disasters.
An additional benefit of data consolidation is that storage management software such as FAST™ allows backup and recovery of the central data store to be accomplished in a manner that is transparent to end users.
Adopting Storage Consolidation Requires Good Planning
Moving to a consolidated storage implementation is not something that can be done overnight. There are several potential pitfalls that IT managers must be aware of and plan for.
For example, if your data will be housed in your own on-premises datacenter, or partly on-premises and partly in the cloud, you'll need to consider the performance of your internal network. Whether it's an in-building LAN, or a geographically dispersed WAN, all storage traffic must pass across the network. If the network can't handle the load without degrading its speed, it can become performance bottleneck that slows down applications (and enrages end users) that depend on real-time access to the stored data.
Secondly, pay very special attention to ensuring that you have a rock-solid backup plan. Since all your company's business-critical data will now exist as a single authoritative copy, unanticipated holes in your backup and restore procedures could prove disastrous.
One of the most serious issues that must be planned for when adopting a storage consolidation solution is how your existing data will be migrated to the new system. You'll need to assess what data must be moved, how it will be moved, and what must be done to vet it before it can be moved.
Often an enterprise's data was originally housed in a disparate variety of local storage implementations. Will you need to filter and possibly reformat data from diverse repositories in order to prevent duplication, overlap, and inconsistencies? How will the data be physically delivered to its new storage location? By tape? Electronically? Your planning must address all these issues.
For most companies, a primary requirement during a data migration process is that it not result in significant system downtime. As the Mesabi Group's David G. Hill observes, "Migrations can take an unbelievably long time, and the longer that it takes, the greater the chance that something will go wrong."
Rick Allen, now CIO at Southern Regional Medical Center in Atlanta, knows the kind of planning storage consolidation requires. While serving at Gwinnett Health System in Gwinnett County, Georgia, he oversaw the consolidation of a storage infrastructure that had about 500 servers, 200 TB of disk storage, and 700 TB of tape backup. He speaks with the voice of experience when he says, "The biggest thing to remember is to do your homework up front. You don't have to buy the most expensive products -- just the right products."
Pick The Right Vendor To Work With!
A further piece of advice from Rick Allen is perhaps the most important of all: "Be sure to invest adequately in vendor support plans. Problems with consolidated storage usually have a large impact on an organization."
The challenges inherent in getting a consolidated storage implementation up and running reliably make the choice of the vendor you'll work with and depend on a crucial decision. Talon Storage, with its FAST™ intelligent file sharing solution, has become the acknowledged leader in the field. As a Microsoft partner, Talon has announced the availability of FAST™ for Azure StorSimple, which Microsoft is now offering to its StorSimple customers.
If you'd like to further explore how storage consolidation can provide significant benefits for your company, please download the Talon FAST™ data sheet.