Recently, I came across a very interesting infographic that exhibited ‘What happens on the Web in 60 seconds’. The Infographic showcased that in just a minute, we create thousands of terabytes of data which is an incredible feat in itself. Add enterprise data to it and the number shoots up exponentially. That’s the thing about data – it is only going to grow, and will continue to grow at breakneck speed.
Today, every organization from a start-up to a large enterprise is grappling with the challenges of data growth and the subsequent management of the storage of that data and its associated costs. With the continuous advent of new technologies and more efficient storage devices, companies are continuously looking to move data from old to new platforms, whether they be local or to the cloud.
Over the years, we’ve spoken to hundreds of customers, both large and small, and deduced that there’s a misconception that exists about data migrations. The most common misconception is: ‘the migration of data from legacy system to a new application is relatively simple’. Well, how hard can it be to move data from A to B? It should only be a matter of data movement and that’s all physics, isn’t it?
As far as the senior management — the CIOs, CTOs — is concerned, data migration is not even a part of their agenda. For them, it’s accepted as a given and something that just gets done ‘magically’ overnight. The Infrastructure Manager or Storage Head rely on their Architects and Technicians to figure out how to deal with data migrations. Alternatively, companies ask the hardware vendor to find a solution and bundle it with the hardware purchase. The Storage Operations team — the foot soldiers — on the other hand believe that data migrations means an end to their personal lives simply because data migrations are perpetual, and typically get done on nights and weekends. The team is always on the lookout for any excuses of not having to deal with data migrations as its painstaking and unrewarding.
Below are the top five myths that we’ve found from our conversations on what data migration is and what is expected out of a migration.
- The hardware vendors will bundle in professional services
- We can hire an army of consultants and they will write scripts to get it done
- Our Business As Usual (BAU) team can take on the additional workload
- Virtualize the storage infrastructure and we don’t need to worry about migrations
- We will migrate everything ‘as is’ and restructure after we get there
Myth 1: Hardware vendors will bundle in professional services Perhaps the most used and common approach in the industry today.
The process starts with identifying a hardware vendor with expertise in hardware, software and professional services to provide a solution that includes a package of products that could address your data migration issue. You want the hardware vendor to figure out the services part to integrate the hardware and software into a working environment. The most important aspect for you is to determine the fastest and most efficient way to utilize your resources given your purchase.
Unfortunately, as it usually turns out, the entire solution is designed around what the vendor wants to sell to you and very little to do with solving your problem. If you perform a gap analysis, you would determine that there is a mismatch in ‘what you want’ and ‘what you get’. As hardware vendors are not experts in professional services, they are typically thinking of professional services as staff augmentation rather than providing true solutions. They sell their hardware, software and professional services to you and more likely than not, the services aspect entails you as the customer to take ownership of its execution. This approach usually leads to finger pointing between the two parties, not to mention a drawn out project timeline and an exponential increase to your expected cost of for the migration project.
Myth 2: We can employ an army of consultants to write scripts and get it done.
Most companies think that by employing an army of consultants who write customized scripts, that the migration will be ‘automated’ and delivered within a confined timeframe. In addition, there is an assumption that the scripts are shared among the team and there is appropriate testing performed to validate script function and scalability.
Performing data migration through scripts can bring value in very specific scenarios, usually when the customer has very customized processes and conventions that need abiding by. However, in the vast majority of the cases what companies are NOT aware of is that the process of data migration performed by a set of scripts is riddled with a lot of conditions.
The challenge with scripts is exactly that, they are scripts written to automate certain functions that the engineers do manually. The team members usually each write their own scripts and that creates a myriad of ways in which the implementation takes place, leading to lack of consistency in the migration process. Any changes require engagement with the individual who wrote the scripts. In addition, most of the time there the scripts are a trial and error as it’s a fully manual process to put them together with limited or no testing except when deployed into a production environment. Wow, now that’s scary!
Myth 3: Our business as usual (BAU) team can take on the additional workload.
The initial thought process for those that have not experienced a migration is ‘I have a ton of resources that manage storage, it will be easy to give the migration work load to them as well. This will save me from spending on services or software to do the migrations’.
Unfortunately, the reality is that the BAU team is barely able to cope with the demands of the business and meeting the operational challenges tied to exponential storage growth. A typical company continues to expand its capital spend on storage but continues to look to reduce operational expenditures including head count. As such, storage teams have limited or no time to take on additional responsibility much less work on large refresh initiatives that require adequate time for planning, design, validation and execution. By adding migrations to the BAU team, we have observed that most teams end up burnt out, cause outages due to simple mistakes made out of pure fatigue and eventually the resources look for new employment, leading to a retention issue.
If you value the smart and experienced operational teams you have and want to ensure a quality of life for them, we would like to suggest something: DO NOT add additional burden of migrations to your BAU team but rather look at alternatives.
Myth 4: Virtualize the storage infrastructure and we don’t need to worry about migrations.
Mention ‘Storage Virtualization’ and even a storage pro would get confused with what you really mean to say. Here’s why: storage virtualization means different things to different people, some of the products virtualize across various storage manufacturers’ products. While other vendors virtualize within their own storage hardware only. The rest of the vendors are adding virtualized storage features to their non-virtualized offerings.
For customers who are upgrading their storage hardware, storage virtualization works like a charm, once integrated, when it comes to data migrations. Although the entire process is pretty straightforward and, depending on the vendor, it may require an outage to insert or remove. The amount of data that can be stored these days can make it a very disruptive affair simply because more the data, more the time it may take to copy the larger disks over to the destination during which the source disk may be unusable.
It is also important to get to know the pitfalls of the storage virtualization technology. Yes, data migration works like a charm however, many storage virtualization products do not do a good job at migrating out of the abstraction layer and the web that they create. It is very important that the product you are going to use can efficiently handle both directions of virtualization. It is NOT always entirely clear which physical device is storing what information, as Storage Administrators are realizing it the hard way that disaster recovery in storage virtualization is a nightmare.
Myth 5: We will migrate everything ‘as is’ and restructure after we get there.
This one’s my favorite, it often comes up over and over again. Storage Administrators understand that by the time an asset comes up for tech refresh, resident data is not ideally and efficiently structured. In an ideal world, the Administrator will analyze his existing estate and structure the target environment for optimal utilization. Some factors to be considered for optimal layout are business alignment, performance characteristics or separation of users from CPUs.
More often than not, the Administrator will want to restructure as part of the move but due to lack of tools to restructure, they employ a brute force method to get data from the legacy environment to the new one. The Administrators have a false hope that they will be able to get a second or third outage from the business to restructure after the move. This rarely happens, once the migration is complete there is rarely an appetite from the business for subsequent outages. Additionally, the administrators will be bound by available capacity on the new environment as swing space required to restructure after a move is rarely factored in. So, what usually happens is a proliferation of poorly structured data containers moving over and over again. The only way to guarantee that at least your data will start out in an optimal layout is to restructure as part of the migration.
Data migration is far from being easy. It is rather complex process and requires planning, design, execution, and validation. The overall success of the process hinges on a veritable combination of rich technology tools, experienced people, and a robust process. It is imperative that the process is blended with advanced project management and coordination skills which engages all critical stakeholders such as business owners, application owners, and various infrastructure teams.
Data Migrations are best executed by teams who have lived and breathed large scale and complex migrations and are equipped with tools, processes, and the right people. In synopsis, organizations can ill-afford to underestimate the complexity and uncertainty of data migrations. Such oversights may lead to significant business interruptions, application outages, loss of reputation, and regulatory exposure.