StorageX 7.0, which our team released in August 2013, included a new StorageX feature — Migration Projects.
This new feature has really caught the eye of our customers and generated lots of good questions. As one of the senior software developers here at Data Dynamics who was heavily involved in the design and creation of this new feature, I thought it might be helpful to share with some of our StorageX customers, both old and new, how this feature came about.
Origin of “Migration Projects” idea/concept
As many of our customers know, Data Dynamics purchased StorageX from Brocade in 2012. During this acquisition, several folks started to take a look at StorageX with new eyes, including Piyush Mehta, our CEO.
Piyush has worked with enterprise storage teams for many years, and has a long-term vision of simplifying enterprise data center migration projects and workflows. As Piyush talked with current, past, and potentially future StorageX customers in his new role as Data Dynamics CEO, he heard a lot of good things about the Phased Migration capabilities StorageX already provides. As most of our existing StorageX customers know, StorageX Phased Migration policies allow storage administrators to move data stored in CIFS shared folders and NFS exports from sources to destinations using an automated, policy-based approach. Phased Migration policies also allow storage administrators to safely preserve or adjust security permissions and file attributes as needed during the migration.
However, Piyush also heard from several of our StorageX customers that there was a “bigger picture” need they had when doing migrations at the device level. Storage teams told Piyush that they were drowning in data migrations due to recent technology and storage platform changes from vendors, and struggling to get their data migrated as a part of their quarterly tech refresh cycles. In talking with them, Piyush saw they they really needed a way to further automate and streamline their migration projects.
In these “bigger picture” device-level migrations, which were typically driven by tech refresh projects, storage teams had many different tasks they needed to perform before they were ready to actually migrate their data. For example, their task list typically included the following items:
- Discovering and analyzing what data was stored on their existing sources and destinations (which was particularly challenging when migrating between different vendor platforms)
- Designing their migration, which typically included the need to model and analyze several different designs before deciding on a final design
- Provisioning items on source destinations, including the following items:
- Volumes, qtrees, CIFS shared folders, and NFS exports when migrating to NetApp Data ONTAP destinations
- File Systems, tree quotas, CIFS shared folders, and NFS exports when migrating to EMC VNX OE for File destinations
- Folders, quotas, CIFS shared folders, and NFS exports when migrating to EMC Isilon OneFS destinations
- Validating, or verifying, prior to migration that destination items have appropriate space, no volume/share/export name conflicts exists, etc.
Only after performing all of these tasks were they really ready to use StorageX to create Phased Migration policies and begin migrating their file data from sources to destinations.
From what Piyush heard, StorageX was great at migrating data from sources to destinations. The phased approach used by Phased Migration policies helped maximize user access to data, and also gave teams the time they needed to validate the migration was successful and had proceeded to plan prior to cutover. However, StorageX didn’t provide any kind of start-to-finish workflow for migrations, which meant they still had to perform hundreds of manual steps related to discovery, analysis, design, design validation, and policy creation before they could start using StorageX policies to actually move their data.
Building the requirements for “Migration Projects”
As Piyush shared this information with our dev team, we all began looking at StorageX with a fresh set of eyes. We saw that StorageX Phased Migration policies could handle migrating data from sources to destinations extremely well. However, Piyush and the rest of our team also saw how we could add some additional capabilities to StorageX that could really reduce the number of manual tasks storage teams had to perform when migrating file data as a part of a tech refresh project.
Starting in October and November 2012, we decided to focus some of our development cycles on adding functionality into StorageX 7.0 that provided a migration project-focused workflow. Our goals was to create a workflow that migration teams could use to help them streamline their migration projects and reduce the amount of time it took them to complete migrations during tech refresh cycles. As a part of this work, we started adding in some additional capabilities into StorageX that supported the following “typical” migration project lifecycle our customers told us that they usually followed:
- Identification of sources and destinations to include in projects
- Discovery and analysis of existing source and target destination environments
- Modeling and validation of different migration scenarios
- Provisioning destinations
- Creation of Phased Migration policies to run on a schedule or on demand to migrate file data from sources to destinations
- Actual movement of file data from sources to destinations using policies
- Validation of file data migration success to the destination locations
- Cutover of users to the data stored in the new locations
Based on these ideas from customers and using the requirements set forth by Piyush, the idea of a StorageX “Migration Projects” feature was born.
Coding “Migration Projects” — Jet Fuel, Jet Fuel, and more Jet Fuel
Over the next several months, our development team spent many, many hours — fueled by many, many cups of Jet Fuel K-Cup coffee — designing, coding, and testing the new Migration Projects feature that is now available in StorageX 7.0.
Our team uses an Agile software development approach, so we all dug in and began defining our sprints and user stories and working on the core functionality of the Migration Projects feature. While we can’t cover all the bits and pieces of the new feature (you can learn more about the nitty-gritty details of Migration Projects in our product documentation or online help, or by personally delivering a six pack of a desirable cold beverage loved by many of our developers to the Houston office on a Friday afternoon), some of the things we added into StorageX to help storage teams migrate their data more quickly and effectively included:
- Support for the latest NetApp and EMC platforms – One of the first decisions we had to make early on was which platforms to suport. Per feedback from StorageX customers, the biggest demand was for migration project support for the latest NetApp and EMC platforms, so we decided we would focus on supporting NetApp Data ONTAP, EMC VNX OE for File, and Isilon OneFS devices right out of the gate. We knew we could always go back and add support for additional platforms in future versions based on customer demand.
- Bulk import – Migration teams are always under the gun and have to deal with data stored on lots of different devices, so we knew we needed to provide bulk storage resource import functionality. Migration teams need this in order to quickly import lists of the sources and destinations they want to use in a project from a .csv file. Our customers told us this capability is a huge time saver when they have five, ten, or 50 resources in a NAS estate that they need to manage as a part of one or more migration projects.
- Source and destination mappings – We wanted to give teams an easy way to specify what sources and destinations to include in a project, and to be able to reconfigure and add and remove sources and destinations easily from projects when needed.
- Migration Project views – This was one of the trickier parts of the new Migration Projects feature, and one piece that we spent a lot of time on. We wanted to give migration teams a way to discover the properties of all of the resources included in a project, including volumes, qtrees and tree quotas, CIFS shared folders, and NFS exports.
The Migration Project views we designed were created specifically to help migration teams discover and understand their environments without having to type in commands and use shell scripts to collect all of this information and then dump it into huge Excel spreadsheets.
We heard from a lot of our customers that doing this kind of discovery work was one of the most painful aspects of trying to get a migration project going, as they had to write and run a lot of customized scripts, which could be a time-consuming and error prone process, or they had to pull in vendor pro services teams to help them with this work, especially if they were moving between storage platforms as part of a migration or tech refresh. Migration teams also told us that they needed to be able to see aggregate information as well as filter on specific pieces of information, so we designed Migration Project views so migration teams can see can both aggregated and filtered information across NetApp Data ONTAP, EMC VNX OE for File, and EMC Isilon OneFS file storage resources.
- Project Design – We heard from many storage teams that there were lots of different ways to approach any given migration, so they spent a lot of time scratching out source and destination mappings and doing a lot of modeling, such as “if I put these volumes over here, what would this look like, and if I moved them over here instead, what would that look like…”. So in Migration Projects, we provided a way for teams to create, view, and model multiple designs for each project right from within StorageX.
- Design Validation – We also heard from many storage teams that not only was designing the migration critical, but also very critical was the idea of design validation, or being able to see if their design was going to succeed or fail in their “real world” environment. Validation was particularly important for migration teams migrating data between vendor platforms, as different storage vendors like NetApp and EMC do not always use the exact same approach or settings. Based on this feedback, we developed a validation rules engine in StorageX Migration Projects that runs behind the scenes and then provides feedback to the migration team on whether their design will succeed or fail in their current environment. Below are just a sampling of a few of the validation checks we perform:
- Warning if the volume size you want to provision on the destination can potentially exceed the available size of the aggregate
- Warning if the source volume is a thin provisioning volume but you are creating a thick provisioning volume on the destination
- Error if the volume name, CIFS share name, or NFS export name you want to provision on the destination is already in use on the destination
- Error if you are copying a SnapLock volume to a destination where SnapLock is not enabled (example of a platform-specific error)
Some of the errors that StorageX detects during validation can be fixed right from within the StorageX user interface. For example, you can typically address naming conflicts right from within the StorageX Console. However, for others you may have to connect directly to the resource using native tools. For example, if you need to add disks to a Data ONTAP aggregate, you must use a Data ONTAP tool such as NetApp OnCommand System Manager. After you add the new disks, you can re-run the design validation again to confirm that you successfully addressed the issue.
- Provisioning – Our customers using NetApp Data ONTAP and EMC VNX and Isilon storage devices told us that it would really streamline their migration time and costs if we could provide a simple user interface for provisioning destinations across the different platforms, so we built in to Migration Projects the ability to automatically provision destination items such as volumes, file systems, folders, quotas, shares, and exports on destinations when you execute a validate migration project design. This saves migration teams from having to perform the literally hundreds of individual manual provisioning steps they typically have to complete in order to get a destination ready for new file data.
- Automated Policy Creation – Our existing StorageX customers were always able to use templates and wizards to create policies to migrate file data. However, we wanted to further streamline and automate migration project workflows for our customers, so we also gave StorageX the ability to automatically create the Phased Migration policies migration teams need to actually move the file data. For example, in the past, if customers had 200 shares or exports they needed to migrate, migration teams could use StorageX to create these policies. However, each one was typically created one-by-one, and StorageX didn’t provide any validation capabilities, such as checking for CIFS shared folder or NFS export name conflicts on the destination. There also wasn’t an automated way to check and see if someone on the team had accidentally left out or missed a source/destination mapping, which could sometimes happen when teams manually created Phased Migration policies from a spreadsheet with 200+ source/destination mappings. Now with Migration Projects, when you execute a validated migration project design, StorageX automatically creates policies for you, helping teams ensure that they don’t accidentally miss a share or leave something else out.
Give Migration Projects a spin, let us know how we’re doing
Now that the new version of StorageX has been out and shipping for a few weeks, its been fun to see how our StorageX customers are starting to explore and use this new feature. We’ve also started gathering some feedback from our “early adopters” on other capabilities they would like to see added to Migration Projects.
If you haven’t had a chance to check out the new Migration Projects feature now available in StorageX 7.0, request a demo or request a trial of the latest version of StorageX and give it a try — we’d love to get your feedback.