Object Storage is the buzz of the storage industry and for good reason. Object storage encompasses file, block and object access portals into the same pool of raw object storage. As compared to traditional file and block storage, object storage offers massive petabyte scalability and built-in availability. Object storage is a distributed design model; thereby, removing the potential of a single drive failure and object storage nodes are combined to enable unlimited capacity and consistent access performance.
At Data Dynamics, we are very excited about the potential for object storage. In the recent release of StorageX 8.0, we unveiled new file-to-object conversion to support S3-compliance object storage. Based on feedback from our customers, file-to-object conversion was the number one requested new feature.
StorageX is the file management solution of choice for 6 of the 12 largest banks in the world. Large banks that actively manage petabytes of data rely on object storage for its massive scalability, availability and economies of scale. One of our customers, whose name we cannot reveal, was kind enough to share how they recently completed a 10 PB filesystem refresh using StorageX.
When measured across all our customers, the benefits delivered by StorageX are truly incredible:
- Reduce storage-related operational costs 50%
- Deploy new storage technology 66% faster
- Modernize applications for 10X productivity improvement
Data Dynamics is proud to work with all its technology partners who share our passion for object storage. We are witnessing a MAJOR shift in the storage industry and we are excited about the future potential of object storage.
If you are in the New Your City metropolitan area next week, I recommend you attend this event being held on Dec 06 by our friends at Scality. Go learn more about object storage and speak with the experts.
To gather insights on the state of DevOps, we spoke with 22 executives at 19 companies implementing DevOps for themselves and helping clients to implement a DevOps methodology. We asked, “How has DevOps changed since you began using the methodology?” Here’s what they told us:
- As I talk to customers and prospects there’s greater awareness of DevOps and what it can do. It’s being taken more seriously. There is sufficient proof of organizations doing well with DevOps. This is a business process change.
- A survey we just conducted reflected the insurgency within companies to the way things are done. One year ago, it was, “what is DevOps?” Today there’s a common understanding with a desire to know how to scale.
- CD has become mainstream. Microservices are more commonplace and are a good way to be successful with DevOps. Being in the cloud gives you more flexibility. Containers are becoming more mainstream. Function as a service is a helpful way to solve scaling issues.
- People are beginning to understand the benefits of DevOps. Best practices have been solidified. Allows you to get code from developers to customers in a fast and secure way.
In a previous article, I stated that StorageX is multi-threaded. I also spent quite a bit of time discussing why I consider this fact to be (mostly) irrelevant to the administrator who is using StorageX to perform his file system migrations. What the user of StorageX really wants is for StorageX to do its job as fast as possible: when he is doing a baseline copy, he wants StorageX to fill his network pipe and move the data as quickly as possible, and when he is cutting over to his shiny new NAS hardware, he wants StorageX to do the final incremental copy within his allotted cutover window.
As I mentioned in my previous article, the techniques StorageX uses to fill the network pipe during a baseline copy are very different from those used to find changed files as quickly as possible during an incremental copy. In this article, I will focus on baseline copies.
With the rapid rise of technology and the borderless nature of the modern digital economy, governments have had to adapt to provide better data protection and improve the fundamental rights of data subjects. On May 25, 2018, the world’s most sweeping data privacy regulation, the European Union’s General Data Protection Regulation (GDPR), will become law.
The GDPR gives EU residents the right to request from organizations whatever personal data is being stored about them and to withdraw consent of its use, thus effectively ordering its destruction. Per Article 12 of the GDPR, this request must be free of charge, easy to make, and must be fulfilled without “undue delay and at the latest within one month.”
The GDPR contains four key mandates:
- Accountability and Governance – Maintain relevant documentation on data processing activities and implement measures that demonstrate compliance, such as audits.
- Storage Limitation – Personal data may not be kept for longer than is necessary for the purposes for which it was originally obtained.
- Breach Notification – A notifiable breach must be reported to the relevant supervisory authority within 72 hours of the organization becoming aware of it.
- Individual Rights – An individual may request the deletion or removal of personal data when there is no compelling reason for its continued existence.
GDPR aims to encourage organizations to be more accountable, transparent and responsible for any personal data they hold. Any entity that stores or processes the personal data of EU residents will be obligated to conform to this new law, regardless of where that organization resides. Further, it empowers EU residents to control the data that an organization may hold on them.
Implications for File Management
GDPR demands improved data governance for files that contain personal information of a customer or employee. File shares may contain millions of files widely distributed across incompatible storage resources making it a challenge to comply with GDPR rules. A file management solution that can work across heterogeneous storage resources and provide the ability to analyze, move and manage files for GDPR compliance is a necessity.
- StorageX Dynamic File Management platform empowers you to analyze, move and manage and your files for GDPR compliance. StorageX is built using industry standards and operates seamlessly across heterogeneous storage resources, freeing your data from technology lock-in, complexity and risk.
- Using StorageX’s integrated analytics, you can quickly analyze files based on file name, type, size, location, creation, last access, attributes, SID and more. Files that contain personal data can be marked with custom tags so they can be easily managed in the future.
- When action is required (move, copy, delete), StorageX’s automated data movement policies facilitate the transfer of SMB/NFS source files to file resources more suitable for GDPR management. Move entire shares or exports to a new location with speed and reliability. StorageX reports record all file actions to document compliance for audits.
To learn how StorageX can help your organization manage its data for GDPR compliance, contact Data Dynamics Sales.
Are you facing a large petabyte scale migration? A job this large requires the right solution. Read one customer’s own story how he used StorageX to successfully manage a ten-petabyte migration. With today’s constant need for maximizing efficiency, while constantly driving costs down, the StorageX story related here is a clear example of how we’re changing the landscape of managing your data.
- Eliminate origin storage issues
- Data growth
- Traditional tools would take 5x the time
- Scale of weekly cutovers (100s)
- CIFS & NFS mixed environment
- Maintain security
- Business Unit buy in
- CIFS and Mixed Mode shares
- 4,000-5,000 total shares
- Up to 40 TB per share
- Up to 180 million files per share
- 10 years of work done in 2 years
- 10 PBs optimized
- 50% cost savings realized
Key StorageX Features
- Phased migration: using GUI interface it was easy to configure migration policies (baseline, incremental and final copy). Reduced downtime window required to migrate the NAS data.
- Security permissions and file attributes: StorageX copied security permissions (in Windows) and mode bits and file attributes (in Unix). No manual intervention or correction was needed.
- Migration summary: detailed summary report of the migration with migration start, end time stamps, bytes copied, number files and folders copied, skipped or deleted files and error logs.
With data growth, equipment end-of-life (EOL), and increasing support costs it is very crucial for organizations to consolidate and eliminate older and more expensive storage systems. The cost of such data migration is significant and migration can present substantial business and technical challenges.
A variety of tools are required to move data between arrays from different vendors or to newer equipment. Traditional migration solutions do little to address these challenges. Host-based tools can reduce the performance of other business applications by consuming CPU cycles and I/O bandwidth. Array-based solutions either do not support heterogeneous storage environments, or offer a one-way transfer that locks customers into a single-vendor solution. Appliance-based solutions require a service technician to enter the data center, install the appliance in the data path between the host and the storage device, and then remove the appliance after performing the migration. The result is an expensive, intrusive approach that is not readily scalable and requires downtime before and after the migration. To efficiently meet ongoing migration needs, storage administrators need a simpler, less disruptive, and more cost- effective way to migrate data between heterogeneous storage arrays.
StorageX File Management
One such solution for file based data management and migration is Data Dynamics StorageX. StorageX is a software based solution which facilitates file based storage management and migration. StorageX distinguishes itself by being a very simple, yet powerful GUI based tool which simplifies the migration, consolidation, and archiving of file data in large, complex, heterogeneous file storage environments. StorageX is fully automated and has a policy driven approach to data migration and management.
StorageX supports various NAS devices (EMC VNX/VNX OE for File and Isilon/OneFS, NetApp/Data ONTAP 7-Mode and Cluster Mode, Windows, and Linux) as both sources and destinations, as well as stand-alone CIFS and NFS file storage resources. StorageX software can be installed on virtual or physical servers. Installation is a simple and straightforward process. We migrated more than 3,000 NAS between variety of source and destinations storage systems/servers. Some of the major benefits of StorageX are explained below.
Benefits of StorageX:
- GUI interface:
The graphical user interface makes it very easy to handle bulk NAS migrations. With the GUI, we can add storage resources (source and destination) and create data movement policies for NFS/CIFS share by specifying the replication and migration options.
- Migration Policies:
Migration policies define the data movement between source and destinations. For CIFS shared folders and NFS exports, there are several configuration options available. Carefully input the data in the migration template as it is the base for deciding the migration options. Options are available to copy directories only, delete orphaned file/folder on destination, copy security settings, choose file attributes, filter files by age, exclude files/folders, etc. After the initial baseline data copy phase, continuous incremental copies ensure replication of new, locked, or recently modified files on the source to the destination.In the final cut-over phase, StorageX options allow you to remove user access to the source, perform a short, final sync to copy any new files recently added or updated, and then share the new destination with users.This phased migration (baseline, incremental and final copy) approach reduces the overall downtime window required to migrate the NAS data.
- Migration Schedule:
Once the baseline copy is completed, subsequent incremental copies can be automated through migration schedules. Migrations can be scheduled every minute, hourly, daily, weekly, etc.
- Automated Emails:
Email notifications can be configured for effective monitoring. We did not need to login to the tool to check the migration status, emails alerted us under different conditions (completed successfully, completed with errors/warnings or cancelled).
- Migration Summary Report:
Summary reports give a detailed status of the migration with migration start, end time stamps, bytes copied, number files and folder copied. If the migration is cancelled with error or if it is unsuccessful, the error logs clearly indicate the cause so we can troubleshoot to fix the issue and retry the data copy. If certain files are skipped or deleted, that is also indicated in the summary.
- Security Permissions and File Attributes:
A major challenge in NAS data migration is copying the security permission (in Windows) and mode bits and file attributes (in Unix). StorageX is able to copy security permissions and file attributes over smoothly from source to destination based on the options specified in the migration template. No manual intervention or correction is needed after the final data copy.
Have you been putting off the need to update or replace thousands of aging NAS volumes containing millions of old files?
If so, you need StorageX.
StorageX is the ONLY file management solution proven capable of managing massive petabyte migrations. It is not wonder that six of twelve world’s largest banks rely on StorageX.
Click Here for a personal demonstration of the power of StorageX. Take back control of Your Data!
Watch this six minute video to learn about StorageX 8.0, the latest release of the industry leading file management solution. Learn about the new file to object conversion feature and more.
If you have been around the computer industry long enough, you have witnessed multiple major shifts in technology. If you were working in the 1970’s you remember the major shift from mainframe computing to workstations. In the 1980’s, we witnessed the shift from proprietary workstations to personal computers. Most recently, we saw the major shift from personal computing to the smart phone. In each example, the incumbent technology never disappeared, but its growth slowed considerably in favor of new technology that pulled the entire market in a complete new direction.
A similar technology shift took place in 1992 when NetApp, Inc. was founded. NetApp pioneered a new enterprise storage technology called Network Attached Storage (NAS). Compared to traditional direct attached storage (DAS) and storage area network (SAN) storage, NAS storage offered access to large amounts of storage in an easy to manage appliance. NAS was enabled by improved network technology and new high capacity disk drives. NetApp combined the new technology in such a way to revolutionize the enterprise storage market.
It has been 25 years since its introduction and NAS remains a highly viable enterprise storage solution. But, 25 years is a long time for any technology and NAS is beginning to show its age. Major changes are taking place in the computer market that are directly impacting NAS. The first is the exponential growth in unstructured data. In each era, data use changed dramatically. In the mainframe era, data belonged solely to the System Admin and it consisted of structured database information. In the Personal Computer era, control of data moved to the individual and unstructured files proliferated in the form of word documents, spreadsheets and graphics. Most recently in the current era of smart phones and Internet of Things (IoT), data growth has exploded in the form of multi-media files and device/machine data.
Click Here to read the rest of this new white paper.
The following interview took place with Piyush Mehta, CEO of Data Dynamics Inc. Read to learn Piyush’s thoughts and predictions regarding data management and the impact it will have on the digital economy.
Q: As you look back on 2017, what were the key challenges organizations faced as they worked to move towards the requirements of the new digital economy?
The digital economy brings with it some great opportunities but the transformation, like any other, is painful. Most organizations want to leverage the digital world to gather data points to make better business decisions and customize products/offerings to as granular a level as feasible. The challenge in 2017 was the realization that unstructured data was dumped into storage silos without any thought or plans of extracting it in the future for value. Those mounds of legacy data today have become an important asset and as such, understanding it becomes a vital first step in the journey. In addition, applications are being re-written to meet the needs of a digitally paced economy but rewriting applications takes time, how do you leverage existing apps to access and manage new storage paradigms that scale to meet the needs of a digital world? This is a repeated challenge that we have seen enterprises face in 2017 and are still trying to address.
Q: Do you think these challenges will remain as we move into 2018? What new challenges will likely emerge?
The challenges from 2017 will remain as transformations take years not months. That being said, companies are adapting and innovating faster than ever and over the next 2-3 years will address the challenge of managing legacy data. As enterprises meet those challenges, new challenges will continue to evolve, among them are:
- Sheer amount of data creation and retention. At what stage does historical data not provide any value and can be deleted? Statutory requirements have no bearing on this question because every enterprise goes far beyond the legal requirement to store data as there is a ‘you never know when you may need it’ question. But with data growing at exponential rates, there will need to be an answer to when the value of the data dissipates and its best to delete.
- Security of the data and who has access. As data volumes grow, so too does the vulnerability to securing it. Managing access control to data will be one of the top challenges facing CEO’s as an enterprise’s greatest asset is its data.
Q: We are hearing more and more about “smart” and/or “intelligent” IT (less and less about speeds and feeds) – how will we see this concept playing a role in 2018 and onwards?
There’s more buzz then bite to this. You can only generate intelligence if you understand what you have, otherwise how you can extract the value from it? Most companies just don’t know the data they have, let alone leverage it to its max. I believe today we leverage 10-15% of the value that data can provide in helping make better and more intelligent decisions as many of the tasks that support and surround data management are still done in an archaic manner. Until we drive a consistent, automated and policy driven method to manage data so we may find it when and where we need it, these words will continue to be strong marketing jargon but slow to adoption.
Q: Can you talk about Data Dynamics and its StorageX software – what enhancements have been recently added to meet the requirements of 2018’s digital business environment?
It is an exciting time for us at Data Dynamics as we just launched StorageX 8.0. Our 8.0 release enables customers to accelerate their digital transformation journey. There is a vast array of new feature/functionality that we’ve added to the product based on years of experience working with some of the largest enterprises in the world. Our customers wanted a better understanding of what they have stored over the years so we added an Analytics module to empower them to know what they have, how it is being stored, who is access it, etc. This enables them to utilize raw meta data into actionable information! Our support for S3-based Object storage, both on and off-prem, provides the flexibility and versatility required to move data across a hybrid cloud, meeting both the need for scale and agility. StorageX 8.0 empowers dev ops as we have a robust Restful API to support applications to automate creation, management, movement and reporting of their storage infrastructure. These functionalities are further empowered because StorageX 8.0 is the first platform that helps manage unstructured data in a native format so that users and applications can have end to end visibility into where and how the physical data is stored. This is an essential differentiator to storage management software developed by other ISV’s who ‘box in’ customers into their proprietary name space or shard the data as they transform it. We will continue to innovate to meet the needs of our customers and we have some great plans ahead for StorageX 8.1 and beyond.
Q: Any parting advice you wish to offer to business and/or IT leaders?
I continue to see enterprises set ambitious goals to transform their infrastructure to meet the needs of a digital economy. Unfortunately, most of these goals are ‘pie in the sky’ dreams as executing them in complex, siloed environments is not an easy task. Start the journey for your transformation with achievable baby steps, set actions and plans that can actually be accomplished and realize that the transformation is not a sprint but a marathon and the digital age is just starting. Understand what you have before even embarking on what you want to accomplish. Find technologies that you can leverage to drive automation into the process, is scalable and can grow with you as you expand your executable goals.
As with the industrial revolution in the late 1800s, the digital revolution is in its infancy, the world as we know it will change with the evolution and advancement of quantum computing and machine learning. This will impact every aspect of our lives and change the course of humanity. It’s a great time to be in technology as each organization can have an impact in this transformational era but must do so understanding operational limitations that exist within their business and adapt and innovate accordingly.
Click Here to IT Briefcase Article
On several occasions, I have been asked a question that is straightforward to answer, but where I’m left with an uneasy feeling about why the question was being asked in the first place. For example, about a year ago I was asked, “Is StorageX multi-threaded?”
That’s a seemingly reasonable question, and a correct answer is easy to give: yes.
I suspect the person who asked the question would have been happy with that basic answer. He would have understood it (“yep, they have multiple threads in their program…that’s good, right?”) and he could easily convey the answer to whoever asked it of him. Most likely the question originated with a piece of marketing collateral for a competing product, touting how the product is “multi-threaded so that it scales to the available hardware” or some such seemingly wonderful claim.