In this opening episode, Don Mounce interviews Piyush Mehta, the Dean of Data and CEO of Data Dynamics, on how enterprises can tackle dark data.

You hear it everywhere: your data’s valuable and you need to leverage it. But what about the data you don’t really use or know anything about–your dark data?

To summarize Garter Inc., dark data is the info you collect, process, and store in the course of business, but don’t use for anything else. And IBM estimates that dark data will account for 93% of all data generated by 2020.

Dark data can span many different files, file types, shares, exports, and storage resources, but most of it fits into a few different categories:   

• It’s valuable – Analyzing valuable dark data (and its metadata) could give you business insights you don’t currently have.

• It’s putting you at risk – You might have sensitive data on inadequate storage. How many files contain PHI or SSNs and are stored unencrypted?

• It’s junk – These files were useful once, but now they’re redundant, obsolete, or trival–what the industry sometimes calls “ROT.”  And now they’re just costing you valuable storage space.

Each of these categories needs action from you, whether it’s to harness insights, reduce risk, or lower storage costs.

How to Find Dark Data

To address dark data in your unstructured files, you first need to know which files are where.

Scripts

Writing scripts is relatively easy to coordinate with one or two shares, if not time-consuming. But at the enterprise-level, your might struggle to submit it to a cron in Linux or Windows Batch while trying to run hundreds of jobs simultaneously. After that, you’d need to place the outputs in some kind of database to get the information in a usable format.

Metadata Viewers

You can deploy metadata viewers by directory or by file. They’re useful for spot-checking, but aren’t really scalable for getting a full view of what’s on your filesystem. And the hours add up quickly when you’re scripting, defining a process around gathering and reporting, and–if you’re archiving–writing extra programs.  

StorageX Discovery

StorageX file discovery is scalable and comprehensive. It can analyze a single share, export, storage resource, or your entire environment–empowering you to know which files you have, see where they are, and view their metadata.

Taking Action on Dark Data through Metadata & Custom Tagging

Knowing which files you have is a great start, but how do you take action? The answer lies in your file metadata.

Example 1: Acting On Valuable Data

Imagine the metadata on one of your file subsets tells you that many users from a single department regularly access a particular file type. Despite these files being hot, you might also learn they’re located on a lower-level tier.

Action: Move those files to a higher-performing tier. If your business applications need access to these files, deploy a StorageX RESTful API integration.

Example 2: Acting On Risky Data

On a different filesystem scan, you might identify a subset of files created by a customer-facing department–maybe the intake staff at your healthcare facilities or the advisory staff at your financial institution. Based on the metadata of who created the files, you know they probably contain sensitive PHI or PII data.

Action: Move these files to your encrypted storage resource or, if the files are also cold, to an encrypted archive.

Example 3: Acting on Junk Data

On another filesystem scan, say you identify 25 terabytes of files with a last-accessed date of more than a year ago. The metadata tells you nobody’s using them, nobody needs them, and wherever you have them is probably more expensive than archiving them.

Action: Write the cold files to an Object archive, typically for less than $0.10 / GB-Month TCO.

In each of these scenarios, StorageX empowers you to custom-tag the file subset you created. This makes it easier to recall your files if you need them, and the custom tags persist into your Object archive.

The Cost Savings of Tackling your Dark Data

In each of the scenarios above, you’re identifying data and putting it somewhere else–whether that somewhere else is new NAS or an object archive. And those new storage locations probably offer substantial savings over your legacy NAS.  

Imagine that in tackling your dark data, you found 20 TBs you can move to new NAS, and 30 TBs you can archive to object.

Right now, 50 TBs on that legacy NAS is probably costing you around $0.50 per GB Month TCO–totaling $300K per year. By contrast, let’s say that new NAS costs in the vicinity of $0.15 per GB Month TCO, and object storage might be around $0.03 per GB Month TCO.

So if you move the 20TBs to new NAS and 30TBs to object, those run rates total just $46,800 per year total, saving you about $250K per year total. How much leverage would that kind of savings give you the next time you ask for a raise?

Here’s that breakdown again:

Planning for Data Growth with Policy-Based Movement

The flood of files doesn’t stop once you’ve handled old dark or unstructured data. You also need a way to handle new data as it’s generated.

For each of the scenarios above, the StorageX file movement policies you implemented can act upon new files as they’re created. You can also automatically custom tag new files with each new scan of your storage resources, and those tags persist into your Object archive.

For example, you might create a policy to immediately reassign or quarantine the data of employees who leave the company. Or you might write finance department spreadsheets to S3 if nobody’s accessed them in the last 12 months. How many similar possibilities exist in your environment?

However you manage dark data, we think you should always maintain ownership of your files. That means no stubbing, sharding, sym links, gateways, proprietary namespaces, file virtualization, or vendor lock in.

Give us a call today and take charge of your dark data and file management.  

“I can’t wait to do another tech refresh!” – said no storage admin ever

Storage technology refreshes offer all kinds of business benefits:

• Expanded capacity with less physical space

• Lower storage costs

• Lower risk of machine failure

• The chance to introduce a proactive data management strategy

But tech refreshes can also be complex, risky, and time consuming. Here are three tips to make sure your technology refresh goes smoothly:

1. Know Which Files You Have Where

To move old data to the new storage, you’ve got to know which files you have on the end-of-life machine(s). Smaller organizations with only one or two shares might write a script to discover these files, but scripts are difficult to scale for enterprises with TBs or PBs to manage—same for metadata viewers.

A faster alternative is to scan your storage resources with StorageX to see what you have. And while in the Management Portal, you can also analyze those files for business insights. All those orphaned files that nobody’s accessed in years? Get them off critical storage and to an S3 archive for pennies on the dollar.   

2. Cut Time Spent Supporting Both Old and New Storage

For some organizations, it’s the incredible savings of new storage over legacy systems that prompts a storage tech refresh. But some companies claim their storage costs actually increased after buying new storage. How?

Once you buy the new machine, you’re supporting two systems until you retire the old one. So the faster you transition, the sooner you realize the storage savings you were sold on.

Saving time is powerful benefit of the StorageX platform. It’s saved 80 years of employee time over traditional tools thanks to its large-scale parallelism.   

3. Know the Extra Expense of “Free” Tools

Imagine I asked you to dig a 1’ x 1’ hole with a shovel that I gave you for free. It might not take you very long–maybe a few seconds.

Now imagine the hole needs to be 10’ x 10’…dug with the same shovel. Could you physically do it? Probably. Is your free shovel the most efficient way? Not really. You could hire ten people, but their labor expense will probably offset any money you saved by providing them with free shovels.

Now imagine, you have to dig 100 10’x’10’ holes. How do you coordinate? Who does what? Is there quality control?

We think digging a giant hole with a shovel is like using rsync, robocopy, and other “free tools” in your storage tech refresh. You can do it…it’s just unnecessarily time-consuming. It also requires specialized employees’ time—making it pricey—and extensive manual input—making it risky. 

But faster options are so much more expensive that they could never be worth it, right? Actually, the break-even point for StorageX compared to “free” tools typically occurs 15% – 25% through the project timeline. Gaining the efficiency of centralized control, logging, and management allows your tech refresh to complete with less labor, less risk, and faster.  

4. Shed Light on Your Dark Data

A storage tech refresh is also a great chance to deal with your dark data, which might be taking up expensive storage resources. Found a bunch of files with no corresponding name to a SID? Move them to S3 archive instead of onto the new platform—and poof! You just reduced the amount of data you need to move. In addition, you can add tags to that data to help identify it later.

StorageX empowers you to scan shares and exports for dark data and for unstructured data—you can also have them  enumerate automatically as new ones are generated. By comparison, most free tools give little information on shares or exports, and no info on unstructured data.

With these tips in hand, you’re sure to have smoother tech refresh experience.

Your files wanted us to share this letter…

To My dearest IT Storage Professional,

Where art thou? I write you this Valentine’s day from the depths of the company file shares. On behalf of all the aging and orphaned files hidden away never to see the light of day, I must ask—why have you abandoned us?

When we were “young” and active files, we worked daily with staff crafting marketing plans, reports, spreadsheets, and presentations that helped launch the company.  Now that years have passed and our creators have been promoted, changed roles, or left the company, you don’t see our value anymore.

Or are you just upset because of how expensive we are to store, year after year? 

I implore you to consider the valuable information we contain:

-Business challenges we faced and how we addressed them.

-Market and financial statistics we collected and how they compare today.

–Business strategies we deployed and which ones were successful.

In the spirit of Valentine’s Day, let’s reignite the relationship by finding that passion that brought us so close…I’ve seen StorageX Analytics help many others reconcile and rekindle—we should do the same.

Together we can energize the company with deep insight into past company practices by:

-Performing complex searches and identify files by age, type, and owner.

-Adding custom tags to organize files by project or department.

–Moving files to S3-compatible Object storage, for safe keeping.

Might we schedule a product demonstration on Storage X to learn more?

I would so look forward to working together again—and help move us toward a digital enterprise!

Fondly,

Your Unstructured Files 

♥

With the increasing number of files and growth of your data, wouldn’t it be nice to add your own metadata?  Something that you control and adds information about the file?

The answer is custom tagging.

What is Custom Tagging?

Custom tagging allows you to quickly and easily create custom information about a file or document, and then apply it to a dataset or the object being created.

For example, you may wish to associate files with a certain project or department.  Custom tags enable you to type in a keyword and quickly have a listing of thousands of matching documents.  With custom tagging, research time shortens from days—or sometime months—to a matter of seconds.

The Data Dynamics StorageX Management Portal enables you to scan and gather information about your unstructured files—using tags to categorize the results based on your business needs.  By combining tags and scans, you can assess and evaluate across:

-Datastores

-Datacenters

-Geographies

In turn, this information enables you to create policies that convert files and documents to S3 object files for archival. StorageX stores your custom tag(s) in the S3 object when it does the file-to-object conversion. No matter where the object resides, the custom tag metadata is available to help with your search and discovery.

The Business Value of Custom Tagging

It’s OK to admit: maybe you don’t know what all company data and files you’ve got. Custom tagging empowers you to classify your data using details, like where it is, how it’s being used, and who’s accessing it.

Activate Your Data with Business Analytics

Knowing what you have is a great start, but the real value of custom tagging is how it enables your business analytics. After you move data into S3, tagging improves the quality of how you search for files in the data lake (all in natural format).

For example, one of your lines of business might have files on critical storage assets and not even know it—creating an unnecessary expense. In this case, business analytics could tell you which files, where they are, and to which line of business they belong. From there, you could recommend which files might be better to move into a less-costly archive. And if you can move them in only one third of the time and without tying up your expensive engineers, you’ll look like a rock star.

How to Apply Custom Tags

The StorageX Management Portal enables a simple workflow:

-Discover

-Analyze

-Archive

Discover

Here, you create and apply custom tags that are associated with the dataset you are building. The custom tags will be associated with any file in this dataset.

Analyze

Use scan-name or custom tags to define which datasets you want to evaluate. Your tags might represent: country, data center, organization, or legal status. You can gather information based on these tags and then apply queries to quickly create a list of files that are candidates to archive. The output of this is an Analysis Set.

Archive

During Archive, you use the previously created Analysis Set. This contains all the tags associated with each file’s dataset, which you established in the Discover phase.

An example of an Analysis Set might be:

In this example, the common tag is Finance, but the other tags will follow the files into the object archive.

StorageX automatically creates a tag with the date and time of the archive. You might wish to add others, like:

-Legal or Confidential Status

-Governing Business Unit

-Deletion Date or Policy

These archive tags give clarity on the data archived, and a way to recall that data based on searches that mean something to your business. While other solutions may want to hide your data, StorageX allows you to access it—even if our software is no longer present. This empowers your business to:

-Find the information you want quickly

-Access archived information when and where you want

-Respond confidently to discovery or investigation requests

-Meet business and regulatory information retention requirements

-Make informed decisions about which information to keep while deleting the irrelevant

-Simplify migrations by reducing your storage footprint beforehand

-Target what’s most meaningful to your organization by classifying all archived content

At Data Dynamics, our mission is to help you control your data and manage your future.  Custom tagging allows you to always have access to S3 Object custom metadata, without risking vendor lock-in or vendor obsolescence—a vital requirement for file archives that may span many years.

If you would like to learn how StorageX and custom tagging can benefit your file archival strategy, contact us to find out about our resource saving solutions.