Hi – for those of you just tuning in, this is part four of a 9-part blog series in which I am describing the Zetta solution, and how it is built from the ground up to host primary, unstructured enterprise data in the Cloud. Here again is the list of requirements; the first two have already been addressed, along with an introduction to the series:

• Accessed like traditional storage
• Easy to use, enterprise features
• Comprehensive data integrity/protection
• Data security/privacy
• Continuous availability
• Non-blocking performance
• Administrative transparency and control
• Good investment value

In this post I’m going to discuss the position that in order for a cloud storage service to be a viable option to host primary enterprise data sets, it must provide comprehensive data integrity/protection.

At Zetta, our most important design consideration was data integrity (or data “protection” – whatever term is used, this is the idea that we won’t allow data to be lost or corrupted), since ultimately a data storage solution is worthless if it allows the loss or corruption of data.

A dirty little secret in the storage community is that data corruption happens all the time – though the relative rate of corruption seems low on its face, the increasing scale of data stored guarantees that corruption events are always occurring. For more on this topic at a deeper technical level, along with a calculator to help you gauge your own data integrity risk, please see JW’s post on Calculating Mean Time To Data Loss (and probability of silent data corruption).

So any solution for storing primary enterprise data MUST assume data corruption will happen, and must be designed to adapt to that reality and repair corrupted data, thereby guaranteeing data integrity.

Here are some of the unique ways the Zetta solution has been designed to automatically detect data corruption and repair it; taken collectively, these tools truly give Zetta an unparalleled data integrity profile.

• Write Receipts — Zetta creates a strong SHA-1 hash of every file that enters a Zetta customer virtual volume, and we do two things with that hash (one of which is optionally available at the customer’s request, one mandatory though transparent to the customer).
  • First, at a customer’s request, we can place these hashes on the customer’s volume, allowing a customer to ensure that what we have stored at Zetta is what was sent by the customer.
  • Second, we store each hash in perpetuity. This allows us to compare a read file with the one that was originally received; if there is any difference, we repair the file before completing the read, guaranteeing that what is read is identical to what was written.

     

• RAIN6 N+3 — Zetta employs a best-in-class RAID algorithm. It is based on RAID 6 (based on Reed Solomon encoding), and adds an additional parity node (RAID 6 traditionally has 2 parity nodes, the Zetta solution has 3 – this is laid out in great detail by JW in his post on Data Integrity in the Cloud). We also refer to it as “RAIN” because we stripe data not just across independent disks, but actually across independent nodes (i.e. storage servers). This level of redundant protection is not available even in traditional storage hardware from top vendors, ensuring integrity (and availability) of data in the event of up to three independent computer failures.

   

• Proactive Error Correction — In addition to creating a SHA hash of every complete file that enters the Zetta storage cloud, the Zetta solution also creates a SHA hash of every “chunk” of data encoded and striped across the disks in our lower-level storage servers. Then, using any spare system processing cycles, a background process on the system traverses all hard drives and compares those stored hashes to the current chunk on disk, proactively detecting and repairing any data corruptions on disk using our triple redundancy RAIN6 encoding.

   

• Snapshots — Zetta cloud storage comes with a full-featured file system (a distributed, clustered, highly parallelized file system that we’ll be discussing in a future post). As with most file systems, the Zetta file system provides full snapshotting capabilities – either scheduled or ad hoc snapshots. And Zetta snapshots are free from the capacity and performance limitations of single devices and fixed size clusters. This provides a customer-controlled protection mechanism – once a snapshot is created, the file system is preserved in that state until the snapshot is deleted, allowing a user to go back and restore filed and directories from the “.snapshot” directory like with any on-premises filer.

   

• Geo-Replication — All customer data stored at Zetta is replicated to another data center. In 2010 we expect to begin to offer full asynchronous replication to our customers who want a fully-mountable volume resident in another Zetta data center, either for performance or for data integrity.

Again, the Zetta solution was designed with the core premise that preventing data loss was our primary charter, and these are some of the unique features we’ve put into the solution to live up to that charter.

Compare this with what is available today from the HTTP cloud storage vendors. I reiterate that this is not a knock on those solutions – they do an excellent job for their customer target, but their target is not the enterprise, and they don’t provide the requisite features to host primary enterprise data. These solutions do not provide write receipts, have no RAID implementations, lack proactive error correction, and offer no file systems with snapshots.

I’ll be back soon to discuss Zetta’s approach to data security & privacy.

Reading: Hosting Primary, Unstructured Enterprise Data in the Cloud – Part 4: Comprehensive data integrity/protectionTweet This

Tags: availability, Cloud Requirements, Enterprise Cloud Storage, Enterprise Features, error correction, hash, RAID-6, RAIN, RAIN-6, reliability, Snapshots

This entry was posted on Wednesday, December 16th, 2009 at 10:16 am and is filed under Cloud Storage, Zetta Product. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.