storage: raid vs backup - AVS Forum
Forum Jump: 
 34Likes
Reply
 
Thread Tools
post #1 of 75 Old 08-25-2014, 03:00 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
storage: raid vs backup

Hello,
I'm planning my system, and was thinking of some more advanced means of storing media (software raid, drivepool, snapraid, ...). But then it hit me: I also backup media, and would back up to blueray or M-disk.
The benefits of having data redundancy would be
  • higher availability
  • easier repair if a drive fails
However, for media, higher availability is not that mandatory, particularly because playback should be possible from the optical disks. My media library is not excessive (compared to some others here, my total current gross storage is approx. 5 TB, net storage is less due to some duplication and raid), so keeping it on optical disks is still possible. I could keep a copy on harddisk, or just keep the copy of the things I'm interested it at that time. It maybe requires some discipline to properly organize it on blurays, but so does organizing diskspace or solutions like snapraid.

Pricewise, it seems that blueray disks and harddisk are at a similar price level (0.03 eur/GB), but the blurays can be bought in the amount necessary when needed.

Any people do something similar, or is the more advanced storage the more popular way?

Thanks!


Jörg

PS: I should add it is a 2 person household, so not that much parallel viewing going on.
V_J is offline  
Sponsored Links
Advertisement
 
post #2 of 75 Old 08-25-2014, 05:23 AM
Advanced Member
 
balky's Avatar
 
Join Date: Oct 2008
Posts: 885
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 77 Post(s)
Liked: 61
I have two FreeNAS boxes with ZFS in both of them, I put data on only one and Rsync takes care of the rest, I have a backup of my backup for important files... music, family photos... etc... on an external 1.5T USB drive that is mostly offline and only brought up when I need to add new data...

I try to imagine using multiple BR discs and I don't see it being practical...

If we assume 50G of data can be stored on BR disc, you will need at least 20 discs to back up 1T of data... now imaging how much effort it will require to write data into the discs and keep track of which data resides in which disc...

5T will require at least 100 BR discs versus 2 X 3T external USB drives...

For me, I think the choice is quite obvious...

And mind you... RAID ≠ Backup
ajhieb and Tesshin like this.
balky is offline  
post #3 of 75 Old 08-25-2014, 08:05 AM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,595
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 430 Post(s)
Liked: 434
Quote:
Originally Posted by balky View Post
And mind you... RAID ≠ Backup
Came here to say that.

Both are great, and both serve a purpose, and while there is some slight overlap, they are most definitely not the same thing.

RAID protects you data in the event of a drive failure. That's pretty much it, from the standpoint of redundancy.

A backup protects your data in the even of a drive failure, several drive failures, accidental deletion, intentional deletion, file system corruption, spontaneous computer combustion, alien invasion, etc.

If your data is mostly ripped stuff from optical media, then it makes sens to backup everything else on optical disk (if you don't mind re-ripping stuff) but if the content also has a lot of other data, then you'll probably be better off, and backing up to a 2nd hard drive (or set of drives) as @balky suggested.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4 of 75 Old 08-25-2014, 08:35 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
Quote:
Originally Posted by balky View Post
I have two FreeNAS boxes with ZFS in both of them, I put data on only one and Rsync takes care of the rest, I have a backup of my backup for important files... music, family photos... etc... on an external 1.5T USB drive that is mostly offline and only brought up when I need to add new data...
Somehow I'm weary of using a harddisk for backup... I'm also doing it now, but important data are also on multiple optical disks. I've seen too many harddisks fail.

Quote:
Originally Posted by balky View Post
I try to imagine using multiple BR discs and I don't see it being practical...
If we assume 50G of data can be stored on BR disc, you will need at least 20 discs to back up 1T of data... now imaging how much effort it will require to write data into the discs and keep track of which data resides in which disc...

5T will require at least 100 BR discs versus 2 X 3T external USB drives...
True... Well, there are 128 GB BR discs... But physically the BR will take up more space and are more cumbersome to put data on it. Keeping track of what is on them is just a matter of proper labeling and discipline when writing the discs. But I immediately agree that that is easier said than done.

Quote:
Originally Posted by ajhieb View Post
And mind you... RAID ≠ Backup
I know... I'm just not sure which is the best for my purpose.

Quote:
Originally Posted by ajhieb View Post
Came here to say that.

Both are great, and both serve a purpose, and while there is some slight overlap, they are most definitely not the same thing.

RAID protects you data in the event of a drive failure. That's pretty much it, from the standpoint of redundancy.

A backup protects your data in the even of a drive failure, several drive failures, accidental deletion, intentional deletion, file system corruption, spontaneous computer combustion, alien invasion, etc.

If your data is mostly ripped stuff from optical media, then it makes sens to backup everything else on optical disk (if you don't mind re-ripping stuff) but if the content also has a lot of other data, then you'll probably be better off, and backing up to a 2nd hard drive (or set of drives) as @balky suggested.
I'm fully aware that RAID is not backup.
I'm just considering what would be the best option to prevent loss of my media in the event of a harddisk/controller failure. Important files (photos etc) are backed up on several harddisks and optical discs (multiple, and regularly rewritten and stored in physically different places :-)).

But media files are different: they are quite static (usually files just get added, rarely removed or modified), so the need to go back to a previous version is not really present. Some benefits of raid are not needed, and some benefits of backup are not needed. Most of my files are ripped from own discs, but ripping is a slow process.

Putting the media files on a RAID (or some post-raid system) should allow for quite a fast fix if a drive fails. If it is a software RAID solution, it even works if the computer suffers a big failure, unless it fries the drives (unlikely). The added benefit of availability is not a huge necessity for media.
Backing up the files (either to HDD or to optical disc) also provides for a quick fix (faster than ripping everything again) if a drive fails, and in the event of a big computer failure. Of course, copying from a backup harddisk would be faster than from optical disks...

My initial plan was a simple 3-disk raid 5 and backups of important files or media that I would really mind loosing. But the raid plan got me a bit worried after reading different topics relating to so called end-of-raid5: http://www.zdnet.com/blog/storage/wh...ng-in-2009/162 . So I then started looking at post-raid alternatives and started wondering if it is really necessary for movies and music... Hence the thread... I'll probably end up doing some mixed solution, which probably is what most people do...
V_J is offline  
post #5 of 75 Old 08-25-2014, 08:56 AM
Advanced Member
 
balky's Avatar
 
Join Date: Oct 2008
Posts: 885
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 77 Post(s)
Liked: 61
Quote:
Originally Posted by V_J View Post
Somehow I'm weary of using a harddisk for backup... I'm also doing it now, but important data are also on multiple optical disks. I've seen too many harddisks fail.


True... Well, there are 128 GB BR discs... But physically the BR will take up more space and are more cumbersome to put data on it. Keeping track of what is on them is just a matter of proper labeling and discipline when writing the discs. But I immediately agree that that is easier said than done.



I know... I'm just not sure which is the best for my purpose.


I'm fully aware that RAID is not backup.
I'm just considering what would be the best option to prevent loss of my media in the event of a harddisk/controller failure. Important files (photos etc) are backed up on several harddisks and optical discs (multiple, and regularly rewritten and stored in physically different places :-)).

But media files are different: they are quite static (usually files just get added, rarely removed or modified), so the need to go back to a previous version is not really present. Some benefits of raid are not needed, and some benefits of backup are not needed. Most of my files are ripped from own discs, but ripping is a slow process.

Putting the media files on a RAID (or some post-raid system) should allow for quite a fast fix if a drive fails. If it is a software RAID solution, it even works if the computer suffers a big failure, unless it fries the drives (unlikely). The added benefit of availability is not a huge necessity for media.
Backing up the files (either to HDD or to optical disc) also provides for a quick fix (faster than ripping everything again) if a drive fails, and in the event of a big computer failure. Of course, copying from a backup harddisk would be faster than from optical disks...

My initial plan was a simple 3-disk raid 5 and backups of important files or media that I would really mind loosing. But the raid plan got me a bit worried after reading different topics relating to so called end-of-raid5: http://www.zdnet.com/blog/storage/wh...ng-in-2009/162 . So I then started looking at post-raid alternatives and started wondering if it is really necessary for movies and music... Hence the thread... I'll probably end up doing some mixed solution, which probably is what most people do...

My only take-home in that article is that you should have a backup and not rely on or consider a RAID (5 or 6) as an equivalent to a backup,
Other than that, there is quite a bit of exaggerated scare mongering about HDD failure rates...

I especially prefer software RAID as it frees you any dependence on HW controllers...

My main zpool has seen the days... I first set it up on FreeNAS 7.2, moved it to ZFS on Linux, moved it to ZFS on FreeBSD and now back to on FreeNAS 9.2... each time replacing mobo, processor and memory...
In each case, I first set the OS up, and simply import my pool back...
balky is offline  
post #6 of 75 Old 08-25-2014, 09:21 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
Quote:
Originally Posted by balky View Post
My only take-home in that article is that you should have a backup and not rely on or consider a RAID (5 or 6) as an equivalent to a backup,
Other than that, there is quite a bit of exaggerated scare mongering about HDD failure rates...
I agree, but still it is an interesting notion (I once had issues rebuilding a raid, but due to different problems). So it never hurts to check alternatives / options...

As an alternative, I was looking at SnapRAID, but it is a bit a mixed bag. Syncs have to happen manually, and reverting to a previous version is possible as long as there was no sync (but the current version is not protected if it was no synched yet). Looks interesting, but more of a hassle to maintain. I read very good comments on DrivePool, which basically does pooling where duplication of data over multiple drives is possible (and everything is readable in any computer).

I have a number of external harddisks (ranging from 500 GB to 2 TB), so I can use those for backup, combined with some optical discs for special data.

Perhaps I'll stick with my original plan: 2-3 HDD (each 3-4 TB), with some redundancy and backups on hdd. Lower priority to backups on optical disc...
V_J is offline  
post #7 of 75 Old 08-25-2014, 10:20 AM
Member
 
pylor's Avatar
 
Join Date: Mar 2014
Posts: 83
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 21 Post(s)
Liked: 18
I'm with the others, backing up to optical disk (unless it's just pictures or something very special) just seems like a huge headache. I just run simple intel-based raid 5 (allows me time to react to a single HDD failure) and also backup to external 4TB HDDs using a simple robocopy script every time I feel I've acquired enough new data that I want to back it up.
pylor is offline  
post #8 of 75 Old 08-25-2014, 11:00 AM
AVS Special Member
 
bryansj's Avatar
 
Join Date: Feb 2004
Location: Atlanta, GA
Posts: 6,446
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 112 Post(s)
Liked: 203
CrashPlan for irreplaceable media and software RAID for the rest.

I've seen too many burnt discs go bad to even consider optical. Not to mention the hassle.

bryansj is online now  
post #9 of 75 Old 08-25-2014, 11:04 AM
AVS Addicted Member
 
stanger89's Avatar
 
Join Date: Nov 2002
Location: Marion, IA
Posts: 17,491
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 173 Post(s)
Liked: 156
The original discs are the backup.
htpcforever likes this.

See what an anamorphoscopic lens can do, see movies the way they were meant to be seen
stanger89 is offline  
post #10 of 75 Old 08-26-2014, 01:33 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
Thanks guys, it seems like there is quite a concensus (on the internet... who would have thought?). And it was my first idea, untill I started overthinking it...

Quote:
Originally Posted by stanger89 View Post
The original discs are the backup.
Ultimately yes, but they are in storage at my parents' place, 1400 km away, as I lack the physical space to put it... So not directly accessible in the event of a failure.
V_J is offline  
post #11 of 75 Old 08-26-2014, 06:53 AM
Senior Member
 
Join Date: Jul 2014
Posts: 376
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 224 Post(s)
Liked: 77
Quote:
Originally Posted by ajhieb View Post
Came here to say that.

Both are great, and both serve a purpose, and while there is some slight overlap, they are most definitely not the same thing.

RAID protects you data in the event of a drive failure. That's pretty much it, from the standpoint of redundancy.

A backup protects your data in the even of a drive failure, several drive failures, accidental deletion, intentional deletion, file system corruption, spontaneous computer combustion, alien invasion, etc.

If your data is mostly ripped stuff from optical media, then it makes sens to backup everything else on optical disk (if you don't mind re-ripping stuff) but if the content also has a lot of other data, then you'll probably be better off, and backing up to a 2nd hard drive (or set of drives) as @balky suggested.
With the type of RAID used by SnapRAID and Disparity you can restore a file you deleted (accidently or on purpose), provided you have not synced the RAID again yet. When using them for movie storage protection, manual updates are a great thing to use since the data should only change when you do it on purpose.

Last edited by htpcforever; 08-26-2014 at 06:57 AM.
htpcforever is offline  
post #12 of 75 Old 08-26-2014, 07:04 AM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,595
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 430 Post(s)
Liked: 434
Quote:
Originally Posted by htpcforever View Post
With the type of RAID used by SnapRAID and Disparity you can restore a file you deleted (accidently or on purpose), provided you have not synced the RAID again yet. When using them for movie storage protection, manual updates are a great thing to use since the data should only change when you do it on purpose.
That's basically no different than Recycle Bin functionality in Windows. And while either can get a recently deleteed file back (so can file recover software for that matter) none of those options is as relaible as a good backup.
balky likes this.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #13 of 75 Old 08-26-2014, 07:57 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
Quote:
Originally Posted by htpcforever View Post
With the type of RAID used by SnapRAID and Disparity you can restore a file you deleted (accidently or on purpose), provided you have not synced the RAID again yet. When using them for movie storage protection, manual updates are a great thing to use since the data should only change when you do it on purpose.
It was SnapRAID that made me rethink everything.
Recovery is possible after sync, returning to a previous version before a sync.So with the displine of properly choosing the moment you sync, and backing up the data that you know will not change (media file after conversion, subtitles, ...).
Problem with SnapRAID is that you can only get it back if the data on the OTHER disk that is used for the parity also has not been changed. And there is the snag: which other data is that? So if you are some time after the last sync, you are not sure if you will be able to go back to the previous version, nor if the file is still recoverable. And this is for files on different harddisks. This situation scares me a bit about SnapRAID. It looks very powerfull, if used properly.

I just haven't figured out a scheme in which SnapRAID would be useful. If all data is static, then yes. Synching regularly (e.g. after modifying data) should be quite fast, but it really requires discipline in usage.


Quote:
Originally Posted by ajhieb View Post
That's basically no different than Recycle Bin functionality in Windows. And while either can get a recently deleteed file back (so can file recover software for that matter) none of those options is as relaible as a good backup.
Well, there is less of a space limitation than with recycle bin. And it is even less sure which files you can get back. A good backup is more reliable, no discussion about that.
V_J is offline  
post #14 of 75 Old 08-26-2014, 03:53 PM
One-Man Content Creator
 
wmcclain's Avatar
 
Join Date: May 2006
Posts: 17,683
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 617 Post(s)
Liked: 375
Quote:
Originally Posted by V_J View Post
This situation scares me a bit about SnapRAID. It looks very powerfull, if used properly.

I just haven't figured out a scheme in which SnapRAID would be useful. If all data is static, then yes. Synching regularly (e.g. after modifying data) should be quite fast, but it really requires discipline in usage.
I think that's fair.

With SnapRAID newly added files do not disturb the array, but they are not protected until you sync. I keep other backups until my weekly sync. And sometimes sync manually on special occasions.

Deleted and modified files require more caution. I keep a /delete folder on each disc outside of the array and move deleted files there temporarily. They can be deleted after the next sync. If you need to recover before then, the fix command actually has an option that reads non-array directories for just this purpose. Or you can move them back into place.

For modified files (rare in a media library, but it happens) you'd want to:
  • move the old one to /delete
  • move the new one into place
  • sync
  • delete the old one

I have this mostly automated. SnapRAID is working well for me. It's best for a static collection of large files. "Static but growing" is not a big problem.

-Bill
ajhieb, morganf and htpcforever like this.
wmcclain is online now  
post #15 of 75 Old 08-27-2014, 01:33 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
Quote:
Originally Posted by wmcclain View Post
I have this mostly automated. SnapRAID is working well for me. It's best for a static collection of large files. "Static but growing" is not a big problem.
Thanks for your comments. I was thinking of first trying with SnapRAID, but I'm still a bit puzzled... suppose you make a SnapRAID of HDD1, HDD2 and one parity disk.
To properly make sure (and keep track of) what is protected and what not, it is probably best to define the SnapRAID on the data of HDD1 and HDD2 that are quite static.
Does it then mean that you have a folders in one of these categories on both HDD1 and HDD2:
\staticdata (this can be one or more folders, which part of the snapraid)
\deleted (excluded from the snapraid, used for deletion of protected files
\dynamicdata (one or more folders like this, not part of the snapraid, and just used for files that modify a lot

Thanks,


Jörg
V_J is offline  
post #16 of 75 Old 08-27-2014, 05:32 AM
One-Man Content Creator
 
wmcclain's Avatar
 
Join Date: May 2006
Posts: 17,683
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 617 Post(s)
Liked: 375
Right, that would work.

In your example, only staticdata is part of the array that SnapRAID protects. deleted is not part of the array but is available to "snapraid fix" just in case you need it for a recovery.

dynamicdata is not covered by snapraid either, and you can use it for whatever transient or often-updated data you have. I do something similar with a daily rsync from my desktops to the file server. Just a place to keep another copy of the live data.

For those same desktops I do a monthly rotating backup by tar-ing (I'm on linux; zip or whatever would work as well) my working directories into one big archive file which is copied to the SnapRAID area. These are immediate term backups which will be deleted after a few months. I should take them off-site as well; to do.

People use various pooling systems that manage the distribution of files across discs so that you don't have to be aware of these structures on each drive, but I haven't done that. I'm used to managing my data per disc. For moving deletes and updated files around, I think you'd want to know where the files are. Obviously you want to move large files within a physical disc rather than copying them between physical devices.

-Bill

Last edited by wmcclain; 08-27-2014 at 05:55 AM.
wmcclain is online now  
post #17 of 75 Old 08-27-2014, 07:55 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
Quote:
Originally Posted by wmcclain View Post
Right, that would work.

In your example, only staticdata is part of the array that SnapRAID protects. deleted is not part of the array but is available to "snapraid fix" just in case you need it for a recovery. dynamicdata is not covered by snapraid either, and you can use it for whatever transient or often-updated data you have.
Ok, thanks! Just wondering if this should be reflected in the folder names.
I read on the SnapRAID forum that it is e.g. possible (on Windows) to put the Recycle Bin as the "deleted" folder. This would allow deletion the normal way still.

Quote:
Originally Posted by wmcclain View Post
People use various pooling systems that manage the distribution of files across discs so that you don't have to be aware of these structures on each drive, but I haven't done that. I'm used to managing my data per disc. For moving deletes and updated files around, I think you'd want to know where the files are. Obviously you want to move large files within a physical disc rather than copying them between physical devices.
It is also the thing that scares me about drive pooling things.
A bit strange, as I have a hardware raid 5 running, and there you also don't decide what goes where, but it feels more like one storage unit. I read about DrivePool (on Windows), and it basically creates one driveletter that is the drivepool. File stored on that one are stored on a hidden folder on the disks that are part of the pool, according to some algorithm (balancing, first filling one disk, ...) and possibly stored on multiple disks (setting for the folder on the pool).

It looks nice, but I also prefer to decide what goes where. Perhaps it is old fashioned, especially if you consider the pool as "one storage unit". Snapraid offers more control (and one dedicated parity disk)... On the other hand, the more transparent the system, the better (I'm not the only one using the computer ).
V_J is offline  
post #18 of 75 Old 08-27-2014, 01:15 PM
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 485
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 230 Post(s)
Liked: 78
Quote:
Originally Posted by V_J View Post
Ok, thanks! Just wondering if this should be reflected in the folder names.
I read on the SnapRAID forum that it is e.g. possible (on Windows) to put the Recycle Bin as the "deleted" folder. This would allow deletion the normal way still.


It is also the thing that scares me about drive pooling things.
A bit strange, as I have a hardware raid 5 running, and there you also don't decide what goes where, but it feels more like one storage unit. I read about DrivePool (on Windows), and it basically creates one driveletter that is the drivepool. File stored on that one are stored on a hidden folder on the disks that are part of the pool, according to some algorithm (balancing, first filling one disk, ...) and possibly stored on multiple disks (setting for the folder on the pool).

It looks nice, but I also prefer to decide what goes where. Perhaps it is old fashioned, especially if you consider the pool as "one storage unit". Snapraid offers more control (and one dedicated parity disk)... On the other hand, the more transparent the system, the better (I'm not the only one using the computer ).

I guess as I read this thread, I just wonder how large your media collection is. I'm talking about the movies, music, shows and stuff - not your important files that need real protection. For "important" stuff, I found that a $45 128 GB thumb drive has sufficed with room to spare, and my other half has probably in the vicinity of 5,000 pictures saved. But that's for the "important" stuff.

As far as media goes, you pointed out that the discs are all in storage cross-country at your parents' place because you don't currently have room for them (I can relate). That would seem to me to indicate though, that your collection is at the point where it is only growing by a few titles once in a while now. You mentioned at the outset of having only about 5 TB of data in total. I was a bit fuzzy as to if that was just the media stuff, or if it was everything. If it is everything (including parity) then RAID might actually be a bit overkill for you. I'm not saying RAID 5 is bad or wrong for you, but it might be way more than you actually need to be considering. I say this for a few reasons.
  1. First, HDD prices are dropping fast these days. You can get a pair of 5 TB externals for under $300 delivered now. 6 TB drives are coming down in price as well and should be affordable soon.
  2. Second, as you pointed out, you won't be the only user, and you want some real transparency/control over where everything is.
  3. 5 TB is a lot of data. Sure, there are plenty of us around here that can reminisce about the days when we thought "5 TB is huge". There's a reason for that though, 5 TB really is a ton of data space - we just happen to be media hogs.
  4. While the various RAID 5 options can be made quite simple, the reality is that there are still multiple parts that can or do require attention.

Unless your media library is growing rapidly, and you can actually see a time in the very near future you will pass 6 TB of media data that you wish to retain (this would not include temporary media like watch-once shows), you might actually be better off forgetting RAID altogether and going with a simple mirror. If you have under 4 TB of static media data, a pair of 5 or 6 TB drives could last you a long time if your collection is not simply exploding. This would give you pretty much everything you seem to be looking for. You would have a very large drive space under one volume letter. You would have 100% transparency with where data goes and how it is saved. You would have a mirror drive for instant recovery in the event of primary drive failure. The frequency of your data mirror back-ups is entirely up to you. It can be instant so that when you write a file, the mirror writes it too. Or, it can be scheduled by you to happen at some predetermined time. There won't be any real performance hit so to speak, so you can do it pretty much whenever.

The RAID 5 and other parity solutions really only make sense if your library is going to continue to expand rapidly to the point where you can pretty much take it to the bank that you will be storing more media than the current generation of drives can hold. Regardless of whether you go with a simple mirror, or if you go with a parity solution, the expense is going to be a minimum of one drive of your largest size (either a parity drive or your mirror). It's not until you get to the point where you would need a second mirror drive where parity solutions really start to come in handy. At that point parity solutions can save you a bundle on drive space. But until then, the cost factor is the same, but mirrored is a much simpler solution which has far fewer things that can go wrong.

You mentioned having a number of drives already. Creating a simple pool out of them and then using a new external that is larger in size than the pool might work for you.

If you are set that you want to employ drive pooling and parity, there are absolutely some very simple but powerful solutions to be had out there. I have an array with 40 TB of data drives. You bet I use a parity solution (1x 4 TB drive). But if I had less than 5 TB of data and my collection was only growing at a somewhat modest rate of less than 25 movies per year, I would almost certainly simply stick to a mirror for my redundancy versus the "hassle" of a RAID-type solution.
Aryn Ravenlocke is online now  
post #19 of 75 Old 08-27-2014, 04:48 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,595
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 430 Post(s)
Liked: 434
Quote:
Originally Posted by Aryn Ravenlocke View Post
If you are set that you want to employ drive pooling and parity, there are absolutely some very simple but powerful solutions to be had out there. I have an array with 40 TB of data drives. You bet I use a parity solution (1x 4 TB drive). But if I had less than 5 TB of data and my collection was only growing at a somewhat modest rate of less than 25 movies per year, I would almost certainly simply stick to a mirror for my redundancy versus the "hassle" of a RAID-type solution.
While I think that replicating your data on a 2nd drive is probably the easiest solution (be it via an actual RAID-1 mirror, or copying data over to a 2nd drive) I disagree with your assertions about RAID-5

I've been running RAID-5 for years, and for day-to-day operations it is completely maintenance free. If a drive fails, I replace it. That's it. But to keep it operational I don't have to tell it to do anything. I don't have to schedule anything, I don't have to check to see if it's doing what it is supposed to . It just plain old works... just like a single drive. Once it its setup, I don't do squat to it.

As far as having control over where your data goes... That may or may not have any real merit. If you have a habit of breaking your array, and removing drives to take specific data to another computer, I can see why you'd want to control what data goes on what drive (I'd think that is a silly way to do things in general, but it would at least justify the desire to have control at the drive level) but if you're just wanting to build an array and leave it alone, then there is no real difference between having control over what drive something goes on, and what folder it goes into.

With my current setup, I have a folder in the root of my RAID array called "My Media" that is shared with Read Only permissions on the network. It contains subfolders called "TV" "Movies" etc, and those folders contain the content you'd expect. Copying TV shows to the TV folder is no different from an organizations standpoint than copying to the TV drive. Plus I don't have to worry about my "Movies" drive filling up while my "Music" drive is nearly empty. It's one common lump of storage.

But if you're keeping your array intact, and don't plan on breaking it to move data from a single drive, then wanting drive level control is a totally arbitrary distinction. Once the array is created, it acts as one device. Wanting drive level control is no different than wanting platter level control. There is no practical reason for it.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #20 of 75 Old 08-27-2014, 05:25 PM
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 485
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 230 Post(s)
Liked: 78
Quote:
Originally Posted by ajhieb View Post
While I think that replicating your data on a 2nd drive is probably the easiest solution (be it via an actual RAID-1 mirror, or copying data over to a 2nd drive) I disagree with your assertions about RAID-5

I've been running RAID-5 for years, and for day-to-day operations it is completely maintenance free. If a drive fails, I replace it. That's it. But to keep it operational I don't have to tell it to do anything. I don't have to schedule anything, I don't have to check to see if it's doing what it is supposed to . It just plain old works... just like a single drive. Once it its setup, I don't do squat to it.

As far as having control over where your data goes... That may or may not have any real merit. If you have a habit of breaking your array, and removing drives to take specific data to another computer, I can see why you'd want to control what data goes on what drive (I'd think that is a silly way to do things in general, but it would at least justify the desire to have control at the drive level) but if you're just wanting to build an array and leave it alone, then there is no real difference between having control over what drive something goes on, and what folder it goes into.

With my current setup, I have a folder in the root of my RAID array called "My Media" that is shared with Read Only permissions on the network. It contains subfolders called "TV" "Movies" etc, and those folders contain the content you'd expect. Copying TV shows to the TV folder is no different from an organizations standpoint than copying to the TV drive. Plus I don't have to worry about my "Movies" drive filling up while my "Music" drive is nearly empty. It's one common lump of storage.

But if you're keeping your array intact, and don't plan on breaking it to move data from a single drive, then wanting drive level control is a totally arbitrary distinction. Once the array is created, it acts as one device. Wanting drive level control is no different than wanting platter level control. There is no practical reason for it.
I was not trying to make RAID 5 sound complicated. Perhaps instead of "hassle" I could have used the term "minor, mostly one-time inconveniences". RAID 5 really is almost as easy to set-up as a mirror. It just becomes a matter of whether or not it is running in real-time or in snapshot mode. If it is running in real-time, then that pretty much does take care of it. If the array is in snapshot mode, then there is some scheduling to be done in order to make sure that snapshots are done with the desired level of frequency. That leads to making sure to keep track of what data has been taken care of through parity and what has not.

These are not big issues, and in the grand scheme of things, they are simple and almost easy to forget about entirely if things are working properly. They are however, one small step more complicated than a mirror. suggestion for V_J was simply that if his media library was of sufficiently small(ish) size, that a parity array might not be his only/best solution given the situation. A parity array from the start certainly does make things easier if there is significant growth in the future though. In those cases, parity always.
Aryn Ravenlocke is online now  
post #21 of 75 Old 08-27-2014, 05:35 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,595
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 430 Post(s)
Liked: 434
Quote:
Originally Posted by Aryn Ravenlocke View Post
I was not trying to make RAID 5 sound complicated. Perhaps instead of "hassle" I could have used the term "minor, mostly one-time inconveniences". RAID 5 really is almost as easy to set-up as a mirror. It just becomes a matter of whether or not it is running in real-time or in snapshot mode. If it is running in real-time, then that pretty much does take care of it. If the array is in snapshot mode, then there is some scheduling to be done in order to make sure that snapshots are done with the desired level of frequency. That leads to making sure to keep track of what data has been taken care of through parity and what has not.

These are not big issues, and in the grand scheme of things, they are simple and almost easy to forget about entirely if things are working properly. They are however, one small step more complicated than a mirror. suggestion for V_J was simply that if his media library was of sufficiently small(ish) size, that a parity array might not be his only/best solution given the situation. A parity array from the start certainly does make things easier if there is significant growth in the future though. In those cases, parity always.
Well, I don't think there is such a thing as snapshot RAID-5, I know I've certainly never encountered it. Setup should pretty much be identical to a mirror. Add drives, initialize, done. The standard RAID levels are a piece of cake to setup and maintain. Say what you will about the cost or performance of such a setup, but getting the things going and keeping them going is a piece of cake. It may have been a little different years ago when all of the hardware implementations were using some flavor of SCSI, and you had to worry about assigning device IDs, and terminating the SCSI bus and whatnot, but on today's hardware its pretty much child's play.

In terms of your advice to the OP, I agree.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #22 of 75 Old 08-27-2014, 08:05 PM
Senior Member
 
Join Date: Jul 2014
Posts: 376
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 224 Post(s)
Liked: 77
Quote:
Originally Posted by ajhieb View Post
That's basically no different than Recycle Bin functionality in Windows. And while either can get a recently deleteed file back (so can file recover software for that matter) none of those options is as relaible as a good backup.
Quote:
Originally Posted by ajhieb
RAID protects you data in the event of a drive failure. That's pretty much it, from the standpoint of redundancy.

A backup protects your data in the even of a drive failure, several drive failures, accidental deletion, intentional deletion, file system corruption, spontaneous computer combustion, alien invasion, etc.

So which is it, do RAIDs such as SnapRAID and Disparity protect your data even in the event of a drive failure, several drive failures, accidental deletion, intentional deletion, and file system corruption or not? (pro-tip, they do)


I was basically saying your descriptions were so out of whack they might as well have been describing different types of cheese. The only thing a data backup does for you that a RAID does not is that it gives you snapshots in time. A RAID is current only as of its last sync, but a backup can be of last month...another from last year...another from yesterday, etc. If you want a file from before your last sync, you cannot get it...but if you have yet to sync, you are golden.


Mind you, backups are vital for the OS drive and dynamic data, but not vital for static or very rarely changing data. For such data a RAID setup such as SnapRAID or Disparity are all you need.
htpcforever is offline  
post #23 of 75 Old 08-27-2014, 08:11 PM
Senior Member
 
Join Date: Jul 2014
Posts: 376
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 224 Post(s)
Liked: 77
I like using SnapRAID or Disparity because I have disk level control over my data. I can also remove and add drives at will. I can lend one of my two Sci-Fi HDDs to a friend for the weekend and, provided I do not do a manual sync, if the drive dies I can recover everything that was on it. I have many terabytes of data and these two programs both work well to protect it.
htpcforever is offline  
post #24 of 75 Old 08-27-2014, 08:25 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,595
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 430 Post(s)
Liked: 434
Quote:
Originally Posted by htpcforever View Post
So which is it, do RAIDs such as SnapRAID and Disparity protect your data even in the event of a drive failure, several drive failures, accidental deletion, intentional deletion, and file system corruption or not? (pro-tip, they do)
Well, no they don't.. not in the same way that a separate backup does. If I have a separate backup of the file that exists independent of the computer, then it is there regardless of what happens on the computer. If the file is deleted, then I have access to the backup. Period. Full ****ing stop. That isn't the case with the RAID products. All of the redundancy comes with qualifications. It protects you from deletion if you haven't synced the parity yet. It protects you from file system corruption... (I dunno on that one... presumably because you think it's immune to file system corruption? but regardless, anything it does with the file system is a feature of the file system, not of the RAID itself) It protects from drive failures but only a few of them...


Quote:
Originally Posted by htpcforever View Post
I was basically saying your descriptions were so out of whack they might as well have been describing different types of cheese. The only thing a data backup does for you that a RAID does not is that it gives you snapshots in time. A RAID is current only as of its last sync, but a backup can be of last month...another from last year...another from yesterday, etc. If you want a file from before your last sync, you cannot get it...but if you have yet to sync, you are golden.
Bwahahahah! That's a damn riot!

So in order for a backup to be effective, you're telling me that I need to make sure I backup as frequently as possible. (so if something goes awry I have the most up to date version backed up) But for RAID products to be effective, I need to make sure that I update as infrequently as possible if I'm to make any use of the "restoring deleted files" feature (which is only effective for a few varieties of RAID) but in doing so, I'm putting all the rest of my changed data in jeopardy. Yeah... that sounds like a great strategy.

No what I said at first was accurate. RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #25 of 75 Old 08-28-2014, 02:21 AM - Thread Starter
V_J
Member
 
Join Date: Jun 2014
Posts: 109
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 12
Quote:
Originally Posted by ajhieb View Post
No what I said at first was accurate. RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
I full agree with this statement.
My initial question could be seen as two questions:
1. If you have a proper backup of your movies, do you really need a raid or similar system? From what I read so far, the answer seems to be: no, you can always recover from the backup, but a proper system can make for an easier to manage system (e.g. transparent expansion, less hassle in the event of drive failure).
2. If you have a raid, do you really need a backup? This seems to be: no, unless you want to be protected from accidental deletion or huge failure (controller, ...).
(huge simplification of the benefits/drawbacks, but I think this is just what matters in my case)

I also agree that raid-5 is really a set-and-forget thing (I have a hardware raid5, and never had to worry about maintenance of it). The snapraid scares me a bit: if you forget the sync, you may be in a situation where you have nothing: not a backup and not a disparity (nothing for some files if files are edited, nothing for the newly added files). I was looking for something other than raid5, as my current raid5 is fully tied to the computer (controller is a PCI-X card), which gives a big point of failure. This made me look at so many options and perhaps made me overthink things.

You are right that in the end if you consider the drives together as a storage unit (raid, pool, whatever), it does not matter on which drive things are stored. I for one do not intend to break the storage unit, all disks that are part of it would be internal. But having data readable when disks are put in another system could be helpful in the event of a big failure.

My collection has been quite static for a while, but is now growing steadily. So I'm looking for a rather transparent solution to store data in a way that allows for a more dynamic growth. I would start with e.g. 4-8 TB of net storage, leaving room for expansion.

Reading the comments seem to have helped me... I will get a few disks and will then play around: I can copy my old raid and break it, perfect playing ground to test how well everything goes. I'll check how comfortable I would be with SnapRaid and how easy it is to recover things, and have a second like DrivePool as alternative. Backups will be mixed: to external hdd or optical, depending on what it is...
ajhieb likes this.
V_J is offline  
post #26 of 75 Old 08-28-2014, 11:23 AM
Member
 
AaronMilwaukee's Avatar
 
Join Date: Jan 2006
Posts: 40
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 24 Post(s)
Liked: 29
You don't need to remember to sync Snapraid, you can just set it up to sync automatically as often as you want. No different than Flexraid in that regard.
AaronMilwaukee is offline  
post #27 of 75 Old 08-28-2014, 02:30 PM
Senior Member
 
Join Date: Jul 2014
Posts: 376
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 224 Post(s)
Liked: 77
Quote:
Originally Posted by ajhieb View Post
Well, no they don't.. not in the same way that a separate backup does. If I have a separate backup of the file that exists independent of the computer, then it is there regardless of what happens on the computer. If the file is deleted, then I have access to the backup. Period. Full ****ing stop. That isn't the case with the RAID products. All of the redundancy comes with qualifications. It protects you from deletion if you haven't synced the parity yet. It protects you from file system corruption... (I dunno on that one... presumably because you think it's immune to file system corruption? but regardless, anything it does with the file system is a feature of the file system, not of the RAID itself) It protects from drive failures but only a few of them...


You claimed it does not protect you from deletion and now you say it does. Which is it?
When the aliens arrive, which is one of your scenarios, how do you know you will have access to your backup? Did you store it off planet? Unless you store it off-site (not in your house), then you are not really even protected from the fire scenario you mentioned either.
RAIDs (such as SnapRAID and Disparity) are not different from Backup Media with regards to media corruption...you seem to think backup media is immune to corruption.
RAIDs protect from as many drive failures as you desire - the limitation is simply on how many parity drives you use.



Quote:
Bwahahahah! That's a damn riot!

Yes, but your statement was funny is a sad way, not funny in a ha-ha way. It is made even sadder because you appear to actually believe what you said is true (even while also stating the opposite as being true at the same time).

Quote:
So in order for a backup to be effective, you're telling me that I need to make sure I backup as frequently as possible. (so if something goes awry I have the most up to date version backed up) But for RAID products to be effective, I need to make sure that I update as infrequently as possible if I'm to make any use of the "restoring deleted files" feature (which is only effective for a few varieties of RAID) but in doing so, I'm putting all the rest of my changed data in jeopardy. Yeah... that sounds like a great strategy.

No, but I can see why your faulty thinking would make you believe this. We all know restoring a file from backup can only be done for the dates of the backup. Let me provide an example even you can follow:


You have a txt file. You type one sentence in it and save it. You then make a backup. You then type four sentences in it and save it. You then delete one sentence and save it. You then make a backup. You now want to restore the txt file with all four sentences in it. Can you do that from your backups?



Quote:
No what I said at first was accurate. RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.

Sigh..then you prove you know nothing about how SnapRAID and Disparity work. I can easily restore a file I accidently deleted after my last sync and before my next sync. I realize you want this to not be true so you can keep lying to people, but it is true. Please stop lying to people.
htpcforever is offline  
post #28 of 75 Old 08-28-2014, 02:32 PM
Senior Member
 
Join Date: Jul 2014
Posts: 376
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 224 Post(s)
Liked: 77
Quote:
Originally Posted by V_J View Post
I full agree with this statement.
Why, it is simply not true.


Quote:
Recover

How can I undelete a just deleted file?


Simply run the "fix" command using a filter for the specified file. Like: snapraid fix -f my_just_deleted_fileTo undelete a directory use:
snapraid fix -f my_just_deleted_dir/To undelete all the missing files use:
snapraid fix -m
http://snapraid.sourceforge.net/faq.html
htpcforever is offline  
post #29 of 75 Old 08-28-2014, 03:41 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,595
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 430 Post(s)
Liked: 434
Quote:
Originally Posted by htpcforever View Post
You claimed it does not protect you from deletion and now you say it does. Which is it?
My claim was that RAID doesn't protect your files from deletion and it doesn't. You brought up an example of a specific product that mimics RAID-like functionality. That particular product does provide limited recovery of deleted file. It isn't complete protection, not in the same way that a proper backup does.

Not sure why that is so hard for you to understand.

Quote:
Originally Posted by htpcforever View Post
When the aliens arrive, which is one of your scenarios, how do you know you will have access to your backup? Did you store it off planet? Unless you store it off-site (not in your house), then you are not really even protected from the fire scenario you mentioned either.
Proper backup is offsite.

Quote:
Originally Posted by htpcforever View Post
RAIDs (such as SnapRAID and Disparity) are not different from Backup Media with regards to media corruption...you seem to think backup media is immune to corruption.
Fair point, but corruption of an offline backup is far less likely than corruption of an online file system. And that still doesn't change the fact that you tried to assert that RAID offers protection against file system corruption, which it doesn't. Some RAID-like products offer some protection, but to suggest that RAID in general (which is what I was talking about) protects against file system corruption is false.


Quote:
Originally Posted by htpcforever View Post
RAIDs protect from as many drive failures as you desire - the limitation is simply on how many parity drives you use.
False. Not a RAID system on the planet can protect your data in the event of 100% drive failure. Only a backup can do that.


Quote:
Originally Posted by htpcforever View Post
No, but I can see why your faulty thinking would make you believe this. We all know restoring a file from backup can only be done for the dates of the backup. Let me provide an example even you can follow:

You have a txt file. You type one sentence in it and save it. You then make a backup. You then type four sentences in it and save it. You then delete one sentence and save it. You then make a backup. You now want to restore the txt file with all four sentences in it. Can you do that from your backups?
What's your point? RAID isn't going to provide any better (or even comparable) solution in that scenario. I can backup as often as I see fit, and I can restore to whatever version I've backed up.


Quote:
Originally Posted by htpcforever View Post
Sigh..then you prove you know nothing about how SnapRAID and Disparity work. I can easily restore a file I accidently deleted after my last sync and before my next sync. I realize you want this to not be true so you can keep lying to people, but it is true. Please stop lying to people.
Yeah and that functionality exists completely outside of your RAID-like products. It's called the Windows Recycle Bin and it's been around for about 2 decades. I'm pretty sure It existed on Macs prior to that.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #30 of 75 Old 08-28-2014, 04:37 PM
Senior Member
 
Join Date: Jul 2014
Posts: 376
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 224 Post(s)
Liked: 77
EDIT: Not worth it, you know you are lying, but since you refuse to admit it there is no reason to continue. It is silly for me to argue with you just as it is silly for me to argue with a 3 year old. I won't bother to do it any longer.

Last edited by htpcforever; 08-28-2014 at 09:37 PM.
htpcforever is offline  
Reply Home Theater Computers

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off