FlexRAID vs unRAID - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #1 of 83 Old 08-21-2009, 03:36 PM - Thread Starter
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 150
I am getting close to putting together my first data storage server. I am currently looking at both FlexRAID and unRAID. From what I have read, they are very similiar.

I know FlexRAID is free and that unRAID is more polished. Other than that, does anyone know the functional differences between the two?

As a note, I am a storage server nub and know nothing about Linux and such.

Thanks!
cybrsage is offline  
Sponsored Links
Advertisement
 
post #2 of 83 Old 08-21-2009, 04:07 PM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
FlexRAID sits on top of an operating system, at least last I checked. It basically calculates parity based on data you provide it. It also doesn't have unified folder views as unRAID and WHS have.

unRAID is basically a stripped down Linux distro (Debian?) with custom code for the storage system and GUI. Assuming you did your research beforehand and all your hardware is supported, installation is as simple as unzipping files to a flash drive.
ilovejedd is offline  
post #3 of 83 Old 08-21-2009, 05:57 PM
AVS Special Member
 
oliverjg's Avatar
 
Join Date: Sep 2006
Posts: 1,760
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
because flexraid runs on an existing o/s you can install it on an existing windows computer. just add drives to the pc and go. you can still use the pc for other purposes. you might not need to add another box to the network.

if you don't already have the pc and you are building something for this purpose, unraid is added on to linux distribution.... that is, it is intended to have a dedicated linux box that doesn't do anything else.

for my purposes, having read the FAQs and some threads around the net, i would use flexraid.

read the faq and see if it describes you ...

http://www.openegg.org/faqFlexRAID.curi#basic

Quote:


FlexRAID™ Basic is suited for data that you only change a few times during the day.
If you continuously change your data during the day, every day, then FlexRAID™ Basic is not a fit for you.
You will have to wait for FlexRAID™ Live! or FlexRAID™ NAS.

Note that we are not talking about having an issue with continuously reading the data.
FlexRAID™ Basic will work just fine even if you read your data continuously 24/7.
It is just that FlexRAID™ Basic needs break periods where no data change is occurring to re-synchronize the RAID.

When I say "data change", I am not talking about adding new data files to the RAID as that action needs no break period at all.
Basically, you can continuously be writing new files to the RAID and FlexRAID™ Basic will run along just fine.

A good strategy would be to think about your data and separate the data you change continuously from the data you don't change continuously.
For almost all users, the data you don't change continuously will be larger and most likely the bulk of your data.
Once that's done, you can move the data that does not continuously change to be managed by FlexRAID™ Basic and move the other set of data to a much smaller RAID 1 or RAID 5 or NAS configuration and save a lot of money in the process.

my data is already separated as suggested.
oliverjg is offline  
post #4 of 83 Old 08-21-2009, 06:39 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by ilovejedd View Post

FlexRAID sits on top of an operating system, at least last I checked. It basically calculates parity based on data you provide it. It also doesn't have unified folder views as unRAID and WHS have.

However, you could use FlexRAID on WHS, and have the functionality of both.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #5 of 83 Old 08-21-2009, 08:07 PM - Thread Starter
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 150
Hmmm...my data is segregated as well. I plan on storing my DVD and BR rips.

Does "no unified view" mean each hdd will be its own drive letter?
cybrsage is offline  
post #6 of 83 Old 08-21-2009, 08:56 PM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by Darin View Post

However, you could use FlexRAID on WHS, and have the functionality of both.

Interesting. Last I checked FlexRAID, I don't think WHS was available. Or maybe it was but FlexRAID hasn't been tested on it at the time. unRAID was the only inexpensive option available and I needed something I can deploy right away. Too bad I'm out of cash (traded in my clunker so now, I'm making car payments). I would have liked to play with this.

Quote:
Originally Posted by cybrsage View Post

Hmmm...my data is segregated as well. I plan on storing my DVD and BR rips.

Does "no unified view" mean each hdd will be its own drive letter?

That would depend on how the underlying file system of your OS manages it. If using Windows XP, then yes, each drive will be its own drive letter.
ilovejedd is offline  
post #7 of 83 Old 08-21-2009, 10:33 PM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
You can always have each drive as a NTFS folder nested on a root folder of your c:\\ drive.

Here's my set up:

c:\\movies\\Drive01
c:\\movies\\Drive02
c:\\movies\\Drive03
etc.

Each numbered "Drive" is actually a physical drive that no longer has a drive letter. There's no limit to the amount of drives you can add this way and all the drives are kept in a very organized way for FlexRAID to protect.
MickeyDora is offline  
post #8 of 83 Old 08-21-2009, 11:20 PM
Senior Member
 
stottle's Avatar
 
Join Date: Jul 2004
Posts: 212
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I think part of the problem is that not many people have used both. I've been using unRaid for several years and have never had any problem. I have no reason to look at alternatives. I like that it is a simple OS run off a flash drive, so all drives are data. No worries about OS crashes/updates or other crap.

I certainly give unraid a thumbs up.

Brett
stottle is offline  
post #9 of 83 Old 08-21-2009, 11:28 PM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I just don't like the drive limit of unRAID... 16-20 drives is OK (unRAID) but I am pushing 30 drives with my FlexRAID set-up.
MickeyDora is offline  
post #10 of 83 Old 08-22-2009, 12:09 AM
Member
 
jouyang's Avatar
 
Join Date: Oct 2003
Location: San Francisco
Posts: 93
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I have Unraid the Pro version for a month now and I must say I cannot recommend it to anyone unless you are building it from scratch. If you have old hardwares, well wish you luck.

The problem that I have with Unraid is that it locks up the server every time I copy a large chuck of data over the network. I have to do hard reboot every time. Sometimes I feel like Unraid as my Vista. I do get helpful hints from Unraid forum members who are very knowledgeable but it's hard to diagnose a lockup issue.

edit:
Also, if you are into running media server like Mediatomb or PS3MediaServer, the Windows PS3MediaServer version is a lot better then the Linux version. I can run both Unraid and PS3MediaServer on the same box but I am not doing it because the better Windows version. So basically I am running 2 boxes. Now if I can get VMware to work on Unraid then that could be a different story. Then again, my Linux IQ is limited.

thanks,
~joy
jouyang is offline  
post #11 of 83 Old 08-22-2009, 04:24 AM
AVS Special Member
 
Willie's Avatar
 
Join Date: Jun 2001
Location: The Bay - Green Bay, WI
Posts: 1,264
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 8 Post(s)
Liked: 43
Quote:
Originally Posted by jouyang View Post


The problem that I have with Unraid is that it locks up the server every time I copy a large chuck of data over the network. I have to do hard reboot every time. Sometimes I feel like Unraid as my Vista. I do get helpful hints from Unraid forum members who are very knowledgeable but it's hard to diagnose a lockup issue.


I have been running unraid for a couple years with no problems whatsoever. I would highly recommend it.

If you have lockup issues, it is a hardware issue. Don't blame unraid because your hardware problems are hard to diagnose.

Willie
Willie is offline  
post #12 of 83 Old 08-22-2009, 07:19 AM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by cybrsage View Post

does anyone know the functional differences between the two?

I don't have either yet. But I've just finished gathering all the hardware for my server build, and I'm about to use a combination of WHS and FlexRAID. I can't profess to completely understand the inner workings and all the differences, but generally speaking, there are two significant operational differences, as I understand it:

1) unRAID essentially

is the operating system. It's a customized Linux build, which is installed on a usb flash drive. The entire machine becomes a file server using RAID redundancy. FlexRAID can be considered more of a program installed on a PC (already running Windows, Linux, etc).

2) unRAID is a more typical RAID setup which builds parity on the fly. The current version of FlexRAID is considered "snapshot" RAID, in that it only runs when invoked, either manually, or by schedule (I believe most people have it running nightly). In my opinion, for my needs, FlexRAID has the definite advantage on item 1. There are other things I'd like to have a server do besides file serving, and I'd rather not have two machines running 24x7. Since FlexRAID is just a program running on a PC, it does not care about what other things you may also have that machine doing. To the best of my knowledge, an unRAID box pretty much needs to do just that. I never did investigate the feasibility of putting unRAID on some type of virtualized server though, but even if that can be done, that comes with its own set of challenges.

On item 2, most people don't like the "non-live" aspect of FlexRAID. If you write data to a FlexRAID server, and don't manually invoke an rsync (parity build), then a hard drive failure between the time you wrote the data and the next automatic (scheduled) rsync could mean data loss up to the amount of data that you wrote since the last rsync. unRAID is calculating parity data live every time you write data. FlexRAID supposedly has a live version in the works, as well as a NAS version. But the current version is "snapshot".

That was almost a deal breaker for me, until I thought more about it. The most significant amount of data going on the server will be movie back-ups. If I back up a movie, forget to run rsync, and a drive failure occurs before the nightly rsync, it's not like I don't have the discs any more. The vast majority of data is still protected. At most, you should only lose data up to the amount that was written since the last rsync. Some people narrow their window of risk by scheduling rsync more often (like hourly). Another factor is that since you can run it in conjunction with WHS, WHS has it's own mirroring function built in. With WHS, you can pick and choose what data you want mirrored, and with FlexRAID you can pick and choose what data you want RAID'd. So the recommended practice is to have small frequently changed files protected by WHS mirroring, and large infrequently changed data protected by FlexRAID. And if you write a big chunk of data, just invoke an rsync.

Although I haven't seen any discussion on this, to me, it seems there would be a hidden advantage to "snapshot" RAID: it is completely transparent as far as data transfers go. Since unRAID is having to calculate parity and write to two disks for every single write, CPU cycles are being taken, and writes will be slower (as well as reads while another write is happening). Although I believe the vast majority of users are happy and get good performance, you do see the occasional "stuttering" post, so I can't help but think that the extra layer between you and the disks can sometimes introduce an additional variable that may have to be troubleshooted. FlexRAID operates more like a back-up program... while it's not running, it's just you reading/writing to the disks like normal.

One other advantage to the FlexRAID setup is that it's not restricted to protecting data on entire disks on the server. You can also include disks (or just specific folders) from other machines, as long as they are powered up when the rsync runs.

Someone may have to correct me on some of my points, but that is my understanding of the significant differences, as well as my thought process that led me to the direction I'm heading. Oh, and FlexRAID is free.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #13 of 83 Old 08-22-2009, 07:57 AM - Thread Starter
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 150
Thanks everyone.

It seems that FlexRAID is what I want. I am going to be building a machine with the parts left over when I upgrade my HTPC (yeah, I have upgrade-itis, those new Intel i5 chips just look too good!), my CPU and motherboard will suddenly be freed up and available for the storage server.

I will be using Vista Ultimate (since I have a spare copy lying around) as the OS, so the OS cost is irrelevant to me.

I am an MCSE, so I have no problems with file shares. What I currently do in my HTPC (which has 4 HDDs right now) is to redirect the My Movies folder from my OS hard drive to the hard drive with all my DVDs on it. The other two hard drives have my BRs on it, and I just put shortcuts to each movie in the DVD hard drive. I then point all my applications which need to see any movie to the My Movies folder and they can see all the movies as if they were all there. It might sound like a pain to do if you have hundreds of movies, but I started out this way, so each rip just included the one extra step of the shortcut...a few seconds.

I can simply point the My Movies folder to the FlexRaid server where the DVDs are at and ensure the shortcuts are all good still and have the same result.

Again, thanks everyone!
cybrsage is offline  
post #14 of 83 Old 08-22-2009, 08:40 AM
AVS Special Member
 
oliverjg's Avatar
 
Join Date: Sep 2006
Posts: 1,760
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
cybrsage,

it sounds like you are headed down about the same path i took a year ago.

some ideas i implemented that you might consider...

using mklink to create your shortcuts can be automated with cmd scripts. i wrote one that creates multiple "views" on drive c:.

the MyMovies meta data is stored in

C:\\MyMovies

each mkv/iso/mpg file has a folder under c:\\mymovies and the folder has a link to the file on whatever disk. ...

c:\\MyMovies\\\\movie.xxx --> x:\\_movies\\m\\movie.xxx

all of the drives with rips have a folder layout like this...

x:\\_movies
x:\\_movies\\a
x:\\_movies\\b
...
x:\\_movies\\z

collections are stored like...

x:\\dirty_harry


the script automatically goes through the folders on the drives and creates new links when files are added.

in addition to the MyMovies view. a unified folder setup is built in my videos folder. this view allows direct access to files using layouts (views) that my movies doesn't provide.

for example

videos\\_movies\\_all_mkv_files
videos\\_movies\\_all_mpg_files
videos\\_movies\\_all
videos\\_movies\\a
videos\\_movies\\dirty_harry
...

when the script runs it effectively merges the folders on all the separate drives into one under videos using mklink.
oliverjg is offline  
post #15 of 83 Old 08-22-2009, 10:41 AM
Member
 
jouyang's Avatar
 
Join Date: Oct 2003
Location: San Francisco
Posts: 93
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Willie View Post

I have been running unraid for a couple years with no problems whatsoever. I would highly recommend it.

If you have lockup issues, it is a hardware issue. Don't blame unraid because your hardware problems are hard to diagnose.

Willie

I guess you didn't read my first paragraph. If someone is building a new box with the hardwares that are known to be working with Unraid, sure go for it. If you have old hardwares, they may or may not give you a lot of headaches.

Also, this same exact hardwares have been running XP for years. Sure, sometime I have to reboot it but that's Microsoft for you. For a Linux based OS that locks up your computer for copying data over a network and you have to do hard reboot. I think that's a bit hard to swallow.

thanks,
~joy
jouyang is offline  
post #16 of 83 Old 08-22-2009, 12:40 PM
Member
 
miglo's Avatar
 
Join Date: Dec 2005
Posts: 100
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I have used Unraid for over a year now, built on the cheap mind you (<$200 not including drives), and it has been rock solid. Its a 6Tb system that has been running nonstop for over a year. I'm looking forward to dual-parity support.
miglo is offline  
post #17 of 83 Old 08-22-2009, 01:31 PM - Thread Starter
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 150
Quote:
Originally Posted by oliverjg View Post

cybrsage,

it sounds like you are headed down about the same path i took a year ago.

some ideas i implemented that you might consider...

using mklink to create your shortcuts can be automated with cmd scripts. i wrote one that creates multiple "views" on drive c:.

the MyMovies meta data is stored in

C:\\MyMovies

each mkv/iso/mpg file has a folder under c:\\mymovies and the folder has a link to the file on whatever disk. ...

c:\\MyMovies\\\\movie.xxx --> x:\\_movies\\m\\movie.xxx

all of the drives with rips have a folder layout like this...

x:\\_movies
x:\\_movies\\a
x:\\_movies\\b
...
x:\\_movies\\z

collections are stored like...

x:\\dirty_harry


the script automatically goes through the folders on the drives and creates new links when files are added.

in addition to the MyMovies view. a unified folder setup is built in my videos folder. this view allows direct access to files using layouts (views) that my movies doesn't provide.

for example

videos\\_movies\\_all_mkv_files
videos\\_movies\\_all_mpg_files
videos\\_movies\\_all
videos\\_movies\\a
videos\\_movies\\dirty_harry
...

when the script runs it effectively merges the folders on all the separate drives into one under videos using mklink.

The My Movies folder I am talking about is the one inside My Documents. I actually use MediaBrowser instead of My Movies.

mklink does sound like it is a wonderful thing, I will look into it. When I finally build my storage server I will have to redo the shortcuts, and that will make it MUCH easier.

Thanks!
cybrsage is offline  
post #18 of 83 Old 08-22-2009, 10:20 PM
AVS Special Member
 
JoshDorhyke's Avatar
 
Join Date: May 2008
Posts: 1,171
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I'd like to hear from anyone who's using flexraid and wrote some data right before having to reconstruct a failed drive. I just don't follow the Flexraid part when say you write data to drive X and then drive Y fails before you sync the parity. If Flexraid uses the data on the other drives to recontruct drive Y then the new data on drive X and the parity data will not combine correctly to reproduce drive Y. If you added say a 12gig rip to drive X right before the failure then there would be 12gig of bad data on drive Y after you recontruct it.

Sure, Flexraid could rebuild drive X if you just wrote to it and then it failed but how does it protect against a case of writing data to drive X (which, to me, seems to break the parity) right before drive Y fails?

Flexraid sure looks like someone started to create a program to emulate unRAID but just never got the live or real-time part figured out. Without the real-time aspect, it appears to me that the protection is broken until the parity is synchronized with any new data. It just strikes me that Flexraid is a "labor of love" development that hasn't yet fully matured into a product to protect many TB of data that get stored on some servers.

In the future, if Flexraid goes "live" then there might be the ability to create a server that can run other apps and also give a very cost effective way to protect all the stored data against a drive failure.

At this time I use unRAID. unRAID is a very simple OS that allows you to create a simple home build NAS device. I very much like the way it is simple and super reliable and it's really one less computer where I have to mess with the OS - just copy the files to the flash drive and go. I spend enough time messing with the HTPC to not want to also worry about the server OS. It's just cost effective way to store a lot of data. If you want fast read/write speeds or the ability to easily run other apps then you should look at other options.

Peter
JoshDorhyke is offline  
post #19 of 83 Old 08-22-2009, 10:42 PM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by JoshDorhyke View Post

I'd like to hear from anyone who's using flexraid and wrote some data right before having to reconstruct a failed drive. I just don't follow the Flexraid part when say you write data to drive X and then drive Y fails before you sync the parity. If Flexraid uses the data on the other drives to recontruct drive Y then the new data on drive X and the parity data will not combine correctly to reproduce drive Y. If you added say a 12gig rip to drive X right before the failure then there would be 12gig of bad data on drive Y after you recontruct it.

Sure, Flexraid could rebuild drive X if you just wrote to it and then it failed but how does it protect against a case of writing data to drive X (which, to me, seems to break the parity) right before drive Y fails?

Flexraid sure looks like someone started to create a program to emulate unRAID but just never got the live or real-time part figured out. Without the real-time aspect, it appears to me that the protection is broken until the parity is synchronized with any new data.

Peter

Not quite sure what exactly you are asking but I will try this example:

Drive 1 contains: 1,2,3,4,5
Drive 2 contains: 6,7,8,9,10

I create a parity on Drive 3 (PPU#1) with two DRU's (#1 for Drive 1 and #2 for Drive 2) to protect the above two drives and their data.

I then change (add/delete) the following files on each drive:

Drive 1: 1,2,4,5,11,12
Drive 2: 6,8,9,10,13,14

So each drive lost a file but gained two files. The parity was not re-synched.

Now Drive 1 has failed and I use Drive 3 (PPU#1) to recover Drive 1 (DRU#1).

When the recovery is done I get this:

Drive 1: 1,2,3,4,5
Drive 2: 6,8,9,10,13,14

I have recovered the original files but lost the added files (11,12). If you create a DRU for each drive when you created your parity, you can recover each drive individually with a single PPU without affecting the other drives.
MickeyDora is offline  
post #20 of 83 Old 08-23-2009, 07:11 AM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Josh,
I posed the same question to the author last week. Yes, you do risk losing data on the failed drive, but only up to the amount of data that has changed (edited or deleted) on the remaining drives. That's why it's really best for data that rarely changes, like dvd backups. That's why my intent is to use it in conjunction with WHS file mirroring for the more frequently changed data (which for me, is generally small files, so the duplication doesn't mean a lot of extra HD space). Check out that thread and the replies.

Mickey,
Your example isn't necessarily true, at least not by my understanding. Since you have changed files 6,8,9,&10 on drive 2, the parity data would be based on "old" data. If you were to try to rebuild drive 1 by combining those files with old parity data, it wouldn't work. Since you didn't change file#7 on drive 2, you should be able to recover some of the data on drive 1. How much depends on how big file#7 is relative to the others. So FlexRAID probably wouldn't be a good choice for someone who changes files in between rsyncs as often as your example.

So, to Josh's point, this can be a serious consideration for someone until FlexRAID live is available. If ALL you want is a file server (or you don't mind having another server for your other needs), and you are storing files that are frequently changed, unRAID would be a better choice. But if your data is relatively static (or if you can use another solution for your non-static data) FlexRAID is a viable (free) alternative that has the advantage of being able to be run on a machine that is also fulfilling other needs.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #21 of 83 Old 08-23-2009, 09:54 AM - Thread Starter
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 150
Let me see if I understand it correctly, using a real world example of my use.

Assume this:

Vista OS running FlexRaid.

1 HDD for the OS
1 HDD for parity (call it drive e) (does it have to be the largest drive?)
4 HDDs for data storage.
-(a) has DVD rips
-(b) through (c) have BR rips
-(d) has both BR Rips and monthly acronis backups of three other computers

If I rip a new BR ISO to HDD (d) and it fails prior to the resync of parity, I only lose the newly ripped ISO. If I resync immediately after the rip and then the HDD dies, I lose nothing. In the slim change the HDD fails during the resync, I might lose everything, correct?

I can easily add an acronis backup of the OS partition so it is protected.


If the Parity drive fails, I simply replace it and then resync?
cybrsage is offline  
post #22 of 83 Old 08-23-2009, 10:43 AM
Member
 
nrk7001's Avatar
 
Join Date: Sep 2005
Posts: 92
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Darin, you posts have helped get me over the edge on which file storage system I will use. This should not be as hard as it is, but I see many with the same issues I have.

Basically, I want a file server primarily for BR rips and my general media (photos and MP3s). The data does not change everyday and I could see myself using FlexRaid Sync feature nightly. No bug deal. I also was thinking of running a server so I can share my files with a few friends (either via FTP or some other easy share program).

So the WHS and FlexRaid scenario seems to fit. I have some old PC parts, and I just bought a Rosewill RSV-S8 case to hold multiple drives, and I will just build a small PC to run WHS 2008 and connect the Rosewill RSV-S8 via the eSata ports.

Anyone see any holes in my scenario? Just need a last vote of confidence.
nrk7001 is offline  
post #23 of 83 Old 08-23-2009, 10:58 AM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Darin View Post

Mickey,
Your example isn't necessarily true, at least not by my understanding. Since you have changed files 6,8,9,&10 on drive 2, the parity data would be based on "old" data. If you were to try to rebuild drive 1 by combining those files with old parity data, it wouldn't work. Since you didn't change file#7 on drive 2, you should be able to recover some of the data on drive 1. How much depends on how big file#7 is relative to the others. So FlexRAID probably wouldn't be a good choice for someone who changes files in between rsyncs as often as your example.

If you noticed I created two different DRU's; one for each drive. This isolates the protection for each drive therefore you can recreate (restore) each drive individually if you have changed the data on the other drives.

You have to remember that the snapshot is taken at the time of the parity build not at the time of the restore. So if you change the data on the other drives after the parity creation it will still have the right parity if you created different DRU's. I have not created a single DRU for all drives since the whole point is to create a DRU for each drive so you have individual protection. If you have 20 drives, you create 20 DRU's. That way you can restore each drive individually in case of a failure.

Basically, if you created different DRU's for each drive and you lose one you can recreate that particular drive. After it is restored then you want to re-synch your parity so that the new data on the other drives are added to youyr parity so you can protect the other drives with their new data. You will still lose the new data on that restored drive but its better than nothing.
MickeyDora is offline  
post #24 of 83 Old 08-23-2009, 12:45 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by cybrsage View Post

If I rip a new BR ISO to HDD (d) and it fails prior to the resync of parity, I only lose the newly ripped ISO. If I resync immediately after the rip and then the HDD dies, I lose nothing. In the slim change the HDD fails during the resync, I might lose everything, correct?...If the Parity drive fails, I simply replace it and then resync?

Yes, I would agree with all that. I'm not sure exactly what would happen if a drive failed during rsync. Though my understanding is that FlexRAID keeps track of what's changed, and only needs to build new parity on the changed/new data. So I would think the window of opportunity for that to happen would be small.

Quote:
Originally Posted by nrk7001 View Post

Anyone see any holes in my scenario?

No, you just need to make sure that external rack will work with your esata port. If it holds multiple drives and connects over a single esata port, then it's relying on esata port multiplication. Not all host ports support that. Also, my understanding is that all drives are accessed simultaneously during an rsync, so performance might suffer if there are a lot of drives being addressed over a single esata port at the same time. Though in that case, system i/o may be the bottlneck rather than port bandwidth. Regardless, if you schedule rsync at night, performance may be moot.

Also, the parity "drive" doesn't necessarily even have to be a single drive. Your parity data needs to be as large as your largest data set you want to protect (which is typically your largest drive, but doesn't have to be). But obviously, spanning your parity data across multiple drives does increase potential points of failure.

Quote:
Originally Posted by MickeyDora View Post

If you noticed I created two different DRU's; one for each drive. This isolates the protection for each drive therefore you can recreate (restore) each drive individually if you have changed the data on the other drives.

You have to remember that the snapshot is taken at the time of the parity build not at the time of the restore. So if you change the data on the other drives after the parity creation it will still have the right parity if you created different DRU's.

Then perhaps this is a question for the developer. I haven't even tried it yet (still putting the hardware together today, but one of the drives isn't here yet). But honestly, I fail to understand how your scenario would work. The parity data is based on the differences between the various DRUs. Your scenario suggests that you could have, for example, 20 drives (each a different DRU), plus one parity drive (PPU), and you could have multiple drives fail and still recover them all as long as you have your parity data. Which is a mathematical impossibility. After all, changing data on a DRU is effectively no different than deleting data (or having the entire drive crash). To rebuild a failed drive (or DRU), you need the data from the other drives (or DRUs), plus the parity data. If you remove, or even just change one DRU since the last snapshot, the "difference" between the parity data and remaining DRUs no longer adds up to the image for the failed DRU.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #25 of 83 Old 08-23-2009, 01:11 PM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Darin View Post

Then perhaps this is a question for the developer. I haven't even tried it yet (still putting the hardware together today, but one of the drives isn't here yet). But honestly, I fail to understand how your scenario would work. The parity data is based on the differences between the various DRUs. Your scenario suggests that you could have, for example, 20 drives (each a different DRU), plus one parity drive (PPU), and you could have multiple drives fail and still recover them all as long as you have your parity data. Which is a mathematical impossibility. After all, changing data on a DRU is effectively no different than deleting data (or having the entire drive crash). To rebuild a failed drive (or DRU), you need the data from the other drives (or DRUs), plus the parity data. If you remove, or even just change one DRU since the last snapshot, the "difference" between the parity data and remaining DRUs no longer adds up to the image for the failed DRU.

I think you are missing the point. The changes are made after you created the parity. The parity does not know you changed the data on multiple drives. It only knows you changed the data on the failed drive. The parity is created once and it will not be re-synched unless you do it manually.

So if you create a parity, change the data on any drive (multiple drives) and NOT re-synch the parity, once the ONE drive fails and you tell it to restore that one drive it will look at that drive alone and not the others. The parity is already created and it does not matter if the other drives have been changed since the parity was not re-synched after the data changed. The parity does not access the other drives while its restoring your failed drive.

Think of it this way: you have 10 drives, you create the parity with multiple DRU's, you change some data on all drives, then you unplug all the drives except for two (meaning they are not accessible by the PC) and then you delete all the files on the single attached drive to recreate a failure (the parity drive is still attached of course) and you tell it to restore that drive with it's own DRU then it will restore the data with no issues, even if the other drives are not attached to the PC.

It will restore the failed drive with the data that was there before the data change. If you restore that drive and later another drive fails, as long as you do NOT re-synch, the parity will be able to restore the old data to any of the failed drives....ONE at a time. You will NOT be able to restore the new data on ANY drives since you did NOT re-synch.

That is why it is called "snapshot". It restores the drives with the data that was created with the snapshot taken at the time not with what is on them now. The parity can replace multiple drives but not at the same time. One drive fails, restore it. The parity still remains intact. A different drive fails, restore it. It will restore ONE drive (at a time) with whatever data it had on it when the parity was created. If two drives fail at the SAME time then you have lost the data on those TWO drives only as the parity cannot restore two simultaneous drives.

The best way to keep your parity up to date is to schedule a re-synch every night or in my case since my re-synch can take several days, once a week or two.
MickeyDora is offline  
post #26 of 83 Old 08-23-2009, 02:14 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by MickeyDora View Post

I think you are missing the point. The changes are made after you created the parity.

But that's exactly MY point.
Quote:


It only knows you changed the data on the failed drive.

It doesn't even know that much.
Quote:


So if you create a parity, change the data on any drive (multiple drives) and NOT re-synch the parity, once the ONE drive fails and you tell it to restore that one drive it will look at that drive alone and not the others.

It HAS to look at the other drives. That's how parity allows RAID to do the magic it does with just a single parity set and theoretically unlimited number of data sets.
Quote:


Think of it this way: you have 10 drives, you create the parity with multiple DRU's, you change some data on all drives, then you unplug all the drives except for two (meaning they are not accessible by the PC) and then you delete all the files on the single attached drive to recreate a failure (the parity drive is still attached of course) and you tell it to restore that drive with it's own DRU then it will restore the data with no issues, even if the other drives are not attached to the PC.

I think you have a significant misunderstanding of how RAID uses parity to create redundancy for a theoretically unlimited number of data sets with just single parity set. Think about this for a second: if what you said was true, you could compress an unlimited amount of data into one drive. If you had 100 drives, and one parity disk, and you did what you said above with just drive#1 and the parity drive, and if you were able to recreate drive #1 with just the parity data available, you should then be able to do the same thing with drives #2-100. In other words, your description would allow you to have EVERY drive fail except the parity disk, and you'd be able to recreate all of them. What you are describing is unlimited compression, even of practically uncompressible data (like movies).

Think of the case of 1+1+0+1=3. 3 is your parity data. If you remove any one of the single digits, you can re-create the others. If you remove the first "1", you still know it's "1" because with the other digits, and the answer, you can re-create it. It HAS to be 1, because your parity is 3, and your other three data sets only add up to 2. But if you also remove the "0", how do you know what the first digit was? was it 0, and the 3rd digit 1, or the other way around? You have no way of knowing. With just one parity set, you can only lose one data set and still be able to re-create it. And the only reason you CAN recreate it is because the parity is derived from ALL the data sets, so with the parity set and the other data sets, you can calculate the missing data sets. But you need ALL the other pieces to do it.

Quote:


That is why it is called "snapshot". It restores the drives with the data that was created with the snapshot taken at the time

Correct. But that's ALL it means. It's a snapshot in time. So if your data changes, your parity is out of date. If your changes are only to one data set, you can re-create what it was at the time of the snapshot from the parity data and the other data sets. But if two ore more of your data sets changed since the snapshot, you can't recreate them. At least not all of them.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #27 of 83 Old 08-23-2009, 02:46 PM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Darin View Post

Think about this for a second: if what you said was true, you could compress an unlimited amount of data into one drive. If you had 100 drives, and one parity disk, and you did what you said above with just drive#1 and the parity drive, and if you were able to recreate drive #1 with just the parity data available, you should then be able to do the same thing with drives #2-100. In other words, your description would allow you to have EVERY drive fail except the parity disk, and you'd be able to recreate all of them. What you are describing is unlimited compression, even of practically uncompressible data (like movies).

You do not understand how unRAID or FlexRAID works. This is EXACTLY how it works.

One drive that is the same size as your largest drive in the pool can then restore any ONE of the drives on your pool. You are saying that it can restore multiple drives at ONCE. It cannot. Only one drive at a time. After one drive is restored if another drive fails separately then it can restore that drive. It could theoretically restore all the drives if they fail ONE AT A TIME (and separately) and you restore each one individually. But only with the data that was protected in the original snapshot.

Quote:


But if two ore more of your data sets changed since the snapshot, you can't recreate them. At least not all of them.

You can restore any of them to the previous state at the time of the snapshot. You will lose all new data. The parity does not care what you add or change as it sees the original data only and restores it.

My current system has 29 drives (1TB & 1.5TB drives) and I have a single 1.5TB drives doing the parity with 29 DRU's. If one drive fails, and they have, I can restore it. I have lost a second drive a week later and it restored that one with the same parity even though I had changed the data in multiple drives. I just lost the new data (on the failed drive) and nothing else since I forgot to re-synch that week. So in the end I lost a total of 4 movies that were added to that one drive after the original parity.

You keep saying that the parity changes as the data changes on the drives. It does not. The parity does not care as that 1.5TB worth of parity has all the information of all the drives in the pool (yes, even 40TB worth of data in my pool). It only cares that you want it to restore the one failed drive to the state it was when you created the parity.

I can tell you that the more drives you have the longer the parity takes and the longer it takes to restore your drive. My last failed drive took almost 72 hours to restore.

But in the immortal words of Jar-Jar "Me gives up".
MickeyDora is offline  
post #28 of 83 Old 08-23-2009, 03:08 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by MickeyDora View Post

You are saying that it can restore multiple drives at ONCE. It cannot. Only one drive at a time.

No, I did not say at once. I don't care if it's at once or one at a time. You NEED the other data sets to create the missing one. Otherwise, you are simply saying that the parity data is a COMPLETE backup of every singe drive. It's not. It's a set of data that can be used to recreate a missing piece, with the requirement that the other pieces are available. If it were a complete backup, then it's size would have to grow with the number of data sets you have. It doesn't. You can have as many data sets as you want, the only requirement is that your parity set must be as large as the biggest data set.

Quote:


My current system has 29 drives (1TB & 1.5TB drives) and I have a single 1.5TB drives doing the parity with 29 DRU's. If one drive fails, and they have, I can restore it.

Quote:


I have lost a second drive a week later and it restored that one with the same parity even though I had changed the data in multiple drives. I just lost the new data (on the failed drive) and nothing else since I forgot to re-synch that week.

Sorry, but I'm skeptical. You don't have rsynch scheduled? If not, you're sure the other drives changed in that period?
Quote:


The parity does not care as that 1.5TB worth of parity has all the information of all the drives in the pool (yes, even 40TB worth of data in my pool).

You don't find it unusual that 1.5TB can hold 40TB of "incompressible" data?

Quote:


But in the immortal words of Jar-Jar "Me gives up".

Yes, we'll have to agree to disagree.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #29 of 83 Old 08-23-2009, 03:32 PM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Darin View Post

No, I did not say at once. I don't care if it's at once or one at a time. You NEED the other data sets to create the missing one. Otherwise, you are simply saying that the parity data is a COMPLETE backup of every singe drive. It's not. It's a set of data that can be used to recreate a missing piece, with the requirement that the other pieces are available. If it were a complete backup, then it's size would have to grow with the number of data sets you have. It doesn't. You can have as many data sets as you want, the only requirement is that your parity set must be as large as the biggest data set.

The other data sets are created on the original parity creation. Once again, the parity is the size of the largest hard drive and it holds the parity for all drives. It does not grow as your pool grows unless you add a hard drive that is bigger than your current largest drive. I have kept all my drives to 1/1.5TB but I will soon add a 2TB drive so I will have to buy an additional 2TB drive to replace my parity drive when I add a 2TB drive to my pool.


Quote:
Originally Posted by Darin View Post

Sorry, but I'm skeptical. You don't have rsynch scheduled? If not, you're sure the other drives changed in that period?

I have an auto re-synch scheduled but I forgot to leave the server on the weekend so the scheduled synch did not take place. So when I re-booted and I lost the second drive and realized by the parity date that the scheduled re-synch had not been done.

I am positive they changed since I delete movies that I don't like and then add new ones in the free space created on that drive. On that particular drive I had deleted 5 movies and added 4 since the last parity re-synch. So when I restored it I ended up with the 5 old movies (the ones I deleted) and lost the 4 new ones.

I had also added a total of 10 movies to other drives in my pool and deleted 12 or 13 movies in that time.

Quote:
Originally Posted by Darin View Post

You don't find it unusual that 1.5TB can hold 40TB of "incompressible" data?

That is how unRAID, FlexRAID, and DisPARITY (the command line application that does the same thing as the other two) work. unRAID is limited by their OS to 20 drives (the current beta, while the current release has a 16 drive limit). Since FlexRAID sits on top of a current OS (Windows, or Linux) it does not have a drive limit. I am at 29 drives (1 OS drive, 29 data drives, 1 parity drive).

The mathematical process to create the parity grows with each additional drive so it will take longer when you add more drives. I/O throughput is also very important so the faster your I/O is the faster the parity calculation is.

If you have ever used QuickPAR then you know how unRAID, and FlexRAID work. It's not actually a RAID system but more of a parity system, just like QuickPAR but in a whole bigger scale. QuickPAR can create recovery parities for smaller files and even a whole drive but it would take forever as it was never meant to do that type of work.
MickeyDora is offline  
post #30 of 83 Old 08-23-2009, 04:55 PM
AVS Special Member
 
MickeyDora's Avatar
 
Join Date: Feb 2005
Posts: 1,354
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
You just jinxed me. One my pool drives is refusing to mount. I have tried various things and the drive is still not showing up.

So I slapped another 1TB drive and the recovery has started. So I guess I will have to wait 3 days to see the drive again. snif, snif.

If you have never tried FlexRAID, load it on your system and play with it. That is how I learned to use it. I tried it with 3 drives of junk and I could not get it to fail with various scenarios.

It is quite easy to use once you get the hang of it.

One tip I can give you is that if you don't chnage the data on your drives too often then you do not need to re-synch that often. The more you change data the more often you should re-synch. I should re-synch once a day but it would tie up my system too much so I do it once a week on weekends since I never watch movies on weekends.
MickeyDora is offline  
Reply Home Theater Computers

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off