SnapRAID: An Open Source alternative to UnRAID and FlexRAID - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #1 of 214 Old 04-17-2011, 09:21 AM - Thread Starter
Member
 
Ben_in_COSprings's Avatar
 
Join Date: Sep 2007
Posts: 32
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Has anybody tried this: http://snapraid.sourceforge.net/compare.html

I've long wanted something that 1) runs on windows, and 2) is open source, to compete with the likes of UnRAID (Linux-only) and FlexRAID (not open source). SnapRAID looks like it might fit the bill.
Ben_in_COSprings is offline  
Sponsored Links
Advertisement
 
post #2 of 214 Old 04-17-2011, 09:39 AM
AVS Special Member
 
whiteboy714's Avatar
 
Join Date: Feb 2009
Posts: 4,595
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 16
Sounds cool, always nice to have more options. I use flex raid, seems to be running fine but I have not had a failure yet.

This one needs some more time to mature before I jump in on it.
whiteboy714 is offline  
post #3 of 214 Old 04-17-2011, 09:43 AM
AVS Special Member
 
jrwalte's Avatar
 
Join Date: Jun 2007
Posts: 2,537
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 11
Quote:
Originally Posted by whiteboy714 View Post

Sounds cool, always nice to have more options. I use flex raid, seems to be running fine but I have not had a failure yet.

This one needs some more time to mature before I jump in on it.

I restored 900gb with flexraid once.
jrwalte is offline  
post #4 of 214 Old 04-17-2011, 09:53 AM
AVS Special Member
 
whiteboy714's Avatar
 
Join Date: Feb 2009
Posts: 4,595
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 16
Quote:
Originally Posted by jrwalte View Post

I restored 900gb with flexraid once.

Good to hear.
whiteboy714 is offline  
post #5 of 214 Old 04-18-2011, 12:29 AM
AVS Special Member
 
hlkc's Avatar
 
Join Date: Feb 2004
Posts: 2,758
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 11
Sounds very good but I agree it needs some time to prove the reliability.
hlkc is offline  
post #6 of 214 Old 04-18-2011, 10:05 AM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Hi, I'm the author of SnapRAID.

If someone of you guys try it, please report your impression. I'm interested to both positive and negative ones

Simply keep in mind that it's an application written mainly for the Unix environment. So, no GUI for Windows users. It's only command line.

On the other hand, a lot of effort was put to have high reliability from the first version. I'm already using it in my Linux system with 7 HD of 2TB.

Ciao,
Andrea
amadvance is offline  
post #7 of 214 Old 04-18-2011, 11:14 AM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Create a GUI and I will try it out. While I can use the command line, I rather dislike it.
cybrsage is offline  
post #8 of 214 Old 04-18-2011, 11:36 AM
AVS Special Member
 
jrwalte's Avatar
 
Join Date: Jun 2007
Posts: 2,537
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 11
I too wouldn't even try it until at least a GUI existed. Worked with command line for too long with FlexRAID and he finally was able to integrate a GUI with his new 2.0 release. I'm not going back to that But keep up the work. It's always good to have multiple choices.
jrwalte is offline  
post #9 of 214 Old 04-18-2011, 05:07 PM
AVS Special Member
 
whiteboy714's Avatar
 
Join Date: Feb 2009
Posts: 4,595
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 16
Quote:
Originally Posted by amadvance View Post

Hi, I'm the author of SnapRAID.

If someone of you guys try it, please report your impression. I'm interested to both positive and negative ones

Simply keep in mind that it's an application written mainly for the Unix environment. So, no GUI for Windows users. It's only command line.

On the other hand, a lot of effort was put to have high reliability from the first version. I'm already using it in my Linux system with 7 HD of 2TB.

Ciao,
Andrea

Ouch only command line huh. That sucks, don't think you will get many folks jumping on it using windows.
whiteboy714 is offline  
post #10 of 214 Old 04-18-2011, 05:44 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by whiteboy714 View Post

Ouch only command line huh. That sucks, don't think you will get many folks jumping on it using windows.

But on the other hand, command-line offers some flexibility that sometimes takes a lot of time to get all programmed into a nice GUI. I've been a long time proponant of FlexRAID, and have been using command-line just because the GUI client doesn't have the flexibility I need for a big WHS implementation. Now, Brahim is only working on 2.0, which is GUI only. There are some issues that need fixing in v1, but he appears to not be working on it any more. I like a nice GUI as much as the next guy, but I'll suffer through some painful learning curve in a command-line if that means it has all the features I need. I also like that this is parity only. There are a lot of things I'd like to see in software like this, but it seems that Brahim needs to spend so much time on FlexVIEW that he can't really focus on improving the core data protection functionality.

All that being said, I need to see more documentation on this to understand it, and feel comfortable in trying it out. I'd also wait for at least dual parity before trying something else out. I'm up to 17 drives in my pool, so single parity just seems too weak at this point.

Looking forward to learning more.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #11 of 214 Old 04-18-2011, 06:21 PM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Agreed, more choices is better, keep chugging along!
cybrsage is offline  
post #12 of 214 Old 04-18-2011, 07:10 PM
Newbie
 
spartan711's Avatar
 
Join Date: Apr 2011
Posts: 6
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
can we get some more details on how it works? Like a link or something? Does it protect data in the same way flexraid does? i.e. doesn't touch any files but creates parity for them on a separate disk?
spartan711 is offline  
post #13 of 214 Old 04-19-2011, 09:34 AM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by whiteboy714 View Post

Ouch only command line huh. That sucks, don't think you will get many folks jumping on it using windows.

Sorry guys. No planned GUI for now. The rationale about it is that only a command line tool can provide the control required to use it in companion with other services.
I mean encryption, virtual filesystem views, SMART/Power control, and any scripting tools to automatize all these operations.

I understand that this not going to please Windows users, but the scope of SnapRAID is to be integrated in an automatic system with such tools, and with a GUI you cannot do it.

Anyway, consider that in normal use you have only to use two commands: "sync" to update the parity, and "check" to check it. It's not so difficult
amadvance is offline  
post #14 of 214 Old 04-19-2011, 09:44 AM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by spartan711 View Post

can we get some more details on how it works? Like a link or something? Does it protect data in the same way flexraid does? i.e. doesn't touch any files but creates parity for them on a separate disk?

To get more details about SnapRAID you can check the snapraid.txt file in the Windows download. It provides a lot of information.

In short, SnapRAID read your data (without changing it in any way) and it writes a parity file in a dedicated disk.

The main difference with other solutions is that SnapRAID stores also integrity checksums at block level. The last time I checked, all the other solutions, like unRAID/FlexRAID/disParity, had only a file level checksum or none at all. (please correct me if I'm wrong here, as I found a bit difficult to get this information from their documentation).

With a block level checksum, you can detect when the HD is reading garbage without returning an error, because each read block has its checksum. This is important when computing parity and recovering. If in these operations you silently read garbage, the whole parity concept fails.

It's like what all the most modern filesystems, like ZFS and Btrfs, are doing. They provide block checksums integrated with RAID 1/5/6 capabilities. SnapRAID does the same, but at higher level.

Another main point of SnapRAID is to be robust. You can stop/kill/terminate it at any time, and expect to always have consistent data. You can also power-down the machine, but do not try it .
amadvance is offline  
post #15 of 214 Old 04-19-2011, 10:12 AM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
After reading what little documentation I could find on it, I have a couple of questions:

It says the checksumming works on a block-level, rather than a file level. What about the parity in general? Is it based on a list of files, or the blocks of entire disks? Is defragmenting irrelevant to the parity (because it only cares about the files, not where the data resides), or does moving data cause the parity to become out of date?

Based on the example, it appears that the data is based on pointing to a specific disk. Is it not possible to define a data unit as a collection of paths like FlexRAID? If, for example, I want to protect the data in the movies, music, and photos folders on disk one, can they be defined that way, or do you have to designate the entire disk, then exclude everything BUT those three folders?

Have you actually used FlexRAID? If so, do you have any comparisons regarding performance (speed) of various operations?

I know these questions may be more appropriate for your forum, but there aren't any posts there yet, so I figured I'd post where there's already some dialogue.

Thanks!

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #16 of 214 Old 04-19-2011, 12:10 PM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Darin View Post

After reading what little documentation I could find on it, I have a couple of questions:

It says the checksumming works on a block-level, rather than a file level. What about the parity in general? Is it based on a list of files, or the blocks of entire disks? Is defragmenting irrelevant to the parity (because it only cares about the files, not where the data resides), or does moving data cause the parity to become out of date?

Based on the example, it appears that the data is based on pointing to a specific disk. Is it not possible to define a data unit as a collection of paths like FlexRAID? If, for example, I want to protect the data in the movies, music, and photos folders on disk one, can they be defined that way, or do you have to designate the entire disk, then exclude everything BUT those three folders?

Have you actually used FlexRAID? If so, do you have any comparisons regarding performance (speed) of various operations?

In SnapRAID parity is at file level. Disk de/framentation is irrelevant.

A not obvious feature is that also renaming or moving files inside the same disk doesn't trigger a new parity recomputation. Only the new name/path of the file is updated. This is achieved identifying the files using their inode/FileIndex. For who don't know, and inode it's simply a number that identify the file in the filesystem, just like an alternate pathname.

About including paths, you don't need to include the whole disk. You can specify a subdirectory of the disk to backup, and inside it, exclude some files or directories using globbing chars, like *.bak, tmpdir/ and so on.

I have only tried FlexRAID one time, so, I do not have speed comparison. Anyway, In my Linux system with SnapRAID I have a throughput of 350 MB/s when syncing/checking 7 disks, but all the data is also encrypted with AES256, and the encryption is the main bottleneck. No idea how this translates to Windows.

If you want more details, you can read the snapraid.txt file inside the Windows download. Or simply ask here or where you like

PS
Some of my messages are delayed by moderation because I'm a new user. Sorry if you get late answers.
amadvance is offline  
post #17 of 214 Old 04-19-2011, 12:55 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Thanks for the reply, and no worries on the delay.
Quote:
Originally Posted by amadvance View Post
About including paths, you don't need to include the whole disk. You can specify a subdirectory ...
Thanks. But what about specifying multiple subdirectories? For a snapshot, I think it's important to exclude any folders that contain data that could be frequently changed. But it's much easier to only include folders that you know are relatively static, as that prevents data from new folders creeping in that you didn't originally plan for. Some of us use FlexRAID in conjunction with WHS. WHS represents a disk as a path under C. So for example, a data disk for me might be c:\\fs\\1a. But there is lots of data within that disk that is transient, some of which isn't even part of the user shares (and therefore not under my direct control). So the data I would want to protect would look something like {c:\\fs\\1a\\de\\Shares\\Movies;c:\\fs\\1a\\de\\Shares\\Musi c;c:\\fs\\1a\\de\\Shares\\Photos}, and that would repeat for each drive in my system, with only the "1a" changing. Is that possible with SnapRAID? It wasn't clear to me if there was any format for a collection of paths to represent a single "disk" (or data risk unit, as FlexRAID calls it, since it doesn't necessarily have to be a single physical disk).

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #18 of 214 Old 04-19-2011, 02:50 PM
Senior Member
 
micksh's Avatar
 
Join Date: Feb 2008
Location: San Jose, CA, US
Posts: 270
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
Hi, amadvance

This is just in time, your solution can be very interesting for WHS v1.
I am long time FlexRaid user, I'm cuttently on 1.4 beta 7 but it is going to expire in a few days. And it seems that Brahim excluded some things from final release that are vital for WHS. So, I will have to find another solution.

Quote:
Originally Posted by Darin View Post
But what about specifying multiple subdirectories? For a snapshot, I think it's important to exclude any folders that contain data that could be frequently changed. But it's much easier to only include folders that you know are relatively static, as that prevents data from new folders creeping in that you didn't originally plan for. Some of us use FlexRAID in conjunction with WHS. WHS represents a disk as a path under C. So for example, a data disk for me might be c:\\fs\\1a. But there is lots of data within that disk that is transient, some of which isn't even part of the user shares (and therefore not under my direct control). So the data I would want to protect would look something like {c:\\fs\\1a\\de\\Shares\\Movies;c:\\fs\\1a\\de\\Shares\\Musi c;c:\\fs\\1a\\de\\Shares\\Photos}, and that would repeat for each drive in my system, with only the "1a" changing. Is that possible with SnapRAID? It wasn't clear to me if there was any format for a collection of paths to represent a single "disk" (or data risk unit, as FlexRAID calls it, since it doesn't necessarily have to be a single physical disk).
Yes, specifying multiple paths for each disk is very important. I believe, currently, with SnapRAID we will have to include full disks like
disk d1 c:\\fs\\1A\\de\\Shares\\
disk d2 c:\\fs\\1B\\de\\Shares\\


and then specify exclude paths separately like
exclude c:\\fs\\1A\\de\\Shares\\Scrap1
exclude c:\\fs\\1A\\de\\Shares\\Scrap2
exclude c:\\fs\\1B\\de\\Shares\\Scrap1
exclude c:\\fs\\1B\\de\\Shares\\Scrap2


Andrea, please correct me if I'm wrong. Is there limitation on number of exclude entries? Can some exclude paths be present in config but missing on hard drive?

I have few other questions:

- Block-level checksum is an addition to parity, right? Can it happen that there will be no room for parity? For example, if I have 2TB parity disk and 2TB data disk that is filled to the end.

- In some cases when one disk is close to be full WHS may move data from that disk to another. So, file from c:\\fs\\1A\\de\\Shares\\S1\\file1.ext will become c:\\fs\\1B\\de\\Shares\\S1\\file1.ext
Will this break parity? (We can't control this and BTW, this was handled in FlexRaid, Brahim developed some code especially for such case)

- Can I specify restore path different to what failed disk was?
For example, if my failed disk was c:\\fs\\1A\\de\\Shares\\ and I replace it with a new disk it will become c:\\fs\\1C\\de\\Shares\\
Ideally, I need to be able to restore files into arbitrary path of my choice, and then copy them to WHS pool in order to preserve WHS tombstone integrity.

- Memory requirements and block size. From documentation:
For example with 6 disk of 2 TiB and a block size of 256 KiB (1 KiB = 1024 Bytes) you have:
RAM = (6 * 2 * 2^40) * 24 / (256 * 2^10) = 1.1 GiB

That means it will not fit into 2GB address space if I have 12 2TB disks, right? I will have to increase block size. And then what happens if I have a lot of small files?

Thanks
micksh is offline  
post #19 of 214 Old 04-20-2011, 11:15 AM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by micksh View Post

Andrea, please correct me if I'm wrong. Is there limitation on number of exclude entries? Can some exclude paths be present in config but missing on hard drive?

Your rules are almost correct. There isn't the need to repeat the path of the disk. Something like the following is enough:

disk d1 c:\\fs\\1A\\de\\Shares\\
disk d2 c:\\fs\\1B\\de\\Shares\\
exclude Scrap1/
exclude Scrap2/


Also, there is no limit of exclude rules, and you can exclude also not existing paths, use globbing chars like * and ?, single dir/file name, or complete paths.

Quote:


- Block-level checksum is an addition to parity, right? Can it happen that there will be no room for parity? For example, if I have 2TB parity disk and 2TB data disk that is filled to the end.

The checksum are placed in the another file called "content", and not in the "parity" file. If you place it in the parity disk, it reduces the space available for parity.
But in my 7x2TB array is 2GB. Not really a problem. In case, you can put it in another location and not in the parity disk.

Quote:


- In some cases when one disk is close to be full WHS may move data from that disk to another. So, file from c:\\fs\\1A\\de\\Shares\\S1\\file1.ext will become c:\\fs\\1B\\de\\Shares\\S1\\file1.ext
Will this break parity? (We can't control this and BTW, this was handled in FlexRaid, Brahim developed some code especially for such case)

If a file is moved to another disk, the parity has to be recomputed. Theoretically you can skip the computation if the allocated parity of the file is not yet used in the destination disk, but I do not see a general solution to always avoid it.
Anyway, it could be an interesting optimization for the cases where it's possible.

Quote:


- Can I specify restore path different to what failed disk was?
For example, if my failed disk was c:\\fs\\1A\\de\\Shares\\ and I replace it with a new disk it will become c:\\fs\\1C\\de\\Shares\\
Ideally, I need to be able to restore files into arbitrary path of my choice, and then copy them to WHS pool in order to preserve WHS tombstone integrity.

If you want to restore in a different disk, you need only to change the disk path in the configuration file.

Quote:


- Memory requirements and block size. From documentation:
For example with 6 disk of 2 TiB and a block size of 256 KiB (1 KiB = 1024 Bytes) you have:
RAM = (6 * 2 * 2^40) * 24 / (256 * 2^10) = 1.1 GiB

That means it will not fit into 2GB address space if I have 12 2TB disks, right? I will have to increase block size. And then what happens if I have a lot of small files?

If you cannot use a Windows x64 version, yes. You are limited by the Windows 2GB limit. There is a Windows boot option to increase it to 3GB that could be useful in your case.

Otherwise you have to increase the block size and you are going to waste some space in the parity that may fill-up before the data disk.
To get an estimation of the wasted space, you need to count the number of files in each disk. You can approximatively assume that in each disk you lose half of the block size for each file.
For example, if in a disk you have 10000 files, and the block size is 256KB, you lose approximatively 1.28 GB in this disk.

An obvious solution is to use a little bigger disk for the parity.
amadvance is offline  
post #20 of 214 Old 04-20-2011, 11:23 AM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Darin View Post

So the data I would want to protect would look something like {c:\\fs\\1a\\de\\Shares\\Movies;c:\\fs\\1a\\de\\Shares\\Musi c;c:\\fs\\1a\\de\\Shares\\Photos}, and that would repeat for each drive in my system, with only the "1a" changing. Is that possible with SnapRAID? It wasn't clear to me if there was any format for a collection of paths to represent a single "disk" (or data risk unit, as FlexRAID calls it, since it doesn't necessarily have to be a single physical disk).

SnapRAID doesn't allow to specify a collection of paths to represent a disk. You can specify only a single path.

Anyway, in your case this is not a problem because all your data is in a single directory hierarchy.
You need only to define some rules to exclude the files or directories you do not want to be included in the parity computation.

For an example of such rules, see the previous message.
amadvance is offline  
post #21 of 214 Old 04-20-2011, 11:43 AM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by amadvance View Post

You need only to define some rules to exclude the files or directories you do not want to be included in the parity computation.

Yes, I understand that's an option. The problem with that is it's just more complex (there are more paths to exclude than inlcude), and every time someone creates a new share, you have to go back and specifically exclude it. If you just want to include certain things (so you don't include anything that might change frequently), it's much easier to say "I want a and b and c", rather than "I want everything but d and e and f and g and h and i", as well as checking every day to make sure someone in the house didn't add some new letters of the alphabet that you never heard of before.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #22 of 214 Old 04-20-2011, 01:23 PM
Advanced Member
 
Calcvictim's Avatar
 
Join Date: Nov 2007
Posts: 859
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10
will snapraid employ other tools, for example to read and monitor SMART data, will you use something like smartmontools or will you rewrite all the scsi commands into snapraid itself?
Calcvictim is offline  
post #23 of 214 Old 04-20-2011, 01:39 PM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Darin View Post

The problem with that is it's just more complex (there are more paths to exclude than inlcude), and every time someone creates a new share, you have to go back and specifically exclude it.

Yes. It could be an interesting addition for the next version. I'll add it at the TODO list.
amadvance is offline  
post #24 of 214 Old 04-20-2011, 01:42 PM
Member
 
amadvance's Avatar
 
Join Date: Apr 2011
Posts: 35
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Calcvictim View Post

will snapraid employ other tools, for example to read and monitor SMART data, will you use something like smartmontools or will you rewrite all the scsi commands into snapraid itself?

SnapRAID only takes care of the parity computation. No SMART control, no power control, no encryption and no virtual filesystem view. All these services are better provided by other tools.
amadvance is offline  
post #25 of 214 Old 04-20-2011, 02:36 PM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Quote:
Originally Posted by amadvance View Post

Sorry guys. No planned GUI for now.

Oh well, not for me then. Best of luck to you, though!
cybrsage is offline  
post #26 of 214 Old 04-20-2011, 06:52 PM
Senior Member
 
micksh's Avatar
 
Join Date: Feb 2008
Location: San Jose, CA, US
Posts: 270
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
Quote:
Originally Posted by amadvance View Post

If a file is moved to another disk, the parity has to be recomputed. Theoretically you can skip the computation if the allocated parity of the file is not yet used in the destination disk, but I do not see a general solution to always avoid it.
Anyway, it could be an interesting optimization for the cases where it's possible.

My guess is that when FlexRaid doesn't find some file on disk where it used to be it tries to look for it in the same folder on other disks. But, I'm not sure. It's probably safe thing to do if you have file checksum.

This is relevant only for WHS v1 but I think in Windows world WHS is the most logical platform for SnapRAID.
It may be relevant for WHS 2011 too, right now there are at least two Drive Extender replacements being developed for WHS 2011.

Quote:
Originally Posted by amadvance View Post

An obvious solution is to use a little bigger disk for the parity.

I bought a bunch of 2TB disks and using 3TB disks is problematic for me. I believe there are other people in the same situation.

Would it be possible to add multiple locations for parity? So when you fill one location you continue in another? I guess, it's similar to what Darin is asking for data disks.
I don't know about other platforms but WHS tends to fill one disk to the end and then switch to another one. So there will be almost full disks.


PS. Don't understand why people feel a need to point to missing GUI. It's a tool in development. Functionality and stability first. Then someone can come and write GUI.

For Windows until recently there were no such tools with GUI anyway. BTW, FlexRAID added GUI not so long ago and it lost flexibility of configuration files, which for me currently is a dealbreaker. IMHO, data safety is more important than GUI.
As I see it, we are talking about editing few lines in notepad and putting couple of shortcuts to desktop - sync and check.
micksh is offline  
post #27 of 214 Old 04-20-2011, 07:45 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by micksh View Post

My guess is that when FlexRaid doesn't find some file on disk where it used to be it tries to look for it in the same folder on other disks. But, I'm not sure.

I don't know what he did internally either, he said he took care of it with no detail. But my guess would be, if something in a typical WHS path (like c:\\fs\\\\DE\\Shares) isn't found, it simply looks for it in d:\\Shares. Regardless of where it got moved to, it should be found there.

Quote:
BTW, FlexRAID added GUI not so long ago and it lost flexibility of configuration files, which for me currently is a dealbreaker. IMHO, data safety is more important than GUI.
As I see it, we are talking about editing few lines in notepad and putting couple of shortcuts to desktop - sync and check.

+1
IF a GUI can provide all the functionality I need, then I'd certainly prefer it over manual typing. But in the case of FlexRAID, the GUI doesn't address all my needs like the command line client did. So unless Brahim can incorporate those into the GUI (or provide a way to edit the GUI's settings), I will have to look elsewhere.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #28 of 214 Old 04-22-2011, 02:08 PM - Thread Starter
Member
 
Ben_in_COSprings's Avatar
 
Join Date: Sep 2007
Posts: 32
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I just did a little testing on Windows:

Test conditions:
Computer used is an Intel i3 530 2.9 GHz w/ 4 GB of RAM, running Windows 7 x32. Data drives are 6 WD20EARS/WD20EADS drives, NTFS formatted, with 2 m2ts files, with a total of about 50 GB per disk. These all reside in a Rosewill RSV-8 device, and which is attached to my computer via dual eSATA connections. My contents and parity files are on a 7200 RPM 1 TB drive, which is attached directly via SATA to my motherboard.

Test:
Sync Function: It took about 35 minutes to generate the parity file, which ended up being about 54 GB. The contents file was 40 MB. I ran with the default block size, and the program's debugging output indicated that I was transferring about 110 MB/s.

Check Function: I started this, and it looked like it would also take about 35 minutes. I didn't feel like waiting for it to finish so I killed it after a couple of minutes. It looks like a Check forces a read of all files, regardless of whether their timestamps indicate that they've been changed since the last sync/check.

Fix Function: I manually deleted one of the m2ts files (A Clockwork Orange!), and the fix correctly detected its absence, and automatically rebuilt it. The entire process took about 35 minutes. I have not been able to do a binary diff yet, but the file sizes do match.

Based on these results, I anticpate that it would take about 24 hours (give or take some error) to build a partity file for my 6 fully-loaded 2 TB drives.
Ben_in_COSprings is offline  
post #29 of 214 Old 04-22-2011, 07:36 PM
Newbie
 
Matthi4s's Avatar
 
Join Date: Nov 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
After losing 500GB of data I've been looking for something that offers data protection while still being able to used variable sized hard drives.

I have a lot of 720p content that I am slowly replacing with 1080p content. The way I go about it is to just copy the new file over top of the old file.

The file name stays the same so does SnapRAID detect this change in the file?
Matthi4s is offline  
post #30 of 214 Old 04-22-2011, 07:50 PM - Thread Starter
Member
 
Ben_in_COSprings's Avatar
 
Join Date: Sep 2007
Posts: 32
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Matthi4s View Post

The file name stays the same so does SnapRAID detect this change in the file?

Yes. It will detect pretty much any change in the file.
Ben_in_COSprings is offline  
Reply Home Theater Computers

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off