SnapRAID: An Open Source alternative to UnRAID and FlexRAID - Page 7 - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #181 of 214 Old 03-20-2012, 07:35 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by MichaelZ View Post

After running sync with NO errors, I ran snapraid check and after 15 minutes I saw 5 different files pop-up with byte differences I stopped the check at that point. I then cmp the "bad files" against the original files (they were the same) then I started snapraid fix, to see what it fixed (I thought it would be the raid file).

I'm not sure why you capitalize "NO" errors. sync generates parity and checksum data, it does not verify or check anything.

Then you ran check and it found 5 files that did not match the checksum data. So obviously if you run 'snapraid fix', it will try to repair those 5 files. But since you had an independent verification that there was nothing wrong with those files, it made no sense to try to fix them!

I'm not sure what you mean by "raid file". Do you mean the parity file? The only way to change the parity file is to run 'snapraid sync'. You will not change the parity file with 'snapraid fix' -- that will change data files.
jim2100 is offline  
Sponsored Links
Advertisement
 
post #182 of 214 Old 03-20-2012, 08:19 PM
AVS Special Member
 
MichaelZ's Avatar
 
Join Date: Sep 2000
Posts: 1,480
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 6 Post(s)
Liked: 13
Quote:
Originally Posted by jim2100 View Post

I'm not sure why you capitalize "NO" errors. sync generates parity and checksum data, it does not verify or check anything.

Then you ran check and it found 5 files that did not match the checksum data. So obviously if you run 'snapraid fix', it will try to repair those 5 files. But since you had an independent verification that there was nothing wrong with those files, it made no sense to try to fix them!

I'm not sure what you mean by "raid file". Do you mean the parity file? The only way to change the parity file is to run 'snapraid sync'. You will not change the parity file with 'snapraid fix' -- that will change data files.

I meant sync did not report any errors, i.e. it finished it job. Yes, I knew five files were ok, but I wanted to see what it was going to fix. As I stated earlier, I've written a program similar to this one and it would check the checksums against the file(s) to determine what was broken before fixing anything - could be parity or some other combinations. I see that this program assumes the file is broke (it wasn't) and fixes it. Very simple and not always right. Yes, I meant parity drive instead of raid - old habits.

Due to your confusion on the subject (as well as my own) I am sorry I even posted that I had an error - good luck to everyone using this software. If I discover whatever the cause of the errors I encountered, I'll post it, otherwise I don't see any point of any further discussion on this subject.
MichaelZ is online now  
post #183 of 214 Old 03-20-2012, 08:49 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by MichaelZ View Post

I see that this program assumes the file is broke (it wasn't) and fixes it. Very simple and not always right. Yes, I meant parity drive instead of raid - old habits.

snapraid does NOT "assume the file is broke". It computes the hash for the file, and compares it to the hash (checksum) stored in the content file. It will not change the file unless that hash does not match.

You have stated that the files that snapraid identified as not matching the stored checksums actually do match with your backup files. So that means that the checksums that snapraid was comparing against must have been bad. The most likely explanation is a read or write error with the content file.

You never mentioned whether you checked all of your content files to see if they were identical. I guess that there was a write error on at least one of your content files (the one that snapraid used when doing the check).

Also, there is something not right with your system if the sync took 30 hours. It should have taken less than 10 hours.
jim2100 is offline  
post #184 of 214 Old 03-21-2012, 10:16 AM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
I changed my config file to add in another location for the content file, but SnapRAID never created it. How do I force it to create another content file?
cybrsage is offline  
post #185 of 214 Old 03-21-2012, 10:44 AM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by cybrsage View Post

I changed my config file to add in another location for the content file, but SnapRAID never created it. How do I force it to create another content file?

Good question. I've never tried that. Did you run 'snapraid sync'? If that didn't work, the next thing I'd try would be adding an empty file to one of the data disks and try 'snapraid sync' again. If that still does not do the trick, I suggest reporting the issue on the snapraid forum:

https://sourceforge.net/projects/sna.../forum/1677233
jim2100 is offline  
post #186 of 214 Old 03-21-2012, 02:07 PM
Newbie
 
Smurf-IV's Avatar
 
Join Date: Jun 2011
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Smurf-IV View Post

What's really annoying, is that I have not narrowed it down as to what part of the code "Feels" it needs to beep! otherwise I would have removed it already..
Work in progress, Thanks for the feedback, and Suggestions for improvements welcome (Over @ Elucidate)

FYI Beeping sorted and other improvements:
http://elucidate.codeplex.com/releases/view/81536
Smurf-IV is offline  
post #187 of 214 Old 03-21-2012, 02:09 PM
Newbie
 
Smurf-IV's Avatar
 
Join Date: Jun 2011
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by whiteboy714 View Post

It only worked after I fixed it. I have tried it twice. Each time it edited my config file. So I just don't do that anymore and its fine.

Can you please add your config (GUI and fixed) to a work Item over @ http://elucidate.codeplex.com please
Smurf-IV is offline  
post #188 of 214 Old 03-23-2012, 06:26 PM
AVS Special Member
 
Killroy's Avatar
 
Join Date: Oct 2008
Location: Ninth Circle of Hell
Posts: 1,730
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 63
Switching from FlexRAID to SnapRAID to see if it fits my need but I do have a quick question for those using it already...

I have three boxes right now that need protecting. Each box has 20-drives that will be protected. With FlexRAID I was doing a single parity drive but reading up on SnapRAID I keep seeing that it is recommended to use 2 parity drives. Any reason why I need to use two instead of one and can I get away with one? With drive prices through the roof I feel the pain of getting another parity drive when the benefit is minimal.
Killroy is offline  
post #189 of 214 Old 03-23-2012, 07:03 PM
AVS Special Member
 
Darin's Avatar
 
Join Date: Aug 2002
Location: Atlanta
Posts: 5,999
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Because with one parity drive, you are only protected for a single drive failure. And the more drives you have, the greater your chances of having multiple failures in a short period of time. It may not seem that likely, but I've had it happen. Sometimes you might buy more than one drive at a time, and drives from the same batch under similar loads have a better chance of failing near each other. Once a drive fails, you need to run a restore to get your data back, but with parity that requires a rather lengthy read of all the data from all your remaining health drives (similar to the initial create). The strain of the restore process can sometimes be enough to cause another drive that might have been on it's last leg to fail. It's not about how many drives fail at the same time, it's how many drives fail before you fully recover from the first failure(s). If a second one dies before you've successfully recovered, you've lost data.

It's all a matter of your comfort level of how much risk you're willing to accept. Personally, I'm currently running FlexRAID with two parity drives for 20 DRUs, and I feel like that's really pushing it. I'll probably be adding a 3rd before my array grows much bigger.

My dual Rythmik Servo sub project (actually quad now, need to update page)
HDM format neutral thanks to the pricing wars of the '07 xmas shopping season :)
Darin is offline  
post #190 of 214 Old 03-23-2012, 07:16 PM
AVS Special Member
 
Killroy's Avatar
 
Join Date: Oct 2008
Location: Ninth Circle of Hell
Posts: 1,730
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 63
I can live with single drive parity then. If two drives fail I figure my karma has finally caught up with me and its time for me to pay the piper.
Killroy is offline  
post #191 of 214 Old 03-23-2012, 08:41 PM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Quote:
Originally Posted by Smurf-IV View Post

FYI Beeping sorted and other improvements:
http://elucidate.codeplex.com/releases/view/81536

Sweet! I will give it a shot.
cybrsage is offline  
post #192 of 214 Old 03-23-2012, 08:41 PM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Quote:
Originally Posted by jim2100 View Post

Good question. I've never tried that. Did you run 'snapraid sync'? If that didn't work, the next thing I'd try would be adding an empty file to one of the data disks and try 'snapraid sync' again. If that still does not do the trick, I suggest reporting the issue on the snapraid forum:

https://sourceforge.net/projects/sna.../forum/1677233

I will have to check it again, I added a DVD folder rip to one of my drives. Hopefully it created the new file. If not, to the forums I go.
cybrsage is offline  
post #193 of 214 Old 03-24-2012, 04:42 PM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
OK, I just looked. To get the program to create the new content files, you have to make a change to the raid content. Once you do, and it resyncs with the new changes taken into account, it creates teh new content files.
cybrsage is offline  
post #194 of 214 Old 03-26-2012, 09:45 PM
AVS Special Member
 
MichaelZ's Avatar
 
Join Date: Sep 2000
Posts: 1,480
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 6 Post(s)
Liked: 13
I downloaded the git 1.9 snapraid and reran it on my system and although it took 35 hours this time to run plus 24 hour to check, I had no errors! So, I am not sure what happened before but at least I had a clean sync and check. I altered a few files to see if snapraid would find them and it did and then I had it fix them and it did, so it seems it is working ok. I am not sure why I had errors on the previous run but I am using the same hardware as before and the only difference is 1.9 vs 1.8 build - same conf, etc.
I will do further testing over the next couple of weeks and report back any errors.
MichaelZ is online now  
post #195 of 214 Old 03-26-2012, 09:51 PM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Where are you all getting 1.9? I only see 1.8 for download.
cybrsage is offline  
post #196 of 214 Old 03-26-2012, 10:14 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by cybrsage View Post

Where are you all getting 1.9? I only see 1.8 for download.

1.9 is not released yet. He downloaded the source code from the git repository and compiled it.
jim2100 is offline  
post #197 of 214 Old 03-26-2012, 11:34 PM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Ah ok. I will wait.
cybrsage is offline  
post #198 of 214 Old 04-04-2012, 08:42 PM
Senior Member
 
wymann's Avatar
 
Join Date: Sep 2003
Posts: 226
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I'm testing SnapRAID 1.9 (command line) on Windows XP (32-bit) for the first time as an alternative to Flexraid. I set up my config file and started a sync, but I've run into two problems:

1) SnapRAID is apparently automatically excluding many files that I don't want to exclude. A few of them are understandable (system files), but many others are not and they are files I need to include in the parity. I had a look at the files listed and some were hidden or locked, but others were neither.

Code:
Ignoring special file...
.jpg, .ini, .dll, .wav, .dat, .pdf, .com, .bak...(and some files without extensions)

I did not use the "nohidden" config option. Please let me know how I make sure that SnapRAID includes all these files and does not ignore them.

2) SnapRAID seems to have choked on an extremely long file/pathname. I have run into this problem before, but other programs have been able to bypass these files and continue on with the other files. Is there an option to skip these types of file? Otherwise, I'd like to request that feature for a future release.

As it is now, I can't run SnapRAID without somehow scanning my drives for all files that might be too long for it to handle and manually adding them all to the config exclude list.

Code:
Unexpected Windows error 3.
Error in stat file/directory 'E:/mydir/myfiles/xyz...'. Input/output error.
My config:

Code:
parity P:\\par\\parity.par
content C:\\snapraid\\content.lst
content D:\\snapraid\\content.lst
content E:\\snapraid\\content.lst
content F:\\snapraid\\content.lst
disk d1 E:\\
disk d2 F:\\
exclude Thumbs.db
exclude \\$RECYCLE.BIN
exclude \\RECYCLED
exclude \\RECYCLER
exclude \\System Volume Information
block_size 256
Thanks for your help.
wymann is offline  
post #199 of 214 Old 04-04-2012, 10:58 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by wymann View Post

I'm testing SnapRAID 1.9 (command line) on Windows XP (32-bit) for the first time as an alternative to Flexraid. I set up my config file and started a sync, but I've run into two problems:

If you don't get any response here, you may be better off posting to the snapraid help forum:

https://sourceforge.net/projects/sna.../forum/1677233

The author monitors that forum, but I am not certain how frequently he reads this forum.

I've never had any problems like you mentioned, but I use snapraid on linux, and your questions seem windows-specific.
jim2100 is offline  
post #200 of 214 Old 04-05-2012, 03:27 PM
Senior Member
 
wymann's Avatar
 
Join Date: Sep 2003
Posts: 226
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by jim2100 View Post

If you don't get any response here, you may be better off posting to the snapraid help forum:

Thanks for the link. I'll have a look over there.

I did find a simple utility for finding long pathnames, in case anyone runs into the same error. It doesn't fix them, but it identifies them, then you can manipulate them by mapping lower level subdirectory paths with "subst".

Path Scan 0.11

It still would be if Snapraid could handle these errors gracefully with a warning and continue on, instead of breaking on the error.
wymann is offline  
post #201 of 214 Old 06-09-2012, 10:23 AM
Newbie
 
smandurlo's Avatar
 
Join Date: Jun 2012
Posts: 2
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Hello,
I am using snapraid, but I want to make it syncs after moving files from downloading location to the storing location. This is "scriptable" even in windows using some move commands before the snapraid command.

But... what if a drive in the "raid" has troubles? I.e. some smart advices or, worst, it has broken and not visible? It can override the whole parity with the new configuration and you cannot recover anything...

I got the point that using other tools for checking the drives is better, but snapraid should know about the results of this check before to start a new sync operation not to make an unrecoverable mess.

What do you think?
smandurlo is offline  
post #202 of 214 Old 06-09-2012, 02:41 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
I'm not an expert on Windows scripting (I run snapraid on linux), but can't you just add a SMART check to the beginning of your script that runs snapraid, and if a drive fails the SMART check, then make the script exit without running snapraid (and maybe send an email or something).
jim2100 is offline  
post #203 of 214 Old 06-10-2012, 10:26 AM
Newbie
 
smandurlo's Avatar
 
Join Date: Jun 2012
Posts: 2
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Well, I am studying how to make it possible...

A batch file that:
1) copies a small file in every disk of the array to wake up them and then delete all of them;
2) checks of every disk of the array with smartmontools, if everything is ok it goes further, otherwise it sends an email with blat, waits 30 seconds and shutdown the PC;
3) moves all the files in the assigned directory;
4) starts the new sync.

It should be safe in this way, I suppose.

Anyway, I am "playing" with snapraid and I found a strange thing: the parity file should be as big as the biggest space used in a drive of the array, i.e. 4 drives in the array, 1TB each and one of them has 80% of the space used. The parity should be around 800GB.

If I remove 100GB from that hard drive and move this space into another drive of the array (30% full) to balace, the parity file should decrease of 100GB. Instead it is the same size as before after the sync.

Is it correct?
smandurlo is offline  
post #204 of 214 Old 06-10-2012, 11:07 AM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
I'm not sure if snapraid is capable of shrinking the parity file. That would be a good question to ask of the author, on the snapraid forum:

http://sourceforge.net/projects/snapraid/forums/forum/1677233
jim2100 is offline  
post #205 of 214 Old 06-10-2012, 05:19 PM
Advanced Member
 
Litlgi74's Avatar
 
Join Date: Jan 2005
Posts: 683
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 16
Anyone using the Simple Scheduling feature of Elucidate? I am not sure how to use these feature... Documentation is quite limited.

Thanks

The world ends when you're dead. Until then, you got more punishment in store. Stand it like a man -- and give some back.

~ Swearengen
Litlgi74 is offline  
post #206 of 214 Old 07-10-2012, 06:19 AM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
Quote:
Originally Posted by Litlgi74 View Post

Anyone using the Simple Scheduling feature of Elucidate? I am not sure how to use these feature... Documentation is quite limited.
Thanks

I am, I just slogged through it. My only gripe is that it opens a CMD window but never closes it.
cybrsage is offline  
post #207 of 214 Old 07-10-2012, 06:22 AM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
News! Version 1.11 RELEASED! Download it here.
cybrsage is offline  
post #208 of 214 Old 07-10-2012, 01:28 PM
AVS Special Member
 
Mrkazador's Avatar
 
Join Date: Jun 2005
Posts: 3,841
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 43 Post(s)
Liked: 250
Mrkazador is online now  
post #209 of 214 Old 10-04-2012, 07:47 AM
 
cybrsage's Avatar
 
Join Date: May 2007
Location: Harrisburg, PA
Posts: 8,074
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 147
News! Version 1.12 RELEASED! Download it here. The default download location did not work for me, but the mirror link did (in the next page after you select the download you want).
cybrsage is offline  
post #210 of 214 Old 10-15-2012, 07:09 AM
Newbie
 
FieldDoc's Avatar
 
Join Date: Oct 2012
Posts: 1
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Here's a hypothetical scenario.

I'm running Ubuntu. I have a four bay hot swappable enclosure for four 2.5" SATA drives that contains four 1TB drives.

1. Say I use mhddfs to merge these four drives into one 4TB virtual volume. Am I right in thinking that I can use SnapRAID to monitor and protect this virtual volume against one (or two) of the disks failing?

2. If question (1) above is true then am I able to pull out one of the 1TB drives, replace it with a 2TB drive and have SnapRAID fix/restore any data that was on the pulled out drive?

Essentially, I'm trying to build a reliable open source version of Drobo for my home,

Is this possible?

Many thanks for the excellent work you've put into this already.
FieldDoc is offline  
Reply Home Theater Computers

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off