SnapRAID: An Open Source alternative to UnRAID and FlexRAID - Page 4

oh yeah, also wanted to mention that I did NOT find any mention of POOLING storage together on the site.

pooling drives as one drive, instead of individual drives is desired by many, and I am curious what commands, if any, are available to pool drives together to appear as one KEYWORD: pool returned nothing with snapraid

cheers

Yep. Not having permission to write a file is an unexpected error for SnapRAID, so it stops the processing, expecting you to revert the stop condition.
This is a safe measure, likely too strict in this case, to prevent to make more damages on a faulty HD. For example if a file commands fails because the HD is not working properly.

A better approach in this case is to exclude hidden files in "sync", and in "fix" remove the read-only attribute if needed.

This is for sure a new feature I'm going to add.


Files are opened in read-only mode in sync/check, and in read-write in fix. In fix a write may be needed to fix the file, so it's opened also for write.
But in fact, it may make sense to reopen them for write if needed.


There are two kind of errors for SnapRAID. Data error are always fixed, and if not possible skipped continuing the processing. HardDisk errors are instead stop conditions, especially in the fix command. The rationale is that if the HD is failing, it's a lot better to try to copy all the readable data from it, instead that trying to fix on it.
So, if you delete a file, or it contains wrong data, the process will continue even if SnapRAID is not able to fix it.
But if the OS is reporting an error in a file operation, the process stop to allow you to address the problem.


Sure

Sorry for the late answer, I was out for holiday in Tuscany near Siena

SnapRAID isn't a pooling storage solution. It's only for backup.

Anyway, you can use it with other tools providing pooling (that was called virtual view on the SnapRAID site), for example for Linux the suggested one in the FAQ is "mhddfs", but I haven't a suggestion for Windows yet.

Hmm. Keep in mind that the speed of SnapRAID is limited by the speed of the slowest HD you are using. If you are using a wireless network HD, even a single one, all the HDs will be slowed down to that speed. This limitation cannot be easily removed. So, better to use only fast networks or USB3 drives.


The "fix" command is likely slower than "check" because it open the files also for writing, and likely it has effects on the network storage protocol.


Any report of use or suggestion is a great help!


Good!

I see. It would be nice to be able to specify data disk for restoring. Normally "fix" operation is supposed to be run after HDD failure and user would want to restore files only on that HDD.
Files on other drives can be opened read-only. Suppose there is a bug, or memory corruption that somehow swaps file handles. Then accidentally good data file may be overwritten with wrong data.
If file is opened read-only this would not be possible.

EDIT: Alternative suggestion: You probably can open file for writing only if it doesn't exist.
I thought you were going to restore only missing files? Or, are you planning to restore edited files to previous version captured in parity?
If latter is the case I believe it's more complicated.

I'm a bit unsure as to how my specific data and usage patterns would sit with this approach to data protection (same as I am with flexraid!) but I guess the obvious answer is to try it and see!

I appreciate the fact that this software, though presumably still in early days, seems much more polished and focused than flexraid. Which is reassuring - at least on the surface. I've never got on with flexraids interfaces or terminology. snapraid is much more straight forward and no-nonsense.

One thing I noticed when prodding about was that it doesn't like broken symlinks (softlinks) :

' Error in stat file/directory '/mnt/disk1/toot'. No such file or directory . '

'toot' is a purposely broken symlink.

This is a blocking error - snapraid stops at this point and I can't find a way to have it ignore / continue with a sync etc.

As a general rule that's probably a good idea on a broken file - but perhaps not if it's a symlink?

Is this a known issue / desired behaviour for a broken symlink?

Once removed and / or the target fixed snapraid could continue as normal.

This is a feature that makes sense Added in the TODO list.


The "fix" command restores anything to the state of last "sync". The only exception is that it doesn't delete new files.

I already planned a new "restore/undelete" command that only restore missing files.

Anyway, at now I changed to open files in read-only mode if they cannot be opened in read-write. In this way you get a failure only if the file really needs to be modified.

PS:
A new release is approaching.

Good spot! Thanks!

In fact it's a bug, as symlinks should be completely ignored by SnapRAID. At now SnapRAID is intended to backup only real files, but maybe in future I could also add a symlinks backup feature.

I'll fix the problem in the upcoming release.

When implemented, make sure there is also same fast command like quickcheck or so which show the list of these deleted/changed (size/timestamps only ?) files.

Because blindly run the restore/undelete would definitely cause issues when some new files messed.

OK, 1st post here, but it is off interest to all you who have been requesting a GUI to drive SnapRAID, please visit "http://elucidate.codeplex.com" and leave a few comments, encouragements etc.

I'm the author of "http://liquesce.codeplex.com/" and it does the pooling that you require, it currently has issues (From the driver layer) with shares on Windows 7 and above, but these are currently being worked around via my software (Not released yet - as performance is a bit poor!)

SnapRAID v1.4 released at http://snapraid.sourceforge.net/

Another maintenance release to fix some issues reported by mkz71 and eztiger.

Now, If no problem is found, I'll start to implement new features In the mean time you can check the TODO list at:

http://snapraid.git.sourceforge.net/...f=TODO;hb=HEAD

Hi
SnapRAID 1.4 restored my 2TB disk. Hard to tell how long it took, I just returned from work. Less than 11 hours, for sure.
Few files that I verified are bit-to-bit perfect. Directory structure is the same as before, no unrecoverable errors in output, so I think it worked.

All issues that I reported before, including read-only files, seem to be fixed.
It now ignores hidden files but I think it's by design.
So, I have encountered no issues, will think of more difficult test cases.

Good job
Thanks

I'm a FlexRaid user and I am not very happy about it. For example, I clicked "Create RAID" 8 hours ago and its still doing something - no progress bar, no speed, no ETA - no nothing, and PC is very slow.

How would it be if I combine FlexRaid Pool Storage (AKA FlexRaid-View) and SnapRAID for parity? Probably it could be a smart move.

I don't see why you couldn't combin SnapRAID and FlexRAID. But just to help with your current situation, how much data is in your array? Across how many disks? A lot of big files, small files? One way you can check progress is to look at your parity disk(s). When you create the array, it first creates all the parity files, then goes back and updates each one as the parity data is calculated. If you look at them, you will see that it starts at the beginning, and works it's way down (in the order that the parity files are numbered). You should be able to scan down the list and find where it is currently working by finding the one that was last updated recently. You can estimate how much longer it has to go by figuring out the percentage it has finished so far. That should be a conservative estimate, as it often speeds up towards the end if your disks are unevenly loaded (i.e., it might be calculating parity on 10 disks in the beginning, but only 1 or 2 towards the end if you only have a couple of completely full disks).

Nice idea, thank you! I tried to follow your guide but it suddenly completed few seconds ago It was something about 2 TB data across two drives.

Hi guys!

I'm trying to build my first "array" using SnapRaid but I'm having problems with content folder/file. This is my snapraid.conf file:

parity V:\\parity

content V:\\content
content C:\\SnapRaid\\content

disk d1 F:\\
disk d2 G:\\
disk d3 H:\\
disk d4 I:\\
disk d5 J:\\
disk d6 K:\\
disk d7 L:\\

exclude *.bak
exclude Thumbs.db
exclude \\$RECYCLE.BIN
exclude \\System Volume Information

block_size 256

And this is what I get when I run "snapraid sync" command:

C:\\SnapRaid\\snapraid.exe sync
Self test...
Loading state from V:\\content...
Error opening the content file 'V:\\content'

I don't understand what's going on here, obviously the folder V:\\content is empty, so I don't know what's looking for in the folder.

What am I doing wrong??? Any clue???

Many thanks for all your help, I really appreciate your patient

Best regards,

Aitor

The parity and content files are actual files. If you have a folder at V:\\content then I don't think it will work. If you do have a folder V:\\content, then delete it (as well as C:\\SnapRaid\\content and V:\\parity if either of those is a folder) and try again. snapraid should create the content and parity files.

Alternatively, if you do want the files stored in folders, then you need to change your conf file to something like:

parity V:\\parity\\parity
content V:\\content\\content
content C:\\SnapRaid\\content\\content

and make sure V:\\parity, C:\\SnapRaid\\content and V:\\content ARE folders that already exist.

Hi erik!!!

That was the problem, after deleting the folders all went fine and the parity file is being created correctly. Many thanks for your help

The problem that I'm having now is that the process is taking too long to complete, although I don't know if this is normal or not. This are the drives to be protected:

1x WD Caviar Green 1Tb
2x Samsung Spinpoint 1.5Tb
2x WD Caviar Green 2Tb
2x Samsung Spinpoint 2Tb

and the parity drive:

1x Samsung Spinpoint 2Tb

The total size of the protected data is 10.9Tb and I have 4Tb free.

The synchronization starts at 3am (in Spain), now it's 6:30pm, so in 15.5 hours it has completed 37% (25:43 ETA remaining). Here you have more details about my resources monitor:



Is this slow speed normal? Any clue?

Many thanks again for all your patient and help, I really appreciate it.

Best regards,

Aitor

The throughput when you took the screenshot is 66 MiB/s. That is the sum of the read rates of all the data drives that it is reading from at the moment. Not necessarily all the data drives, since if one data drive is 100% full and all the rest only 10% full, then snapraid will only need to read from one data drive once it gets past 10%. So if you want to see the top speed, you need to give the throughput when the sync first starts (well, give it a few minutes after starting before recording the throughput). But 66 MiB/s does seem slow. My system with 8 data disks gets over 500 MiB/s at the start. What is the throughput (sum of read rates) of your data disks if you read them all at the same time with another program?

This program is JUST what I wanted. I had no issues with editting a conf file but I also wanted to look at Elucidate and that seemed to show errors being chucked out left and right so I went to CMD line instead. If I ran from Elucidate it seemed to stop with red errors. Hence CMD line.

My conf (if anyone can spot any bloopers that might cause Elucidate to freak)

# Configuration for snapraid via Elucidate
parity K:\\par\\parity
content K:\\par\\content
content C:\\Program Files\\SnapRAID\\content
disk d0 D:\\
disk d1 E:\\
disk d2 F:\\
disk d3 G:\\
disk d4 H:\\
disk d5 I:\\
disk d6 J:\\
exclude \\$RECYCLE.BIN
exclude \\System Volume Information
exclude \\My_Backups (DON'T PUT FILES HERE)\\
exclude \\My_Uploads\\
exclude \\My_Downloads\\
exclude \\My_Roms\\PD Torrents (Amiga & ST)\\
exclude *.bak
exclude Thumbs.db
exclude *.tbn
exclude fanart.jpg
exclude folder.jpg
block_size 256

Its showing 250MB/s or more and estimated 9 hours to build for me but I am concerned it is chucking some "warning: special file" on a few INI files it shouldnt. Only just started me build on 7+1 two TB drives!

Is there a list of files it DOES auto exclude (i.e Desktop.ini files and so on) or atttributes?

Is there a recommended way to schedule updates to Parity file(s) - Like most people I'd have it update around 1am or some time I don't expect to be using PC. I have my SABnzbd directory excluded but will probably set SABnzbd to pause at update time to stop it processing video files and putting them out to storage drives when parity is being updated.

PS: Down to 8hrs now (sped up) and creating a 1.65TB parity file

This is exactly why i moved form FlexRaid to SnapRaid

SnapRaid provided very complete information about it is doing and ETA and percentage.

I think this is normal for your first use.

The "content" file is created at the end of the syncronisation process. So you don't have any content file initially.

It seems quite slow...
I run with 9 * 2To drives (slow 5400RPM green drives) and the process finishes with a speed close to 250Mb/s.

I've come across an odd issue.

I've created 4 test drives of 8G each all on ext4 :

/dev/mapper/disk1 7.9G 860M 6.7G 12% /mnt/disk1
/dev/mapper/disk2 7.9G 146M 7.4G 2% /mnt/disk2
/dev/mapper/disk3 7.9G 146M 7.4G 2% /mnt/disk3
/dev/mapper/disk4 7.9G 6.5G 1.1G 86% /mnt/disk4

The first three disks are data disks with the parity and content being held on disk4.

Snapraid config file is thus :

parity /mnt/disk4/par
content /mnt/disk4/content
disk d1 /mnt/disk1
disk d2 /mnt/disk2
disk d3 /mnt/disk3

Can you spot the problem? The parity file on /mnt/disk4 is 6G in size! But there is minimal (data) usage on the actual data disks.

How can this be?

A few caveat / thoughts :

- The data drives are not accessed directly but via an mhhdfs layer which I had imagined would make no difference, but perhaps it does odd things when writing and allocating data?

- In order to test how the size of the content file grows relative to the file density there are a number of files on disk 1 (copy of /usr/src) that are very small. Am I tripping over something to do with the block size parameter in snapraid here and wasting a lot of space in parity with the small files?

Or am I just being silly?

If this is expected behaviour then it's quite alarming if it's an indication of how much overhead you might need for parity - and seems to be a bit broken to me. I was expecting to have sizing problems with the content file when point snapraid at a dense filesystem - not the actual parity file itself!

OK some more testing.

ACross the three disks I scattered ~200,000 5k files with random data contained within. Distribution was primarily on one of the drives (100,000 files worth).

And then tried to sync. Snapraid failed telling me it needed 30G for the parity file.

This will make it difficult (/impossible?) to judge how much parity space is required if it depends on the existing and future density of the filesystems being monitored.

Very pleased so far. About 6 hours to make my parity. Now takes a couple of minutes run each day for updating a few Gig of changed data. If nothing changed then its under 2 minutes.

This has to be a more economical system than RAID5 which spins drives all the time you're using it. This only spins the drives you need plus all of them for a few minutes a day to re-sync. Has to save electricity and noise/heat.

Also although happy to schedule Snapraid.exe sync for 1am what I wanted to know was how long it takes so I wrote a little script to log the times started and finished to a logfile and I have my SABnzbd paused at that time to ensure it doesnt unpack and move a video file to the HDDs whilst parity is being updated.

Need to play with restoring files next.

If you have hundreds of thousands of 5KB files, SnapRAID is probably not the program for you. The default block_size is 256KiB. You could make it smaller, but that would increase the memory usage:

"SnapRAID requires about TS*24/BS bytes of RAM memory. Where TS is the total size in bytes of your disk array, and BS is the block size in bytes."

You are probably having additional problems because you are putting the content file on the same volume as the parity file. If you have a lot of files, the content file will be large, and if any of your data disks are full and the same size as the parity disk, then the parity file would be expected to take up all of the parity disk, leaving no room for the content file.

For thousands of files of moderate to large size (say, 256KB or larger), SnapRAID should work well. But I do not recommend it for a huge number of tiny files.

Thanks - block size was one of my suspects. I'll play about and have a look.

I don't have hundreds of thousands of 5k files 'in real life' (it was a semi extreme test, again mostly to see how the content file size scaled with number of files). However I do have 3-400k files of various sizes on the array (including down to 5k and below). It's general purpose.

And of course the bigger the array you're protecting, the more files you are likely to have so I suspect this problem is inevitable (at some level) for most people especially with such an aggressive default.

So this makes it problematic (in my eyes) to measure how much you will need space wise for parity unless you can align the snapraid blocksize with the filesystem block size (?). I can't glibly say 'my largest disk is 3TB therefore I need a 3TB parity disk' as actually I may need more (potentially much more) than 3TB.

As mentioned this was kicked off by wondering about the scaling of the content file. I don't really know what the solution to that is? other than have it on another disk as you mentioned. But then you'd need to protect that disk as well as without the content file you're in trouble (or at least have invalid parity?).

If the content file can't be stored on a data disk then it has to go on a separate disk, which also shouldn't be used for parity in the above scenario. So I'm back with how to protect it? Granted losing the disk with only the content file on it means your data is intact so you can replace and regenerate parity but it doesn't seem ideal.

Am I missing the point?

Are you missing the point? Maybe. As I said, I don't think SnapRAID is for you if you have a large number of tiny files. I think SnapRAID is ideal for a media file server, where even 256KiB files are unusually small.

In your test case, it would have been easy to predict that you would have problems. 200,000 files, at 256KiB per file (each file needs at least one block), comes to about 52GB.

I disagree that it is inevitable that people will have trouble with the size of the parity file. As long as the typical file size is several times the block_size, then the parity file will be about the same size as the data on the drive with the most data (a little larger: wasted space will be approximately equal to half the block_size times the number of files). If you want to store the content file on the same drive as the parity data, then you will need to avoid having any completely full data drives with the same capacity as the parity drives. But I never fill my drives beyond 95% anyway, so that is not a problem for me.

Also not a problem because I keep my content file on a different drive than my parity file. SnapRAID is ideal for lots of large media files that are WORM (write once read many). But for files that are small, or frequently modified in place, snapshot RAID is a poor choice. So I have separated out those files on my server and put them on a RAID 1. The RAID 1 volume is also where my content file resides.

If you like, you can store multiple copies of the content file. Just add additional content lines in your configuration file to store as many content files as you like.

If most of your data files are relatively large, the content file consists almost entirely of lines like this:

blk 3844890 99febcb6592eff39c7fe727ff14b611b

That is about 45 bytes per block. So you could estimate the content file size as 45B x TS / BS , where TS is the total size of all your data, and BS is the block_size. But if you have a lot of small files, the content file could be bigger than that estimate, since it also contains a line for each file such as:

file r2c1 31924031388 1296183732 132 path/to/the/file/filename