SnapRAID: An Open Source alternative to UnRAID and FlexRAID - Page 5

Presumably though (and I don't have my test laptop with me today so can't poke at it) the slack space wasted by using a 5 kilobyte file in the 256 kilobyte block size will be as bad as storing a 261 kilobyte file?

You will completely fill one block within snapraid but the next block will still be largely wasted.

This clearly isn't as bad as the worst case scenario I instigated with my 5k file tests which will be wasting space in *every* block but it does mean the more files you have, so long as they don't fall perfectly on a snapraid block boundary (unlikely?), the more wasted space you'll inevitably have. This is backed up in the snapraid manual


The more drives and more storage you're trying to protect - the more files you will likely have.

So even with large files you will eventually hit a problem.

As I said my real concern is that I can't just say 1:1 in terms of largestdatadrivesize:snapraidparityspacerequired in fact I can guarantee that the parity space required will always be more than the largest drive of data you have. And that's before we discuss the space required for the content file.

Even if 1:1 data to parity space holds true at some point in your arrays life it may not hold true once you shove another 3 or 4 data drives in and fill them with data and update parity.


My reasons for using snapraid are to try and remove the need for traditional raid and have protection with the minimum amount of overhead possible. So having to worry about additional protection for the content file (particularly in any sort of raid array) doesn't really suit me.

For reference I'm coming from unraid (where your parity drive is the same size as your largest data drive though unraid obviously works at the block level meaning you're ducking any of the issues snapraid is presenting) and looking at either flexraid or snapraid as potential replacements.

I've done minimal testing of flexraid but my first impression was it doesn't suffer from this problem despite also using file based protection - but I wouldn't like to say for sure

I'd agree it's looking like snapraid may not be for me, though I would suggest I am perfectly in the target market for it having a mid sized array mostly of large media files but also with a scattering of smaller files (metadata files, general day to day documents, photos, ebooks / papers etc). But I'm not sure how scalable it will be if you're protecting a large amount of data / files.

The easy answer, as I've mentioned on this thread previously, is to just try it against my existing data but would require me to buy an additional drive. It's also making spec'ing a new storage system difficult as I have no idea how much storage I'll need for parity.

Perhaps people on this thread running snapraid in anger could report back on their total file count and data size against their snapraid parity and content file sizes?

Thanks for the discussion though - I'm going to drop the snapraid block size to match the filesystem (which for ext4 is 4k by default I think) and see what the results look like. The sums suggest I'll need 3G ram to handle that - which is fine.

Did you forget to multiply by 24? Or do you only have 500GB of data?

I wouldnt claim to be running in anger but I am running it.

I have 8x2Tb drives (one is parity) plus an SSD.

Content file is stored on parity drive and on SSD. Up until yesterday I had 5 of those drives pretty full (90%). The total of PAR file + content file (ie the usage of the Parity drive) was only a few Gb larger than the fullest drive. So unless I chose to fill a single HDD to more than around 97% I'd say there would be no issues. Filling that full would be silly as it would make any defrag operation a nightmare!! Memory usage peaked around 1.5Gb for SnapRAID during creation of PAR file.

I re-balanced my HDD capacity loadings which was a little pointless as it wont make the PAR smaller unless I create a new PAR file! Duh!

Updating PAR file takes around 2 mins each night at 1am.

Personally I decided to check out those small files before I created PAR. I dont care about meta files and thumbnails XBMC generates/needs. In the case of a data loss I would just recreate those by reimporting the media into a manager like Sickbeard or similar. So I excluded *.TBN and folder.JPG from SnapRAID. I also noticed I had many small files from Rom sets for old emulators. Since I rarely use them I chose to zip them up - saves space and adds no real delay if/when I need them (if ever!). I also have my Documents etc backed up via Cloud but I pruned them and archived old ones.

I guess it requires a little thought about your data before you use this but thats IMHO of course.

And then I had a worry when my son was trying to convince me that if a HDD died I might lose more than the data that had changed since last PAR update. He suggested that any files XOR'd with any file that had been deleted that day would also be at risk. My old brain is too slow to work that out so I chose to ignore him. But if anyone can work it out please LMK!

It sounds reasonable, snapshot raid is good for "write once, read many" strategy, and if you had to delete many files it might be a smart move to make a sync.

I don't know - I did the sums as per the docs

I also calculated for my actual 'real' data set of ~9T and ~405658 files I'd likely lose 50G to parity 'wastage'. i.e I'd have a parity file 50G larger than the data used on my most full drive.

I'd also need ~800 megs memory to run snapraid.

That's at the 256k blocksize. At 128k I'd lose 25G to parity and need 1.6 gigs working memory.

This is before we discuss the content file size for 400,000 files.


That works for me


Thanks that's useful info. Do you know how many files you might have roughly?


That's a good idea - and certainly an option. But yes, it means putting more thought into your data management both at initial setup and ongoing as your data usage changes / evolves. All of which I'd prefer to avoid. Though I'd like a lot of things I won't get


Also if you take a snapshot successfully and then edit or change a file contained in the snapshot before trying to restore. The parity will generate invalid data from the changed file when reconstructing the data to be restored. How much is invalid will depend on how much data you changed.

Thanks for the input all.

I've read 4 pages of this thread and could not find an answer to the following, probably due to my lack edumacation:

I have been running chkdsk on a somewhat old and abused HDD which stores my iTunes music and I am seeing errors at a block level, errors which are identified for specific files, i.e., specific songs. So if I use this program, would the program be able to identify that a piece of data for a song has been corrupted and correct the corrupted data from parity data of the last sync (assuming that the song was present when synched)?

It seems that one of my HDD is about to fail and chkdsk is finding several errors everytime I run it. So I am wondering if this program would fix errors caused by hardware degradation of the HDD. Of course, I plan to replace that old HDD with a new HDD, but I would like to prepare for the eventuality when that new HDD starts to fail, too.

I am asking because I would like to backup my music but leary of using a program (like Acronis) which, I think, would back up the corrupted data and I don't think that I'd be able to permanently save every full back up (and I don't like incremental backups, either), not to mention trying to figure out which backup has the uncorrupted copy of the song.

I have read your faq and your website and is my situation what you call "silent corruption"?

Thanks.

Quick question as I'm thinking about using SnapRAID but it will be my first venture into anything RAID-like.

I've read that you need a parity drive as big as your largest drive. That is no problem as I have 4 2TB drives and I intend to use 3 of them for storage and one for parity. Now in Windows 7 I can transform the storage drives into dynamic drives and then span them so it appears as one volume. If I do this, can I still have 3 drives for storage (i.e. a 6TB spanned drive) and 1 2TB drive for parity, or would I now need a 6TB parity drive too?

Thanks in advance

I dont think your plan will work without a 6Tb parity drive as the OS will just see one drive and SnapRAID is a file based RAID if i understand it. If you need a single volume (why?) why not add some storage pooling s/w like DriveBender or Liquiese (sp?)?

I did not spot anything obviously wrong in the conf file.
What sort of errors, Can you zip the log file up with the relevant information for me please, and create an issue @ "http://elucidate.codeplex.com/workitem/list/basic"
Also, what version where you using ?, Does this still happen with the latest maintenance release "http://elucidate.codeplex.com/releases/view/70494"?
Cheers

I posted this weeks ago and gave up soon after with Elucidate - sorry but these days you normally get a response quicker if you are going to get one at all.

Still happily running SnapRAID with my hand editted config.

This may be a bug in Elucidate, I'm not sure... Installed, set the location of snapraid.exe and my snapraid.conf. I select Diff and get this error


I check my conf file and noticed that my Data disks were changed. Originally it was set as
disk d1 E:\\
disk d2 F:\\

After starting Elucidate for the first time it was changed to
disk d0 E:\\
disk d1 F:\\

After fixing my snapraid.conf, Elucidate works normally now. Under recovery options, if I were to select sync zeros, does it start syncing immediately with no warnings? If so, it may be best to add a "Are you sure?" dialog under recovery options once you select your option.

Sync Zeros is something I am looking into, It shouldn't do anything yet (Should remain disabled), But thanks for the suggestion, I think All of the Buttons on that page should have a "Are you Sure" style message.
I'll have to look again at the SnapRaid usage, But I think that the names of the disks can be anything, As long as they remain in the same order and obviously have the same name. I'm glad it was obvious for you to fix.

So I just started using snapraid today, and I am finding it excruciatingly slow. I am running ubuntu 11.4, with a 2tb parity and a 2tb and 750 gb that are 90 percent full. I have a core i3 and 3 gigs of ram on my server.

When I run "snapraid sync" it just says "syncing". After an hour, the "parity" file is only 44.4 gbs. At this rate it will take a couple days to finish. I have specified in the snapraid.conf file to ignore some small files I have on the hdds, so I dont think that is the issue. My drives are plenty fast, getting over 100 MBs writes. I dont know why it is taking so long...flexraid on windows ran much quicker.

Also, ctrl-C doesnt seem to stop the syncing.

my $.02
@sound dropouts:
Have a look at the following thread "http://sourceforge.net/projects/snapraid/forums/forum/1677233/topic/4607751" it gives some test areas to see what / which drive might be making things a bit slow. Please add your results and drive specs to that to help the developer come up with a plan and to give some advice.

i was planing to build a FLEX HTPC/NAS and then i saw this solution...
reading more about it here (still at page 3) but i saw this (quote) the same system i'm building (right now only 4 2TB hdd's)



sounds good... a good pair to the snapraid, i you'll work out the problems i'll use it with my HTPC/NAS... why have 3 2TB hdds when you can have 1 6TB
but i use WIN7 so i'll have to wait


i'll be back soon for more help from the forum as soon as i get my E350 mobo and mem

Any update on the development?

About what, specifically?

I have been using snapraid for months now. It works well.

Anyone got any feedback. Thinking of switching over from flexraid. Is the setup pretty easy for this? Hopefully the GUI is decent.

EDIT* Looks like a lot more work than flexraid. I suck at CMD line.

For the SnapRAID GUI, )Windows only) there is Elucidate "http://elucidate.codeplex.com/documentation"
Pooling is mentioned above

What is your reasoning for switching from Flexraid?

I'm a cheap *** and they are starting to charge $ this month.

I am contemplating moving to snapraid due to flexraid going commercial. Does the parity drive in snapraid work as it does in flexraid - in other words, do I need one parity drive the size of my largest hard drive and can have as many other drives as I please, all protected by the system via that one parity drive?

Yes. Although if you have a lot of drives, you may want to consider dual-parity.

Also, snapraid works best if none of your data drives are completely full. I try to keep all of mine less than 95% full. But if you want to push it closer to the edge, I would recommend leaving at least 20GB free on every data drive. That leaves some space for the content files, and also room for overhead from block_size that causes the parity file to be a little larger than the data on your drive with the most data.

Thanks. I never fill a drive up fully...that way the drives can use the space to recover from physical errors on the disc as needed while I RMA them. Habit from the old days.

Time to learn a new raid program...I feel a bit betrayed by FlexRAID, I helped him out as a tester from the very beginning and now he wants me to give him money for it.

How dare someone ask for money for software they wrote, when it's the core of your multi-hundred/thousand dollar storage system. The insanity...

Serious! Who cares about the hours upon hours he spent developing it. And helping out people using it.

Fortunately, people have a choice. They can use SnapRAID, which is free and open-source. The author of SnapRAID cannot suddenly decide to withdraw all free versions of the program and charge people to continue using it -- so far, all versions of SnapRAID are GPL'd.

It started out free. Suddenly - POP - not free. Had it always been said it would not be free, then you would have a point. That was not the case.

It would be no different than SnapRAID suddenly pay (provided he removed the parts he could not use then and replaced them with his own code).

It actually would be different, because the source code for all SnapRAID versions released to date is under GPL. Even if the author decided tomorrow to keep all future code private, we could still go on using any version of SnapRAID that has already been released under GPL, and we could even take over development of the already released code and release updated versions (also GPL'd).

As far as I know it was always in beta. So you were never even using a final version. Plus he said he would hook people up that gave their time to the project. Unless by helping him test you mean you just used the software like everyone else.