or Connect
AVS › AVS Forum › Video Components › Home Theater Computers › SnapRAID: An Open Source alternative to UnRAID and FlexRAID
New Posts  All Forums:Forum Nav:

SnapRAID: An Open Source alternative to UnRAID and FlexRAID - Page 5

post #121 of 214
Quote:
Originally Posted by erik71 View Post

I disagree that it is inevitable that people will have trouble with the size of the parity file. As long as the typical file size is several times the block_size, then the parity file will be about the same size as the data on the drive with the most data (a little larger: wasted space will be approximately equal to half the block_size times the number of files).

Presumably though (and I don't have my test laptop with me today so can't poke at it) the slack space wasted by using a 5 kilobyte file in the 256 kilobyte block size will be as bad as storing a 261 kilobyte file?

You will completely fill one block within snapraid but the next block will still be largely wasted.

This clearly isn't as bad as the worst case scenario I instigated with my 5k file tests which will be wasting space in *every* block but it does mean the more files you have, so long as they don't fall perfectly on a snapraid block boundary (unlikely?), the more wasted space you'll inevitably have. This is backed up in the snapraid manual

Quote:
Originally Posted by snapraid View Post

As approximation, you can assume that half of the block size is wasted for each file.

For example, with 10000 files and a 256 KiB block size, you are going to waste 1.2 GiB.

The more drives and more storage you're trying to protect - the more files you will likely have.

So even with large files you will eventually hit a problem.

As I said my real concern is that I can't just say 1:1 in terms of largestdatadrivesize:snapraidparityspacerequired in fact I can guarantee that the parity space required will always be more than the largest drive of data you have. And that's before we discuss the space required for the content file.

Even if 1:1 data to parity space holds true at some point in your arrays life it may not hold true once you shove another 3 or 4 data drives in and fill them with data and update parity.

Quote:
Originally Posted by erik71 View Post

So I have separated out those files on my server and put them on a RAID 1. The RAID 1 volume is also where my content file resides.

My reasons for using snapraid are to try and remove the need for traditional raid and have protection with the minimum amount of overhead possible. So having to worry about additional protection for the content file (particularly in any sort of raid array) doesn't really suit me.

For reference I'm coming from unraid (where your parity drive is the same size as your largest data drive though unraid obviously works at the block level meaning you're ducking any of the issues snapraid is presenting) and looking at either flexraid or snapraid as potential replacements.

I've done minimal testing of flexraid but my first impression was it doesn't suffer from this problem despite also using file based protection - but I wouldn't like to say for sure

I'd agree it's looking like snapraid may not be for me, though I would suggest I am perfectly in the target market for it having a mid sized array mostly of large media files but also with a scattering of smaller files (metadata files, general day to day documents, photos, ebooks / papers etc). But I'm not sure how scalable it will be if you're protecting a large amount of data / files.

The easy answer, as I've mentioned on this thread previously, is to just try it against my existing data but would require me to buy an additional drive. It's also making spec'ing a new storage system difficult as I have no idea how much storage I'll need for parity.

Perhaps people on this thread running snapraid in anger could report back on their total file count and data size against their snapraid parity and content file sizes?

Thanks for the discussion though - I'm going to drop the snapraid block size to match the filesystem (which for ext4 is 4k by default I think) and see what the results look like. The sums suggest I'll need 3G ram to handle that - which is fine.
post #122 of 214
Quote:
Originally Posted by eztiger View Post

Thanks for the discussion though - I'm going to drop the snapraid block size to match the filesystem (which for ext4 is 4k by default I think) and see what the results look like. The sums suggest I'll need 3G ram to handle that - which is fine.

Did you forget to multiply by 24? Or do you only have 500GB of data?
post #123 of 214
Quote:
Originally Posted by eztiger View Post

Perhaps people on this thread running snapraid in anger could report back on their total file count and data size against their snapraid parity and content file sizes?

I wouldnt claim to be running in anger but I am running it.

I have 8x2Tb drives (one is parity) plus an SSD.

Content file is stored on parity drive and on SSD. Up until yesterday I had 5 of those drives pretty full (90%). The total of PAR file + content file (ie the usage of the Parity drive) was only a few Gb larger than the fullest drive. So unless I chose to fill a single HDD to more than around 97% I'd say there would be no issues. Filling that full would be silly as it would make any defrag operation a nightmare!! Memory usage peaked around 1.5Gb for SnapRAID during creation of PAR file.

I re-balanced my HDD capacity loadings which was a little pointless as it wont make the PAR smaller unless I create a new PAR file! Duh!

Updating PAR file takes around 2 mins each night at 1am.

Personally I decided to check out those small files before I created PAR. I dont care about meta files and thumbnails XBMC generates/needs. In the case of a data loss I would just recreate those by reimporting the media into a manager like Sickbeard or similar. So I excluded *.TBN and folder.JPG from SnapRAID. I also noticed I had many small files from Rom sets for old emulators. Since I rarely use them I chose to zip them up - saves space and adds no real delay if/when I need them (if ever!). I also have my Documents etc backed up via Cloud but I pruned them and archived old ones.

I guess it requires a little thought about your data before you use this but thats IMHO of course.

And then I had a worry when my son was trying to convince me that if a HDD died I might lose more than the data that had changed since last PAR update. He suggested that any files XOR'd with any file that had been deleted that day would also be at risk. My old brain is too slow to work that out so I chose to ignore him. But if anyone can work it out please LMK!
post #124 of 214
Quote:
Originally Posted by Methanoid View Post

He suggested that any files XOR'd with any file that had been deleted that day would also be at risk.

It sounds reasonable, snapshot raid is good for "write once, read many" strategy, and if you had to delete many files it might be a smart move to make a sync.
post #125 of 214
Quote:
Originally Posted by erik71 View Post

Did you forget to multiply by 24? Or do you only have 500GB of data?

I don't know - I did the sums as per the docs

I also calculated for my actual 'real' data set of ~9T and ~405658 files I'd likely lose 50G to parity 'wastage'. i.e I'd have a parity file 50G larger than the data used on my most full drive.

I'd also need ~800 megs memory to run snapraid.

That's at the 256k blocksize. At 128k I'd lose 25G to parity and need 1.6 gigs working memory.

This is before we discuss the content file size for 400,000 files.

Quote:
Originally Posted by Methanoid View Post

I wouldnt claim to be running in anger but I am running it.

That works for me

Quote:
Originally Posted by Methanoid View Post

Content file is stored on parity drive and on SSD. Up until yesterday I had 5 of those drives pretty full (90%). The total of PAR file + content file (ie the usage of the Parity drive) was only a few Gb larger than the fullest drive. So unless I chose to fill a single HDD to more than around 97% I'd say there would be no issues. Filling that full would be silly as it would make any defrag operation a nightmare!! Memory usage peaked around 1.5Gb for SnapRAID during creation of PAR file.

Thanks that's useful info. Do you know how many files you might have roughly?

Quote:
Originally Posted by Methanoid View Post

Personally I decided to check out those small files before I created PAR. I dont care about meta files and thumbnails XBMC generates/needs. In the case of a data loss I would just recreate those by reimporting the media into a manager like Sickbeard or similar. So I excluded *.TBN and folder.JPG from SnapRAID. I also noticed I had many small files from Rom sets for old emulators. Since I rarely use them I chose to zip them up - saves space and adds no real delay if/when I need them (if ever!). I also have my Documents etc backed up via Cloud but I pruned them and archived old ones.

I guess it requires a little thought about your data before you use this but thats IMHO of course.

That's a good idea - and certainly an option. But yes, it means putting more thought into your data management both at initial setup and ongoing as your data usage changes / evolves. All of which I'd prefer to avoid. Though I'd like a lot of things I won't get

Quote:
Originally Posted by whelp View Post

It sounds reasonable, snapshot raid is good for "write once, read many" strategy, and if you had to delete many files it might be a smart move to make a sync.

Also if you take a snapshot successfully and then edit or change a file contained in the snapshot before trying to restore. The parity will generate invalid data from the changed file when reconstructing the data to be restored. How much is invalid will depend on how much data you changed.

Thanks for the input all.
post #126 of 214
I've read 4 pages of this thread and could not find an answer to the following, probably due to my lack edumacation:

I have been running chkdsk on a somewhat old and abused HDD which stores my iTunes music and I am seeing errors at a block level, errors which are identified for specific files, i.e., specific songs. So if I use this program, would the program be able to identify that a piece of data for a song has been corrupted and correct the corrupted data from parity data of the last sync (assuming that the song was present when synched)?

It seems that one of my HDD is about to fail and chkdsk is finding several errors everytime I run it. So I am wondering if this program would fix errors caused by hardware degradation of the HDD. Of course, I plan to replace that old HDD with a new HDD, but I would like to prepare for the eventuality when that new HDD starts to fail, too.

I am asking because I would like to backup my music but leary of using a program (like Acronis) which, I think, would back up the corrupted data and I don't think that I'd be able to permanently save every full back up (and I don't like incremental backups, either), not to mention trying to figure out which backup has the uncorrupted copy of the song.

I have read your faq and your website and is my situation what you call "silent corruption"?

Thanks.
post #127 of 214
Quick question as I'm thinking about using SnapRAID but it will be my first venture into anything RAID-like.

I've read that you need a parity drive as big as your largest drive. That is no problem as I have 4 2TB drives and I intend to use 3 of them for storage and one for parity. Now in Windows 7 I can transform the storage drives into dynamic drives and then span them so it appears as one volume. If I do this, can I still have 3 drives for storage (i.e. a 6TB spanned drive) and 1 2TB drive for parity, or would I now need a 6TB parity drive too?

Thanks in advance
post #128 of 214
Quote:
Originally Posted by soldave View Post

Quick question as I'm thinking about using SnapRAID but it will be my first venture into anything RAID-like.

I've read that you need a parity drive as big as your largest drive. That is no problem as I have 4 2TB drives and I intend to use 3 of them for storage and one for parity. Now in Windows 7 I can transform the storage drives into dynamic drives and then span them so it appears as one volume. If I do this, can I still have 3 drives for storage (i.e. a 6TB spanned drive) and 1 2TB drive for parity, or would I now need a 6TB parity drive too?

Thanks in advance

I dont think your plan will work without a 6Tb parity drive as the OS will just see one drive and SnapRAID is a file based RAID if i understand it. If you need a single volume (why?) why not add some storage pooling s/w like DriveBender or Liquiese (sp?)?
post #129 of 214
Quote:
Originally Posted by Methanoid View Post
This program is JUST what I wanted. I had no issues with editting a conf file but I also wanted to look at Elucidate and that seemed to show errors being chucked out left and right so I went to CMD line instead. If I ran from Elucidate it seemed to stop with red errors. Hence CMD line.
My conf (if anyone can spot any bloopers that might cause Elucidate to freak)
I did not spot anything obviously wrong in the conf file.
What sort of errors, Can you zip the log file up with the relevant information for me please, and create an issue @ "http://elucidate.codeplex.com/workitem/list/basic"
Also, what version where you using ?, Does this still happen with the latest maintenance release "http://elucidate.codeplex.com/releases/view/70494"?
Cheers
post #130 of 214
Quote:
Originally Posted by Smurf-IV View Post
I did not spot anything obviously wrong in the conf file.
What sort of errors, Can you zip the log file up with the relevant information for me please, and create an issue @ "http://elucidate.codeplex.com/workitem/list/basic"
Also, what version where you using ?, Does this still happen with the latest maintenance release "http://elucidate.codeplex.com/releases/view/70494"?
Cheers
I posted this weeks ago and gave up soon after with Elucidate - sorry but these days you normally get a response quicker if you are going to get one at all.

Still happily running SnapRAID with my hand editted config.
post #131 of 214
This may be a bug in Elucidate, I'm not sure... Installed, set the location of snapraid.exe and my snapraid.conf. I select Diff and get this error

Quote:


2011-09-07 16:11:52.5264[10][] WARN Elucidate.Form1: Start StdErr
2011-09-07 16:11:53.9665[8][] INFO Elucidate.Form1: Process StdOut[Self test...]
2011-09-07 16:11:53.9665[10][] WARN Elucidate.Form1: Process StdErr[Disk named 'd2' not found in 'C:\\snapraid\\content' at line 4884530]

I check my conf file and noticed that my Data disks were changed. Originally it was set as
disk d1 E:\\
disk d2 F:\\

After starting Elucidate for the first time it was changed to
disk d0 E:\\
disk d1 F:\\

After fixing my snapraid.conf, Elucidate works normally now. Under recovery options, if I were to select sync zeros, does it start syncing immediately with no warnings? If so, it may be best to add a "Are you sure?" dialog under recovery options once you select your option.
post #132 of 214
Quote:
Originally Posted by Mrkazador View Post

This may be a bug in Elucidate, I'm not sure... Installed, set the location of snapraid.exe and my snapraid.conf. I select Diff and get this error
I check my conf file and noticed that my Data disks were changed. Originally it was set as
disk d1 E:\\
disk d2 F:\\
After starting Elucidate for the first time it was changed to
disk d0 E:\\
disk d1 F:\\
After fixing my snapraid.conf, Elucidate works normally now. Under recovery options, if I were to select sync zeros, does it start syncing immediately with no warnings? If so, it may be best to add a "Are you sure?" dialog under recovery options once you select your option.

Sync Zeros is something I am looking into, It shouldn't do anything yet (Should remain disabled), But thanks for the suggestion, I think All of the Buttons on that page should have a "Are you Sure" style message.
I'll have to look again at the SnapRaid usage, But I think that the names of the disks can be anything, As long as they remain in the same order and obviously have the same name. I'm glad it was obvious for you to fix.
post #133 of 214
So I just started using snapraid today, and I am finding it excruciatingly slow. I am running ubuntu 11.4, with a 2tb parity and a 2tb and 750 gb that are 90 percent full. I have a core i3 and 3 gigs of ram on my server.

When I run "snapraid sync" it just says "syncing". After an hour, the "parity" file is only 44.4 gbs. At this rate it will take a couple days to finish. I have specified in the snapraid.conf file to ignore some small files I have on the hdds, so I dont think that is the issue. My drives are plenty fast, getting over 100 MBs writes. I dont know why it is taking so long...flexraid on windows ran much quicker.

Also, ctrl-C doesnt seem to stop the syncing.
post #134 of 214
my $.02
@sound dropouts:
Have a look at the following thread "http://sourceforge.net/projects/snapraid/forums/forum/1677233/topic/4607751" it gives some test areas to see what / which drive might be making things a bit slow. Please add your results and drive specs to that to help the developer come up with a plan and to give some advice.
post #135 of 214
i was planing to build a FLEX HTPC/NAS and then i saw this solution...
reading more about it here (still at page 3) but i saw this (quote) the same system i'm building (right now only 4 2TB hdd's)

Quote:
Originally Posted by Shaaden View Post

I've used Snapraid V1.0 on a Windows7 64bits (NAS+HTPC+torrent dl) for about a month and totally love it, so I'm pleased to see this new release 1.1 with a few new minor features and performance increase on the Windows side. Looks great !

I've just download it, reconfigured the conf, and started a new sync form scratch.

Configuration:
_ AMD Zacate APU E-350 1.6GHz (low power CPU!) on a Sapphire Pure Fusion Mini E350 motherboard.
_ 4Go DDR3
_ Windows 7 64-Bits on a Vertex SSD
_ 5 WD 2To Disk (WD20EARS) for Data
_ 1 WD 2To Disk (WD20EARS) for Parity

Quote:
Originally Posted by Smurf-IV View Post

I'm the author of "http://liquesce.codeplex.com/" and it does the pooling that you require, it currently has issues (From the driver layer) with shares on Windows 7 and above, but these are currently being worked around via my software (Not released yet - as performance is a bit poor!)

sounds good... a good pair to the snapraid, i you'll work out the problems i'll use it with my HTPC/NAS... why have 3 2TB hdds when you can have 1 6TB
but i use WIN7 so i'll have to wait


i'll be back soon for more help from the forum as soon as i get my E350 mobo and mem
post #136 of 214
Any update on the development?
post #137 of 214
Quote:
Originally Posted by RamanRB View Post

Any update on the development?

About what, specifically?

I have been using snapraid for months now. It works well.
post #138 of 214
Anyone got any feedback. Thinking of switching over from flexraid. Is the setup pretty easy for this? Hopefully the GUI is decent.

EDIT* Looks like a lot more work than flexraid. I suck at CMD line.
post #139 of 214
Quote:
Originally Posted by whiteboy714 View Post

Anyone got any feedback. Thinking of switching over from flexraid. Is the setup pretty easy for this? Hopefully the GUI is decent.

For the SnapRAID GUI, )Windows only) there is Elucidate "http://elucidate.codeplex.com/documentation"
Pooling is mentioned above
post #140 of 214
Quote:
Originally Posted by whiteboy714 View Post

Anyone got any feedback. Thinking of switching over from flexraid. Is the setup pretty easy for this? Hopefully the GUI is decent.

EDIT* Looks like a lot more work than flexraid. I suck at CMD line.

What is your reasoning for switching from Flexraid?
post #141 of 214
Quote:
Originally Posted by Lars99 View Post

What is your reasoning for switching from Flexraid?

I'm a cheap *** and they are starting to charge $ this month.
post #142 of 214
I am contemplating moving to snapraid due to flexraid going commercial. Does the parity drive in snapraid work as it does in flexraid - in other words, do I need one parity drive the size of my largest hard drive and can have as many other drives as I please, all protected by the system via that one parity drive?
post #143 of 214
Quote:
Originally Posted by cybrsage View Post

I am contemplating moving to snapraid due to flexraid going commercial. Does the parity drive in snapraid work as it does in flexraid - in other words, do I need one parity drive the size of my largest hard drive and can have as many other drives as I please, all protected by the system via that one parity drive?

Yes. Although if you have a lot of drives, you may want to consider dual-parity.

Also, snapraid works best if none of your data drives are completely full. I try to keep all of mine less than 95% full. But if you want to push it closer to the edge, I would recommend leaving at least 20GB free on every data drive. That leaves some space for the content files, and also room for overhead from block_size that causes the parity file to be a little larger than the data on your drive with the most data.
post #144 of 214
Quote:
Originally Posted by jim2100 View Post

Yes. Although if you have a lot of drives, you may want to consider dual-parity.

Also, snapraid works best if none of your data drives are completely full. I try to keep all of mine less than 95% full. But if you want to push it closer to the edge, I would recommend leaving at least 20GB free on every data drive. That leaves some space for the content files, and also room for overhead from block_size that causes the parity file to be a little larger than the data on your drive with the most data.


Thanks. I never fill a drive up fully...that way the drives can use the space to recover from physical errors on the disc as needed while I RMA them. Habit from the old days.

Time to learn a new raid program...I feel a bit betrayed by FlexRAID, I helped him out as a tester from the very beginning and now he wants me to give him money for it.
post #145 of 214
How dare someone ask for money for software they wrote, when it's the core of your multi-hundred/thousand dollar storage system. The insanity...
post #146 of 214
Quote:
Originally Posted by Lars99 View Post

How dare someone ask for money for software they wrote, when it's the core of your multi-hundred/thousand dollar storage system. The insanity...

Serious! Who cares about the hours upon hours he spent developing it. And helping out people using it.
post #147 of 214
Quote:
Originally Posted by Lars99 View Post

How dare someone ask for money for software they wrote, when it's the core of your multi-hundred/thousand dollar storage system.

Fortunately, people have a choice. They can use SnapRAID, which is free and open-source. The author of SnapRAID cannot suddenly decide to withdraw all free versions of the program and charge people to continue using it -- so far, all versions of SnapRAID are GPL'd.
post #148 of 214
Quote:
Originally Posted by Lars99 View Post

How dare someone ask for money for software they wrote, when it's the core of your multi-hundred/thousand dollar storage system. The insanity...

Quote:
Originally Posted by whiteboy714 View Post

Serious! Who cares about the hours upon hours he spent developing it. And helping out people using it.

It started out free. Suddenly - POP - not free. Had it always been said it would not be free, then you would have a point. That was not the case.

It would be no different than SnapRAID suddenly pay (provided he removed the parts he could not use then and replaced them with his own code).
post #149 of 214
Quote:
Originally Posted by cybrsage View Post

It would be no different than SnapRAID suddenly pay (provided he removed the parts he could not use then and replaced them with his own code).

It actually would be different, because the source code for all SnapRAID versions released to date is under GPL. Even if the author decided tomorrow to keep all future code private, we could still go on using any version of SnapRAID that has already been released under GPL, and we could even take over development of the already released code and release updated versions (also GPL'd).
post #150 of 214
Quote:
Originally Posted by cybrsage View Post

It started out free. Suddenly - POP - not free. Had it always been said it would not be free, then you would have a point. That was not the case.

It would be no different than SnapRAID suddenly pay (provided he removed the parts he could not use then and replaced them with his own code).

As far as I know it was always in beta. So you were never even using a final version. Plus he said he would hook people up that gave their time to the project. Unless by helping him test you mean you just used the software like everyone else.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Home Theater Computers
AVS › AVS Forum › Video Components › Home Theater Computers › SnapRAID: An Open Source alternative to UnRAID and FlexRAID