SnapRAID: An Open Source alternative to UnRAID and FlexRAID - Page 2

For sure it could be done, but you still need to have additional storage, and it could be an inconvenience. I see a better solution to reduce the memory usage and allow to use a small block size.

But I'm still unsure if it's really needed. In fact, it's a problem only for the Windows 32 platform and when you have a lot of small files. And the new WHS is x64, correct ?

Guys, how many files do you usually have in a single disk ? Just to understand the typical scenario.

110 MB/s is a bit slow for 6 disks :/ Likely some performance tuning is needed in Windows.

This value is for real data only, not counting the parity file, but it's still too slow. I would have expected something in par with Linux with 300 MB/s.

Can you please run "snapraid -T" and report the MD5 speed result ? (you do not need the array, it's an in memory test only)

If you still have the test array in place, please also try the "snapraid dry" command, and report the speed. (It only read, no danger on it)

Thanks!

I just spot-checked two of my 2TB disks: one has 4750 files, and the other has 9011. This is just counting the files in my user shares. They are 89% and 87% full, respectively. It's going to vary a lot for me, as I keep everything on my server, not just media (though media is the vast majority of it). I tend to rip DVDs as folders, and Blu-rays as ISOs, so the number of files is going to vary greatly depending on how much of what each disk happens to have on it.

Just for a point of reference, I just had to re-build my FlexRAID array yesterday. I have about 21TB of data spread between 17 drives (two of which are brand new/empty). It took just a hair under 21 hours, building a dual parity ("RAID6") array.

Technically, it's 18 "drives", but one of them is small (the "d:\\shares" partition of my system drive), and due to FlexRAIDs ability to span data units (as well as parity units), it is spanned with another <2TB drive to count as one data unit. Two combined into one is supposed to be marginally faster than if the two drives were counted as two separate data units, and if they both were to die it only counts as one failure.

Amadvance,

Thanks for a real open source (and GPLv3) project for non-striping RAID solution.
Not beeing able to code review or knowing someone else can maintain the code on single-person-spare-time-projects is a little bit to scary for me.
Also I'm all in for your design philosophy of "Write programs that do one thing and do it well", and combine or intergrate it with existing proven solutions.

Questions/comments:


Amadvance, what diskdrives are you running and are you running on hardware RAID or SAN/NAS? When doing a fast scanning of your code I can't see where you are reading files from disks in parallell. 110 MB/s read speed is quite normal for a standard disk. To me 300 MB/s sounds as a SATA-300 limit.

Regards
/XAD

I'm using 7 independent disks directly connected to a SATA controller. Without any hardware or software RAID. With this configuration I get something like 350 MB/s in Linux.

It's possible to get this throughput even if Internally SnapRAID does all the read operations not in parallel, because the operating system is smart enough to recognize the sequential read pattern and read-ahead the data. This happens in Linux almost automatically, but in Windows I have to open the file in a special way to trigger this behavior. I'll implement this trick in the next version.

About the 300 MB/s SATA limit, it's a single device limit. When you read from multiple devices you should be able to get more.

Thanks for reporting the data!

Part of the bottleneck in your case is also the way SnapRAID access the files. In Windows you have to explicitly hint the operating system that you are reading the data in sequential way, to allow it to do the necessary read-ahead to maximize the throughput.

It's something I'll put in the next version. Not so far away as I've already finished the RAID6 support .

OK. For less than 10.000 files, the wasted space is really irrelevant. It's less than 0.1%.
Even with 100.000 files you waste less than 1%, that I think is something acceptable.

Obviously, allowing to span the parity in more drives, and allowing a smaller block size are still good features to implement, but not the most urgent

Yes, new WHS is 64-bit but the old one is 32. And the new one doesn't have Drive Extender so it's too limited. But even with small block size parity can be bigger than maximum data size, correct?

I normally have 500-10000 files on my 2TB disks but there is one that has ~28,000. I think all my mp3s are there.

As a reference for speed tests FlexRaid takes ~12-16 hours to rebuild parity on 9TB of data spread to 9 disks. It depends on parity size mostly - if data is distributed evenly the parity size will be around 1TB and it may take less than 12 hours.
It takes ~30-35 minutes to update parity after adding 45GB blu-ray ISO. And I have Core 2 Duo 2.4GHz. CPU usage is rarely more than 80%. It's interesting to see if this CPU is enough for SnapRaid.

I will build a PC from spare parts to test this, probably next weekend.

I expect this wouldn't be an issue in the vast majority of cases, due to exclusions, spare space, etc. For example, I specifically exclude the metafiles and cover jpgs that are in my movie folders, because those could get updated without my knowing it. I use MyMovies, so that stuff is automatically updated, but I don't want it inadvertently "outdating" my parity without my knowledge (and a handful of small updates spread throughout a disk can trigger a lot of parity updating). Since those files are easily recreated if they are lost, I don't see the need to include them in the parity. I also exclude hidden system files like thumbs.db. And of course, there are shares on my server that aren't included in the RAID, because they contain non-static data. And last but not least, I leave about 100GB of empty space on each drive, just to ensure defragmentation doesn't run out of swap space, and to ensure I always have plenty of free space margin in case I suddenly need to remove a drive from the pool.

Media files do consume the vast majority of space on my server, and there's not guarrantee of how much space these things will all save (I could have a disk that has nothing but Blu-ray ISOs). But still, since my data drives are never 100% full, I wouldn't think an extra 1% of parity space would cause issues on a same-sized parity disk. Besides, 1% of a 2TB drive is 20GB, which just so happens to be the margin that WHS leaves as a default (instead of the 100GB I use) before it starts putting the data elsewhere.

I have tried it. Looks promising. Confirmed the bug:
https://sourceforge.net/projects/sna...8/index/page/1

Strange that this not get noticed by snapraid. I've tried to play with blocksize, etc. but it doesn't matter - even if file becomes hundreds megabytes from being under kilobyte check shows it is the same "touch" not helps as well. Manual edit get noticed however. Something about how Linux works with file modification time, perhaps. And, yes, disk mounted with noatime option as most storage system are. It may be the root of the issue.

Started initial parity creation (test run) from eight 2 Tb drives, 5 of them filled with ~20 Gb left, remaining partially filled. Default 256kb blocksize. The process shows ETA 80+ hours already and increasing

Perhaps because I use WD green (20EARS) as parity drive... But on other hand: according to the "dstat" I see ~230 Mb read and just ~5 Mb write. The RAM is 2 Gb (64bit) but it occupied already.

Would further syncs be faster? For example if I add another drive and/or remove existing?

What if I change ~100 Gb of data on single drive?

What happens to files which are sitting in renamed directory? E.g. it was dir1/file1, dir1/file2, etc. and I've renamed dir1 to dir2 leaving it on the same drive - I believe additional resync is not necessary, e.g. it should be pretty fast by updating new locations/names, am I correct?

The initial is always the longest. Re-syncing are typically short as it's only having to add/modify what has changed since the last one - like an incremental backup.

The bug is in the check command that doesn't detect extra data over the expected end of the file. This is a condition to check, and the file has to be truncated to the correct size to restore its original state.

Hopefully this bug doesn't invalidate the disk failure protection of SnapRAID, but it's sure something to fix for the upcoming 1.1 version.

Touching the file doesn't work intentionally, because the check command verify only the data and not the metadata. Otherwise a fixed file would be always wrong due the different modification time.

Today I've completed the sync of 8 2Tb drives to single parity drive. All of them are WD20EARS.

Sync was paused after ~5% and then restarted. Since restart it took ~24 hours to sync.

Then I've run sync for another time to check how long it would take to load the data, etc. (nothing was changed on the drives). It took ~9 min.

Another test was to remove ~33 Gb of data from one of drives and then run sync. It took ~27 min.

I didn't run fix to recover these 33 Gb of data because I believe it would scan everything and it take a lot of time, correct me if I'm wrong. Would be interesting to know how long does it take to recover comparing to resync.

Finally I've renamed big directory with ~43 Gb of data and moved it to specially created directory (so inodes are to be saved) and run sync. It took ~16min and I was told that Nothing to do

The sync speed is always the same, it only depends on the amount of data. Also, the check and fix commands will likely take the same amount of time of the first sync.

But to speedup the recovering process, you can use the -f option and specify only the files or directory/ to restore. It can be used to undelete, or undo unwanted changes for example.

Anyway, I think it's a pretty slower in scanning the disks. 16 minutes to do nothing is too much Maybe, are you running out of memory and it starts to swap ? How much RAM do you have ?

with virtual filesystem you mean to have a new virtual disk with a share, and that share is the "sum" (combination) of several shares with the same name in several disks?

e.g.

having

D:\\pictures\\pic1.jpg
E:\\pictures\\pic2.jpg
F:\\pictures\\pic3.jpg

with a "virtual filesystem" tool you could create a new disk+share (e.g. X:\\pictures), where you can find the three pic1.jpg, pic2.jpg and pic3.jpg?

If so, which "tools" can achieve it?

Thanks!

I use mhddfs to achieve this on my media server. It works great and my 6 1T drives can be accessed as if they were a singe 6T filesystem.

Tracy

thanks! but mhddfs is unix based. any option for win?
sorry not to be more specific before...

FlexRAID had something like this when I looked at it last... I can't get to the download now, though. Windows Home Server used something they called "Drive Extender" to present your storage as a single large virtual filesystem, as well. (Although they're supposed to be removing it from the next version of WHS.) I didn't have much luck with WHS when I tried it on a little test system, though. It was probably just that the system was underpowered, but the WHS server spent all of its time at near 100% CPU while it was indexing and moving files around - after a week of the thing staying pretty much unusable, I gave up on it. (P4/2.4GHz, 2G RAM, 4 1T SATA drives connected via external USB 2.0 drive enclosures. Just didn't cut it for my particular data set.)

Sorry I couldn't offer more help but this is about all I encountered while experimenting with Windows as a home media server. Even if I wasn't already a Linux maven, mhddfs would have made my mind up. I just hate having, say, 6 different "Television" folders and trying to remember which one "Lost Girl" is in. (shudder)

Good luck.

Yes, I've started fix yesterday (removed single ~40 Gb folder), 10 hours passed already and ETA is 16 more

Btw, why fix is so slow? Is it because it verifying each file to prevent silent corruption? You may wish to have some kind of quick fix which compare saved state with actual state and then restore (if needed) by computing the difference. You also would be required (uh, I mean advised to ) to create something like validate in flexraid, e.g. the quick check option. One absolutely not required to read ALL data from ALL drives with every check, it may be done quarterly while validating may be run weekly/monthly.

So, such quick check/fix would be great options to consider.

Regarding the speed, yes, server is 2Gb RAM and it looks like it is not enough:
total used free shared buffers cached
Mem: 2010 1958 51 0 1 28
-/+ buffers/cache: 1928 82
Swap: 8197 1368 6828

snapraid is in top with:
9647 root 20 0 2616m 1.6g 224 D 41 82.1 442:11.33 snapraid

So it consumed 1.6 Gb of real memory. Mine block size is 256kb, 5% of files are hundreds times huge than remaining (movies in BDMV format with some big m2ts files and tens/hundreds of smaller files of playlists, etc.)

The current read from drives is ~300-312 Mb/sec so it is very close to SATA II limit (on other hand half of them on different controller). I'm using deadline scheduler as cfq is too slow on massive reads.

Your advise?

P.S. I also having hard time using your filtering syntax. I've drives as /home/drive1, /home/drive2, ..., /home/drive8 with parity as /home/drive9. I'm trying to recover directory from /home/drive7/movies/new/DELETED/ by using this syntax:
time /opt/snapraid/bin/snapraid -c snapraid.conf.test -f "/home/drive7/movies/new/" fix
Unfortunately it doesn't work while man says if I use trailing slash it filter whole directory. Could you guide me to test this kind of restore? I'll try to remove /home, then /home/drive7 from beginning, perhaps paths are not absolute.

To get an estimation of the wasted space, you need to count the number of files in each disk. You can approximatively assume that in each disk you lose half of the block size for each file.
For example, if in a disk you have 10000 files, and the block size is 256KB, you lose approximatively 1.28 GB in this disk.

How this is being calculated exactly? E.g. if one disk has 10,000 file and other 100,000 files should I multiply 110,000 to half of block size to get estimate on loss?

Do you plan to have some workarounds for disks with large number of files?

I have this in my snapraid.conf file.

parity L:\\par\\parity
content L:\\par\\content
disk d1 C:\\Users\\mkz71\\
disk d2 E:\\
disk d3 H:\\
disk d4 K:\\
disk d5 U:\\
exclude *.bak
exclude tmp/
block_size 256

After I ran it, I got this error.
What have I done wrong about this?

C:\\snapraid-1.0-windows-x86>snapraid sync
Self test...
Scanning disk d1...
Error in stat file 'C:\\Users\\mkz71\\/AppData/Local/Apple Computer/iTunes/Cache.db
'

I have a newbie question about snapraid and flexraid. If I were to build a gaming pc/backup server, does this software run at all times or only when I set it to check/verify/sync the raid?

I don't agree with this. Its perfectly possible to have -

1. A core engine with a programmable api that runs as a service in Windows or daemon in unix
2. Cmd line tools that interact with it
3. GUI tools that interact with it

OR

1. cmd line tools that implement all the functionality
2. GUI that invokes cmd line internally and displays results graphically

Quite a few tools work in exactly this fashion - including all of Linux There is no inherent disadvantage of a GUI, quite the contrary in fact. I can understand you focusing on cmd line though.

Btw I really like your project page and goals - very well thought out.

Yes. Both "fix" and "check" always verify ALL your data. They are intended to be used after a disk failure, and a such check is really mandatory.


Yep. I think it has sense to have a new "restore" command, that brings the state back at the last sync.


I'm unsure about this. In fact, I do not really understand that the validate command is really doing. If it doesn't validate the data, what is it validating ? Only the metadata ?



If think that you have to switch to 512 kB block size, or add more memory to the system. I'm planning to review this issue after the SnapRAID 1.1 release. The upcoming 1.1 version is already more conservative in memory usage, but maybe other improvement are possible.

To evaluate the space that you waste for the block size, you have to count half the block size for each file. You have to keep this space free, or use it for files excluded from the parity.


All the paths are relative to the mount point. You should use only "/movies/new/". Or you can also specify only a subdirectory, like "movies/new/" or "new/".
I got this syntax from the rsync utility.

Likely this file has some attribute or permission that doesn't allow SnapRAID to read it.
You can exclude it adding the following rule in the configuration file:

exclude Cache.db

It's a good thing to exclude any file that often changes, and "Cache" files are surely this kind of files.

SnapRAID run only when executing a command, it's most like a backup program. You run it only to backup or to restore.

It seems that the error keeps changing.
I tried this time and it shows different error.

C:\\snapraid-1.0-windows-x86>snapraid sync
Self test...
Scanning disk d1...
Error in stat file 'C:\\Users\\mkz71\\/AppData/Local/Application Data'