A first look at NZFS and replacing unRAID with NZFS’s Transparent RAID (tRAID) - Page 8

Yes, that's it. I didn't pay attention to the username, lol. The fact that you are having deletes completely fail leads me to believe that you have a hardware or software problem. This isn't really the place to troubleshoot it, though. If you want to pm me or start another thread I'd be happy to help. Reiserfs is generally a very stable file system despite it's creators history. Deletes failing is definitely not normal.

Looking at the Flexraid vs NZFS comparison that Spectrum posted I'm almost tempted to say that Flexraid is a better product for us media junkies.

What makes you think RAID-F is better than tRAID for media junkies? Most of the advantages it has in the comparison are sort of odd things. Why would you want to exclude content from protection, only recover certain files, or have networked drives part of the array?

I would think that the lack of official support for real-time RAID and inability to operate in a degraded mode in the event of drive failure would be bigger deals. Not to mention the ability to recover from multiple drive failures (if you set it up that way).

In a previous posts I used term "enthusiast" to describe to whom UnRaid in my experience fits. When starting with UnRaid I had zero experience on Linux but 20 years of computer experience ranging from MSX, Amiga, 386 PC and so on. I saw UnRaid as a way to familiarise my self with Linux world. I built the server from scratch with all new components and I had my server running during the evening I started building it. Of course this was with stock features but I did have the first 3TB available to my htpc's. Later on I added feature by feature new things like APC UPS support, powerdown, smart history, rtorrent and cache-dirs. In my mind all of these require more general IT knowledge than Linux knowledge.

To get most of these kind of solutions you really need to have the proper background and an interest to learn more all the time. But what isn't like this in the htpc world? Lets take a few examples:

1. Smooth playback and AV-fidelity. Standard, non-enthusiast couldn't care less about things like judder and tearing, he just downloads movies from torrent, gets a codec pack and uses whatever player to play the movie. If the movie's audio is in the correct language, that's sufficient. An enthusiast searches for the holy grail of perfectly smooth playback studying forums and trying out different combinations in the play chain. An enthusiast wants to see that Dolby TrueHD sign light on his receiver. Personally I have most likely spent hundreds of hours (could be even thousand or more) studying things like reclock, ffdshow, different renderers and players configurations in my ten year htpc history.

2. Media center software. Both XBMC and Plex are quite easy to install and get running for the first time. But to take advantage of all the cool features you have to really dig into wiki and forums. And to have a perfect library displayed, you have to manage it properly (=spend quite some time). For a standard, non-enthusiast user it might even be sufficient to just be able to double click a movie file in the file explorer to play it.

In my mind UnRaid (and I presume also current FlexRaid) require same kind of enthusiasm level as other parts in the advanced htpc world. As said before I really hope nzfs or some other product would provide a polished, out-of-the-box, ready-to-use storage product. Unfortunately I don't believe any single person can pull this off. And for larger companies there just isn't large enough market potential. Plex is now doing this for the media center scheme eg. they already have their client integrated in TVs. I just hope they don't get bought soon...

Reggie 14, can you point to where Tom has said "he acknowledges write speeds could be ~50% faster if he fixed up the code"? I just don't see how it is even theoretically possible to achieve this with online parity calculation with any product.

I disagree
Makemkv should rip across your LAN at the same speed as it does to your desktop. The 2x read speed of your blu ray drive is less than write speeds across your network
Interestingly, the size of your collection in disk space doesn't matter for metadata scraping. It's the number of folders your scraping. 25TB of isos might only be 500 individual titles which would actually go faster than 2TB of 1000 crappy quality titles.

Since you use the premium version of MCM, try modifying the thread count. 4-6 will speed up the auto-scan

Leave the auto-scan on, but lock your titles that are complete. It will get to it pretty quick, and if you see a "recent" addition in your library with the wrong cover you can go back and fix it manually
Edited by Dark_Slayer - 2/24/13 at 12:03am

Well said about the enthusiast level knowledge that most people have to have to set these things up. But if I may... there are quite a few nearly plug-and-play ZFS distros now. No need to wait for "nzfs", when you can get the real-deal now. FreeNAS, NAS4Free, ZFSguru, NexentaStor, SmartOS, and OpenIndiana with Napp-it are about as easy as it gets for setting up a media server.

Bad wording on my part, it is better because it is available today and has a good track record. Real-time RAID is a good feature. I could have sworn RAIDf had multi-parity support. tRAID is in beta and who knows if it will work perfectly when it comes out. It should work perfectly, I'm not trying to put anyone down but new software normally has growing pains.

Being able to exclude content from the RAID to me means that you can have a scratch disk.

There are already numerous threads here that discuss the pros and cons of each program as they've been compared in great detail ad infinitum. I've been using unRAID for about five years and it's done everything I could possibly need in a media server app. I've read through every tutorial and feature description on FlexRAID and haven't seen anything that overwhelms me enough to switch. The only clear advantage to using FlexRAID, at least from my perspective, is the ability to use multiple parity drives and also to be able to read the drives from a Windows PC, unless, of course, you're using FlexRAID on a Linux platform. That's great for data protection, but not all that necessary for a media server. While you may not be able to replace missing data, you can almost certainly replace lost media unless you trust the server hardware never to fail and decide to place your only copy on one of the drives. Irreplaceable data of any kind probably shouldn't be entrusted to a server for safekeeping, IMHO. FWIW, I have successfully recovered data as a result of a single drive failure on more than one occasion with unRAID. Since I tend to purchase drives one at a time I don't have to worry about multiple drives failing from the same batch simultaneously.

I prefer unRAID because it is simplicity personified and the closest thing to an actual plug and play setup. There's no OS to install nor is there any software RAID app that requires installation. You prepare a USB flash drive, copy the files to it, plug it into the PC, and boot it up. It just doesn't get any easier than that. You can have it up and running before you even get halfway through a WHS installation. I particularly like the idea that I don't have to sacrifice a SATA port for an OS drive. UnRAID has more support than any of the current software RAID apps as clearly evidenced by the unRAID support forums. Hardware issues are far fewer with unRAID than they've been in the past, although there are still more than with FlexRAID, but only when used on a Windows platform. Cost for a 24-drive setup between the two is just about dead even. The cost of an unRAID Pro license plus a flash drive will run you about the same as a FlexRAID license, WHS license, and a small OS drive. FlexRAID will work with more than 24 drives, but if you're building a server that large you should probably look into something a bit more robust than either program.

One thing I don't recall seeing mentioned about unRAID is that you can plug a drive into the server and run the pre-clear utility in the background without affecting the server operation. When the pre-clear is complete simply stop the array, add the new drive, and start it back up.

Before you judge unRAID as a "horrendously inferior and incompetent (to some levels) product," do some homework by reading through the unRAID wiki and the FlexRAID tutorials. You should then be able to come to an informed decision as to which product better suits your needs. People here tend to be passionate about the particular product they use and their opinions are somewhat skewed as a result, including mine.

That is ok. We can agree to disagree.


I think you missed a couple pieces of the problem. Yes you are correct that MAKEMKV can rip across my LAN at the same speed. I even specifically mentioned that even two streams at once is under my LAN performance.
The size of my collection does matter when I do auto scan, or episode level auto scan. It's much slower and locks up. I do run 4-6 threads. Sometimes I run 8 threads. That is good advice but I am already doing it.

The piece you are missing is that the performance of 20TB collection over LAN just sucks. It's both speed and size. For me to "lock" all 20TB of my media alone would take forever, and then I would have to unlock it every time I want to change something. That is not going to save me anytime. Also, simply searching out the items that need attention burried within 20TB of other media is not saving me time and not desirable. I'm a bit anal about my media mgmt and I like to sit and do it and watch it, confirm and make sure it's done perfectly like I want. I like to control it. I find it just simply easier to control when working with a smaller folder full of only media that needs attention. I tend to do only 10 or so at a time. It's pretty easy and it goes pretty quick. When I am done I move it over to the server. The entire process only takes me a few minutes. It would take much longer if I did it your way.

I have a folder that I made a library with on my scratch disk. The folder and the library shortcut are called "- Mediamaster" I put the - in front to make sure it was the top library. So any file I can drop it into that folder and tune it up. That is the folder I have my mediacentermaster set to scan as the scan folder. It rarely changes.



It's a root level folder on my RAID0 scratch disk. So regardless if the media file is a download or a rip or wherever it comes from I can drag and drop it into that library and it get worked by medicenter master. There is no copy paste at all. It just relocated or repopulates. I have my 2TB scratch disk set up so that happens. All I do is open mediacentermaster and I am good. There is no changing scan folders either.

Under your suggestion I would be constantly changing the scan folder of mediacentermaster to the appropriate library collection. I have HD movies, Non HD movies, Blurays, DVD's, Disney, TV shows, etc... I would have to rip directly into those libraries to avoid a copy paste. So I would be constantly changing my MakeMKV program and where it rips to.. and I would be changing the scan folder on mediacentermaster chasing new material. If I set mediacentermaster to read all libraries it would literally take 20 minutes to scan all 20TB and for the program to open.

If I ripped over LAN to my server but not inside the final resting place and library collection I would still need to do a copy and paste relocation which is slower over LAN than if I just did it locally on my desktop and put it on the server once it was done.

lastly,

There is many times I drop a bluray in and rip it, and walk away like go to work or leave. When I return I don't always organize or run mediacenter master at that time. Often media might sit in my mediamaster folder which is my "to do" folder for a few days or more before I get around to doing it. I tend to do about 10-15 at a time only once or twice a week. So that means I was have incomplete media on my server and inside my collections - which effects my HTPC and extenders all over the house. Not good.

I know you mean well in your suggestions. I have no doubt smaller or more simple set ups and less anal users your method would work fine. But for me it does not. I have tried it.
My original point was just the open idea and acceptance of individual methods and desired performance levels.

Just because 40MB/sec is good enough for many does not mean it's good enough for all. Just like how your suggested method is probably perfect for many but not perfect for me.
I'm confused why I get so much resistance on the issue of server performance. It's like perhaps some are jealous my server is faster, or they need to convince themselves what they have is fast enough. I just don see how having a faster server or network performance could ever possibly be a bad thing- so I don't understand the resistance I get when I show my appreciation for it. I never said everyone needed it. I just said personally I appreciate it and listed a few reasons why I felt that way.

Seems odd to have to defend this as much as I have.

Edit: Not talking directly to you in all this but more in general.

I have tried a whole assortment of software RAID solutions before settling onto FlexRAID. FreeNAS was great but didn't fully meet my specific needs and I currently don't have the money for a separate box. I have tried unRAID too and my experience with it have been far less than satisfactory. I wouldn't consider unraid to serve my storage needs.

Like captain_video said, people tend to be too attached to the product they use, which causes their opinions to be skewed.

He said it here.

I know the issue of maximum theoretical speed comes up regularly on the unRAID forums. I've seen a couple different formulas thrown around, but I don't understand completely how they derived them. With DMA, the reads/writes to data and parity drives can occur in parallel (well, mostly parallel). So it seems like the maximum write speed should be:
Minus, of course, some amount of overhead for the CPU to send the DMA requests and perform the parity calculation, which should be fairly minimal. That would still leave a lot of headroom for write performance improvements.

Update:
Nevermind. This post explains the other formula used, which says (ReadRate+WriteRate)/4. Nonetheless, Tom still seems to think he could get a pretty good performance increase. I'm not quite sure how.
Edited by reggie14 - 2/24/13 at 9:06am

amarshonarbangla do you ever actually post anything or do you just make blanket statements without any rhyme or reason to them?

FYI to all:

Apparently SSD's are supported. Here is someone using a Samsung 830 as a cache drive on Unraid: http://lime-technology.com/forum/index.php?topic=5754.msg192955#msg192955

To summarize 118MBps is possible. Others are reporting around 110MBps. Apparently my old athlon x2 just isn't fast enough to push my cache drive faster in UnRAID. It looks like an i3-2100 is a good minimum.

You should check my post history and look up my reasons for not going with FreeNAS
And you have pointed out the reasons why I didn't go with unraid in this thread. You have done my work for me. How convenient is that?

I hope you enjoy your arguably slightly better RAID solution.

Thank you again for not adding anything to the discussion while you make blanket statements. I tried FreeNAS also and experienced many crashes on the same piece of hardware that I ended up installing Unraid on so I share your frustration with that.

It's like making excuses. It's not making sense.

If I was a fan of a football team I don't think I would make any excuses for my QB if he could only throw the football 20 yards. I'd want him to be able to make all the throws needed properly and would be frustrated at anything less.
It''s really the same thing.

Yes you could probably win a game without throwing longer than 20 yards, but is that the most desirable ?

No one is saying you can't use something that is slower or less performance so I don't understand the resistance against the appreciation of better performance.

And- I do not mean this only about the Flexraid vs Unraid thing. I admit unraid is a perfectly acceptable solution that works well for many. I have no hate on Unraid or it's users.
I meant more about the general idea that higher performance is not good or not needed. It's always good. If your not the type of person or application to appreciate the higher performance than just go with "good enough".


How many people on this forum have HTPC's with i3's or higher in them ??? Why? Everyone knows the basic Celeron chip for $35 can do 1080p and HTPC duties without problem.
It's same thing on HDD choice, server set up choices etc... One size does not fit all. That's my only point.
Edited by Mfusick - 2/24/13 at 10:56am

The problem with SSDs is TRIM support. The cache drive will work, but without TRIM its performance will likely degrade over time. ReiserFS, the filesystem used by unRAID, doesn't support TRIM. You can hack something together by running a cache drive on an unsupported filesystem, but you need to script some things to get it work work.

As I understand it, Tom is looking to add support for using the btrfs filesystem for the cache. That should bring TRIM support. Who knows when that will be. I wouldn't hold my breath, though.

Huh? What makes you think its CPU-bound? That's the whole point of DMA. More likely its a bottleneck with your NIC or SATA controller.

Samsung controllers, OCZ version of Sandforce, and also the new Vector controllers all do real time garbage collection and will function perfectly without TRIM.
It's really only the older Marvel controllers that might have issues long term. And even then its going to take a while.

I'd have no issue employing an OCZ or Samsung SSD long term without trim. I ran two Vertex3's in RAID 0 for an OS over 6 months without Trim and without any issues at all.

TRIM = Overated.

Most modern SSD's do it on their own in the background without windows assisted TRIM function and it's proven OCZ and Samsung do a fine job without trim.

I will

My point exactly. Not sure why the resistance. I am trying to figure that out myself.
Edited by amarshonarbangla - 2/24/13 at 11:20am

Fair enough. The garbage collection they do now seems to be enough if it has enough idle time to run. For something like an unRAID cache drive it almost certainly would.

For what it's worth, BSD (what FreeNAS is based on) is extremely stable, but very picky about the hardware it supports. I'd say this isn't much different than UnRAID.

I have to admit this was over 2 years ago so, I'm sure things are better now. I have run other FreeBSD based software on the same platform in the past and I don't remember having many problems.

Reiserfs is really meant for small files. When using large files it apparently isn't very efficient. My dual-core processor has one core maxed out when I copy to my cache drive so I'm fairly certain that with a modern processor my writes would be much faster(this is what the thread I linked is showing). Using a modern hdd for my cache drive would probably help also.

I don't believe Apple has a single product that supports trim and they sell a lot of SSD's with there products.

If you look at the link I posted it goes into the reasons better.

BTW, WTF is NZFS?

Am I the only one who sees no or little relation to ZFS proper? This tread contains comments that imply NZFS is somehow related to ZFS proper. Unfortunately, this confusion has neither been corrected nor confirmed by the author.

Best I can tell, the product aims to be some type of multi-parity Raid 4 kernel driver. I welcome any correction, refutation or expansion of this assumption.

From my admittedly limited knowledge of NZFS, it appears to share ABSOLUTELY NO code with the CDDL from Oracle. Nor does it appear to share ZFS' principle architectural goal (filesystem checksums in order to combat bit rot) found in all current and historic ZFS variants (Open Solaris/Oracle/Open Indiana/illumos/FUSE/et al.)

Please answer this question: will NZFS implement ANY of the major features of ZFS as originally proposed by Sun [1]?

Dynamic striping across all devices to maximize throughput
Copy-on-write design makes most disk writes sequential
Multiple block sizes, automatically chosen to match workload
Explicit I/O priority with deadline scheduling
Globally optimal I/O sorting and aggregation
Multiple independent prefetch streams with automatic length and stride detection
Unlimited, instantaneous read/write snapshots
Parallel, constant-time directory operations

[1] http://web.archive.org/web/20071015014209/http://www.sun.com/2004-0914/feature/

Thanks for providing any clarity.

I will admit that I haven't read this whole thread. I have read most of it, and I've tried to read what I could from the developer. What I see is that he is obviously trying to piggyback on the reputation of ZFS based on it's name. I believe ZFS to be the best file system for a storage server. It does usually take a higher level of technical expertise to use, and I think a slightly larger budget to set up and run. This stems mainly from not being able to add single drives, but in groups. I personally have been using Unraid for a number of years. I am happy with it, and it has kept my 27TB of media safe and accessible. It does what I need, lets me add drives as my budget allows, and lets me read individual drives in the event of a catastrophe.

This does not mean I don't hear the siren call of greater speed and a new project to tinker with. That's why I want to set up ZFS as a virtual server. Then it would be accessed by WHS over a virtual NIC. That way I would get all the speed available. I would have my data protected, but I would still have the other advantages of WHS. This is what I would like to do. I just don't have the budget for a new controller, and the extra hard drives I'd need for it. The other fact is the Unraid works for me. I just want to play with ZFS for the challenge.

So the reason I've laid all this out is to show in some small sense that I'm open minded and willing to explore new ideas. What I would like to see is how this new product compares with ZFS. I think since he's using the name people are going to compare it to ZFS. So far short of the name I haven't seen how this has any of the advantages of ZFS. ZFS is a serious file system developed by a large company and thoroughly tested by the open source community. This gives me a level of comfort that I don't get from a small shop. So before I would consider it I certainly want a lot more information. I would also need some sort of reassurance about the safety of my data. Honestly though I don't see how that would be accomplished short of a substantial amount of time, and a large user base finding all the problems. So while I'm interested, I will stick with my proven solution for now. I do believe competition is good though. If there are more players in this game perhaps they can push each other to our advantage.

Edit
The poster above beat me to my main question.

That may be true, but there are other aspects involved that make your example irrelevant. I have an i3 3225 in my HTPC, but I didn't get it for the higher CPU performance. I got it for the HD4000 graphics, which is only available in this CPU or higher. A Celeron won't give you the same graphics performance without the need for a dedicated graphics card.

With respect to the whole unRAID vs. FlexRAID vs. NZFS discussion, not everyone needs the extra performance or features that FlexRAID or NZFS will provide. There's little point in being better than "good enough" if the extra features and added performance aren't being used. My needs are fairly basic so I tend to opt out of the extra bells and whistles because I know I'll probably never use them. As long as my server does the job there's no need to upgrade to anything else. That's not to say that I won't give FlexRAID another shot or maybe even try NZFS when it becomes available.

In my case it ends up being more about time and money than anything else. I already know it will take me quite a while to migrate from my current setup to one that uses a different file system. If it turns out that I need to get different hardware to make it work then chances are it's not going to happen.

For someone just starting out, it's a tough decision to make regarding which software RAID solution to use. With unRAID, you only have to get a compatible flash drive to try the freeware version. With FlexRAID, you pretty much have to commit right off the bat because it requires an OS license and a spare hard drive just to use the trial version.

Exactly. Which is what sets the HTPC forum here at AVS apart from anandtech, overclockers, hardforum, etc. Like it or not this is a HTPC forum and there are those people who are building a HTPC and not a top of the line performance PC (we have this argument here often and some will disagree which is fine. When I get confused where I am I look at the Forum Title and note that it says "Home Theater Computers". Yep, still a HTPC forum )

If everything was exactly the same price then sure --- get the Ferrari.

But unfortunately increased performance comes at a $$$ cost (not to mention usually heat, noise, larger size, etc)

if it's the same price then I agree with you.

I'm inclined to agree with spectrumbx here. At least, I don't see a huge advantage of one over the other. If you happen to have all the equipment you need for your server, but can't decide between unRAID or FlexRAID, then you could install Linux (potentially on a flash drive), or use install Windows and just hold off on activating it.

If you don't have a server yet,it might actually be easier to test FlexRAID, since you could just use the Windows box you probably already have on your desk. I suppose you could basically try out unRAID on that machine, disconnecting any drives with data on them, but that's going to be more work (and you'd need spare drives).

But better does not have everything that good enough has and vice versa. In which case, if the good enough is "good enough" whilst having features that you prefer over the "better" solution, then the decision is based on feature set.