A first look at NZFS and replacing unRAID with NZFS’s Transparent RAID (tRAID) - Page 6

I'd rather light my reproductive organ on fire and then pound it flat with a sledgehammer in an effort to extinguish the flames than suffer those speeds.

You've pretty much summed up why someone with 20TB of media would want Flexraid over Unraid; albeit unintentionally.

I get mad when I get only 75MB/sec. It not uncommon to see over 100MB/sec at all with Flexraid. It's usually the bottle neck of your HDD or LAN that you see for a performance factor.

...
Edited by PobjoySpecial - 5/16/13 at 2:29pm

Cache drive with 5.0rc4 and I only get around 50MBps. I'm not network limited either.

but I copy and paste from my server with regularity to the point my impatience makes the slower transfer "unacceptable" to me.

Ever sit and wait for a transfer or copy paste ??? I hate it. It's not weather or not it can sustain a fast enough transfer to stream 1080p. That's nothing to me.

It's the significant amount of time I spend organizing and building my collection. Ripping. Moving. Storing. I do most work locally on my desktop then move to the server. I can not tolerate those speeds on a daily basis.
Managing my media isn't something I do once in a while and the forget it. It's a near daily ritual for me. I like being able to do it while I sip a beer or coffee then I am done. Waiting for a 40MB/sec paste function to complete versus my 110MB/sec now would literally take almost 3 times a long.


Let's put it into perspective:

Moving 4 bluray rips to my server @ 100MB/sec would take about 17 minutes assuming the total of the 4 bluray rips was 102400 MB (that is 100GB or 25GB per rip)
(my math sucks for this crap so please correct me if I am wrong anyone)

Here is how I think it works: 100GB = 102,400 MB. 102,400MB divided by 100MB/sec is 1024 second. 1024 second divided by 60 is 17 minutes.

Lets do the same with 35MB/sec.

102,400 MB divided by 35MB/sec = 2925 seconds. Divided by 60 = 48 minutes.

17 minutes or 48 minutes?

That's for only 4 bluray rips.

I'm not sure I understand the point people make when they say "good enough". It must be my personality because to me it's a huge deal. I have 20TB of media on my flexraid server. I move stuff all the time.
Such a deal breaker for me.

Eh. Who cares?

I move mine when I sleep. I also don't care what its doing in the background. I got better things to do I guess than lament over the transfer speed on my home media server.

To each their own.

50MBps write is plenty fast enough for most people. The 100MBps read is nice

For anyone complaining that unraid isn't based on a full linux distro: http://lime-technology.com/wiki/index.php?title=Installing_unRAID_5.0_on_a_full_Slackware_Distro

For the initial build 50-60MBps can be limiting and I can understand this being seen as a limitation in this case. When drives get near the end they slow down to 70-80MBps anyways(slower still for some drives).

It takes very ideal circumstances(read not normal) for Flexraid to surpass the read/write of a single drive. You are still limited to gigabit speeds if you are using it as a NAS like most people.

How do the write speeds look on NZFS? or Flexraid real-time?

Sounds like you know what your talking about

When people say stuff like "it's good enough" I get very frustrated because all things can stand to be improved. Even excellent Flexraid.

It's far more frustrating when users decided what is "good enough" for others. Just because one person is willing to settle for a lower level or performance does not mean everyone should or it's appropriate for all.

These guys with a couple plain HDD's and 10TB of basically stagnant media don't have a clue what the 20TB+ guys go through, especially those that have transitioned off cable broadcast TV.

Did some more reading on FlexRaid just out of curiosity. Few questions:
1. Does FlexRaid do some kind "striping" to the data/files or how would it ever go beyond the speed of a single drive?
2. Does FlexRaid support online parity updating?
3. If it does support online parity updates, will it achieve near single drive write speeds and how it is doing it?

Add back in the time you waste by not just ripping straight to your server

The OP in this link is providing an inacurate SW/HW raid comparison

Their desktop motherboard is running a software assisted raid from the proprietary Asus sata controller which should in no way be compared to a true raid controller card

Exactly.

Pot. Meet kettle.

Bias is a dangerous thing.

For multi-access you are limited by gigabit speeds anyways or even worse wireless. Multi-access scenarios on a single drive could greatly degrade performance.

Do you have a prediction for writes on NZFS? I saw writes are pretty bad on Flexraid Real-time(unraid level bad).

Mfusick, maybe you should look at setting up a scheduled task to move your data at night so you don't have to wait on the copy.

I don't have good luck ripping directly to my server. In the end it requires a copy paste anyways.

I keep all my media very organized. I'm willing to bet I have one of the larger and better organized collections here on AVS. I have been collecting media since before I join this forum.
<---Join Date

Everything on my server is highly organized. Every movie is named/renamed exactly with mediacentermaster. It includes the year and resolution in every name. Every video file is inside a video folder of the same name. Every folder contains all the metadata, album art, backdrops and such.
I have mediacentermaster set to build PLEX/XBMC and mediabrowser info as I have all three installed on my HTPC and desktop. I find the performance of the front end is superior when your libraries are well put together and organized. I also find that as your library grows larger and larger that is where the performance and read speed of your server comes more and more into play. I couldn't run my 20TB collection with 40MB/sec read speeds.

Back on topic- I know what Dark Slayer is saying. But I don't think you have tested this yourself. If so you'd know why I prefer my way. I't might be a limitation of how I do it but I am not willing to change that.

Details:
When you rip directly to a server that is fine. Works fine. The read speed of my optical drives is far under the write speed of my network and server. But I choose not to rip directly into my organized library for many reasons. The biggest reason is I don't want anything in my library collection folders that is not 100% complete. Named perfectly and all organized with accompanying meta data and art. I know this is different from how most noobs do it. My realization that the vast majority people probably just dump raw video files into a folder called movies and let XBMC metascraper sort it out makes me think my explanation is going to fall on deaf ears. I don't do that. I do it all perfectly manually and then and only then do I place a finished video folder into the appropriate libary where it belongs. I have 8+ libraries or root folders for categories. My collection is very tight.

So that means I would need to create a folder on the server perhaps called "rips" and then point my mediacentermaster scraper to that folder. It would run (over LAN) do it's thing and then I could move the finished product over to the end resting place and library. I've done this. But it also requires a copy paste no different than if I did it locally on my desktop storage scratch disk. With flexraid it still requires a copy paste- it does not just repopulate into the other area in most cases. So now I am waiting to copy and paste over LAN (worse) than from local to server. It's not saving me any time. In fact it's taking more time. Mediacentermaster sucks over LAN as compared to a locally folder in performance. So that process (which I am watching live and controlling ) also takes longer.

Your right- I often set a copy paste to go and then walk away. I don't sit there an wait for it like your all thinking I do. But that changes nothing about my appreciation for a 100MB/sec transfer speed verus a 40MB/sec transfer speed. Again my original comment rings true. I am not understanding the resistance I am getting on this matter. What the heck is wrong with appreciating good performance from your server ?

90% of the noobs that dump raw video files into a folder they obtained from torrents and use XBMC to sort it out probably have no clue what I am talking about. (not saying anyone is that chillax )

Here's a snap shot of just a small portion of my collection of rips




It it don't look like that then it does not go into my library. I don't rip directly into my library. That works fine for a single movies folder type set up but not for an advanced organization set up.
I have many, many collections that I organize to have top level folders that go into sub folders.

Example:



goes into:







Ripping into my collection or library would be harder for me because I'd have to change the default library location or storage location for the rip in makeMKV each time. If I wanted it to go into DVD or bluray, or Horror, or TV Show etc... That's not saving me anytime.
Then I would have to change mediacentermaster to scan and organize those individual folders. So I would be constantly changing the scan folders trying to chase new added material and get is squared away. Again- that's not saving any time.
Anyone every try using mediacentermaster over LAN on 20TB ???? I have. It's not desirable. I prefer a local RAID0 scratch disc that reads near 200MB/sec on my i7 machine. Just way better

My sole point in any of this is not saying my way is right or anyone else way is wrong. Just different ways for different people.
My way and my personality appreciates the the 100MB/sec speed. I'm not sure what is "wrong" with that.

Ripping to a 200MB/sec RAID0 scratch disk locally installed on my i7 workstation saves me tons of time. It's not the copy paste time over LAN that is the issue. It's the superior performance of my metascraper on the local disk installed in my i7 workstation that performs way better locally on a small library or folder (like just the new media I have ripped) versus over LAN on a 20TB collection.

Your theory is flawed. If I did not rip directly into the collection folder library then it would still require a copy and paste on my flexraid server. Except this time over LAN (read over LAN by my desktop and moved.)
If I rip directly into my library two bad things happen:
#1- MediacenterMaster sucks on huge collections over LAN. I get frustrated with degraded performance. I have it set to verify everything with my manually with a pop up window. I like to choose if the movie is the OMEN from 1976 or 2006 for example.
#2. My collection has incomplete stuff in it not organized up to the level I want. This effects the HTPC and extenders that read from those libraries.

For this reason I like to do the work then place the finished completed movie folder into the library for storage.

Movie meta data management, my favorite topic Though a bit of a sidetrack but since it's at least slightly related to media storage perhaps I can continue.

I too have a rather large collection of DVD and Blu-ray movies stored on my server (Unraid in my case). And I too take keeping my collection organised rather seriously. But my approach is totally different from Mfusick's and perhaps from quite large portion of other users too. I've used DVD Profiler for almost a decade to manage all the meta data for my movies. All the meta data is stored in a centralised database and can be changed very easily and efficiently. I don't have any meta data stored with movie files, only in the database. All my Blu-ray rips are stored as ISO-images with a filename matching exactly the movie title in DVD Profiler. These are all accessible from a single share on my server, "\\unraid1\blu-ray" in this case. No subdirs, just single file per movie. DVD's are stored in Video_TS format in similar manner except the movie name is reflected in the directory name since the format contains multiple files per movie. So I have again a single share called "\\unraid1\dvd" with each movie in it's own subdir. There is no meta data embedded into directory or file names, no genres, no collection info, nothing. Just a movie title in the file/directoryname as a unique key. Some are using ISBN or similar numbers as the unique identifier but I find that counter-intuitive since I can easily create unique movie titles.

Ok, so how do I get the metadata from the DVD Profiler's database to my media center software? Well I was using Meedio for more than 8 years with an excellent XML2Mee plugin which directly supported DVD Profiler. With a few clicks I could export my whole collection's meta data to predefined location and have Meedio either manually or automatically process it for changed items and match them to the media files. As a result I would get "bit-perfect" meta data for my movies. I was looking for a replacement for quite some time but wasn't at all thrilled about meta data management concept of the most potential candidate, XBMC. Having all the meta data distributed all over my file server and having individual xml-nfo files for each media seemed like going back to the dark ages. Also scraping based meta data management might work wonders for some but for me with a lot of non-English titles the max hit rate was 50%. Also the quality of scraping based content eg. for genres was awfull with no consistency. Since I had my perfect meta data for my whole collection, there had to be a better way...

God bless for me to find Plex. The centralised media server concept with separated clients is just perfect for my case. Well, at first, almost perfect since there seemed to be no way of getting my DVD Profiler data into Plex. So I wrote my own meta data agent for DVD Profiler (find it here). It pretty much does what XML2Mee did in Meedio; it let's Plex find any new movie files and then using the file/directoryname as the key searches for meta data in DVD Profiler's XML-export. The movie covers are directly fetched from DVD Profiler's database directory. As the result I have all the meta data elements supported by Plex fed from DVD Profiler. As a bonus I can combine background art from online sources which is the only element missing from DVD Profiler. Oh but wait some might say, Plex does not support image formats like iso or video_ts. Well it does if you really want to (see here).

And closing to the punch line and getting back to the topic, this is what I do when I have a new batch of 4-5 Blu-ray movies:
1. Start ripping the first one using AnyDVD HD. Ripping is done directly to the \\unraid1\blu-ray share (or actually to the "\\unraid1\disk11\blu-ray" since all my user shares are write protected). Filename is automatically filled using the usually somewhat descriptive disk's volume name.
2. In the background add the new movies to DVD Profiler. This takes ~1min/movie if the movie is found on the online database and 5 mins if I have to create the profile from scratch.
3. Run the export wizard in DVD Profiler which exports the meta data in xml-format to a predefined location where Plex can find it. Takes about 2-3 mins to process.
4. Rip the rest of the movies
5. Rename the iso-files to match the movie titles in DVD Profiler. No copy-paste, just rename in place.
6. Update the Plex movie library (either manually or as part of automatic nightly update)

So definitely no copy-paste when adding content. And since I very rarely remove anything from my collection, there is no need to move data around. I just fill up disks one by one moving my ripping target directory to the next disk. And what about TV-Shows, for those I use Filebot (see here) and follow Plex's naming convention (see here) to the letter

Looks like you are missing all the back covers for your media. MyMovies can help you grab those...

An interesting approach.

While I am not sure it's for me I appreciate your precision and passion

I am mostly on mediabrowser. I find the same issues that make mediabrowser suck for many ( like slow LAN, Slow server, and crap hardware ) make it better on larger collections with good hardware and performance systems. My desktop is an i7 2600k. My server is mostly 7200.14 3TB seagates that rock over 100MB/sec religiously, and my HTPC is 3570k i5. Everything is run with wired CAT6. On collections about 15TB I prefer the performance of mediabrowser (and the look) over XBMC or PLEX. But I have all three installed. Plex server is installed on my server, and the media center is on the desktop. I also have the PLEX plugin for XBMC as well as the PLEX plugin for WMC.

Side note:

Have you tried Peter's MediaCentermaster for PLEX plugin ???

Sounds like something you possibly might be interested in.

Missing?

Is that the same as "removed" ?

Why would I want the back covers ??? For what ?

I'd be curious to see how your set up or using them.

...
Edited by PobjoySpecial - 5/16/13 at 2:29pm

Have you tried this ???

http://forums.mediacentermaster.com/viewtopic.php?f=2&t=6811

It does all that for you. And if your have good hardware and speed it does it even faster

I tried out MyMovies after I switched to Win8 and liked it better than MediaBrowser at the time. MyMovies includes both the front and back covers. It can obtain the meta info using the scanned barcode. Not a big deal, but just pointing out something you could go back and add to 20TB of movies.

http://www.mymovies.dk/media/13066/21.jpg

Is this a screen shot from inside WMC ???