or Connect
AVS › AVS Forum › Video Components › Home Theater Computers › Mfusick's How to build an affordable 30TB Flexraid media server: Information Requested.!
New Posts  All Forums:Forum Nav:

Mfusick's How to build an affordable 30TB Flexraid media server: Information Requested.! - Page 15

post #421 of 3346
Thread Starter 
Quote:
Originally Posted by duff99 View Post

You realize there are cables that take 4 SATA ports from you motherboard to a 8087 cable right. They're reverse breakout cables. Just make sure you get the right one. There are forward breakout cables also, they look the same, but they won't work.

Yes. Thanks for heads up.

I needed two cables to go from the two SAS ports on my IBM SATA card to the boards in the Norco 4220. One cable per row . 4 HDDs per row get output as single SAS port. So 8 total HDD from my IBM card.

Soon I'll grab another IBM card for the remaining 8 drive bays.

For motherboard I was going to just remove the board and run normal SATA wires.

I guess you just got me rethinking this. Any advantage other than one cable vs 4 ??? I guess that means 4 SATA power cables too. So 8 cables vs 2.

I might grab one. Thanks.
post #422 of 3346
Thread Starter 
Quote:
Originally Posted by duff99 View Post

You realize there are cables that take 4 SATA ports from you motherboard to a 8087 cable right. They're reverse breakout cables. Just make sure you get the right one. There are forward breakout cables also, they look the same, but they won't work.

Yes. Thanks for heads up.

I needed two cables to go from the two SAS ports on my IBM SATA card to the boards in the Norco 4220. One cable per row . 4 HDDs per row get output as single SAS port. So 8 total HDD from my IBM card.

Soon I'll grab another IBM card for the remaining 8 drive bays.

For motherboard I was going to just remove the board and run normal SATA wires.

I guess you just got me rethinking this. Any advantage other than one cable vs 4 ??? I guess that means 4 SATA power cables too. So 8 cables vs 2.

I might grab one. Thanks.
post #423 of 3346
Quote:
Originally Posted by Mfusick View Post

I do. That's 2 minutes of my life. My time is limited and the work I do on my collection is substantial and near daily.

I do all my management on my desktop and then move and store the finished product folder to the appropriate library on server.

I'm sorry to bring back up something that finally died. Why not just rip it straight to your file server? Even at 50MBps that's still much faster than any bd-drive I know of can rip. Alternatively you could use a cache drive that moves everything at midnight so you get the full 100MBps and would hopefully reduce on/off cycles on your hdd's.
post #424 of 3346
Thread Starter 
Quote:
Originally Posted by macks View Post

I'm sorry to bring back up something that finally died. Why not just rip it straight to your file server? Even at 50MBps that's still much faster than any bd-drive I know of can rip. Alternatively you could use a cache drive that moves everything at midnight so you get the full 100MBps and would hopefully reduce on/off cycles on your hdd's.

I like to manage things myself. The performance is much better locally too.

I find mediacentermaster struggles more with huge libraries over network.

I also like to control where the movie goes. I have many libraries. Example: HD movies, non HD movies, bluray rips, Disney/kids, TV shows .....

I like to move and store the final and finished video folder on these libraries for permanent storage. Each folder goes into the appropriate location Only 100% complete is put here.

So my process is to rip to my RAID0 scratch disk on my desktop. I have dual Bluray drives so I tend to rip two at a time when I'm doing it. Then I run mediacentermaster on it all. Change the names of files, add metadata and art etc...

Once it's done I copy paste to server.

Mediacentermaster would cry if I sent it at my 5TB TV show folder over LAN ... Lol.

Plus I like to manage it manually myself. I find it easier to work with only a handful of videos at a time.

This process also ensures everything on my library is perfect and complete at all times. These libraries are the directories my HTPCs are pointed at. I also have PLEX set up on my server for these directories.

If I did it your way it would either be a mess at times- or I'd end up doing it in a separate folder on my server then copy paste finished product just as I do now- except at lower performance over LAN vs a local HDD RAID0 scratch disk. I hate waiting for stuff and I find the performance over LAN is lower. Mediacentermaster works best on local drives and smaller amount of files at one time IMO.

I use it over LAN at times to tune up or search out duplicates / rename TV shows sometimes but it's not ideal.

I have libraries set up in windows called media master - my mediamaster program reads and directs to that directory as default. I also have library set for mkv rips folder. My makemkv rips to that. Once done I just drag drop from the rip folder to the mediacentermaster folder. It doesn't copy paste or take time- it just relocates instantly since both are in the root level of my scratch disk. They would copy paste from one folder to another if I did the same on my server. That would take time.

You can't rip to the same folder mediacentermaster is working on at same time because it wants to mess with Mkvs that are currently being ripped. No good.

I'm all about speed and efficiency. I'm quite sure I can manage triple the amount of media in the same time the average AVS poster does. I can't spend hours everyday on this stuff. That's biggest challenge in HTPC IMO.

It's not hardware, building or even setting up software. It's the constant maintenance of media that's most difficult I find.

I'm not sure others have as much media as I do. It seems like most are still on the small NAS box or a few external USB drive program.

I'm at 30TB and 14 HDDs... Getting there had taken a long time. But I've figured out a few ways to do it quicker and better. One way is for sure ripping and managing locally on a fast desktop and dual monitors. I just get way more done easier and faster.
post #425 of 3346
I'm not a fan of MCM so I guess that's the difference. I just rip it and forget it. If it doesn't show up as the right title with the correct scrapes then I rename the folder. MCM seemed to create more work for me. To each their own.
post #426 of 3346
Quote:
Originally Posted by Mfusick View Post

You can't rip to the same folder mediacentermaster is working on at same time because it wants to mess with Mkvs that are currently being ripped. No good.

So this part stood out to me

I do this all the time, but I use the advanced view in makemkv and I always make the movie title match the folder title. The whole process involves me choosing the "Destination folder" creating a new folder and pasting the complete title/year from IMDB. Since I've already got the name I just go ahead and paste it into the "File Name" option in Makemkv as well.

It's very useful (in my mind at least) to take the extra second and do this in case I delete a couple of the mkvs. This way they aren't all Title00 or Title01 (even though their restore path should show their origination anyway)


All that aside, when MCM scrapes it only populates the folder with extra files. If you have the correct filename (or you aren't letting MCM change your filenames) then it doesn't interfere with your rip. Sometimes it will have scraped the title and populated my XBMC library before the rip is complete, but I've never had a problem with MakeMKV finishing the rip correctly
post #427 of 3346
i wanted to get the IBM card too, but my mb had a x4 slot only. it is open ended as someone suggested it works, but the bios battery is there, so while open ended, the unused portion of the card would not be able to sink down enough for the card to go into the slot.

i got a rr2320 which is working well for the past week until yesterday. one drive, a WD black, a refurb kept getting disconnected. i thought ok at least move the drive to a mb port and get the data off. turns out the wdb is still disconnecting, so hopefully the rr card is actually ok. time to get another 3tb seagate to go with the the first two 3tb seagates, two 2tb f4, two 500gb samsung and 250gb seagate 2.5 drive as temp storage for sorting. the sdd and two opticals are on the mb.

pm me a good ebay seller if you get the ibm card again.
post #428 of 3346
Quote:
Originally Posted by Mfusick View Post

Yes. Thanks for heads up.

I needed two cables to go from the two SAS ports on my IBM SATA card to the boards in the Norco 4220. One cable per row . 4 HDDs per row get output as single SAS port. So 8 total HDD from my IBM card.

Soon I'll grab another IBM card for the remaining 8 drive bays.

For motherboard I was going to just remove the board and run normal SATA wires.

I guess you just got me rethinking this. Any advantage other than one cable vs 4 ??? I guess that means 4 SATA power cables too. So 8 cables vs 2.

I might grab one. Thanks.

I think you answered your own question.
post #429 of 3346
Thread Starter 
Y
Quote:
Originally Posted by duff99 View Post

I think you answered your own question.

Yup!
post #430 of 3346
What kind of read/write speeds do you all get on your massive FlexRAID pools?
post #431 of 3346
Thread Starter 
Quote:
Originally Posted by amarshonarbangla View Post

What kind of read/write speeds do you all get on your massive FlexRAID pools?

I am consistently about 100mb/ sec with hitachi and Seagate drives.

About 60 with WD greens.

My network performance fluctuates around 105mb and this matches up to HDD speeds close enough I haven't tinkered much.

If I allow larger packets and disable bandwidth throttling I can probably hit 120mb I bet.

Gigabit networks can do 125mb. This beats slow HDDs.
post #432 of 3346
Thread Starter 
Real world and full drives are slower than new empty drives.

A new empty drive could exceed network speeds. But not by much to cause much issue.
post #433 of 3346
Is 60-80MB/s fine for let's say five people in the network streaming 1080p Blurays simultaneously? Assuming all the machines are connected via gigabit ethernet.
post #434 of 3346
Thread Starter 
5 might have issue but I'm sure 4 might work
post #435 of 3346
Umm, why 4 and not 5 frown.gif
post #436 of 3346
Thread Starter 
Quote:
Originally Posted by amarshonarbangla View Post

Umm, why 4 and not 5 frown.gif

lol.

I think most 1080p HD streams use 10-20MB - so it's close. 60MB or a poor performing HDD might struggle with 5 full 1080p steams depending on the bitrate and the type of stream it is. Or it might work fine.
You won't know unless you test it.

If your HDD is doing 80MB+ as it should... then your fine.

New drives will do it fine. Old slower and full drives might not.

Either way it's rarely an issue. And 4 streams are usually fine for most anyways... I never do more than 2 so that is a non issue for me.
post #437 of 3346
Quote:
Originally Posted by Mfusick View Post

5 might have issue but I'm sure 4 might work

Quote:
Originally Posted by amarshonarbangla View Post

Umm, why 4 and not 5 frown.gif

5 streams will be fine even with a Green drive. Some of us have tested this and even have had it reviewed independently.

But don't take my word for it. Here is one of my builds recording 4 HD streams while playing back a 5th by an independent reviewer.

http://dvr.about.com/od/mediacenterpcs/fr/A-Solid-Start-To-A-Great-Htpc-Experience.htm

(Also for those interested here is his "long term" take on this build after using it as his family's daily machine for 3 months: http://dvr.about.com/od/capturetvwithacomputer/a/Assassins-Elite-Htpc-Three-Months-In.htm )

post #438 of 3346
Thread Starter 
Good post^

Bottom line is yeah 4 is never going to be a problem. 5 is still no problem if your HDD is up to speed. Any HDD that can do 80MB/sec will do it. Even Green drives are around 100MB/sec in the 2TB and 3TB sizes. It's only the speed drop when the drive is old and full that is the uncertainty.

I just don't think it's an issue at all and certainly 4 streams at once is never going to be a problem. If you need more than 4 streams then you need to be focusing on building a more powerful performance media server- which is not always the same as a simple and affordable media server.

I think simple, affordable, high value media server that can do 4+ streams is much more common than a 6+ stream monster that costs thousands, uses dual teamed LAN- and hardware RAID. That's not appropriate for HTPC IMO.
post #439 of 3346
Quote:
Originally Posted by Mfusick View Post

lol.

I think most 1080p HD streams use 10-20MB - so it's close. 60MB or a poor performing HDD might struggle with 5 full 1080p steams depending on the bitrate and the type of stream it is. Or it might work fine.
You won't know unless you test it.

If your HDD is doing 80MB+ as it should... then your fine.

New drives will do it fine. Old slower and full drives might not.

Either way it's rarely an issue. And 4 streams are usually fine for most anyways... I never do more than 2 so that is a non issue for me.

Blurays are 10-20 mbps (bits, not bytes). Green drives are 100+ MBps (bytes, not bits).

100MBps / 20mbps = 40 theoretical* streams.

* Take into account seeking, fragmentation, etc. you should still easily get 10+ streams.
post #440 of 3346
Quote:
Originally Posted by vladd View Post

Blurays are 10-20 mbps (bits, not bytes). Green drives are 100+ MBps (bytes, not bits).

100MBps / 20mbps = 40 theoretical* streams.

* Take into account seeking, fragmentation, etc. you should still easily get 10+ streams.

Good science. Great post.
post #441 of 3346
I try biggrin.gif
post #442 of 3346
Thread Starter 
I always thought that 1080p streams could take over 10MB/sec and that is why if you did not have gigabit switch or router you could have issues???

Am I confusing that with wireless or something ???
post #443 of 3346
Thread Starter 
Quote:
Originally Posted by duff99 View Post

Monoprice has them for $9.90.

http://www.monoprice.com/products/product.asp?c_id=102&cp_id=10254&cs_id=1025410&p_id=8189&seq=1&format=2

They'll hit you hard on the shipping, but they are generally good quality products.

Thanks again for this I just bought 2 more so I will have when I need them.

http://www.ebay.com/itm/Norco-Reverse-SATA-Breakout-Cable-4xSATA-SAS-SFF8087-/120909472854?pt=LH_DefaultDomain_0&hash=item1c26c42456

This is a reverse right ??

The only difference is it says reverse ???

Or am I missing something else ???
Edited by Mfusick - 1/18/13 at 12:42pm
post #444 of 3346
Quote:
Originally Posted by Mfusick View Post

Thanks again for this I just bought 2 more so I will have when I need them.

http://www.ebay.com/itm/Norco-Reverse-SATA-Breakout-Cable-4xSATA-SAS-SFF8087-/120909472854?pt=LH_DefaultDomain_0&hash=item1c26c42456

This is a reverse right ??

The only difference is it says reverse ???

Or am I missing something else ???

Looks like the right ones. There are forward and reverse breakout cables. You need the reverse breakout cables since you are effectively reversing the breakout by combining four SATA ports into one SAS cable. I unfortunately need the forward breakout cables to convert the SAS port on my card to hook up to four SATA ports on my Norco 4020. It's pretty confusing. They look identical, they aren't interchangeable though. If you use the wrong one it won't work. This has bitten many a server builder in the past. It's usually one of the first trouble shooting questions you see when people can't connect to their drives.
post #445 of 3346
Quote:
Originally Posted by Mfusick View Post

I always thought that 1080p streams could take over 10MB/sec and that is why if you did not have gigabit switch or router you could have issues???

Am I confusing that with wireless or something ???
Must be thinking of wireless. A single Bluray should stream easily on a 100mb switch.
post #446 of 3346
Blurays are usually mastered at 2x these days, but I'm almost certain they all playback 1x

1x Bluray reads at 36 Mbit/s which is 4.5 MB/s

Check the wiki page (right hand table info)

http://en.wikipedia.org/wiki/Blu-ray_Disc
post #447 of 3346
And I've said it a lot, but I'll repeat again. My linksys e4200 and wet-610n have no problem sustaining blu ray playback wirelessly
post #448 of 3346
Quote:
Originally Posted by Dark_Slayer View Post

Blurays are usually mastered at 2x these days, but I'm almost certain they all playback 1x

1x Bluray reads at 36 Mbit/s which is 4.5 MB/s

Check the wiki page (right hand table info)

http://en.wikipedia.org/wiki/Blu-ray_Disc
A optical drive will read at 36Mbps (the upper limit of the Bluray spec) but the actual video stream can be much less. Source readers (LAV, etc) only need to read the data fast enough to present it to the renderer in a timely manner with a small read ahead buffer. So reading is done in chunks and not at a continous 36mbps.
Quote:
Originally Posted by Dark_Slayer View Post

And I've said it a lot, but I'll repeat again. My linksys e4200 and wet-610n have no problem sustaining blu ray playback wirelessly
And the same router might perform at half the speed if even just moved to another room let alone another house. There are simply too many environmental factors involved with wireless for it to be reliable in every installation for time critial communication like video streaming.
post #449 of 3346
Quote:
Originally Posted by Dark_Slayer View Post

And I've said it a lot, but I'll repeat again. My linksys e4200 and wet-610n have no problem sustaining blu ray playback wirelessly

Wireless sucks. 100mb wired will run blurays 24/7/365. 300mbps wireless might run blurays if you are close enough and noone in the immediate area has a microwave turned on and noone else is connected to it, so on and so on.
post #450 of 3346
Quote:
Originally Posted by macks View Post

Wireless sucks. 100mb wired will run blurays 24/7/365. 300mbps wireless might run blurays if you are close enough and noone in the immediate area has a microwave turned on and noone else is connected to it, so on and so on.

Or if you don't bother with 2.4 GHz band, or manage a dual band router with a dedicated non-broadcast SSID, or don't try to use the dinky built-in laptop wireless one time without ever trying a decent wireless bridge and forever declare that wireless sucks
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Home Theater Computers
AVS › AVS Forum › Video Components › Home Theater Computers › Mfusick's How to build an affordable 30TB Flexraid media server: Information Requested.!