Originally Posted by Aryn Ravenlocke
Losing the USB 3.0 in favor of some other cable connection means ditching the enclosures. Outside of a few similar enclosures, such as the one ajhieb mentioned he is using, I have not been able to many up to the task. I suppose I could invest in one of those monster-sized full towers with plenty of empty drive slots, but even then I need to find a way to connect them all and the biggest capacity I have seen lately only hold 15 drives. That means adding an external enclosure by February.
You DO NOT WANT USB connections. You just do not.
Trust me when I tell you I tried it all, I was rocking the USB externals, I tried the full tower chassis route too... You want to employ a proper solution. A proper solution IMO is SAS connected backplanes with minimal power and signal cabling. It's so easy and neat and it works so well you'll be hard pressed to find anything that can compete with it, even if it's a tad bit cheaper.
Your best bet if you want 10+ HDD's is a server chassis, you can get them cheap if you don't need backplanes and hotswap bays. But again, the backplanes is worth it. A NORCO 4220 get's my recommendation. I am adding a second one, nothing I can find gives you the same bang for your buck.
Are you aware of the SAS8087 connections and the inexpensive flashed HBA sata cards that can add on 8 SATA ports via x8 slot to your system ? Those are affordable ($60-$100) and the connections are full speed, no bottle neck. A much better solution for Flexraid. Once you understand how flexraid works, you'll see the obvious problems with your design and plan. Flexraid accesses all your hard drives at the same time for parity calculations, this includes updates, or whatever.
It does not matter how much data you have your parity process times will always be dependent on your weakest link, the amount of time it takes to read all the required data from your hard drive. Whichever hard drive has the most data, or reads the slowest will be your weakest link.
Let's imagine a perfect scenario where you have 20 data drives of 3TB, let's imagine them to an average read of 100mb/sec and lets pretend this is constant and the same across all drives. That means your parity time is going to be:
Easy simple math for illustration purpose only:
3000000 MB divided by 100MB/sec = 30,000 seconds. (yeah I know the 1024 thing
That is like 8.3 hours. Plus some extra time for the parity drive itself to finish it's thing after that. But all in all it would take 8.3 hours to do 20 data drives if you could maintain that 100mb/sec speed constantly across all drives (you can't in the real world it takes longer)
This is the same time if you had 2 data drives, as if you had 20! All of your HDD get read at the same time. So if they can all go full speed at the same time, you'll enjoy a much reduced time frame when your parity processes run like update, validate, verify, etc...
If you share 4 or 8 hard drives over a slow connection, then at means perhaps one or two drives might go full speed solo but if you run all the drives at the same time your are bandwidth bottle necked. NOT GOOD.
Reduce down to 50mb/sec per drive and it's now double the time (16 hours). Of coarse in the real world I would not be surprised to see your drives when full take 30-48 hours to do it in reality. (YES I AM SERIOUS)
Without going into too much detail about difference speeds of different drives, or why drives are slower when they are full, or how it only takes one drive to slow down that effectively brings the average of all of them down, just take my world on all this. There is 100 posts on these topics in my server thread, and I'm too tired too debate it all out here ad nauseam. Cliff notes is all drives slow down when they are full, and also get slower when reading certain kinds of files like software, pictures, mp3, versus reading big 25GB MKV files for movies. If you have a particularly slow drive it will bring down the average of them all, they work as a team towards completion. Weakest link.
For these reason I personally am not a big fan of huge hard drives, or slow hard drives for flexraid. 5TB and 6TB drives are great on paper, but imagine that it takes twice as long to read 6TB of data as it does 3TB and it's not hard to see the problem. My goal personally is to have my server complete it's parity processes in the middle of the night when I am sleeping and be ready for me the next day when I am ready to use it again. If your personal tolerance for an extended calculation time during which you should not be modifying files (good luck tell MB3 server to stop updating metadata during this time) is greater than mine you might be happy, but I would not be. Just my advice. During these extended periods your server performance will be greatly reduced as well.
My first couple Flexraid servers used cheaper weaker CPU (celeron and Pentium) and also rather slow WD GREEN hard drives (5400rpm like RED) and having upgraded to the i7 CPU and replaced all my 5400rpm drives with 7200rpm drives I can tell you from personal experience and with certainty there is an advantage to it. It works better. If you are considering big drives (4TB+) really pay attention to your drive speeds, because it will be a big factor in your time.
If you filled a WD RED full of 4TB worth of MP3, sofware and pictures I would not be surprised if it took you 25 hours to read all that sequentially with a bottlenecked USB connection shared with drives. There is no way you'll average 100MB/sec consistently at that task, and that is a lot of data to read. Remember this is all based on weakest link, so it's also possible one of your drives would be impeded or slow for the first 25% of it's data, and another drive could be slower during the next 25%-50, and then a third drive might be your weakest link for the 50%-%75 range... etc... It's always your weakest link in real time. There is so many factors that will play into this like the speed of your drives, the speed of your connections, and even the kind of data you are trying to read or where that data is located on the platter.
My advice is design your system with as few bottlenecks as possible. You'll find maximum happiness long term. For me the big bottle neck if my LAN speed, my LAN doesn't seem to want to go much faster than 105MB/sec most of the time. I'm ok with that.