Time to Build the Server - Final Checklist - AVS Forum
Forum Jump: 
 35Likes
Reply
 
Thread Tools
post #1 of 74 Old 08-11-2014, 09:10 PM - Thread Starter
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 452
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 204 Post(s)
Liked: 70
Time to Build the Server - Final Checklist

The time has finally arrived. I’m actually getting a bit giddy. Next Wednesday, I will be purchasing the equipment to build my home media server. Some of this was easy to figure out. Some of it was a bit tougher for me to decide on. And some of it I still hope I am making the right call. I’d appreciate any feedback that anyone has between now and then, especially if anyone notices where I am either wasting money or worse, cutting a major corner that I will later regret.

For the software, it’s pretty simple. I’ll be going with:

Windows 8.1 (this is for simplicity’s sake given the number of Windows machines already in the house on the network)
Media Browser 3
FlexRAID (f)
PowerChute
Chrome
--I’m fairly certain that is all that I will have on the machine in terms of software. I want to keep it as simple and streamlined as possible.

For the disc farm I have elected to go with:

2x Media Sonic 8-bay enclosures (hooked up to the PC via USB 3.0) Although, I am still considering 4x 4-bay enclosures since they cost far less than half as much. I wonder about power usage though.

2x 4 TB WD Red HDDs to be used for parity
10x 3 TB WD Red HDDs to be used for data storage

--This should serve as enough room for my initial data dump and leaves me with 4 empty drive bays for future expansion. I am going with 4 TB parity drives now as price/GB is always coming down and by the time I need to buy new drives (probably late January or early February) the cost/GB on 4 TB drives is likely to be closer to what 3 TB drives are now.


For the PC portion I have elected:

ASUS M11AD-US009S

I was looking at building from scratch, but between my lack of expertise in the area, and the prices I was able to find for parts, this actually came out cheaper than I was going to be able to do it myself. I do have some concern that the power supply might not be up to the task, but as long as these guys aren’t pulling a Lenovo stunt and using an odd 14-pin motherboard or some such, I’m fairly confident I can swap out the PSU if needed recommended.

The PC is almost certainly going to be overkill for 90-95% of daily demand. However, there does exist the outside possibility that 3 people could be watching on Android mobile devices, and another on an iPad, all while I am streaming directly to my HTPC (direct play). It’s unlikely, but it could happen, and since this is a major investment for the household, I want to be sure that the server PC is at least somewhat likely to be able to meet any reasonable demand. The entire library has been converted to uncompressed mkv files to help, but I suspect that only helps a little.


I'm open and eager for any feedback on this. I basically have a week to alter anything before the money goes out and the pieces come in. I figure that gives me enough time for one of the uber-techies around here to bash my skull in with a 1990s PSU for being an idiot and missing something obvious.

Last edited by Aryn Ravenlocke; 08-11-2014 at 09:11 PM. Reason: Fix link
Aryn Ravenlocke is online now  
Sponsored Links
Advertisement
 
post #2 of 74 Old 08-11-2014, 10:25 PM
Advanced Member
 
Defcon's Avatar
 
Join Date: Sep 2001
Posts: 919
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Liked: 39
For the pc I'd recommend a Dell XPS 8700 from their outlet store, if you're going for prebuilt.

http://outlet.us.dell.com/ARBOnlineS...=2202&fid=5991

If you look on slickdeals.net, they regularly have deals where Dell has a 25-30% off coupon for these. I just bought one for <$600.

Advantages :-

- has a better cpu (4770) and PSU
- newer, better motherboard (based on Intel Z87 chipset, I think Asus is a H81/87). e.g. Dell has 6x USB 3.0 ports. I also think you can go for a cheaper option with Core i5 cpu since servers don't need much horsepower.

As for chassis, look at posts by ajhieb in my thread here - IS there a company that builds affordable media servers?, where he recommends a SE3016 enclosure with a SAS extender card. It might be more complicated to setup (or less, since its less external components) and is cheaper.
Defcon is offline  
post #3 of 74 Old 08-11-2014, 10:36 PM - Thread Starter
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 452
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 204 Post(s)
Liked: 70
Quote:
Originally Posted by Defcon View Post
For the pc I'd recommend a Dell XPS 8700 from their outlet store, if you're going for prebuilt.

http://outlet.us.dell.com/ARBOnlineS...=2202&fid=5991

If you look on slickdeals.net, they regularly have deals where Dell has a 25-30% off coupon for these. I just bought one for <$600.

Advantages :-

- has a better cpu (4770) and PSU
- newer, better motherboard (based on Intel Z87 chipset, I think Asus is a H81/87). e.g. Dell has 6x USB 3.0 ports. I also think you can go for a cheaper option with Core i5 cpu since servers don't need much horsepower.

As for chassis, look at posts by ajhieb in my thread here - IS there a company that builds affordable media servers?, where he recommends a SE3016 enclosure with a SAS extender card. It might be more complicated to setup (or less, since its less external components) and is cheaper.
I'll have a look-see at the Dells. I'm not locked in to buying pre-built. But given what I was finding in the way of prices, and then adding a couple of days to build the PC and hoping I get it right, I wasn't going to be saving a dime by building myself compared to the prices I was finding for pre-built. Maybe I was just looking in the wrong places. As for i7 vs i5, I'm probably going to stick with i7 47xx processors. For most of the time it will be overkill, but I want to make sure I can transcode at least 2-3 streams at once if necessary.
Aryn Ravenlocke is online now  
post #4 of 74 Old 08-11-2014, 10:56 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,484
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 362 Post(s)
Liked: 406
Quote:
Originally Posted by Defcon View Post
As for chassis, look at posts by ajhieb in my thread here - IS there a company that builds affordable media servers?, where he recommends a SE3016 enclosure with a SAS extender card. It might be more complicated to setup (or less, since its less external components) and is cheaper.
The SE3016 can be a great way to shave a few bucks off of a storage build, but it certainly isn't for everybody. It might be worth a look in Aryn's case though.

Comparing it to the Mediasonic enclosures it breaks down like this...

With the Mediasonic enclosures, you'll be limited in bandwidth to 2x5Gb/s which is still plenty to saturate a gigabit network (reading or writing) but it might take longer for FlexRAID maintenance tasks (Update, verify, etc) The Mediasonic cases are more expensive, but they're also new and look nicer. (I have two similar 8 bay Sans Digital boxes right now) Basically the setup will be a little more polished.

If you go the route of the SE3016, the case would be cheaper, even if you have to get a SAS controller for it. If you go the popular (and cheap) route of getting one of the LSI based cards like the IBM M1015 and flashing it with th eLSI firmware, you'll still likely get in for less than the cost of one of the Mediasonic enclosures. The SE3016 uses a 4 lane SAS expander running at 3Gbps for a total bandwidth of 12Gbs which would be slightly faster than USB3 configuration.Plus it would be on its own dedicated bus instead of sharing a USB3 bus. The downside is you're going to have a used enclosure and a used controller card flashed with unsupported firmware. If you're the adventurous techy type, then that probably isn't a big deal, but some people might be scared off by that.

Either one is viable.
ilovejedd and Aryn Ravenlocke like this.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #5 of 74 Old 08-12-2014, 08:00 AM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by ajhieb View Post
With the Mediasonic enclosures, you'll be limited in bandwidth to 2x5Gb/s which is still plenty to saturate a gigabit network (reading or writing) but it might take longer for FlexRAID maintenance tasks (Update, verify, etc).
Definitely, definitely take this into consideration when building.

Another thing to keep in mind, the USB 3.0 bus on motherboards is shared so unless the OP gets a USB 3.0 PCIe card (can be unreliable), bandwidth is going to be limited to 5Gb/s. That's still a considerable ~400 MB/s throughput with UASP but divided over 12 drives, that's just ~33.33 MB/s per drive. Consider if you have all drives full, parity calc will take:
(3TB * 1000 GB/TB * 1000 MB/GB) / 33.33 MB/s = 90,000 seconds or ~25 hours

Having each enclosure as a separate array should improve things a bit since then you'd have ~400 MB/s divided by just 6 drives (or ~66.67 MB/s per drive).
(3TB * 1000 GB/TB * 1000 MB/GB) / 66.67 MB/s = 45,000 seconds or ~12.5 hours

P.S.
Forgot to mention, with USB 3.0, the more drives you add to the array, the slower parity calc will be because of the bandwidth constraint.

Last edited by ilovejedd; 08-12-2014 at 12:45 PM.
ilovejedd is offline  
post #6 of 74 Old 08-12-2014, 09:57 AM - Thread Starter
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 452
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 204 Post(s)
Liked: 70
Quote:
Originally Posted by ilovejedd View Post
Definitely, definitely take this into consideration when building.

Another thing to keep in mind, the USB 3.0 bus on motherboards is shared so unless the OP gets a USB 3.0 PCIe card (can be unreliable), bandwidth is going to be limited to 5Gb/s. That's still a considerable ~400 MB/s throughput with UASP but divided over 12 drives, that's just ~33.33 MB/s per drive. Consider if you have all drives full, parity calc will take:
(3TB * 1000 GB/TB * 1000 MB/GB) / 33.33 MB/s = 90,000 seconds or ~25 hours

Having each enclosure as a separate array should improve things a bit since then you'd have ~400 MB/s divided by just 6 drives (or ~66.67 MB/s per drive).
(3TB * 1000 GB/TB * 1000 MB/GB) / 66.67 MB/s = 45,000 seconds or ~12.5 hours
With nearly 30 TB of data to begin with, I'm not sure there is really much I can do in terms of cutting down the initial time for the array to populate and configure parity. 25 hours isn't so bad if that's only the first time through. If it is going to take 25 hours every time I add a file, then there's a problem. I don't mind installing a USB 3.0 card. Chances are I will need to eventually anyway. I absolutely will have to if I go with 4x 4-bay enclosures instead of 2x 8-bay (the four enclosures come in at $140 less for some odd reason).

I'm open to other suggestions of course. That's the point of this thread, to make sure I get things right the first time. I do have a budget though. I'd like to stay close to $3,000. I absolutely need to stay under $4,000. That price needs to include drives, PC, programs, and ancillary items.
Aryn Ravenlocke is online now  
post #7 of 74 Old 08-12-2014, 10:31 AM
AVS Special Member
 
EricN's Avatar
 
Join Date: May 2002
Posts: 1,225
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 55 Post(s)
Liked: 198
At current Amazon prices, if you go with the 8-bay enclosure, the 4TB drives are a slightly better deal than the 3TB drives. Once you factor in the cost of the bays (since naked drives aren't very useful) the prices are close enough that 3TB drives may not be worth bothering with at all.

Code:
Enclosure  Bays   Amazon  $/Bay
H82-SU3S2  8     $270.00  $33.75
HF2-SU3S2  4     $100.00  $25.00
                    
                           | Drive Only | Drive+4Bay | Drive+8Bay
NAS Drive   | TB |  Amazon |   $/TB     |   $/TB     |   $/TB
ST3000VN000 | 3  | $120.00 |   $40.00   |   $48.33   |   $51.25
WD30EFRX    | 3  | $122.00 |   $40.67   |   $49.00   |   $51.92
ST4000VN000 | 4  | $170.00 |   $42.50   |   $48.75   |   $50.94
WD40EFRX    | 4  | $173.00 |   $43.25   |   $49.50   |   $51.69
WD60EFRX    | 6  | $296.00 |   $49.33   |   $53.50   |   $54.96
                             
                           | Drive Only | Drive+4Bay | Drive+8Bay
PC Drive    | TB |  Amazon |   $/TB     |   $/TB     |   $/TB
ST3000DM001 | 3  | $102.00 |   $34.00   |   $42.33   |   $45.25
ST4000DM000 | 4  | $147.00 |   $36.75   |   $43.00   |   $45.19
ST5000DM000 | 5  | $209.00 |   $41.80   |   $46.80   |   $48.55
ilovejedd, ajhieb and dfkimbro like this.

Last edited by EricN; 08-12-2014 at 02:32 PM.
EricN is offline  
post #8 of 74 Old 08-12-2014, 10:54 AM
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 23,505
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 915
Quote:
Originally Posted by Aryn Ravenlocke View Post
With nearly 30 TB of data to begin with, I'm not sure there is really much I can do in terms of cutting down the initial time for the array to populate and configure parity. 25 hours isn't so bad if that's only the first time through. If it is going to take 25 hours every time I add a file, then there's a problem. I don't mind installing a USB 3.0 card. Chances are I will need to eventually anyway. I absolutely will have to if I go with 4x 4-bay enclosures instead of 2x 8-bay (the four enclosures come in at $140 less for some odd reason).

I'm open to other suggestions of course. That's the point of this thread, to make sure I get things right the first time. I do have a budget though. I'd like to stay close to $3,000. I absolutely need to stay under $4,000. That price needs to include drives, PC, programs, and ancillary items.
There was some good talk about parity calculation times and how it works in my server thread. Brahim (Flexraid dev) stopped by and gave a nice explanation how it works.

Using USB connections is a huge mistake IMO. I'd suggest taking a few minutes to really understand how parity calculation works. I can crunch my 50TB of drives faster than most can do their smaller 10TB of data. It's not the amount of drives or space total, it's the pipeline or the weak link. If you need a link or question shoot me a PM, I'm not around thank forum as much these days and I might miss it.
ilovejedd likes this.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #9 of 74 Old 08-12-2014, 11:07 AM - Thread Starter
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 452
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 204 Post(s)
Liked: 70
Quote:
Originally Posted by Mfusick View Post
There was some good talk about parity calculation times and how it works in my server thread. Brahim (Flexraid dev) stopped by and gave a nice explanation how it works.

Using USB connections is a huge mistake IMO. I'd suggest taking a few minutes to really understand how parity calculation works. I can crunch my 50TB of drives faster than most can do their smaller 10TB of data. It's not the amount of drives or space total, it's the pipeline or the weak link. If you need a link or question shoot me a PM, I'm not around thank forum as much these days and I might miss it.
Losing the USB 3.0 in favor of some other cable connection means ditching the enclosures. Outside of a few similar enclosures, such as the one ajhieb mentioned he is using, I have not been able to many up to the task. I suppose I could invest in one of those monster-sized full towers with plenty of empty drive slots, but even then I need to find a way to connect them all and the biggest capacity I have seen lately only hold 15 drives. That means adding an external enclosure by February.
Aryn Ravenlocke is online now  
post #10 of 74 Old 08-12-2014, 11:09 AM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by Aryn Ravenlocke View Post
With nearly 30 TB of data to begin with, I'm not sure there is really much I can do in terms of cutting down the initial time for the array to populate and configure parity. 25 hours isn't so bad if that's only the first time through. If it is going to take 25 hours every time I add a file, then there's a problem.
That's going to be for the initial parity build and every time you verify.

Quote:
Originally Posted by Aryn Ravenlocke View Post
I don't mind installing a USB 3.0 card. Chances are I will need to eventually anyway. I absolutely will have to if I go with 4x 4-bay enclosures instead of 2x 8-bay (the four enclosures come in at $140 less for some odd reason).
Just a heads up, I haven't had much luck with USB 3.0 cards. The ones I've tried tend to drop out with long, heavy loads (like, less than 2 hours of file copies). So far though, I've had good luck with native Intel USB 3.0 paired with a powered Anker USB 3.0 hub. I haven't done anything as stressful as the initial parity calc will be, though.

Quote:
Originally Posted by Aryn Ravenlocke View Post
I'm open to other suggestions of course. That's the point of this thread, to make sure I get things right the first time. I do have a budget though. I'd like to stay close to $3,000. I absolutely need to stay under $4,000. That price needs to include drives, PC, programs, and ancillary items.
Unfortunately, if that includes HDD costs, that doesn't really leave you much for the base server. To do it right, ideally, you really want to stick to SATA for better throughput. It's possible to stay within budget if building your own or buying used hardware but that opens up a whole other can of worms for someone not used to building PCs.
ajhieb likes this.
ilovejedd is offline  
post #11 of 74 Old 08-12-2014, 11:11 AM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by Aryn Ravenlocke View Post
Losing the USB 3.0 in favor of some other cable connection means ditching the enclosures. Outside of a few similar enclosures, such as the one ajhieb mentioned he is using, I have not been able to many up to the task. I suppose I could invest in one of those monster-sized full towers with plenty of empty drive slots, but even then I need to find a way to connect them all and the biggest capacity I have seen lately only hold 15 drives. That means adding an external enclosure by February.
There are plenty of cases which have plenty of drive bays. Most of them are geared towards servers and are rackmounted. The Norco 4220 and 4224 are decent inexpensive options.
ilovejedd is offline  
post #12 of 74 Old 08-12-2014, 11:19 AM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by Mfusick View Post
There was some good talk about parity calculation times and how it works in my server thread. Brahim (Flexraid dev) stopped by and gave a nice explanation how it works.
Was this on your 30TB server thread? Do you know what page Brahim's post is? Thanks!
ilovejedd is offline  
post #13 of 74 Old 08-12-2014, 12:03 PM
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 23,505
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 915
Quote:
Originally Posted by ilovejedd View Post
Was this on your 30TB server thread? Do you know what page Brahim's post is? Thanks!
lol.. yeah. It's in there someplace! I really should get around to organizing all that later great info that was buried. That was just my thread I started when I first played with the idea of rebuilding my rather cheap and small flexraid server into something better. It kind of just kept going, along the way I learned a lot, we figured out the cheap HBA sata card x8 flashing so it was affordable to go 20+ drives with minor cabling and complications. And, I learned a lot about flexraid. Brahim actually offered me to test out T raid, offered a free license, his free set up support to convert over etc.. but I was just too comfortable at that time with F raid. I'm still really happy with it actually. It works perfectly for me. I don't need real time, having an update every 24 hours is fine by me.

If I find it I will dig it up for you.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #14 of 74 Old 08-12-2014, 12:34 PM - Thread Starter
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 452
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 204 Post(s)
Liked: 70
This thread was supposed to help me refine what I had selected, as I thought I had selected rather wisely. Now it is beginning to sound like I took the wrong approach to this project entirely. Ugh. Grumble....grumble...

:head-desk:

So, I'm going through my list looking at where I am spending and where I can save. Depending on who I ask, I could be saving myself a very huge chunk of money on HDD if I opt for economy drives over WD Red. It's pretty much 50/50 from those I ask whether or not investing in the WD Reds is just drinking the kool-aid and investing in a snake-oil product.I can pick up 3 TB Seagates for about $90/each. If I move up to 4 TB I can move down to an initial purchase of 10 drives at about $135/each instead of an initial outlay for 12 drives like I had been looking at with the WD Reds I have listed in my OP.

The Norco 4220 will run me about $350, which is anywhere from $50-190 less than the Mediasonic solution I was looking at.



Not quite back to square-one, but I'm closer to that than I am to a finished product.
Aryn Ravenlocke is online now  
post #15 of 74 Old 08-12-2014, 12:37 PM
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 23,505
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 915
Quote:
Originally Posted by Aryn Ravenlocke View Post
Losing the USB 3.0 in favor of some other cable connection means ditching the enclosures. Outside of a few similar enclosures, such as the one ajhieb mentioned he is using, I have not been able to many up to the task. I suppose I could invest in one of those monster-sized full towers with plenty of empty drive slots, but even then I need to find a way to connect them all and the biggest capacity I have seen lately only hold 15 drives. That means adding an external enclosure by February.
You DO NOT WANT USB connections. You just do not.

Trust me when I tell you I tried it all, I was rocking the USB externals, I tried the full tower chassis route too... You want to employ a proper solution. A proper solution IMO is SAS connected backplanes with minimal power and signal cabling. It's so easy and neat and it works so well you'll be hard pressed to find anything that can compete with it, even if it's a tad bit cheaper.

Your best bet if you want 10+ HDD's is a server chassis, you can get them cheap if you don't need backplanes and hotswap bays. But again, the backplanes is worth it. A NORCO 4220 get's my recommendation. I am adding a second one, nothing I can find gives you the same bang for your buck.

Are you aware of the SAS8087 connections and the inexpensive flashed HBA sata cards that can add on 8 SATA ports via x8 slot to your system ? Those are affordable ($60-$100) and the connections are full speed, no bottle neck. A much better solution for Flexraid. Once you understand how flexraid works, you'll see the obvious problems with your design and plan. Flexraid accesses all your hard drives at the same time for parity calculations, this includes updates, or whatever.

It does not matter how much data you have your parity process times will always be dependent on your weakest link, the amount of time it takes to read all the required data from your hard drive. Whichever hard drive has the most data, or reads the slowest will be your weakest link.

Let's imagine a perfect scenario where you have 20 data drives of 3TB, let's imagine them to an average read of 100mb/sec and lets pretend this is constant and the same across all drives. That means your parity time is going to be:

Easy simple math for illustration purpose only:

3000000 MB divided by 100MB/sec = 30,000 seconds. (yeah I know the 1024 thing )

That is like 8.3 hours. Plus some extra time for the parity drive itself to finish it's thing after that. But all in all it would take 8.3 hours to do 20 data drives if you could maintain that 100mb/sec speed constantly across all drives (you can't in the real world it takes longer)

This is the same time if you had 2 data drives, as if you had 20! All of your HDD get read at the same time. So if they can all go full speed at the same time, you'll enjoy a much reduced time frame when your parity processes run like update, validate, verify, etc...

If you share 4 or 8 hard drives over a slow connection, then at means perhaps one or two drives might go full speed solo but if you run all the drives at the same time your are bandwidth bottle necked. NOT GOOD.

Reduce down to 50mb/sec per drive and it's now double the time (16 hours). Of coarse in the real world I would not be surprised to see your drives when full take 30-48 hours to do it in reality. (YES I AM SERIOUS)

Without going into too much detail about difference speeds of different drives, or why drives are slower when they are full, or how it only takes one drive to slow down that effectively brings the average of all of them down, just take my world on all this. There is 100 posts on these topics in my server thread, and I'm too tired too debate it all out here ad nauseam. Cliff notes is all drives slow down when they are full, and also get slower when reading certain kinds of files like software, pictures, mp3, versus reading big 25GB MKV files for movies. If you have a particularly slow drive it will bring down the average of them all, they work as a team towards completion. Weakest link.

For these reason I personally am not a big fan of huge hard drives, or slow hard drives for flexraid. 5TB and 6TB drives are great on paper, but imagine that it takes twice as long to read 6TB of data as it does 3TB and it's not hard to see the problem. My goal personally is to have my server complete it's parity processes in the middle of the night when I am sleeping and be ready for me the next day when I am ready to use it again. If your personal tolerance for an extended calculation time during which you should not be modifying files (good luck tell MB3 server to stop updating metadata during this time) is greater than mine you might be happy, but I would not be. Just my advice. During these extended periods your server performance will be greatly reduced as well.

My first couple Flexraid servers used cheaper weaker CPU (celeron and Pentium) and also rather slow WD GREEN hard drives (5400rpm like RED) and having upgraded to the i7 CPU and replaced all my 5400rpm drives with 7200rpm drives I can tell you from personal experience and with certainty there is an advantage to it. It works better. If you are considering big drives (4TB+) really pay attention to your drive speeds, because it will be a big factor in your time.

If you filled a WD RED full of 4TB worth of MP3, sofware and pictures I would not be surprised if it took you 25 hours to read all that sequentially with a bottlenecked USB connection shared with drives. There is no way you'll average 100MB/sec consistently at that task, and that is a lot of data to read. Remember this is all based on weakest link, so it's also possible one of your drives would be impeded or slow for the first 25% of it's data, and another drive could be slower during the next 25%-50, and then a third drive might be your weakest link for the 50%-%75 range... etc... It's always your weakest link in real time. There is so many factors that will play into this like the speed of your drives, the speed of your connections, and even the kind of data you are trying to read or where that data is located on the platter.

My advice is design your system with as few bottlenecks as possible. You'll find maximum happiness long term. For me the big bottle neck if my LAN speed, my LAN doesn't seem to want to go much faster than 105MB/sec most of the time. I'm ok with that.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #16 of 74 Old 08-12-2014, 12:40 PM - Thread Starter
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 452
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 204 Post(s)
Liked: 70
Hmm...looking at the enclosures I had selected again, it seems they also have e-SATA connections. But even those only run at 6Gb/s instead of the 5.0 Gb/s of the USB. That's faster, but not substantially and would also require me to find an open expansion port to add an e-SATA card.

So many things are not going the right way here.
Aryn Ravenlocke is online now  
post #17 of 74 Old 08-12-2014, 12:45 PM
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 23,505
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 915
Quote:
Originally Posted by Aryn Ravenlocke View Post
This thread was supposed to help me refine what I had selected, as I thought I had selected rather wisely. Now it is beginning to sound like I took the wrong approach to this project entirely. Ugh. Grumble....grumble...

:head-desk:

So, I'm going through my list looking at where I am spending and where I can save. Depending on who I ask, I could be saving myself a very huge chunk of money on HDD if I opt for economy drives over WD Red. It's pretty much 50/50 from those I ask whether or not investing in the WD Reds is just drinking the kool-aid and investing in a snake-oil product.I can pick up 3 TB Seagates for about $90/each. If I move up to 4 TB I can move down to an initial purchase of 10 drives at about $135/each instead of an initial outlay for 12 drives like I had been looking at with the WD Reds I have listed in my OP.

The Norco 4220 will run me about $350, which is anywhere from $50-190 less than the Mediasonic solution I was looking at.




Not quite back to square-one, but I'm closer to that than I am to a finished product.
My thread is longer than yours, it it was a lot longer than this one before I think I got it figured out Don't bang your head against the desk too hard. You'll get there.

As for drives, you'll see I started buying those 3TB seagates back in 2012 and I have NEVER sent one back to RMA yet. Not one. I'm 2 years strong on all of them, and I have 20+ of them just in my desktop and server. There was a popular study done that also showed consumer drives held up just as well as expensive enterprise drives a while back too. If I find it I'll link it up for you.

I think with hard drives it is very hit or miss. Vote with your wallet. You'll have that leftover cash to buy new ones some day, and at that time they will be bigger, faster and cheaper.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #18 of 74 Old 08-12-2014, 12:48 PM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by Aryn Ravenlocke View Post
Hmm...looking at the enclosures I had selected again, it seems they also have e-SATA connections. But even those only run at 6Gb/s instead of the 5.0 Gb/s of the USB. That's faster, but not substantially and would also require me to find an open expansion port to add an e-SATA card.

So many things are not going the right way here.
Lol, maybe you can convince Mfusick to build ther server for you for a fee. Caveat, shipping on the completed server is bound to be killer.
ilovejedd is offline  
post #19 of 74 Old 08-12-2014, 12:50 PM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by Mfusick View Post
lol.. yeah. It's in there someplace! I really should get around to organizing all that later great info that was buried. That was just my thread I started when I first played with the idea of rebuilding my rather cheap and small flexraid server into something better. It kind of just kept going, along the way I learned a lot, we figured out the cheap HBA sata card x8 flashing so it was affordable to go 20+ drives with minor cabling and complications. And, I learned a lot about flexraid. Brahim actually offered me to test out T raid, offered a free license, his free set up support to convert over etc.. but I was just too comfortable at that time with F raid. I'm still really happy with it actually. It works perfectly for me. I don't need real time, having an update every 24 hours is fine by me.

If I find it I will dig it up for you.
Given my experience with unRAID, I reckon I already have a pretty good idea what he said. Would just be interesting to read it from a developer's perspective.
ilovejedd is offline  
post #20 of 74 Old 08-12-2014, 12:53 PM
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 23,505
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 915
Quote:
Originally Posted by ilovejedd View Post
There are plenty of cases which have plenty of drive bays. Most of them are geared towards servers and are rackmounted. The Norco 4220 and 4224 are decent inexpensive options.
+1

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #21 of 74 Old 08-12-2014, 12:55 PM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by Aryn Ravenlocke View Post
Hmm...looking at the enclosures I had selected again, it seems they also have e-SATA connections. But even those only run at 6Gb/s instead of the 5.0 Gb/s of the USB. That's faster, but not substantially and would also require me to find an open expansion port to add an e-SATA card.

So many things are not going the right way here.
Even with e-SATA, you're still faced with the same issue, it just bumps your ceiling slightly higher. If you go native SATA/SAS via HBA card, you get 6Gb/s bandwidth per drive. That means you're pretty much just limited by HDD speed for parity calculations.
ilovejedd is offline  
post #22 of 74 Old 08-12-2014, 12:59 PM - Thread Starter
Senior Member
 
Aryn Ravenlocke's Avatar
 
Join Date: Aug 2006
Location: Tempe, AZ, USA
Posts: 452
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 204 Post(s)
Liked: 70
Quote:
Originally Posted by ilovejedd View Post

Unfortunately, if that includes HDD costs, that doesn't really leave you much for the base server. To do it right, ideally, you really want to stick to SATA for better throughput. It's possible to stay within budget if building your own or buying used hardware but that opens up a whole other can of worms for someone not used to building PCs.

Well, having a professional build a unit very close to what I am looking at would run about $3650. Now, if they can make a profit at that price, then I should be able to build something in my price range, even if I am paying a slight bit more for raw parts than the pros.
Aryn Ravenlocke is online now  
post #23 of 74 Old 08-12-2014, 01:19 PM
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 23,505
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 915
Quote:
Originally Posted by Aryn Ravenlocke View Post
Well, having a professional build a unit very close to what I am looking at would run about $3650. Now, if they can make a profit at that price, then I should be able to build something in my price range, even if I am paying a slight bit more for raw parts than the pros.
I built one for someone for $2000. I didn't make a profit though...

Quote:
Originally Posted by Mfusick View Post


-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."

Last edited by Mfusick; 08-12-2014 at 01:27 PM.
Mfusick is online now  
post #24 of 74 Old 08-12-2014, 01:20 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,484
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 362 Post(s)
Liked: 406
Quote:
Originally Posted by Aryn Ravenlocke View Post
Hmm...looking at the enclosures I had selected again, it seems they also have e-SATA connections. But even those only run at 6Gb/s instead of the 5.0 Gb/s of the USB. That's faster, but not substantially and would also require me to find an open expansion port to add an e-SATA card.

So many things are not going the right way here.
For about the same price as your proposed solution you can get the Sans Digital cases from Newegg. They have 2 eSata ports instead of just one that the MediaSonic has. Granted you're still using a Sata Port Multiplier but you'd be using 4 drives per eSata channel instead of 8, which would double your throughput, and half your parity calculation time (compared to the eSata of the MedaSonic) That should give you in the neighborhood of 190MB/s per drive which should be close to full speed for most drives.

6Gb/s * 1024Mb/Gb * 1Mb/8MB * 1eSata/4sata drives = 192MB/s/drive

However, if you use the included card you'll still be limited to ~60MB/s because of the PCIe x1 bandwidth restrictions. (8 drives on 1 PCIe x1 connection @ 500MB/s)

If you went with a single 4 port PCIe x4 eSata card like this http://www.newegg.com/Product/Produc...82E16816115122 it would double your throughput back up to around 120MB/s which is not going to be full speed, but still respectable. (16 drives on 1 PCIe x4 connection @ 2000MB/s)

Edit: Find a better HBA that is PCIe x8 and supports port mulitpliers and your back to full speed again.

Edit again: get two of these: http://www.newegg.com/Product/Produc...82E16816115119 and you won't have bandwidth limitations. (only use two ports per card)

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.

Last edited by ajhieb; 08-12-2014 at 01:27 PM.
ajhieb is online now  
post #25 of 74 Old 08-12-2014, 01:30 PM
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 23,505
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 915
all that trouble... not worth IMO. Buy a flashed x8 card from Techmattr or Andy Steb here on AVS. Under $100 shipped. You have 8 full speed sata ports, no installation required. Just physically push it into the slot and turn the machine on, it auto installs itself with windows native drivers. The end.

A SAS8087 to SATA plug wire is like $5 on ebay.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #26 of 74 Old 08-12-2014, 01:32 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,484
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 362 Post(s)
Liked: 406
Quote:
Originally Posted by Mfusick View Post
There was a popular study done that also showed consumer drives held up just as well as expensive enterprise drives a while back too. If I find it I'll link it up for you.
I'm looking forward to that. If it was this study that sparked all of the discussion, it wasn't comparing consumer drives to enterprise drives, it was mainly comparing consumer drives to other consumer drives in an enterprise environment, and a lot of the consumer drives did very poorly.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #27 of 74 Old 08-12-2014, 01:34 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,484
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 362 Post(s)
Liked: 406
Quote:
Originally Posted by Mfusick View Post
all that trouble... not worth IMO. Buy a flashed x8 card from Techmattr or Andy Steb here on AVS. Under $100 shipped. You have 8 full speed sata ports, no installation required. Just physically push it into the slot and turn the machine on, it auto installs itself with windows native drivers. The end.

A SAS8087 to SATA plug wire is like $5 on ebay.
That's funny... I thought there was a great deal of angst over getting those cards working with Windows 8.1 I seem to recall the drivers being a bit fiddly with the latter versions of Windows. Anybody using a flashed HBA with Windows 8.1 want to chime in?

All of what trouble? Buying a couple of $66 dollar HBAs to get enough bandwidth to run 16 drives?

As opposed to your alternative: Buy a pair of $50-$100 HBAs, additional cabling, a server chassis, building a complete system...

But I can see why you think that building a complete server from the ground up is a lot more trouble than installing a couple of HBA cards.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.

Last edited by ajhieb; 08-12-2014 at 01:38 PM.
ajhieb is online now  
post #28 of 74 Old 08-12-2014, 02:11 PM
AVS Special Member
 
ilovejedd's Avatar
 
Join Date: Aug 2006
Posts: 3,725
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 59
Quote:
Originally Posted by ajhieb View Post
For about the same price as your proposed solution you can get the Sans Digital cases from Newegg. They have 2 eSata ports instead of just one that the MediaSonic has. Granted you're still using a Sata Port Multiplier but you'd be using 4 drives per eSata channel instead of 8, which would double your throughput, and half your parity calculation time (compared to the eSata of the MedaSonic) That should give you in the neighborhood of 190MB/s per drive which should be close to full speed for most drives.

6Gb/s * 1024Mb/Gb * 1Mb/8MB * 1eSata/4sata drives = 192MB/s/drive

If you went with a single 4 port PCIe x4 eSata card like this http://www.newegg.com/Product/Produc...82E16816115122 it would double your throughput back up to around 120MB/s which is not going to be full speed, but still respectable. (16 drives on 1 PCIe x4 connection @ 2000MB/s)
That's actually not a bad option. Correction on the calculation, though.

6Gb/s * 8 bit/10 bit encoding * 1MB/8Mb * 1 e-SATA/4 SATA = 150MB/s

Mind, I priced out a server build from Newegg:
  • Case: Norco RPC-4220, $330
  • PSU: SeaSonic SSR-650RM, $100
  • MB: SuperMicro X10SLH-F, $215
  • CPU: Intel Xeon E3-1230V3, $250
  • RAM: Crucial 16GB DDR3 1600 ECC, $180
  • HBA: SuperMicro AOC-SAS2LP-MV8, $110 x3 = $330 (if you don't want to take a chance with flashing IBM 1015s)
  • Misc: SFF-8087 Cables, $15 x6 = $90
  • TOTAL: $1,495 (before OS, storage, etc)

Even after factoring in the cost of drives, OS and FlexRAID license, I reckon the above is doable on a $3K budget.
Mfusick and ajhieb like this.

Last edited by ilovejedd; 08-12-2014 at 02:16 PM.
ilovejedd is offline  
post #29 of 74 Old 08-12-2014, 02:13 PM
AVS Special Member
 
kapone's Avatar
 
Join Date: Jan 2003
Posts: 4,429
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 83 Post(s)
Liked: 126
Quote:
Originally Posted by ajhieb View Post
That's funny... I thought there was a great deal of angst over getting those cards working with Windows 8.1 I seem to recall the drivers being a bit fiddly with the latter versions of Windows. Anybody using a flashed HBA with Windows 8.1 want to chime in?

All of what trouble? Buying a couple of $66 dollar HBAs to get enough bandwidth to run 16 drives?

As opposed to your alternative: Buy a pair of $50-$100 HBAs, additional cabling, a server chassis, building a complete system...

But I can see why you think that building a complete server from the ground up is a lot more trouble than installing a couple of HBA cards.
ajhieb - I'm with Mfusick on this. eSATA is a dying technology. Trust me, I was probably one of the first ones here that built a 48 bay server based on port multipliers... And while it certainly worked just fine, speed and throughput were not its high points.

A $66 eSATA card vs $100 for an M1015? I'll take the M1015 any day. SAS is where it's at.

Now, what you guys haven't mentioned yet is, SAS expanders.... With a single M1015 with dual 6gb/s ports, you could cascade quite a few expanders and still get wire speed for HDD throughput. A spinning hard drive maxes out at 150MB/s sustained. Even with 3gb/s expanders (remember the expander port is x4, i.e. 12gb/s on a 3gb/s expander), which are dirt cheap right now, you would almost never face a throughput issue for media serving purposes (or parity for that matter).

Now, if you wanted to run a RAID 0 array of SSDs (like Mfusick is fond of doing...), that's a different issue.
Mfusick and ilovejedd like this.
kapone is online now  
post #30 of 74 Old 08-12-2014, 02:21 PM
Senior Member
 
rc05's Avatar
 
Join Date: Oct 2011
Location: San Luis Obispo, CA
Posts: 365
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 30 Post(s)
Liked: 53
I've been running an M1015 on Windows server 2012 R2 with no problems. Not exactly the same as Windows 8 though
Mfusick likes this.
rc05 is online now  
Reply Home Theater Computers

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off