Massive storage server build - advice on planned H/W appreciated

Just a couple thoughts:
You wanted protection for one or two drives. I recommend picking between those, as it will drive future recommendations, but if you are going to put 14 drives in a system, 2 parity drives is recommend. If you want the ability to spin down drives, you're looking at a software solution instead of a raid controller. If you want 2 drive protection, you're looking at FlexRaid.

As far as sata ports go, you're going to be limited to a X4 (+) port for anything more then adding a single sata port. If you're going to add a graphics card, which isn't necessary at all, that should go into your mb considerations.

I would run a version of Linux or Unix with ZFS (Debian, OpenIndiana, etc) and do all your arrays as zpools. Not sure how proven SnapRaid is yet, and I never liked Win7 as a server.

also, if you're interested in virtualization (win7 64, etc) you may also want to look at proxmox which is based on debian.

Hello,

If you're going with the kind of storage you're looking at (40 TB+) I would strongly recommend looking at something like the Supermicro boards with IPMI. I have a similar build supplying my home media server needs. I set out with the idea that I wanted to support 22 drives, 20 data + mirrored boot drives. I wanted it rock solid and essentially self healing with redundancy.

As I work in enterprise IT I immediately went to Solaris 10 on an X64 platform (Intel Xeon 3440 to be precise) on a Supermicro X8SIL-F and 8GB ECC RAM. The machine is currently only running 12 drives, 2 boot + 10 data, as I'm waiting for drive prices to dropbefore I buy the next 10 data drives.

The MoBo has 6 SATA ports, I use 2 for mirrored boot, 2 for data then 8 extra ports on a Supermicro AOC-SAT2-MV8. I'm using 5 into 3 X 5.25" SATA backplanes for the drives in an Antec 1200 tower with a 750W PSU.

The O/S is Solaris 10 U8 with ZFS RAID Z2 on my 10 data drives (essentially RAID 6) with a hot spare, so I have 7 data drives, the drives in question being Samsung HD204UI 2 TB drives.

I've been running this system around a year now, and use a script to shut it down at 3.00AM every morning and then I start it up when I get home after work using the IPMI functionality. Given that most drives are rated for thousands of startup / shutdown cycles, this aspect doesn't worry me and means the servers power usage is less of a concern, though even with the current 12 drives it draws just over 250W.

I would strongly recommend a Solaris or OpenIndiana / BSD with ZFS server OS, it's far more stable, resistant to malware and viruses, and it's what's behind most big data storage systems before you go to the lunatic fringe of EMC / NetApp storage.

I hope this helps.

Best wishes,

Dave

Sounds like unraid fits your needs nicely.

unRaid doesn't support a second parity disk.

ZFS does not fully meet your use case. It's non-trivial to gradually increase the storage capacity of a protected ZFS zpool and it's not really designed to add single disks.

unRaid would meet most of your requirements and run well on your hardware - but it lacks dual parity. This is somewhat mitigated because data can be recovered from individual drives in the event of a complete array failure.

Windows 8 Storage Spaces holds some promise for your use case, but it's not ready for prime time.

Similar to W8, FlexRaid holds promise, but it is having some growing pains and it was not polished when I tried it last year.

With your use case, RAID6 on Debian with XFS might be an interesting solution.

FlexRaid is in full production.

Not to hi-jack this thread, but it appears FlexRaid is no ticky -> no laundry.

Do you know if there is a trial or limited version available?

I've no idea what this phrase means, but I believe the latest update to Windows FelxRaid will support a trial version.

Take it away Jack:

http://youtu.be/97TW1Zm1uMI

Yes, looks like this is so:

Changes in 2.0 Update 6 (to be released April 20th 2012):
- Added trial feature (new users will now be able to activate the software for a 14 days trial)

http://www.flexraid.com/2012/04/18/release-notes/

Why would you need a second parity disc ?

Some users --- from what I have read and gathered --- feel that a 8:1 or 9:1 data to parity ratio should be used.

So 8 HDD's for data to 1 parity drive?

yup...
makes sense (from a playing the odds point of view) for a real RAID setup where parity+1 drive fails and all your data goes Poof!!!

for any of the jbod+parity setups (aka unraid/snapraid/etc) it is nice and all... but not a big deal really since you can always recover the individual drives separately...

I think some people confuse protected arrays with data back up. These systems are meant to protect you from a single random drive failure. None of these truly offer great protection from multiple failures without mirroring. And honestly, if two drives fail at the same time, it was likely due to an event (power spike), not just old age. Unraid offers great protection against a single drive failure, and is relatively cheap and simple. If you're looking for Fort Knox security, you need your data duplicated in a second location.

I don't think anyone posting in this thread has that confused.

Any one drive (including parity) can be rebuilt. I didn' mean to imply that those posting here don't understand the systems. It's simply my opinion that if protection from a single drive failure isn't enough for you, how is any of these systems enough? If you need more protection than that, you should be looking at back-up.

I'm not suggesting it as an alternative. With prices like they are, people who can do 1:1 mirrors can probably hire a network administrator to do all the work as well. Protection from single drive failure is enough for me. All of these systems give you that, and all have their own advantages and disadvantages. I don't know which (if any) will allow you to easily mix PATA and SATA though. Personally I'm keeping my fingers crossed that the W8 storage spaces will work well.

Well the options are truly endless. Cabling in one box can be a lot of fun with all those hard drives (and I mean fun...I love modding my own cables). One option that has not really been explored in this thread is the external enclosure options.

There are dozens of really good options. One web site that can give you some good ideas is Addonics. They have several different enclosure options. They have a fifteen bay hot swap (no screws needed either) enclosure that uses port multipliers (which you can buy from them) to attach 5 drives into a single output (and the outputs available include esata, USB, USB 3.0). Basically you can put in fifteen drives and mix and match sizes and then plug those drives into the back of your motherboard. Use the one esata on the MB and then get a internal to external esata converter bracket to plug in two more esata cables. I would then use intel storage manager to configure the drives any way you want (it's a software RAID), but more than enough for any home system.

RAID controller cards are great to work with...but they can also be a headache. They also kill reboot times!

It's a nice option and easily expandable. You can also unplug the enclosure and plug it into another computer more easily.

Wesena...a European company that distributes in the US through Perfect Home Theaters...has some wick nice looking enclosures that house up to ten drives. All brush aluminium...very sleek. Just some options to consider.

I have an Areca ARC-1260 (16 channel SATA) controller in my server and am very happy with it.

A good high capacity HBA is going to need an x8 slot for decent bandwidth, and you only have room for one of those. I would recommend you look at the IBM M1015 cards, which can be flashed to various LSI variants. You can typically find server pulls on ebay for <$100. That will give you 8 ports on the card itself, which you can expand with SAS expanders. One of these cards + two HP SAS expanders can give you 64 ports with good performance. The SAS expanders are actually more than the m1015, but this way you can start small and cheap, and at least have the capability of plenty of expansion down the road. It supports spin-down and SMART, and has great support on a wide range of OSs.


I think you may be confused about the reason behind higher levels of fault tolerance. It's because a single random drive failure isn't your only vulnerability. If you have a lot of drives in your system, there's a decent chance you may have some that you bought at the same time, from the same batch. When you have a drive fail, and have to rebuild the data from parity, that's a fairly intensive task. It requires reading all the data on all your remaining drives. This can be an entire day (or even more) of all of your drives continuously reading. If you had one drive fail, and have others in your system of the same age, chances increase that other drives that may also be towards the end of their life don't survive that stress test. Or, you could have any number of glitches that cause issues on more than one drive at a time. Or maybe it just takes a while to get a replacement installed before you can recover the data. With only a single parity drive, if you lose another before you've fully recovered from the first loss, then you've lost data (from both).

It's simple statistics. The more drives you have, the greater your chances of something going wrong, and the more often it happens. The more parity drives you have, the more protection you have against Murphy's Law. I have 21 drives in my server, and two parity drives. And to be honest, if 2TB drives were still $60, I'd probably have a 3rd by now, and will be adding one when the prices come back down. It's cheap insurance. 1:1 backups are even safer, but would require 42 drives, which is crazy. Multiple parity drives allow you to scale your fault tolerance to whatever level you feel is appropriate for the number of drives you have, and your budget (and space constraints, ports, etc.)

I do understand the reduction in fault tolerance, but I still don't agree with the logic. Dual parity drives protect you from two random failures ( long shot) and only then if one of the drives happen to be the parity drive (longer shot). Two failures in the array loose data even with three parity drives. I'm not trying to pick a fight here. All things being equal, more protection is always better than less. I'm just saying that I wouldn't go with something less suitable for my needs in order to gain extra parity drives. It just seems that it's a long way to go to cover one "what if" while you're still vulnerable to so many others.