Hi everyone,

I have been lurking around here for a while and have learned a lot. My hard drive died in my make shift server (an old PC running freenas) and I lost all of my ripped music and movies. Of course, my family photos and videos were backed up to an external drive and will continue to be backed up to an external drive :). I’ve decided to build a proper server before I re-rip my collection.

My objectives are as follows:
1. storage of some personal computer files, family photos, family videos, ripped music and movies;
2. stream photos and movies to three HTPC, and music to either a sonos or squeezebox (haven’t decided which one to get yet);
3. not having to re-rip all of my music and movies if there is a drive failure;
4. expandable in the future by 1 or 2 drives at a time (I recently purchased a blu ray machine so I’ve begun to purchase blu ray discs);
5. downtime is not truly an issue given that this is just music and movies. Worst case scenario, I’ll physically go and get the CD or DVD and put it into the player).

I’ve been reading up and learning about the different options and have been able to set up a fully functional test server with openfiler and CentOS on an old Pentium 4 desktop (thanks to the tutorials on howtoforge.com) and believe that I’m comfortable with setting up a server with Linux.

From the HTPC Guide thread, this will be my server:
(E) = Existing hardware (O) = on Order

(O) GIGABYTE GA-MA780G-UD3H AM2+/AM2 AMD 780G HDMI ATX AMD Motherboard
(O) AMD Athlon 64 X2 5200 Brisbane 2.7GHz 2 x 512KB L2 Cache Socket AM2 65W Dual-Core Processor
(O) A-DATA 2GB (2 x 1GB) 240-Pin DDR2 SDRAM DDR2 800 (PC2 6400) Dual Channel Kit Desktop Memory Model ADQVE1A16K
(E) Ultra LSP650 650-Watt Power Supply - ATX, SATA-Ready, SLI-Ready, 135mm Fan
(E) A generic case able to hold 8 hard drives
(E) 4x 1TB Seagate Barracuda 7200.11 model ST31000340AS

Future Purchase when need to expand:
NORCO RPC-4220 4U Rackmount Server Chassis w/ 20 Hot-Swappable SATA/SAS Drive Bays
CORSAIR CMPSU-850TX 850W ATX12V 2.2 / EPS12V 2.91 SLI Ready CrossFire Ready Active PFC Power Supply
SUPERMICRO AOC-SAT2-MV8 64-bit PCI-X133MHz SATA Controller Card

I have made a decision of not going the hardware RAID route due to cost. I rather spend the money I save on building a high end HTPC.

My Questions:
(1) Should I install CentOS or should I just go with Openfiler?
(2) Given that blu ray takes up a lot of disk space, I can envisage adding more and more drives to this server:
(a) Should I go with software RAID 6 instead of 5?
(b) As I expand, what is the maximum number of drives on a RAID array before I should set up another array? or can I just have one huge array?
(c) Would set up LVM now help with the addition of drives in the future either in 1 or multiple arrays in the future?
(d) What to do with backup of music and movies or do I really need to worry about it? Am I over thinking it?

Thanks for all of your help in advance.
Cheers

Have you considered using a version of OpenSolaris or FreeBSD so you can use the ZFS filesystem (http://en.wikipedia.org/wiki/ZFS)?

I'd go with Nexenta if you're comfortable with the command line or StormOS if you want a desktop.

1. I know nothing about CentOS or Openfiler. I personally use Debian, which makes for a fine, stable server.
2a. Since you're not storing critical data, I would go with RAID 5 rather than sacrifice an extra terabyte for redundancy. However, if you plan on having a large number of drives in a single array, the chance of a double drive failure goes up, especially during the rebuild process, since that tends to stress the hard drives the most. So if you plan on having a large number of drives, then going with RAID 6 might be prudent.
b. The maximum number of drives in a single array is theoretically unlimited. However, you shouldn't put 100 drives in a single array, because the likelyhood that a second drive will fail during an array rebuild becomes too high.
c. If you use a single array, I don't think LVM would be of much benefit. You don't need LVM to add drives to an array. If you have multiple arrays, however, you could use LVM to combine the arrays into a single, huge filesystem.
d. The only cost-effective way that I know of to backup many terabytes of data is to buy more hard disks, copying the data to them, which are preferably located on another computer somewhere, in case something happens to your main server, knocking out all drives at once. But me personally, as soon as I bought all those extra hard drives, I'd want to fill them up with more data, not use them for backups. :)

Hope this helps!
-- Kevin

Hi TomKooze,

thanks for the suggestion. I actually spent a couple of hours reading up on ZFS and was quite excited about it .... that is until I got the expandability part. With my present 4x1TB drives, with a RAID-Z, I'll have ~3TB in the pool. If I want to expand, I have to purchase 4x1TB drives, RAID-Z it then add it to the pool. To me, this is an inefficient use of drives.


Kevin,

I think that you're right on with the backup. I was hoping there there was something that I missed in my research .... but If I'm going to buy more drives, I'll probably going to add it to the server. I'll just backup my music and keep my finger crossed on the movies.

I'm still undecided on the RAID 5 vs. RAID 6. I agree that RAID 6 seems to be a waste now given that I only have 4 drives. I think that at the most, I'll have 20x1TB drives (this is due to the case that will house the drives).

I know nothing about how servers work in real life ... so I'm not sure if 20x1TB is a lot of drives or not in an array and wether RAID 5 is sufficient.

For now, I'm probably not going to LVM it unless I'm missing something obvious.

Thanks
Eric

I was just doing a little research, and it seems Linux 2.6.30 introduced the ability to convert a RAID5 array to a RAID6 array, and back, on the fly, without destroying your data.

So if you're going to be using Linux, start with RAID5, and then when you get around the 8-10 drive point, add an extra drive and convert to RAID6.

Also, I forgot to mention this before, but whatever you do, don't use ext3! It forces an fsck every once in a while, which takes forever on large, multi-terabyte filesystems. Use BtrFS or ZFS or XFS or Reiser.

Also, I forgot to mention this before, but whatever you do, don't use ext3! It forces an fsck every once in a while, which takes forever on large, multi-terabyte filesystems. Use BtrFS or ZFS or XFS or Reiser.

You can disable this check in fstab, just set last two options to “0 0”...
But yes, if you want to use 'ext' for data drives I would use ext4 …

Doesn't RAID5 use one of the drives for parity, also? You might be thinking about RAID0 (which can be done in ZFS, too).

BtrFS is the closest thing in linux, but it's not stable yet, so I'd stick with XFS if you don't think that ZFS would work for you. And maybe get a small, fast SSD to run the OS from on ext4.

Kevin,

I was thinking about RAID5 to RAID6 conversion too ... but does it work well in real life?

TomKooze,

I was thinking about RAID5 not RAID0 ... was questioning about whether 1 drive parity is sufficient or whether I need to go with dual parity given that this may turn into a 20TB server at some point.

I think that I'll go with XFS ... would jump on ZFS if expandability is not an issue.

May I ask why run the OS on ext4 vs. XFS?

Sorry to change usernames on you mid-convo.

OK, I see what you mean about expanding the array on ZFS. It wouldn't work if you just add one drive at a time, you'd have to add 3 or 4, make a new RAID-Z, then add it to the zpool.

Also, you want the filesystem your OS is running on to be able to read/write tons of small files as quickly as possible (think logs and package management), and ext4 is much better suited for this than XFS.

May I suggest JFS?
XFS can in some circumstances corrupt data when it doesn't get shut down cleanly (power failure, HW/OS freeze) and ext4 is still fairly immature, the fact that the ext4 code base is still changing doesn't inspire enough confidence in it for me.

I have a 4x1TB raid5 under Linux with JFS (no lvm just md raid) and despite several unclean shutdowns (power failures and even a motherboard failure and subsequent replacement) I haven't lost a single bit of data, I know for sure as I keep md5 checksums of all my media files.

There was a thread about this here a while ago:
http://www.avsforum.com/avs-vb/showthread.php?t=1066367

Also, you want the filesystem your OS is running on to be able to read/write tons of small files as quickly as possible (think logs and package management), and ext4 is much better suited for this than XFS.

True, for this reason it makes most sense to have a separate partition (or better a separate dedicated hdd) for the OS.
Then you can use different filesystem types for the OS partition and the data partitions.

Just another note on XFS vs. JFS: I've run them side-by-side, and with the exception of file-deletion (which won't be an issue for you since you aren't running mythbackend), XFS is much faster than JFS. I've never experienced any data corruption, either (except when hardware starts to fail, of course). It seems like the people who have had data corrupted with XFS were almost always using external drives, and the data loss occurred during a power outage. Yet another fine reason to make sure you have ample UPS power.

On ext4, I've heard quite a bit of good feedback on other forums regarding its stability. I haven't used it, though.

I've done a little research on ZFS and RAID. If you don't use RAID-Z, but instead use a single vdev, which is on top of an md array, you can expand the md array, and expand the vdev to fit it. I've read it can be done, but I haven't actually tried it myself.

As for XFS, I've used it on many machines since it was first released by SGI as patches, before it was officially accepted into Linux. The only time I've lost data is when I had a drive go bad. But I have heard of other people losing some data if they don't do a clean shutdown, but if memory serves, it was only in the files that were being written, or were very recently written, but not yet completely flushed. YMMV.

But I do know this about XFS, since I have done it myself: you can easily run it on a RAID5/6 array, and add single drives as needed, and expand the filesystem to take up the extra space.

Hope this helps!
-- Kevin

Also, you want the filesystem your OS is running on to be able to read/write tons of small files as quickly as possible (think logs and package management), and ext4 is much better suited for this than XFS.

Also, even after all this time, grub still doesn't play well with XFS. So if you have XFS on your boot partition, you should use LILO.

But I do know this about XFS, since I have done it myself: you can easily run it on a RAID5/6 array, and add single drives as needed, and expand the filesystem to take up the extra space.


You can do this with JFS too (I have done it twice already as for various reasons I first set up my raid5 md array with only 2 disks and then added the other two later).
In fact this should be possible with any Linux filesystem that can be resized.

Thanks for everyone's help ... A bit more reading for me now on JFS, Ext4, and ZFS on RAID array, and LILO <- I hard of this but not really sure what it is, more googling.

Cheers

I keep the OS on a totally different controller and file system. I like running raid1 with a simple controller using server grade u320 drives.

http://www.scsi4me.com/scsi.html

http://www.scsi4me.com/fujitsu-mas336rnp.html

I keep the OS on a totally different controller and file system. I like running raid1 with a simple controller using server grade u320 drives.

http://www.scsi4me.com/scsi.html

http://www.scsi4me.com/fujitsu-mas336rnp.html

thanks for the info, this is quite interesting (for me anyway) ... So the suggestion this will be a hardware RAID1 for OS. and software RAID 5 or 6 for the data.

One thing I haven't yet seen mentioned in this thread is the advice to use drives from different vendors in your RAID setup. Drives from the same "batch" often have very highly correlated failure rates, which makes you vulnerable to a 2nd drive failing while you are rebuilding your array.

I second this. I didn't go as far as using different manufacturers, but I did space out the purchase of each drive by two weeks, as opposed to buying them all at once. In the semiconductor industry (where I worked for a while before selling my soul), wafers are started on and grouped by the week in which they are started. I am assuming that this is somewhat similar to how the hard drive platters are manufactured. By spacing each purchase out by a week or two, you can probably get hard drives with platters from different batches (and starting material), possibly limiting your risk. Also, buying your drives from a high volume seller may also help with getting drives from different lots.

Good advice on drives, you guys. Thanks.

Thanks for the advice on the drives. I was under the impression that I should buy all the drives from the same manufacturer. But what you suggest make a lot of sense.

If you do end up going with different manufacturers/models, remember that your RAID partitions can only be as big as the smallest drive. There may be some small variations in the size of the drive between models/manufacturers.