or Connect
AVS › AVS Forum › Video Components › Home Theater Computers › HTPC - Linux Chat › Media Server - How to Procede?
New Posts  All Forums:Forum Nav:

Media Server - How to Procede?

post #1 of 38
Thread Starter 
So my current media/file server has failed (RAID controller failure). I have 2 arrays: a RAID 1 for the OS and important files (that data is fine as it's a mirrored pair), and a 5.5TB RAID 10 for my movies (that data may be unrecoverable). I have about 2/3 of my movies backed up, but will have to re-rip and re-transcode the rest. UGH.

My server is an IBM x3500 enterprise server with 2 Xeon quad-core processors (running CentOS 6). Overkill to the max but I got it free from my previous job. Why not use it right? It worked awesome with Handbrake (it could transcode an entire DVD movie in about 45 minutes).

So here's my dilemma: my co worker has a spare chassis of this same server. Should I take and use that to get things back online, or just build another server? Keep using hardware RAID or switch to software RAID? I do know, at this point, I'm going to cease using any RAID levels that involve striping or parity. So I'll just do a bunch of RAID 1's from here on out.

I don't know if I can recover my RAID 10 data, even with replacing the RAID controller with the exact same model.
post #2 of 38
With modern computers, software RAID usually is the better choice, often with software RAID performing better. Another advantage is that then you can move the RAID array from one computer to another without having to worry about on-disk format compatibilities. Unfortunately, IBM makes this difficult by often using LSI RAID HBAs which don't include the JOBD option. (At the lab where I work, we usually have to verify that the order has the right HBA part number so we get the JOBD option, although our salesman knows that now.)

RAID6 with a hot spare usually is the best choice for capacity (i.e. only 3 overhead disks) and it'll automatically start rebuilding on the spare when one of the active array disks fails.

Of course, whatever RAID choice you make, you should make sure you're using disks from different manufacturing batches. Too often, when using disks made in the same batch, when they have failures, several disks tend to fail very close together in time.

And do monitor the array health., of course. The extra reliability provided by RAID does no good if you don't notice that a disk has failed and don't replace it promptly.

Edited to add:

I'd suggest using the identical hardware configuration initially, if only so you can try to recover as much as you can from the damaged array(s). Then copy them to software RAID arrays.
post #3 of 38
...
Edited by quantumstate - 12/25/13 at 10:59am
post #4 of 38
Thread Starter 
Quote:
Originally Posted by Selden Ball View Post

With modern computers, software RAID usually is the better choice, often with software RAID performing better. Another advantage is that then you can move the RAID array from one computer to another without having to worry about on-disk format compatibilities. Unfortunately, IBM makes this difficult by often using LSI RAID HBAs which don't include the JOBD option. (At the lab where I work, we usually have to verify that the order has the right HBA part number so we get the JOBD option, although our salesman knows that now.)

RAID6 with a hot spare usually is the best choice for capacity (i.e. only 3 overhead disks) and it'll automatically start rebuilding on the spare when one of the active array disks fails.

Of course, whatever RAID choice you make, you should make sure you're using disks from different manufacturing batches. Too often, when using disks made in the same batch, when they have failures, several disks tend to fail very close together in time.

And do monitor the array health., of course. The extra reliability provided by RAID does no good if you don't notice that a disk has failed and don't replace it promptly.

Edited to add:

I'd suggest using the identical hardware configuration initially, if only so you can try to recover as much as you can from the damaged array(s). Then copy them to software RAID arrays.

Makes sense. I definitely planned to buy an extra 2TB drive or two to back up the rest of my collection to once I get this back online (IF I do).

Question: if I replace the controller and am able to get the array back online, should I roll with it? Or should I assume I just bought some time and plan do do something else? This server does have an internal USB flash port, primarily for booting a VM hypervisor, but it should boot anything that will run on the server. What if I throw an OS on a flash drive to boot from there, and then setup software RAID with that? Something like FreeNAS or whatnot. I do want whatever OS to be robust enough to run HandBrake and all that. I do love the horsepower this box has with HandBrake to transcode.

The hot swap bays on the front of this server (where all 8 of my drives are) all run through the RAID controller. So if I don't run with the new controller and current config, I guess I'd just set up all the drives as Volumes in the controller. That way the OS can see them as 8 individual drives, and I can implement whatever software solution I choose.

I don't have the money to build a new server unfortunately. frown.gif

Thank you for your post. smile.gif

Quote:
Originally Posted by quantumstate View Post

If you can't get your data with the same model controller, then that would disqualify the whole hardware line AFAIC.

Problem with hardware raid is its lack of transportability, as noted above. I don't really like software raid either, as it's an extra layer. Any Complex System Can Never Be Secure. (Bill's Third Law)

I use ZFS raid for my arrays. You have to patch the driver for the blkid problem (for now), and you can't dynamically add drives, and there is a learning-curve, but otherwise it is superb.

I don't use btrfs anymore, as it thrashes the drives as bad as ext4.

Hey QS! Long time no talk-to. How you been?

I'll look into ZFS. Heard of it, but never did any research on it. Thanks man. smile.gif
post #5 of 38
...
Edited by quantumstate - 12/25/13 at 11:00am
post #6 of 38
QS, is there any particular reason why something like FreeNAS for ZFS is a bad recommendation?

Also, do you run FreeBSD, or a native kernel port for Arch? Or are you still using Arch?


I never could go with FreeNAS for two basic reasons
  1. FreeBSD seemed to leave a lot of applications in need of ports (namely Plex Server) - this could be eradicated if the native kernel ports are stable
  2. Can't add a single drive to an existing array's parity protection. Growing your array either requires replacing existing drives for larger drives or adding additional drives along with their own parity drive

I always try to talk myself out of the last one, but it's been a handy feature of snapshot raid over filesystem. Out of space, add a drive, prosper tongue.gif
post #7 of 38
...
Edited by quantumstate - 12/25/13 at 11:00am
post #8 of 38
While we're on the subject of storage configurations, I have to say that I'm giving strong consideration to running NexentaStor in a VM, passing the drives through to it directly.

http://www.nexentastor.org/projects/site/wiki/Tour

@shane:

As others have noted above, I agree that software RAID (although a bit slower) is the better option first and foremost because of the problem that you are experiencing right now.

If you buy another controller and decide to switch methods, then you're gonna have to find somewhere to back up everything first.

It might actually be easier to re-rip your entire collection onto a software RAID configuration.
post #9 of 38
Thread Starter 
Quote:
Originally Posted by sysadmin View Post

@shane:

As others have noted above, I agree that software RAID (although a bit slower) is the better option first and foremost because of the problem that you are experiencing right now.

If you buy another controller and decide to switch methods, then you're gonna have to find somewhere to back up everything first.

It might actually be easier to re-rip your entire collection onto a software RAID configuration.

So, when I get my hands on the spare server chassis and swap controllers, if I am able to recover the array, should I run with it, or consider it a miracle, backup and start over?
post #10 of 38
...
Edited by quantumstate - 12/25/13 at 11:01am
post #11 of 38
Yeah, you've already waded into the deep end and swimming is going to be difficult

While ZFS is great, I'm not sure you can start with data on your drives?? Seems like a trivial piece of knowledge, but I simply don't know if there is a method out there letting you bring drives into zfs with data on them

As to your conundrum, I'd personally recommend obtaining the controller and a couple of 3TB drives (usually you'll find them $90/each if you keep an eye out here). Then mount them as jbod, copy your data off the Raid 10, try to resell the controller, and reformat/mount the existing drives as jbod. It's certainly not ZFS level integrity, but snapraid is a pretty good solution. Symlink pooling (read-only), writes need to specify drives, snapshot parity backup, up to 2 parity disks. Snapshot based software raid gives you r/w performance equal to your disk except during scheduled parity calculations
post #12 of 38
QS, out of curiosity, can you replace a drive's partition with a full drive?
post #13 of 38
Dunno that I'd go with JBOD, but DS has the right idea.
post #14 of 38
If you're going to use software RAID, then you have to configure the hardware RAID controllers to treat all of the disks connected to them as JBOD -- Just a Bunch Of Disks.

The claim that "software RAID is slower than hardware RAID" assumes slow CPUs and fast RAID controllers. The benchmarks that have been performed at the lab where I work have shown that this is a false assumption with modern multi-core CPUs in our environment. Software RAID is faster than hardware RAID for us. This is completely irrelevant for home audio applications, though. Streaming audio and video files does not need very high performance and those small performance differences wouldn't even be noticed.
post #15 of 38
Thread Starter 
Yeah, the server I'm dealing with has 8 cores at its disposal (2 quad-core xeon processors...older ones, but xeon quads nonetheless). So horsepower might not be much of an issue for software RAID.

So if I get the arrays back online, I will ve backing up the rest of the data (obviously). And just so I'm clear, I should not continue to run with this, and instead should assume I'm on borrowed time and prepare to wipe and start over right?
post #16 of 38
That's your own call, but I would go with software RAID. And since you have 8 cores to work with, I would seriously consider delegating 1 or 2 of them to running a storage appliance in a VM such as FreeNAS or NexentaStor. This would take away from Handbrake, though.

So, you have the opportunity to really consider some different options, at least.
post #17 of 38
...
Edited by quantumstate - 12/25/13 at 11:01am
post #18 of 38
Quote:
Originally Posted by sysadmin View Post

Dunno that I'd go with JBOD, but DS has the right idea.
Quote:
Originally Posted by Selden Ball View Post

If you're going to use software RAID, then you have to configure the hardware RAID controllers to treat all of the disks connected to them as JBOD
Right. There are good guides out there for flashing IBM m1015 cards into IT mode or Dell PERC 310 cards to firmware that allows JBOD. Necessity for all software raid AFAIK

Quote:
Originally Posted by quantumstate View Post

I think you mean if that partition is part of an array. Incisive question, and IDK. I've only pooled full drives. Part of the issue is that you should set up the pool by-id (/dev/disk/by-id) or by uuid (blkid) rather than sdb-sdc-etc in case you ever physically reconfigure, or the kernel rejiggers them. #zfsonlinux will ablely advise.
Yeah, forgot about that. I do have a hard limit on my HDD spots in my case, so I was just trying to think of a clever way to have all spots filled later on. Obviously you got the idea already, but for anyone that didn't it would be creating 4 partitions on a single drive and replacing each individual partition with a full drive. This way you could create your vdev in zfs with a single drive partitioned as many ways as drives you plan on extending. Then begin replacing each partition with a full drive. This wouldn't be good long term since your UoR would be n-1 partitions, and the death of that single drive would toast your zpool. In the short term it would be ideal since you could begin copying data over and delete partitions (less than your UoR) then replace with full discs and grow your pool out to the intended number of drives you want to add.

Sounds good but fatally flawed from basic setup (as noted by QS above) - guess that was too much explanation for something that is ill-advised disregarding whether it would even work at all.


Shane, IMO hardware raid is always on borrowed time unless you have spare identical controllers. Software RAID - either realtime or snapshot - is controller independent (though they can assist in providing additional HDD sata spots with the right firmware) and your backup is UoR based (unit of risk). You'll be able to recover from as many HDD failures as your UoR. Multiple, simultaneous HDD failure is a potential threat to your data in software RAID, but it seems like more of a remote possibility (to me) than a future controller failure where an identical controller is no longer available.
post #19 of 38
Thread Starter 
Quote:
Originally Posted by Dark_Slayer View Post


Right. There are good guides out there for flashing IBM m1015 cards into IT mode or Dell PERC 310 cards to firmware that allows JBOD. Necessity for all software raid AFAIK

I can set up each of the drives in this controller as "volumes," meaning just a disk. Is this the same thing as JOBD?

Quote:
Originally Posted by Dark_Slayer View Post

Shane, IMO hardware raid is always on borrowed time unless you have spare identical controllers. Software RAID - either realtime or snapshot - is controller independent (though they can assist in providing additional HDD sata spots with the right firmware) and your backup is UoR based (unit of risk). You'll be able to recover from as many HDD failures as your UoR. Multiple, simultaneous HDD failure is a potential threat to your data in software RAID, but it seems like more of a remote possibility (to me) than a future controller failure where an identical controller is no longer available.

Understood. Thank you for the info.
post #20 of 38
...
Edited by quantumstate - 12/25/13 at 11:01am
post #21 of 38
How much CPU do you guys think an appliance like FreeNAS or NexentaStor would take? Could it run in a VM on one core?

It would be easier to set up ZFS that way (assuming HW compatibility). I like the health-monitoring tools in NS, plus it runs on a fork of OpenSolaris, so pretty much native ZFS support.
post #22 of 38
...
Edited by quantumstate - 12/25/13 at 11:01am
post #23 of 38
I was addressing Shane's issue. It might be easier for him to run NexentaStor in a VM than to set up ZFS in linux.

He could then export the array (volume) to his core linux OS via iSCSI.

EDIT:

This is such an awesome idea -- I'm gonna do it myself. :-)
Edited by sysadmin - 12/24/13 at 11:24am
post #24 of 38
...
Edited by quantumstate - 12/25/13 at 11:02am
post #25 of 38
You follow. He has a separate volume that boots, as will I.
post #26 of 38
Thread Starter 
But I don't. The RAID controller took that out. I have 2 RAIDs: a RAID 1 which houses the OS and other misc files like pictures and music. Then I have a RAID 10 made up of 6 2tb drives (8 drives total) for the movies/tv shows. All of these drives are connected to hot-swap SAS/SATA bays on the front of the server. That backplane runs through the RAID controller.

I could take one of the mirrored drives and hook it up to one of the SATA ports on the motherboard, but there's not really room to mount the drive inside the case. It's an IBM x3500 server (in rack mount config). Here it is (with only one drive at the time...that whole bay is full of drives now):

2012-09-29112833.jpg
post #27 of 38
If the controller handles the hot-swap functionality and you can't cable the drives directly to the mobo, then you should definitely replace the controller with one of the same model.

You can then access and backup your data before proceeding to setup SW RAID (directly or via an appliance VM) which is better according to the consensus here, for a number of reasons.
post #28 of 38
Thread Starter 
So it appears the RAID controller was fine all along. There is something wrong with the server chassis itself. Either the motherboard or backplanes or something.

I got the spare chassis from my friend, swapped its RAID controller into my chassis, and got the same error: controller failed to start. I then installed my RAID controller into the spare server chassis, moved all the drives over and the controller booted, found the arrays and is in the process of syncing them now (gonna take forever).

But at least I know it works (or will once it's done)!
post #29 of 38
Maybe you can find a board that will run both of those CPU's, then you'll be handbraking and running VM's like there's no tomorrow. :-)
post #30 of 38
Thread Starter 
Que? I did. It's the spare server chassis my friend had. It's the exact same server complete with RAM, 2 CPUs and all. Once the RAID controller finishes syncing the arrays, I'll be back in business with all 8 cores running.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: HTPC - Linux Chat
AVS › AVS Forum › Video Components › Home Theater Computers › HTPC - Linux Chat › Media Server - How to Procede?