[stgdz]

So I have a bunch of hard drives but they are not all the same size. I back up allmost all of my TV shows to CD-rs and speed is not really important because I am limited to my line speed through my network.


Why is RAID used so much and why would it benefit me?

 

[Teran]

1. Data redundancy and uptime. If a drive goes down data isn't lost and the computer can continue to operate.


2. Speed. Depending on the level of RAID used data transfer speeds, especially reads, can be increased.


Normally you want to use as identical drives as you can.

 

[Mac The Knife]

There's also the convenience factor; the RAID controller makes all those little disks look like one big disk (although some non-RAID controllers also can do this).


RAID controllers also tend to be built with higher quality components than the non-RAID conterparts (again there are exceptions).


To expand on what Teran already mentioned about 'speed':

All forms of RAID will reduce your average latency (this will help you even though you are thruput limited by your network). Some forms of RAID will also increase the thruput, but in your case you don't care.

 

[evans036]

most raid controllers do not have the problem with writing beyong the 137GB partition size


for random access read, raid can slow you down (ie 2 rotational delays - but this depens of stripe - chunk - size). raid 0 really shines for large sequential transfers. But this is not required for video apps since most video

raid 1 (mirroring) is good for fault tolerance (ie one disk goes bad, the other keeps everything running) its also faster for reads becuase 'smart' controllers use whichever drive has the head closer to the data.


raid5 sucks for sequential writes (raid 5 is rare on consumer level controllers) but is good for fault tolerance (like raid1) but requres about 20% addition disk (instead 100% disk for raid1).


for most of the kind of stuff we do with video, no raid or raid 1 is best


my 2c


steve

 

[Hugo C]

Four types of RAID you'll come across (there's more, but you won't realistically encounter them):


RAID 0: stripes data across all of your drives. If you had three drives, instead of writing bit 1, then 2, then 3, each drive would get one bit and they'd be written at the same time. Theoretically, this multiplies your drive performance by the number of drives you have - subject to the throughput of your interface, controller/software overhead, and so forth. The downside is that if one drive in the array goes down, you have lost the data on EVERY drive as there's no redundancy.


RAID 1: Mirrors the data on one drive onto a second. This speeds up drive read but not write actions, but the main benefit is that if one drive fails, you haven't lost any data as it's all duplicated. However, if you have 2 120GB drives you're only going to get 120GB storage out of them which isn't the best MB per $.


RAID 0+1: basically, takes two drives (e.g.) to make a RAID 0 stripe set and then duplicates that onto two more drives in RAID 1 fashion. The downside is that, like RAID 1, you're doubling the disks, but in this you need at least 4 which can get expensive.


RAID 5: uses a stripe set as RAID 0 combined with parity bits to provide some redundancy. It's not quite as fast as RAID 0 and not quite as reliable as RAID 1, and the array needs to be controlled with hardware (dedicated RAID card) and can't be done in software through the OS, at this time. I don't know of any low-end cards that provide RAID5, so you'll need to get a mid-to-high-end SCSI RAID controller which, together with a few SCSI drives, will probably set you back well over a thousand dollars. You can easily build a complete, well-specced computer for less than that.


As to how RAID would benefit you, I'd say it wouldn't. As Steve and Mac said, streaming DVD video requires only the throughput of a 1x DVD-ROM, which is way below even that of a 10-year-old hard drive. RAID 1 will give you redundancy but unless that data is mission-critical and irreplaceable, you won't need that. You can expect around 1m hours MTBF on a modern drive which means, if you ran it 24/7, you could reasonably expect it to last over a century without a failure. I used to be a hardware technician and in two years in the field I probably saw 2 or 3 physical drive failures. You don't really need to take that into consideration unless your data is literally priceless.


Hope that helped!

 

[Mr.Poindexter]

Hugo, not to pick on you, but you have stated several things incorrectly.


1. RAID 5 can be done in software in Windows 2000 and Windows 2003 server. It can also be done in Linux.


2. You wouldn't need SCSI to do RAID 5. There are many IDE RAID cards that support RAID 5, using either PATA or SATA IDE drives. 3Ware is a good option that most people here use. RaidCore is a new card that will probably be the card of choice when SATA drive costs come down.


You can run a hard drive non stop, but it will not last 100 years. Looking on Maxtor's site, their 200 GB drive lists a minimum of 50,000 start/stop cycles and a component design line of 5 years minimum. Their MaxLine II drives list 1 million hours MTTF in low I/O use. Don't expect one to work with a TiVo for a century.


I have not had a drive last 100 years yet, but I did have a SCSI drive at work die after 4 years - one of 5 drives in the RAID array. I also had 3 hard drives on the desktops die in 4 years out of 6 computers at work. I had a hard drive on my computer in college die after 3.5 years. At any rate, I would NEVER count on a drive lasting for more than 5 years.


All in all, I have suffered with 5 hard drive crashes and survived one on the sole RAID system I had. Now I will NEVER EVER EVER run a server and not have RAID.


Evans036, RAID 1 is not the best RAID solution for a media server, RAID 5 is. That is why almost everyone with a large server uses it. If I ran RAID 1 on my media server, I would have 2 TB of storage. With dual RAID 5 arrays, I have 3.5 TB of storage. If I had the Raidcore cards and SATA drives, I would have had 3.75 TB. The overhead for RAID 5 is 20% if you have 5 drives in your array. The actual overhead is one drive out of the array.

 

[cutman0122]

If there are benefits to RAID, is there a proper and improper way of implementing it? If you have RAID 0, would the setup be compromised if you add another drive not part of the array? With an array plus an independent drive, which of the two would be optimal for the OS? Thanks.


Gerald

 

[Mr.Poindexter]

First off, RAID 0 is not really RAID. It is just striped drives, but there is no redundancy and no fault tolerance. I have a RAID 0 partition and I just use it for file swaps, memory paging, etc. I don't use it for the media server's storage.


Online expansion is not very common on RAID arrays. The best way is to get a controller and fill it up with some big drives and make that your array. If you must have online expansion, buy a card that supports it, but realize that you will be out more money when you are done compared to just getting a RAID controller that doesn't offer expansion.


RAID 5 servers that have terabyte capacities are not cheap. There really isn't a way to get into it on the super cheap, so be prepared to spend some money on it. If you cannot afford to, then you are probably looking above your pay grade, to use military speak.

 

[BrianH33]

If your are just backing up movies or music that you own or data that can easily be re-created, then a raid 5 system is not really justified for the cost.


If you are backing up data that cannot be re-created, then redundancy is mandatory with either a raid 1 (2 drive mirror) or a raid 5 system.


If you have a bunch of different drives that you want to rip movies to, that you own, then I would highly recommend using Firewire to IDE convertors.

 

[madpoet]

Quote:

If your are just backing up movies or music that you own or data that can easily be re-created, then a raid 5 system is not really justified for the cost.

As someone (I think it was Elvis) said yesterday... when you're talking a terrabyte or more of data to have to re-rip, it is well worth the cost of a drive to give yourself some protection with RAID5. Drives really aren't expensive, and sacrificng one for the parity is a small price to pay for the hundreds of horus that goes into really large file volumes. I set up my Win2K server software RAID5 last night. 8x160gb maxtors, using 2 IDE controller cards. Software raid5 across them all, giving me 1.16tb of data with some fault tolerance. Is it perfect? Nope, all sorts of things could still go wrong. But it beats the heck out of nothing. There's not a lot of cost to it, and even less if you use Linux and it's freeware tools.


-MP

 

[stgdz]

But why not just get another drive when your drive is full and duplicate it and just put it in the closet? That way if your drive ever dies you have a fresh one.

 

[madpoet]

Er... because RAID5 is a lot cheaper than that.


-MP

 

[BrianH33]

I currently have 1.2TB (6 x 200g) full of movies as a JOBD. If I lose a drive I would only lose the data on that drive... not all 1.2TB's of data. Even if 2 drives go down at once, I would "only" lose 400g. If a RAID 5 loses 2 drives... game over i believe. (????)


So if I lose a drive (200g ... actually 186G or 40~45 movies), I replace it and re-rip my movies. No big deal, might take me a month or 2. No big hurry to re-rip my movies.


Plus, if I run out of space, simply buy another drive, format and start using it. I am limited by drive letters only.....


Let's see

A,B,C and D are used

E and F are for shared drives from other computers


G thru Z is available. That's 20 x 200g = 4TB of drives. If you use 250g then 5TB. Should be more than enough for awhile. And the drives keep getting bigger. If I need to replace a drive (200g), I could then get a bigger drive (500g) that will be available by then.


RAID systems do not provide that much in flexability.

Again, I stress, that I would only do this with "recoverable" data!

 

[Teran]

Quote:

Originally posted by stgdz But why not just get another drive when your drive is full and duplicate it and just put it in the closet? That way if your drive ever dies you have a fresh one.

RAID isn't for backup purposes. If you had a software or user error that destroyed data on a array you would still lose that data and have to rely on backups to be able to restore that data. For backup purposes external firewire drives are quite popular.


RAID is for protection against hardware failure. With either RAID 1 or 5 you can lose a drive and keep on running. In the case of RAID 5 you would have a significant performance hit. But, you would continue to run until you could schedule downtime. And, you would not lose a minute's worth of data.

 

[madpoet]

Quote:

Originally posted by BrianH33 I currently have 1.2TB (6 x 200g) full of movies as a JOBD. If I lose a drive I would only lose the data on that drive... not all 1.2TB's of data. Even if 2 drives go down at once, I would "only" lose 400g. If a RAID 5 loses 2 drives... game over i believe. (????) So if I lose a drive (200g ... actually 186G or 40~45 movies), I replace it and re-rip my movies. No big deal, might take me a month or 2. No big hurry to re-rip my movies. Plus, if I run out of space, simply buy another drive, format and start using it. I am limited by drive letters only..... Let's see A,B,C and D are used E and F are for shared drives from other computers G thru Z is available. That's 20 x 200g = 4TB of drives. If you use 250g then 5TB. Should be more than enough for awhile. And the drives keep getting bigger. If I need to replace a drive (200g), I could then get a bigger drive (500g) that will be available by then. RAID systems do not provide that much in flexability. Again, I stress, that I would only do this with "recoverable" data!

Heh, you are limited also by space aren't you? What kind of monstrosity are you going to keep putting these drives in? RAID5 isn't flexible, you're right. At BEST you might be able to dynamically expand it, but that is typically with hardware raid controllers and only in some circumstances. What it is for is peace of mind. If I lose a drive, I haven't lost ANYTHING. I pop in another 160, it rebuilds the parity, and I'm done. If I lose 2 drives at once? Well, the chances of this happening are slim . If I do, then oh well. I'm screwed.I just make sure I have a well ventilated case with adequate power and a UPS.


-MP

 

[dguns]

RAID 5 question:


So how do you know which drive died in the array? BIOS? Eg. "HD connected to port#3 failed."

 

[BrianH33]

Quote:

Heh, you are limited also by space aren't you?

No more than a raid setup

Quote:

What kind of monstrosity are you going to keep putting these drives in?

I am using FW enclosures ... 1 per drive. Each comes with thier own PS and takes very little power. Just need plugs


For me, they are in the basement on a shelf under my HTPC setup connected by a single FW cable. Then daisy-chained from there. You can daisy up to 64 drives together.


I am not saying that RAID 5 systems are not good.... in fact they are excellent but expensive and some-what inflexable. But for storing music and movies, which is very static data, raid 5 is unnecessary. If you use your raid for movies, music and real non-recoverable data then you are being smart. (How many people out there have no backup of importance data on their C: drive? )


I do use a RAID 1 - Mirror to store all my data files (ie mail, letters, backup of OS's and other non-recoverable data).

 

[madpoet]

Right. It comes down to what your time is worth . One thing I did hear when I was researching this, but have never quite understood the rationale behind, is that the more drives you have in a Raid5 array the faster the array becomes. I assume it's because they are all spinning, and therefore writing the stripe is that much faster?

 

[Mr.Poindexter]

Quote:

Originally posted by BrianH33 I currently have 1.2TB (6 x 200g) full of movies as a JOBD. If I lose a drive I would only lose the data on that drive... not all 1.2TB's of data. Even if 2 drives go down at once, I would "only" lose 400g. If a RAID 5 loses 2 drives... game over i believe. (????) So if I lose a drive (200g ... actually 186G or 40~45 movies), I replace it and re-rip my movies. No big deal, might take me a month or 2. No big hurry to re-rip my movies. RAID systems do not provide that much in flexability. Again, I stress, that I would only do this with "recoverable" data!

Yeah, re-ripping 40~45 movies isn't that much work. Try re-ripping 600 CD's. Or doing the work it takes to edit out DVD extras and join split disc titles into one seemless title and you are looking at much more work. I can rip 40 movies in about 15 hours. An extra hard drive cost me $110, so I would be working for $7.33/hr. when a hard drive fails if it has movies on it. If it has CD's, then I am working for much less. Doesn't sound like a good deal to me.


If two drives fail at once on the same array, then yes, game over on that array. I have two RAID 5 arrays right now, so even then I would not lose everything, just half.


Recovery of recoverable data is only one side of it. You still have lost the work of putting that data in the form you want.

 

[BrianH33]

When writing data to one drive at say 500K (yes I know it's very low, example for calculation purposes) per second


To write a 10mb file, in theory, that's (10,000K / 500K) = 20 seconds.


If you have 2 raided drives, effective thru put becomes 1000K. This is because 1/2 the data gets sent to drive A and the other half get sent to drive B. Therefore, time required is (10,000K / 2 / 500K) = 10 seconds.


So for 4 drives, (10,000K / 4 / 500K ) = 5 seconds.

So for 12 drives (10,000K / 12 / 500K) = 1.67 seconds


That is the theory. In actual pratice, the gains are not so great but still pretty. The real speed increase is in reading!


Now to play a movie, all we need to around 9~10mb per second. This is actually VERY slow for a hard drives. Most stand alone IDE-ATA 100 can do 30 to 40MB easily!


Oh, this is my understanding of it and I am not any kind of expert on this.