Guide To Building A Media Storage Server - Page 54

Only issue is I don't think they support Raid6. Or maybe the latest FW added it? What's the difference between Raid5 and Raid6?

If I use 3 of those cards can I raid together each card with 8 drives and then do one logical drive with the 3 different arrays? Also if I do something like this is it possible the expand it? Seems like it would make adding new drives more complicated.

How about this LSI card? It has good specifications, 16 ports, the usual online capacity expansion and RAID level migration, is PCI-e, has the nice SFF-8087 multilane internal cables AND supports Chenbro expanders if you ever need to go past 16 drives.

It may not be quite up to the level of the Areca 1680 series in performance, but I think it is pretty close. Far better than 'fake RAID' at least. Search the last few pages of this thread for benchmarks. If you get a good deal it is just over $600. What a bargain

Oh, as far as it 'takes a day' to expand an array . . . it really isn't much of an issue as your array remains online during the whole process. It doesn't take your system down. On my Highpoint card I can even adjust the priority of the task to prevent slowing other operations, or speed the expansion.

You've got to spend some money to get decent hardware. The LSI solution is barely half of Areca.

Obviously you know by now I much prefer RAID to WHS drive-pooling or JBOD, but I think my reasons are sound. I would recommend RAID over WHS.

RAID6 has two parity drives instead of one, so can suffer two drive failures and survive. Not a bad idea for larger arrays, as your odds of unrecoverable bit errors increase with size. Actually, they use a fancier encoding scheme than simple parity called Reed-Solomon, but the result is essentially the same: redundancy.

Is your interest in software RAID based on cost? If so check the LSI products. You may be able to afford true HW RAID performance.

They're Dell versions of a very expensive hardware-RAID LSI card. They had compatibility issues with plenty of non-Dell motherboards early on, but someone discovered that taping a pin on the PCIe connector would make them work with more boards. Most people flash directly to the most recent LSI firmware. They're quick cards. Not as fast as Arecas, and without RAID-6, but still plenty quick.

That's the good. The bad is that quite a few people seem to have weird problems with the damn things. Quite a few don't, but enough do that I wasn't comfortable buying one and trusting my storage to it. I opted for a RAIDCore card instead. I lost the battery backup and the hardware chip, but I gained rock-solid stability and excellent drivers. But now that RAIDCore is defunct, those Dell cards are the only game in town for that price and performance bracket.

If you're serious about software RAID, I'd still lean toward the SuperMicro cards. The IOP333 chip won't do you any good, and you might as well choose cards that don't require a sharpie to function properly.

I'd recommend you just use RAID6 from the start and do away with LVM. That way you are more protected - any two disks can fail. LVM also has it's own headaches and will reduce performance.

I've just set up another server so have some kind of clue how to get SMB, NFS and AFP working - all insecure though.

Is anyone using ZFS with a raid-z pool instead of a more typical RAID (be it hardware or software via mdadm)?

I'm considering going with Solaris or OpenSolaris for my new server and putting 7x1TB drives in a raid-z pool.

ZFS is pretty awesome and offers a lot of nice things RAID doesn't, and is especially competitive on a budget since no expensive RAID controller is needed. One major downside of it is that it currently doesn't support growing the array size, but that should be coming as a future feature.

As I mentioned earlier, I'm planning to evaluate solaris/opensolaris with ZFS/Z-RAID2 using my adaptec cards as I think the hardware stack on them is rather dodgy. Essentially I'll be using them as glorified HBAs with just JBODs or single volumes being handled by ZFS.

From what I understand, you could just create a new Z-RAID2 and add it to the pool as a sort of workaround for expansion, but I'm not sure what the performance implications of that would be. If expansion of the Z-RAID2 will be implemented in the future, you'd just have to hope that it will be done by the time you get new disks.

Cockroach:

Thanks for replying. I had a couple of reasons for considering LVM.

1. Increase total space in increments
2. Each increment is its own RAID 5/6 array
3. New arrays can use higher capacity HDD's as they become available
4. Each array is limited to 4 drives, so rebuilds aren't as long as they'd be if each array had 8-12 drives
5. Should one array fail, the others would have less chance of corruption during the failed array's rebuild

It was my understanding that I could meet these goals with mdadm, while using LVM to have my desire of one volume fulfilled. Is there some other method of JBOD-ing seperate arrays that won't cause gray hairs? BTW, are there any GUI frontends for mdadm?

Also, please elaborate on your SMB/NFS setup. Do these services need to be established before setting up mdadm? Feel free to PM me if you don't want to clutter up this thread with my requests. Thanks.

What if I can get one of those Dell Perci 5E cards for 30 bucks shipped? haha might be worth a try?

Sure, but be aware that the "E" stands for "external," so you'll have to mount your drives outside of the case that holds the card.

These cards have dropped in price since the last time I looked. I see the 5/i version selling for $100, including the memory and the BBU. That's a good deal, though it doesn't include the specialized SATA fanout cables that you would need.

The problem is that I'm still not convinced it's the most reliable controller, and the tape modification doesn't work for everyone.

http://forums.2cpu.com/showpost.php?...&postcount=771

That's typical. Check the rest of "mainboard compatibility" thread to see if someone has your board. Some additional reading:

http://www.overclock.net/hard-drives...card-tips.html

Well I didn't win it so no big deal lol. Just figured for 30 bucks I'd give it a shot. I haven't purchased the rest of the system yet. Still not sure if I want to reuse the Dell server I have thats a dual xeon 2.1ghz or just build from scratch. The Dell is really noisy.

LVM does kill performance, but that said, if all you care about is maxing out a GbE, LVM won't hurt that much if you have a fast underlying array.

On another note, I'm doing a format on the new adaptec controller under windows server 2008 (2 8x1Tb RAID6 arrays). The controller is giving me a few warnings about "bad block discovered" and a hex block number, but none of the actual drives is showing an error. How can I tell which drive is taking the error? As these are the same drives that I used to run linux raid on, I am concerned that one of the drives has issues, and before I put any data on it, I'd like to pull it and get a replacement.

The controller doesn't seem to show a lot of low level drive data.

On the plus side, windows server 2008 installed on the hardware really easily (better than any install process I've ever seen with windows), and runs lickety split on the Quad core hardware. The system disk is a 500 GB 7200.10 drive and not connected to the adaptec for simplicity's sake.



thx
mike

I am gonna give ZFS a whack as soon as everything shows up. I will be running the following:

- Norco 4020
- AMD 9950 x4
- ASUS M3A79-T DELUXE AM2+
- 8 gigs RAM
- 4 gigs iRAM SATA RamDisk (for the ZIL)
- 8 port LSI SAS HBA
- 4 port Intel PT Gig Nic
- 4 port Intel MT Gig Nic
- 1000w Antec PSU
- 1.5T Seagates x 20 (30T)
- Dell 24 port Gig managed switch

I definitely will be using raidz2, and mostly using it to host VMs and will be doing iSCSI for the interface to the VM host box. Am taking a lot of advise from this guy, http://jmlittle.blogspot.com/ who has a lot experience with ZFS (in an enterprise environment) and has a lot of good info on SAS HBA compatability for Solaris.

I had planned on using Nexenta for the OS and ZFS management, but they recently upped their price from $800 for unlimitted storage to $14,000. OUCH. So much for that idea.

I am still waiting on the nics and the LSI card, once I have that I will post progress.

So anyone ever actually get their hands on an actual SAS expander card and try it out? They almost seem like things of myth and legend. I am dying to get a few but don't want to go and spend the money to be the first to find the compatability problems. I can't really afford to take one for the team like that.

Wish I could...

k.

According to my e-mail from provantage, they received the expander at their ohio warehouse and are in process of forwarding it to my address. It's the 28 port chenbro expander. I plan to test it with my adaptec 51645 and 31605 cards. I'd test it with more cards, but those are the only SAS cards I have. I'm gonna lay off getting any new cards for evaluation as SAS-2 6GB/s is right around the corner, and I suspect vendors will be coming out with new SAS-2 HBAs and RAID cards this coming year, e.g. LSI Logic and Adaptec.

There's a white paper over on the LSI page expounding on the benefits of SAS-2, and they did a little comparative graph of their SAS vs SAS2 performance results - it looks like their write speeds improved tremendously, so if they come out with a new RAID card, I might pick that up to replace my adaptec. It looks like ATTO already has some SAS2 HBAs available, but only available for OEMs.

Looks like Seagate is coming out with new SAS2 drives as well, e.g. the next generation Cheetahs and Savvios, with higher STR. I'm still not convinced of the long term reliability of SSD drives, so I'm probably going to stick to old fashioned disks for the near future, until prices go down and all the kinks are worked out.

Basically, the double-bandwidth is handled pretty seamlessly because the new cards are using PCI express 2.0 rather than the 1.0/1.1 all current cards are using. And SAS2 offers port aggregation, so you can combine 2 SAS1.0 links across a single SAS2.0 link. So your typical expander link would offer 2400MB/sec rather than the current 1200MB/sec, assuming your expander was SAS2 compliant. Sounds like fun... now if only cards were available now...

Please keep us up to date with this. Will you be running a single external cable from the card to the Expander?

Yes, I plan to run a single SFF8088 to SFF8088 cable from the 51645 to the external port of the SAS expander. At the very least I'll be able to connect 16 disks, but I think 20 might be possible if the cascade ports are just redirected back to the norco backplane.

I forgot that the 31605 doesn't have any external ports, but I can still probably rig something up using various converters. It might not be good for signal integrity though - that will turn up in testing, I suppose.

The expander is scheduled to arrive on tuesday next week, and I ordered a 2 meter SAS cable from ebay but I have no idea when that's going to arrive.

Also, it looks like provantage jacked up the price on it - it was closer to $230 when I ordered it, and now its more like $280... still special order only.

Most any of the distributors that carry chenbro can get it, ASI, Malabs, etc..., but it's still special order. Malabs will sell it to you for about $211, but it's not returnable because of the special order.

I'm very interested in how well it works for you, as I may try one myself if you have good luck for the 4 extra ports on the norco that are not covered by my 31605. Of course, if you can't daisy chain the 8087 through, you have you use 8 of the expander ports since the 8087 cable used supports 4 drives already.

If this works real win will be for folks who have 8 port cards like the 5805 or 3805 as it'll really add a lot of density to those configs.

BTW, on your adaptecs, do you see occasional bad block errors in the storage manager log file? What do you do about them?

thx
mike

I generally don't get bad block error messages unless there's some physical problem with the connection - e.g. dodgy backplane, loose power cable, bad drive. These problems also cause "aborted commands" to occur as well, which in my case tends to freeze up the system.

I looked up the error message, and the help text said "This is an informational message. This requires no user intervention," or something to that effect. That's a lot of help. That might be true, but normally you'd like to proactively fix those type of problems, and like you said, the error message is rather vague on details for troubleshooting. So sorry I don't have much more to offer than that, but I'd suspect a connectivity issue. Tighten the cables, etc. See if it's a particular drive or bay giving you problems.

It's a little odd that the utility doesn't tell you which drive is having the issue. I would expect the system to report all bad blocks as it's doing a build/verify operation to create the array, and I have formats running on the server as well. So every block is being hit multiple times, and with 16 1 TB drives, that's a LOT of blocks.

Still, if the bad blocks were all clustered on one drive, I'd surely like to replace it now before it goes into service.

By the way, I'm not sure why Windows wouldn't let you update the driver - you did manually select the newer driver, right? What did it complain about?

Just FYI, the newer drivers available for the 5 series do work with the 3 series cards, but you get a lot of spurious power management events in the ASM, so it might not be worth cluttering up your event log to use the newer driver. This might be because the 3 series cards do not support power management whereas the 5 series cards do. Also, I think the 3 series cards do not support JBOD whereas the 5 series cards do. I think the "create JBOD" option won't show up for 3 series cards, and it's certainly not there in the BIOS.

So you can run this 28 port SAS expansion on a 8port SAS card? Or maybe even a 4 port card? And for only around 211 bucks? If so this may be a nice deal for me Unless there is some kind of limitation. I've been considering getting a 1680ix lately.

Quick note for those following our current flirtation with Expanders...

I have an LSI 84016E (16 ports) and the Chenbro CK13601 36 port expander on order (both should be here early next week) to go along w/ a handful of Samsung 1TB F1's (trying to get more in-hand for the initial build/testing).

I'll post more info once it is warranted.

I looked at it last year during some hard core network throughput testing. I got Solaris installed, got the pool set up ok, but as I vaguely recall, I went insane trying to edit files to allow FTP connectivity and add "users" to read the pooled data. At that time it was all command driven to setup and maintain the pools too. In the end I punted, reinstalled Linux & a FTP host, installed Filezilla on the client, tweaked, and called it a day. Maybe Solaris things have changed to make it a little easier, don't know.

Granted I don't do Solaris admin work daily, but I work on a "non-Z" Solaris box daily and do a fair amout of command line work on it. I also have worked with Linux for a long time.

FWIW, Intel Pro LAN cards and fast hard drives like the 640B WD made the biggest bang, over 100MByte/sec transfers were possible to a single drive. Some onboard LAN chips sucked too.

Where'd you order the 36 port variant from? How much did it run you?
Also, what chassis are you planning to use it with - a 4020?
Any reason for the 84016E versus the 8888ELP?
The 8888ELP has 2 external ports on it, so it would be easier to use externally with SAS expanders. Don't tell me you have some huge 7U 48 hotswap monstrosity and they're all going in one internal chassis...

Windows refused to install it. It's like it thought was an unsigned driver. I tried the 5XXX series driver, and it installed fine and seems to work. This was important, because the builtin driver on server 2008 wouldn't allow an array to be created that exceeded 2 TB in size.

I think things are working pretty well now. The formats and builds are almost done (at 90% or so - it's been about 24 hrs since I started the operation), so I'll see how well it performs.

Also, when the initialization is done, I'm going to halt the seagate 7200.11 array, and cycle the disks though a firmware upgrade cycle before they brick on me. :-) See here: http://seagate.custkb.com/seagate/cr...p?DocId=207931

Hm, so you're not getting any power-management messages in your log? Maybe it's because I had a 5 series card in my system and it left some vestiges of its driver or something. Are you using the older version of ASM or the newer version for the 5 series card?

I guess that seagate article is about the "self-brick" bug that all the news sites are reporting about? You know which firmware revisions are affected? From what I gather it's drives from Thailand (pretty much all of mine) and particular firmware revisions...

This, I suppose, is on top of the other stuttering problem with the 1.5TB drives... so I guess there's going to be another firmware update on top of that?

I am using 5.x version of ASM, not the 6.x that is spec'd for the 5XXX cards. No power management messages in the ASM log as far as I can see.

thx
mike

Most probably already know, but the Seagate 7200.11 drives including the enterprise ES series have a serious bug that can cause them to brick themselves during a power cycle (reboot). If you have windows update set to automatic, it might be a good time to turn it off until you can flash the drives. It tends to reboot itself after applying updates.

Seagate bulletin

Affected Models include the 1TB & 1.5TB models

Barracuda 7200.11
ST31000340AS
ST31000640AS
ST3750330AS
ST3750630AS
ST3640330AS
ST3640630AS
ST3500320AS
ST3500620AS
ST3500820AS
ST31500341AS
ST31000333AS
ST3640323AS
ST3640623AS
ST3320613AS
ST3320813AS
ST3160813AS


Barracuda ES.2 SATA
ST31000340NS
ST3750330NS
ST3500320NS
ST3250310NS

DiamondMax 22
STM31000340AS
STM31000640AS
STM3750330AS
STM3750630AS
STM3500320AS
STM3500620AS
STM3500820AS
STM31000334AS
STM3320614AS
STM3160813AS

For all of us that have them running in raid arrays, the drives will need to be pulled from the arrays, connected to a SATA port and flashed with the firmware fix.

Now would be a good time to start backing up your hard earned data
I dunno what people with 20TB of it will do.

Seagate's been having a really miserable time with their drives lately. Their CEO just resigned.
But they seem to be taking a step in the right direction by offering free data recovery for all those who's drives bricked.
That is course, if you want them to be messing with your p0rn

Did I read previously that the 1680ix does not support the SAS expanders?