Guide To Building A Media Storage Server - Page 56

Pretty sure the expander doesn't use the PCI slot for anything. There would be SAS cable that goes to the SAS Raid card.

It doesn't actually use the PCI/PCI-x/PCI-e bus for anything - it's just a means to an end; i.e. being mounted securely.

The card (at least the 36p model) has a molex connector on-board for power, so in either type of config [all-in-one or external storage] you just connect it to the PSU like you would a fan or harddrive.

Tried posting some URLs here, but the forum won't allow me to, so I'll come back with those after I have at least three posts. Guess it would have made sense me remembering the username / password from years back

Sorry for double-post, but it should allow me to post the URLs now

Dunno if the manual for the "card" has been posted earlier (and no, I'm not about to re-read all 56 pages) but it's available here. There are some promotional photos of the card available over at my "buildlog" here (should be the first two snaps in the gallery).

Ok, if it isn't glaringly apparent, I'm kind of feeling my way through this.

Alright, so that drive I had mentioned as being bad was indeed bad [sidenote: if anyone has any experience RMA'ing Samsung drives, please point me to your resource as I can't seem to find out how to get this drive exchanged].

I now have a 4 x 1TB RAID-5 array (approx. 3TB usable space) that has finished its initailization. I'm now doing a Consistency Check on it (about 50% done in 1.5 hours). No warnings in the log file.

Also, to answer some other questions: I have this all in a 4020 case (full system) and the Expander card has a dummy PCIish slot on the bottom of it -- it doesn't electrically plug into a bus slot on the motherboard, you just put it into a free slot for installation convenience. It gets its power from a standard 4-pin molex connection and I have a SFF-8087 to SFF-8087 cable that runs from one of the MegaRAID 84016E's ports into the internal input port on the Expander card. The drives are then connected to the Expander card via SFF-8087 to SATA fan-out cables (each cable has 4 SATA connections).

Once the consistency check finishes, I'll start w/ other tests. I will do a few bandwidth tests and then I'll unplug the drives from the Expander and move them directly to the 84016E card to see if it still sees the array.

Also, looking through the MegaRAID Storage Manager, I don't see where it provides information on the reallocation of bad sectors or any other S.M.A.R.T. info. Anyone know where to look for this info?

I don't have that hardware/software, so can only guess . . .

'Array status' or similar, possibly individual drives are listed within the array?

emigrating, that build is looking nice What cpu's are you going with? I was considering going dual quad core.... way over kill I'm sure but it's crossed my mind none the less. I'm sure that if I end up going hardware raid though I won't need anything better then a low end xeon or core 2 duo.

The Chenbro card looks nice, is there wan SAS input on it? And I guess 8 of the ports are external via the 2 SAS connectors on the back? So I guess in theory at least one 2 port SAS card (2 SAS ports) could support 64 hard drives? If I ever needed more then 20 hard drives it would sure make things cheaper just add another 4020 case and another expander card. Of course I'm not so sure about running 64 drives off one SAS card haha.

Thanks for that. I've just checked my LSI whitepapers and in theory it should support drive roaming no problem, but I really don't feel like going russian roulette with my existing array

Oh, I'd also be interested in how long it actually took you to run that consistency check. I did one a few weeks back (8x1.5TB drives in R6) and whilst it told me it was 50% after a couple hours the counters reset themselves at 100% and in actual fact it took over 24h before it logged "complete".



I went with the Xeon L5420 CPU's (same as E5420 but half the energy consumption). There's also 24GB ram in there as I'm running a bunch of VM's for various tasks - but I may have to reconfigure it to run Linux rather than Windows Server 2008 as I've been playing around with Xen under Linux, and with VT-d support I can actually do PCI Passthrough on a TV Tuner card and run Vista Media Center (or Win7) as a VM taking care of all my television recordings. That's for a slightly different thread though I guess - but it finally looks like I can have a central tunerfarm in true server style.

As for the card, yeah, one of the back connectors is "in" and the other is "out" so you can daisychain these together if you like. I've done some benchmarking on expanders in the past and I haven't really come across any bottlenecks to speak of with some 40-odd drives running of a single 4 port SAS controller. I'm guessing if you have a lot of arrays on that one controller which are being accessed by a lot of users you would see the speed dropping "per request" but for the type of use we're discussing here I really don't think it'll ever be a problem.

Awesome, sounds like this is the way I should go for sure. Now to look for a nice SAS controller for around 300-400 bucks. If such a thing exists lol.

We also need a retailer that stocks or even sells the 36 port model as the 28 port will not fill out a 4020 due to the redundant 4 port inputs and the 4 port output for daisy chaining. That takes 12 ports already leaving you with 16 on the 28 port model and 24 on the 36 port model.

I have always been weary of buying special order parts as well.

Aren't the external ports for input and output?

Either way buy.com has it, unless your not comfortable buying from them?

http://www.buy.com/prod/36-port-sas-...209882580.html

Ok now I see, so the 36 so called port version actually only does 24 HD's. I mean that's cool it's still plenty but a bit misleading Still a great budget option. If you can get an 8 port LSI SAS card or something for 300 that puts this combo at about half of what the Areca 1680ix goes for.

Yes thanks...

I had not set up the winodws backup feature in Windows 2008 at first. I did so monday, but first attempt to do a backup for the OS and VHD holding volume failed with a VSS error -- probably related to the settings you mentioned above. thanks for the link.

I take it the image recovery choice on chosing a repair of the windows 2008 install is referencing the WB images created by the native application

I do use Acronis for server on my windows 2003 server and acronis home on my desktop..

I gues I need to pay the money for new Acronis version that will run on windows 2008. I tried to test install my version 8 copy and nearly trashed my windows 2008 install as it put me in a endless boot loop -- between the issue of the unsigned driver for the supermicro 8 port sata card and the acronis driver causing this reboot loop during windows statup -- I nearly had to reinstall, but finally figure a way into the command prompt and delete and rename drivers until I recovered things and did not in the end have to restart from scratch.. this was 3 hour detour - LOL

--

additional question -- have you tried virtual disk mapped to added scsi virtual controllers versus the whole physical disk mapping ( I beleive these are called pass-thru disks) in WHS VM for putting more drives into the storage pool??

mapping the physical disk to the virtual controller basically shuts off the snaphsot stuff in the Hyper-V manager

Is there any infomration on number of drive on a virtual controller and using more than one virtual scsi controll and balancing drives

any details on best performance practices on this type things..

thanks again

I've been using VMware workstation for a while on another machine at home and their are many mature features in VM management in that product...

I enjoy learning in this area....

Is this the registry patch you pointed too??

"If you're using Windows Server Backups, a registry key must be set on the Hyper-V host to enable VSS support. In the location HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows NT\\CurrentVersion\\WindowsServerBackup\\Application Support\\{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}, set the REG_SZ value for Application Identifier to Hyper-V."

windows backups completed after I enabled shadow copies on the drive volumes, but I have not yet applied the registry edit??????????????

So the main problem I see with the cards that are in the ~300 dollar range is they don't have a lot of cache. Is 128mb of cache adequate? This is one that I found

http://www.ewiz.com/detail.php?p=LS-...2a812a61183aa8

Very misleading but a good marketing ploy. I went into detail about this issue several pages back.

What really bugged me is that the expander has a 4 port external input and a 4 port internal input accounting for 8 of the cards ports while only being able to use one of the inputs at a time. hurray for marketing.

This is one of those things that I would get pissed about if I got the 28 port card and then could only use 16 out of the 20 drives the case can support.

Yeah I agree, I was looking more and I think this card is probably the better buy.

http://www.provantage.com/lsi-logic-...7~7LSIG074.htm

If you need the 8 port it's like 150 more though. But this one is actually an x8 card so a bit more bandwidth available especially if your running the bigger arrays. As long as there is no draw back from 128mb of cache which is a good bit more then a lot of lower end cards I've seen.

I'm wondering with these 500mhz PPC cpu's on these how long would it take to expand an array with a new drive?

Cache only helps random writes. For sustained, sequential writes it is the RAID processor and disk speeds.

You might look at one of the LSI 16-port cards that supports port expanders (I linked one a few pages earlier). That would get you a long ways down your storage road before even needing to consider buying an expander. I think they were in the $600 range.

I'm not sure of your storage needs, but you might find it takes years (or never) to fill a 16-drive array.

Yeah I suppose that is a possibility. I'll have to look into it more thanks.

Why on LSI's website do they not have anything over an 8port SAS card?

Unplugged the drives from the Expander and plugged them directly into the 84016E -- it complained upon boot that something was amiss. I let it boot into Windows and the volume was accessible in Explorer. I deleted a file from it w/ no problem. In the MegaRAID app, it shows the array is degraded, etc, etc., but I stopped playing with it there.

Before doing that tho, I ran some benchmarks...

CrystalDiskMark 2.2 tests -- 4x1TB Sansung F1 drives in RAID-5 array, all numbers are in MB/s

5 tests @ 50MB
Seq: Read: 724.5 / Write: 157.2
512K: Read: 686.2 / Write: 117.6
4K: Read: 65.32/ Write: 5.969

5 tests @ 100MB
Seq: Read: 738.8 / Write: 158.7
512K: Read: 687.6 / Write: 107.1
4K: Read: 65.60 / Write: 4.441

5 tests @ 500MB
Seq: Read: 321.4 / Write: 58.37
512K: Read: 57.34 / Write: 21.80
4K: Read: 0.842 / Write: 0.322

5 tests @ 1000MB
Seq: Read: 303.6 / Write: 55.87
512K: Read: 40.79 / Write: 17.79
4K: Read: 0.0632 / Write: 0.300


HD Tun Pre v3.50 Tests:

Benchmark tab, Read test:
Transfer rate Minimum: 226.7 MB/s
Transfer rate Maximum: 338.3 MB/s
Average: 285.3 MB/s

Access time: 11.0 ms
Burst rate: 315.2 MB/s
CPU usage: 2.0%

Those benches seem to confirm my speculation about the poor write performance of the LSI cards. In my experience crystaldiskmark isn't very good and cache memory on raid cards tends to throw it off quite a lot. But, even looking at the 1GB sustained test, you can see that it only manages about 55MB/sec write speed, on FOUR drives. That's even worse than a single drive write speed, and there's no SAS expander in the equation either.

Compared to that, my Adaptec and Areca cards can write to RAID5 and RAID6 at roughly the same speed as it can read. The Adaptec and Areca are pretty similar in terms of performance, and I think the effect of additional cache memory is negligible at best - your main bottleneck will be the access times of your drives, and how many IOPs you can squeeze out of them.

Also, just swapping drive order doesn't trigger drive failure and rebuild, which seems to be the LSI behavior. I believe the LSI is set the most conservative setting, where even moving drives around marks the drives or array as "bad"/degraded. The Adaptec card notifies you that drives changed positions, but chugs ahead and works fine. I haven't tested it with the Areca card, but I believe it doesn't care about drive order either. You do have to keep tabs of drive order if you need to rescue the raid in case the raid metadata gets corrupted, however.

I finally managed to solve my DPC problems with the Adaptec, I believe, once and for all. The solution seems to be simply, just fully connect each multilane cable. If only a few drives are connected to a multilane cable, the card seems to scan the other ports for drives periodically, which causes the DPC spikes. It's kind of dumb, but I suppose they assume you'll fully populate the port capacity of the card.

Also, if anyone is looking for a cheap 16 port SAS card, I have a spare 31605 kicking around with a BBU attached - PM me if interested. I'm planning to do a little testing of the 31605 with the SAS expander beforehand, though.

Just to clarify, for those benchmarks the drives were attached to the Expander which was in turn attached to the 84016E.

Also, currently doing a network transfer test (~500GB of data) and am seeing a sustained transfer rate of ~33% to 50% network utilization on a 1 Gbps link. This test is mainly to see how the array handles a sustained transfer.... seems to be doing well so far (only a few PD errors in the log).

LSI Logic announced 6GB/s SAS2 raid-on-chip & HBA for OEMs (Supermicro etc). Looks like actual availability for regular folks is the end of the year.

The press release just came out yesterday:
http://www.lsi.com/news/product_news...09_01_20a.html

They also have a whitepaper that's been there for a while introducing SAS2:
http://www.lsi.com/documentation/sto...y_Overview.pdf

Of interest are the last 2 slides comparing their current SAS megaraid with the new SAS2 megaraid. They don't put any hard numbers on the graphs, but the write graph looks like it almost doubles for sequential writes. That would, at least, have the LSI cards catch up to the current Areca and Adaptec offerings, and the LSI cards are known for their compatibility, so they might be better than going with the finicky intel IOP348 based SAS solutions from other vendors. I wonder if there's going to be also a SAS2 based intel IOP, hopefully with better SATA compatibility...

Plasticquart,

I think LSI claims better performance than you are experiencing, although still slightly below Intel IOP performance. Not 50 MB/sec, tho. Did you say the SAS expander was in between the drives and the RAID card?

Perhaps the RAID is operating in degraded mode, and that explains the poor performance. Really, 55MB/sec is really quite piss poor, even my highpoint fakeraid can manage faster than that.

Yep that's why it failed. Once you understand that taking a backup image is like trying to take a photograph in a specific moment in time of many objects all moving at hyper speed. Once you enable VSS, it allows the system to prepare that moment in time to be photographed without interupting the all the moving ojects.


Yes, native WB in server 2008 is quite advanced over NT backup, in that it now takes image based backups over NT's file level backups. However Acronis is still further advanced in the sense of granular backup and recovery which WB lacks.



I believe you can download a 30 day trial of Echo Server for Windows to see how you like it. You should also be able to find it floating around the net (for testing purposes).



If you mean creating an additional VHD and attaching it to the VM, yes I have. It works fine. In this case I suppose you'd put the drive online with the 2008 host. Then in hyper V create a blank VHD the size of the entire physical drive and attach it to the VM. IMHO, it's not worth it tho. Snapshots would be unmanagebly huge.

Besides one of the greatest benefits to WHS is that each of the drives in the storage pool are readable independantly in any system. You'd lose that if you turned the drive into a virtual one.


Of course, you cannot snapshot something that's partly virtual and partly not. But I wonder if you really need VM snapshots, when you can take an incremental image using VSS. Just restore the VM and go.


Not sure if this is what you mean, but a single virtual scsi controller allows you to add 64 drives (ports 0 - 63). I would think this is plenty.


I use VMware workstation on my notebook as well for other VMs and to try some prebuilt VMware appliances from time to time. We have also remember that Hyper-V is in v1, while VMware has been at it a long while. But we can't compare VMware workstation (application layer) to Hyper-V (kernel layer).
A more even comparison is VMware ESX to Hyper-V, and Hyper-V sitting on server 2008 is considerably easier to manage as a host.


Yes that's correct. You can easily check if Hyper-V's VSS writer is properly enabled. Just open a command prompt and type vssadmin list writers

It might take a few seconds, but Hyper-V VSS should show on the list.



Yes WB will still run once you've enable shadow copies on the host and take an image, but the image will be based on what was on the host and the VM wasn't imaged in a safe state. Without the reg entry, WB has no idea how to communicate with Hyper-V's VSS writer, which would in turn communicate with the VM to prepare itself to be imaged/backed up to provide safe image.

When both VSS is enabled on the host drive volume and the WB VSS reg entry, after a backup you can actually RDP into the VM, and check it's event viewer where you'll find that the VSS service started inside it, prepared a state to be backed up, then resumed all seamlessly. Without this reg entry, the VM was never aware it needed to prepare to be backed up.

Part of me wants to say just screw it and go with the Areca 1680ix lol.

Yes, those benchmarks are with the drives attached to the expander. They were not directly attached to the LSI card. And the array was not degraded when running the test.


Just an fyi, but SATA drives aren't "supported" on the 16xx Areca cards. Their 12xx series is for SATA drives... which leads me to where I'm torn...

No this LSI 84016E/Expander solution didn't tear apart that benchmark, but then again that was one set of numbers from a simple application testing an array setup by someone who is literally fumbling their way through all of it. (Not even sure this meager 4-drive array is optimally configured.) So, do I spend $1200 on an Areca 1280-ML, which can handle a full compliment of drives in a 4020 case but is a dead end when it comes to expansion beyond that one box, or do I stick w/ a lower-performing solution that easily will grow into at least 3 additional 4020 cases simply by purchasing an ~$300 card....??

Given my specific needs (a rather obscene amount of data to store), I absolutely do love the fact that this 84016E/Expander solution is working. Given the complete lack of information out there, I did have my doubts. But now I have no doubt now that I can get 3 more Expander cards and put them in separate chassis, all hung off of this one 84016E card with performance that is workable.

Hopefully continued tuning with bring about some performance increases.

Network transfer benchmark between my two servers.
This is the media server backing up 48.7GB of data to the Hyper-V VM server in a little under 8 minutes





Network was pretty much pinned at 100% except the small blips that indicate the start and stop of the files transfered. It's all working extremely well

What are you talking about it doesn't support SATA drives? Lots of people run SATA drives with it. Including the guy that did the 45TB media server.

For Chrissake, how many different RAIDs do you have? I thought one was bad enough!