Mfusick's How to build an affordable 30TB Flexraid media server: Information Requested.! - Page 142 - AVS Forum | Home Theater Discussions And Reviews
Forum Jump: 
 144Likes
Reply
 
Thread Tools
post #4231 of 4731 Old 02-23-2015, 07:32 AM
 
ajhieb's Avatar
 
Join Date: Jul 2009
Location: The Commonwealth, not the Jelly.
Posts: 2,696
Mentioned: 15 Post(s)
Tagged: 0 Thread(s)
Quoted: 1037 Post(s)
Liked: 698
Quote:
Originally Posted by EricBurns View Post
So is flashing these the same process as the M1015 and Dell?
If by "these" you mean the Intel res2sv240 mentioned above, no. They aren't SAS HBAs they are SAS expanders.
Mfusick likes this.
ajhieb is offline  
Sponsored Links
Advertisement
 
post #4232 of 4731 Old 02-23-2015, 07:55 AM - Thread Starter
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 29,681
Mentioned: 23 Post(s)
Tagged: 0 Thread(s)
Quoted: 608 Post(s)
Liked: 2729
They give you more SAS or SATA ports off a single card.
Mfusick is offline  
post #4233 of 4731 Old 02-23-2015, 08:11 AM
Member
 
Join Date: Feb 2015
Location: Roc, NY
Posts: 48
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 34 Post(s)
Liked: 4
So it has to be used in conjunction with Dell or m1015? or as a standalone and just connect the hdd to it directly?

Sorry for all my questions.
EricBurns is offline  
Sponsored Links
Advertisement
 
post #4234 of 4731 Old 02-23-2015, 08:28 AM
Senior Member
 
NNate's Avatar
 
Join Date: Nov 2006
Location: Minneapolis, MN
Posts: 358
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 57 Post(s)
Liked: 29
Quote:
Originally Posted by EricBurns View Post
So it has to be used in conjunction with Dell or m1015? or as a standalone and just connect the hdd to it directly?

Sorry for all my questions.
In conjunction with
NNate is offline  
post #4235 of 4731 Old 02-23-2015, 08:31 AM
Member
 
Join Date: Feb 2015
Location: Roc, NY
Posts: 48
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 34 Post(s)
Liked: 4
Thanks NNate. I was just going to post that I answered my own question with a little googling.
EricBurns is offline  
post #4236 of 4731 Old 02-23-2015, 08:47 AM - Thread Starter
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 29,681
Mentioned: 23 Post(s)
Tagged: 0 Thread(s)
Quoted: 608 Post(s)
Liked: 2729
Quote:
Originally Posted by EricBurns View Post
So it has to be used in conjunction with Dell or m1015? or as a standalone and just connect the hdd to it directly?

Sorry for all my questions.
If you go back a bit in my thread you will see I went through this too.

Cliff notes:

I started with one IBM 1015 (I flashed it, made a guide based on another guide I found)
Soon need more- filled up those 8 sata ports (2 SAS ports on IBM) with HDD, and populated my motherboard ports too.
Added a second card. Ended up bricking an IBM card. I removed it or messed with with the PC on. (anyone want it? Free. Did not test it much but I think it's dead.)
So after bricking the IBM by being a bonehead- I just got a DELL flashed from Andy who hooked me up and because I was in a rush.

I like the DELL card and the IBM card equally. Once flashed both are basically the same thing.

I added a third card- when I need more. Again. (imagine that).

I then used a SAS expander from the third card to go from 8 drives to 16 drives.

So basically what the SAS expander does is allow you to run more hard drives off a single card. Example you have a DELL/IBM card and you typically get 8 HDD from that (2 SAS ports). If you run those two SAS ports (or even one of them) into a SAS expander you can get more than 8HDD from the same card.

I found that as long as I had available x8 PCI lanes and slots on my mobo it was cheaper and easier to just add a second card. But once I ran out- the SAS expander was a good option. Depending on how many drives and your hardware will determine if you are better with a single card, two cards, or a combo with expander. Generally if you want more than 20 hard drives you probably are looking at an expander, and under that it's not necessary but still can be used.
Mfusick is offline  
post #4237 of 4731 Old 02-23-2015, 08:58 AM
Member
 
Join Date: Feb 2015
Location: Roc, NY
Posts: 48
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 34 Post(s)
Liked: 4
Quote:
Originally Posted by Mfusick View Post
If you go back a bit in my thread you will see I went through this too.

Cliff notes:

I started with one IBM 1015 (I flashed it, made a guide based on another guide I found)
Soon need more- filled up those 8 sata ports (2 SAS ports on IBM) with HDD, and populated my motherboard ports too.
Added a second card. Ended up bricking an IBM card. I removed it or messed with with the PC on. (anyone want it? Free. Did not test it much but I think it's dead.)
So after bricking the IBM by being a bonehead- I just got a DELL flashed from Andy who hooked me up and because I was in a rush.

I like the DELL card and the IBM card equally. Once flashed both are basically the same thing.

I added a third card- when I need more. Again. (imagine that).

I then used a SAS expander from the third card to go from 8 drives to 16 drives.

So basically what the SAS expander does is allow you to run more hard drives off a single card. Example you have a DELL/IBM card and you typically get 8 HDD from that (2 SAS ports). If you run those two SAS ports (or even one of them) into a SAS expander you can get more than 8HDD from the same card.

I found that as long as I had available x8 PCI lanes and slots on my mobo it was cheaper and easier to just add a second card. But once I ran out- the SAS expander was a good option. Depending on how many drives and your hardware will determine if you are better with a single card, two cards, or a combo with expander. Generally if you want more than 20 hard drives you probably are looking at an expander, and under that it's not necessary but still can be used.
Awesome, now I fully understand. I don't think I'll be needing an expander then as I won't be running that many drives (I say this now). I just need to flash the Dell that I've got and then I'll be where I want to be.

I'll post some pictures of what I've got once my new mobo gets here. I decided on keeping everything the same and just replacing that, for now.
EricBurns is offline  
post #4238 of 4731 Old 02-23-2015, 09:20 AM - Thread Starter
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 29,681
Mentioned: 23 Post(s)
Tagged: 0 Thread(s)
Quoted: 608 Post(s)
Liked: 2729
I like pictures!

And yeah you can always upgrade or add an expander when you need it.
Mfusick is offline  
post #4239 of 4731 Old 02-24-2015, 10:39 AM
AVS Forum Special Member
 
DotJun's Avatar
 
Join Date: Dec 2012
Posts: 1,681
Mentioned: 10 Post(s)
Tagged: 0 Thread(s)
Quoted: 598 Post(s)
Liked: 163
Quote:
Originally Posted by Mrkazador View Post
Someone on ebay is selling the res2sv240 real cheap, offered $55 and he countered with $60! He has about 80 available so get them while you can. Even though I don't need one right now, its too good of a deal to pass up.







http://www.reddit.com/r/homelab/comm...xpanders_6075/

Wish I would have seen this before I bought the 36 port chenbro!
DotJun is offline  
post #4240 of 4731 Old 02-26-2015, 05:55 AM
AVS Forum Special Member
 
Axel's Avatar
 
Join Date: Mar 2002
Location: Metro Detroit, MI, USA
Posts: 5,044
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 219 Post(s)
Liked: 63
Quote:
Originally Posted by Mrkazador View Post
Someone on ebay is selling the res2sv240 real cheap, offered $55 and he countered with $60! He has about 80 available so get them while you can. Even though I don't need one right now, its too good of a deal to pass up.



http://www.reddit.com/r/homelab/comm...xpanders_6075/
Mine arrived yesterday. I expected the bare bone card, but to my surprise it came with full height and low profile bracket, as well as 6 SAS cables. Wow!



Now I want to set it up:

I currently run a dedicated I3 with Win8.1 Pro 64 with FlexRAID (RAID-F) and currently 10 data + 2 parity drives (all SATA II).
The drives are connected to (2) M1015 flashed to a LSI9211-IT mode.
The OS SSD and the BD ROM drives are connected directly to SATA ports on the mobo.


What is the best way to integrate the expander in my set up, regarding a) system stability or b) performance/speed during parity calculation (verify, validate, etc.)?

Scenario 1:
Leave everything as is and once (if) I run out of ports, add drives to the expander.

Scenario 2:
Connect the expander to one of the M1015 and move all FlexRAID drives to the expander. I could remove the second M1015 for the time being.

Scenario 3:
Like 1 or 2, but also connect the OS and BD ROM drives to either M1015 or expander

I am looking forward to your comments.
____
Axel

Axel is offline  
post #4241 of 4731 Old 02-26-2015, 07:31 AM
Member
 
Quaraxkad's Avatar
 
Join Date: May 2004
Location: Port Orange, FL
Posts: 125
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 63 Post(s)
Liked: 52
Quote:
Originally Posted by Axel View Post
Scenario 1:
Leave everything as is and once (if) I run out of ports, add drives to the expander.

Scenario 2:
Connect the expander to one of the M1015 and move all FlexRAID drives to the expander. I could remove the second M1015 for the time being.

Scenario 3:
Like 1 or 2, but also connect the OS and BD ROM drives to either M1015 or expander

1. You still have 4 free ports. Once they're full, add the expander and move the drives from one of the M1015's to it.

Option 2 will be too slow for FlexRAID.
Quaraxkad is offline  
post #4242 of 4731 Old 02-26-2015, 07:39 AM
AVS Forum Special Member
 
Axel's Avatar
 
Join Date: Mar 2002
Location: Metro Detroit, MI, USA
Posts: 5,044
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 219 Post(s)
Liked: 63
Quote:
Originally Posted by Quaraxkad View Post
1. You still have 4 free ports. Once they're full, add the expander and move the drives from one of the M1015's to it.

Option 2 will be too slow for FlexRAID.
Interesting!
I had thought that during parity calculations I would see a performance increase with Option 2, as all drives are directly connected to only a single controller (the expander controller), rather than to multiple controllers and addl. data bus transfers.
____
Axel

Axel is offline  
post #4243 of 4731 Old 02-26-2015, 07:54 AM
Member
 
Quaraxkad's Avatar
 
Join Date: May 2004
Location: Port Orange, FL
Posts: 125
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 63 Post(s)
Liked: 52
Quote:
Originally Posted by Axel View Post
Interesting!
I had thought that during parity calculations I would see a performance increase with Option 2, as all drives are directly connected to only a single controller (the expander controller), rather than to multiple controllers and addl. data bus transfers.
____
Axel
That controller and/or expander can't handle that much simultaneous drive access.
Quaraxkad is offline  
post #4244 of 4731 Old 02-26-2015, 08:10 AM
AVS Forum Special Member
 
Axel's Avatar
 
Join Date: Mar 2002
Location: Metro Detroit, MI, USA
Posts: 5,044
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 219 Post(s)
Liked: 63
Quote:
Originally Posted by Quaraxkad View Post
That controller and/or expander can't handle that much simultaneous drive access.
Thanks for the explanation.
____
Axel

Axel is offline  
post #4245 of 4731 Old 02-26-2015, 11:39 AM
AVS Forum Special Member
 
Mrkazador's Avatar
 
Join Date: Jun 2005
Posts: 4,352
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 308 Post(s)
Liked: 428
Quote:
Originally Posted by Axel View Post
Mine arrived yesterday. I expected the bare bone card, but to my surprise it came with full height and low profile bracket, as well as 6 SAS cables. Wow!
Really? I just bought a sas cable for mine, should be here tomorrow

Quote:
Originally Posted by Quaraxkad View Post
That controller and/or expander can't handle that much simultaneous drive access.
I thought the exapander could do 24 Gbps (total) if using one uplink from the m1015?

_______________________________

DIY Sound Group: Information and Tips
Mrkazador is offline  
post #4246 of 4731 Old 02-26-2015, 11:44 AM
AVS Forum Special Member
 
Axel's Avatar
 
Join Date: Mar 2002
Location: Metro Detroit, MI, USA
Posts: 5,044
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 219 Post(s)
Liked: 63
Quote:
Originally Posted by Mrkazador View Post
Really? I just bought a sas cable for mine, should be here tomorrow
yep, really.

I should be getting mine I bought separately also either today or tomorrow.

______
Axel

Axel is offline  
post #4247 of 4731 Old 02-26-2015, 01:26 PM
Member
 
Quaraxkad's Avatar
 
Join Date: May 2004
Location: Port Orange, FL
Posts: 125
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 63 Post(s)
Liked: 52
Quote:
Originally Posted by Mrkazador View Post
I thought the exapander could do 24 Gbps (total) if using one uplink from the m1015?
In theory. And it may do that up to a certain point, but the more drives it's reading from simultaneously the slower it will perform per-drive. You can test it by running a benchmark on just one drive, then two drives simultaneously, then three, etc. And see how many drives it can handle at full speed before it starts to slow down each drive.
Quaraxkad is offline  
post #4248 of 4731 Old 02-26-2015, 05:08 PM
 
ajhieb's Avatar
 
Join Date: Jul 2009
Location: The Commonwealth, not the Jelly.
Posts: 2,696
Mentioned: 15 Post(s)
Tagged: 0 Thread(s)
Quoted: 1037 Post(s)
Liked: 698
Quote:
Originally Posted by Quaraxkad View Post
That controller and/or expander can't handle that much simultaneous drive access.
That's kinda vague.

Yes, you can run into bottlenecks using an expander, but unless you have insanely fast drives, (as in: if your drive has platters it isn't insanely fast) you aren't likely to run into a bottleneck with 12 drives running at 6Gb/s.

And since the Intel controller allows for 2 uplinks, you could set it up in a 2x4 configuration which would make any bottlenecks or performance impact practically nil.

Quote:
In theory. And it may do that up to a certain point, but the more drives it's reading from simultaneously the slower it will perform per-drive. You can test it by running a benchmark on just one drive, then two drives simultaneously, then three, etc. And see how many drives it can handle at full speed before it starts to slow down each drive.
Well, actually your statement is "in theory." In practice spinning platters don't transfer data fast enough to saturate most SAS buses, and since the expander does bandwidth aggregation, you can add a lot of drives before you start to see any significant impact on performance.

Seagete 4TB drive for example has an average transfer rate of 146MB/s or 1168Mb/s. With 24Gb/s bandwidth available (in a 1x5 configuration) you could add 20 drives without seeing any noticeable performance impact (which as it turns out is the number of drives you could connect to the expander in that configuration)

If you had 40 of those drives and 2 of the intel expanders connected to a single m1015 (absolutely maxing out everything) then the m1015 would be the bottleneck as the PCIe 2.0 x8 connection is going to be 32Gb/s and you would have the potential to transfer 48Gb/s if reading from all 40 drives simultaneously.

If you were using a 3Gb/s expander then you would encounter a bottleneck with fewer drives.

You're right that you can eventually see a performance drop, but it isn't going to be in more hypothetical scenarios than in real life uses.

Theoretically in the 20 drive scenario listed above which is pushing the limits of what a setup like that can do, you would see some write penalties as the drive cache would be limited by the SAS bandwidth, but in a FlexRAID setup I don't know why you'd be writing to all (or nearly all) of the drives at one. Yeah, benchmarks might suffer, but real world performance should be fine.
ajhieb is offline  
post #4249 of 4731 Old 02-26-2015, 05:33 PM
Member
 
Quaraxkad's Avatar
 
Join Date: May 2004
Location: Port Orange, FL
Posts: 125
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 63 Post(s)
Liked: 52
Quote:
Originally Posted by ajhieb View Post
Theoretically in the 20 drive scenario listed above which is pushing the limits of what a setup like that can do, you would see some write penalties as the drive cache would be limited by the SAS bandwidth, but in a FlexRAID setup I don't know why you'd be writing to all (or nearly all) of the drives at one. Yeah, benchmarks might suffer, but real world performance should be fine.
FlexRAID processes essentially *are* benchmark scenarios. Real world scenarios involve accessing one drive at a time... Maybe two...

You're looking at the theoretical SAS and PCI-Express bandwidth and assuming that's the only thing that matters. You can't just add up the individual theoretical max bandwidth and multiply by the number of ports and assume that's what you're going to get. If you're so certain, go ahead and test it... I have tested for my own purposes, and I plan on doing a more thorough test to find exactly what the maximum drive limit is before performance suffers. I can't say for sure whether the issue is the controller or the expander, but I promise you the combination is absolutely not fast enough for a full array of drives simultaneously running at full speed reads with no slow-downs.

Again, we're talking about ALL drives being read SIMULTANEOUSLY, as this is what snapshot RAID requires. Obviously there is no mechanical HDD on the planet that would be bottlenecked by itself by any of these SAS controllers or expanders.

Last edited by Quaraxkad; 02-26-2015 at 05:37 PM.
Quaraxkad is offline  
post #4250 of 4731 Old 02-26-2015, 05:46 PM
 
ajhieb's Avatar
 
Join Date: Jul 2009
Location: The Commonwealth, not the Jelly.
Posts: 2,696
Mentioned: 15 Post(s)
Tagged: 0 Thread(s)
Quoted: 1037 Post(s)
Liked: 698
Quote:
Originally Posted by Quaraxkad View Post
FlexRAID processes essentially *are* benchmark scenarios. Real world scenarios involve accessing one drive at a time... Maybe two...

You're looking at the theoretical SAS and PCI-Express bandwidth and assuming that's the only thing that matters. You can't just add up the individual theoretical max bandwidth and multiply by the number of ports and assume that's what you're going to get. If you're so certain, go ahead and test it... I have tested for my own purposes, and I plan on doing a more thorough test to find exactly what the maximum drive limit is before performance suffers. I can't say for sure whether the issue is the controller or the expander, but I promise you the combination is absolutely not fast enough for a full array of drives simultaneously running at full speed reads with no slow-downs.

Again, we're talking about ALL drives being read SIMULTANEOUSLY, as this is what snapraid RAID requires. Obviously there is no mechanical HDD on the planet that would be bottlenecked by any of these SAS controllers or expanders.
Depends on what benchmarks you're talking about. FlexRAID everyday usage, is going to be just like accessing a single drive. FlexRAID is essentially transparent when reading and writing to drives. The only time FlexRAID is a factor is when doing parity related activity. When that happens you are performing a bunch of sustained, simultaneous reads. In that situation drive cache becomes practically irrelevant. It's a matter of sustained transfer rates. And yes there is some protocol overhead involved with SAS, and there is additional overhead when you use SATA drives on a SAS bus, but neither is very significant.

That said, I'm not arguing that you'll never hit a bottleneck. It's absolutely possible and I even outlined how.

However you asserted that an M1015 controller with an expander can't handle 12 drives, and you did so without any factual basis. If you have relevant benchmarks for such a scenario feel free to share, but based on my experience (and the experience of others) you aren't even approaching the limits of that setup with 12 drives.
ajhieb is offline  
post #4251 of 4731 Old 02-26-2015, 06:34 PM
Member
 
Quaraxkad's Avatar
 
Join Date: May 2004
Location: Port Orange, FL
Posts: 125
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 63 Post(s)
Liked: 52
Quote:
Originally Posted by ajhieb View Post
FlexRAID everyday usage, is going to be just like accessing a single drive. FlexRAID is essentially transparent when reading and writing to drives. The only time FlexRAID is a factor is when doing parity related activity.
Yes, I'm well aware of that. That's why I specified that I'm talking about FlexRAID processes/tasks such as update and verify. Aside from that, you could put 100 drives on one of these controllers and it'd be fine...

Quote:
Originally Posted by ajhieb View Post
However you asserted that an M1015 controller with an expander can't handle 12 drives, and you did so without any factual basis. If you have relevant benchmarks for such a scenario feel free to share, but based on my experience (and the experience of others) you aren't even approaching the limits of that setup with 12 drives.
He's talking about expanding beyond 12. As I said, I don't know exactly where the cutoff is, but the more drives you have the closer you get to slowing things down to a crawl. Potentially up to 24 on a single card and expander. I speak from experience (and I'm certain everyone else who's done it can back it up if you're so adamant about it) that this will have a significant and unavoidable impact on FlexRAID tasks performance.

Last edited by Quaraxkad; 02-26-2015 at 06:41 PM.
Quaraxkad is offline  
post #4252 of 4731 Old 02-26-2015, 07:16 PM
 
ajhieb's Avatar
 
Join Date: Jul 2009
Location: The Commonwealth, not the Jelly.
Posts: 2,696
Mentioned: 15 Post(s)
Tagged: 0 Thread(s)
Quoted: 1037 Post(s)
Liked: 698
Quote:
Originally Posted by Quaraxkad View Post
He's talking about expanding beyond 12. As I said, I don't know exactly where the cutoff is, but the more drives you have the closer you get to slowing things down to a crawl. Potentially up to 24 on a single card and expander. I speak from experience (and I'm certain everyone else who's done it can back it up if you're so adamant about it) that this will have a significant and unavoidable impact on FlexRAID tasks performance.
I'm assuming he bought the expander so he could use more drives than what the controller natively supports. That's the point of an expander. Suggesting that he not use it because it might cause problems at some point because of some limitation you've allegedly encountered at some point isn't very helpful.

His current setup (single M1015 and 6GB SAS Expander) has plenty of room for expansion before he sees any significant slowdown. He could expand up to 16 drives on a single M1015 and not even begin to approach a possibility of a chance that he might start to hit performance bottlenecks. (in 2x4 configuration) Suggesting otherwise is flat out wrong.

As stated repeatedly, yes the potential for a bottleneck does exist, no one is denying that. Suggesting the possibility is fine. Flat out stating that it will happen is wrong. Suggesting that his setup can't handle "that much" (yet unspecified) drive access is wrong. That controller and that expander absolutely will handle that much drive activity. He could end up limited in any number of places on his system, but the proposed storage system (single M1015 and and expander configured as 2x4) is more than adequate to handle his current needs (12 drives) future expansion (4 more drives) additional expansion (adding his 2nd M1015 for 8 more drives) and if he wanted to get a 2nd expander that would allow him to add 8 additional drives, without ever having anything close to a performance bottleneck. That's a total of 24 drives with his existing hardware, and 32 if he wanted to get a 2nd expander. If he does start running into bottlenecks at that point, they are going to be related to his motherboard's ability to push data back and forth, his CPU's ability to calculate parity, or some other external influence that has nothing to do with the controller or expander.

That configuration allows for at minimum 66% overhead at every single point.

It's also possible that if his motherboard is running 1 of his M1015 cards at less than x8 that moving everything to one card running at full speed could increase his performance. (which is unlikely, but still more probable than any situation Alex would run into where he has too many drives plugged into his expander for it to work effectively)
ajhieb is offline  
post #4253 of 4731 Old 02-26-2015, 09:03 PM
Member
 
Quaraxkad's Avatar
 
Join Date: May 2004
Location: Port Orange, FL
Posts: 125
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 63 Post(s)
Liked: 52
I never suggested that he not use either the controller or the expander. I merely told him that he will not see increased performance by simply adding the expander, and if you argue against that then you're just a lunatic.

This is not the first time you have argued with me over what is largely and clearly semantics. You say yourself that you are not denying that a potential bottleneck exists. You take one motherboard, one SAS card, one expander, put 24 drives on it (or even drop to 20 with two SAS connections between the two), and then tell me that every drive is going to perform at its 100% peak, solely because the individual theoretical max throughput of a single port is fast enough for each individual mechanical drive, obviously you have not tested this theory.

I'm telling you with absolute certainty that the expander and/or controller and/or motherboard (it makes absolutely no difference which one) simply and absolutely can not handle a scenario where every port is occupied and all drives are simultaneously being read without *severe* bottlenecks. Frankly I don't care if you believe that or not, clearly you think you've got this all down and have all the facts. But you go ahead and keep telling people that they can get 100% performance from their mechanical drives during snapshot operations with a full expander, you're going to have some unhappy people when they realize how wrong that was. It doesn't matter how many times you tell me I'm "flat out wrong", it doesn't mean that these devices can suddenly perform miracles. I want you to provide some evidence to back up your claim that any of these involved components can handle the full load scenario required by snapshot RAID duties. It will still "work" no matter how many drives you hook up, and if using SnapRAID then you will quite possibly see a higher total MBps reading for each and every drive that you add (even once you're beyond the "max-drives" threshold). In FlexRAID the MBps reading will decrease as it essentially measures the weakest link whereas SnapRAID reports the total combined throughput. However, it's undeniable that the performance of each individual drive will be degraded once you pass a certain count and the hardware can no longer keep up. And that limited will be reached before the expander is full. Deny it all you want because you have the theoretical numbers to back it up, it's a fact. Test it yourself. Show the results. Prove me wrong.

EDIT: Once again, because I know you're going to completely miss the point and bring it up again... This is referring *only* to operations where all drives are being accessed simultaneously (as in those required by snapshot parity systems). Individual single-drive read/write performance will be essentially untouched no matter how many drives are connected.
Mfusick and htpcforever like this.

Last edited by Quaraxkad; 02-26-2015 at 09:07 PM.
Quaraxkad is offline  
post #4254 of 4731 Old 02-26-2015, 10:44 PM
 
ajhieb's Avatar
 
Join Date: Jul 2009
Location: The Commonwealth, not the Jelly.
Posts: 2,696
Mentioned: 15 Post(s)
Tagged: 0 Thread(s)
Quoted: 1037 Post(s)
Liked: 698
Quote:
Originally Posted by Quaraxkad View Post
I never suggested that he not use either the controller or the expander. I merely told him that he will not see increased performance by simply adding the expander, and if you argue against that then you're just a lunatic.
"Option 2 will be too slow for FlexRAID."

Yeah, I guess you didn't say that thing that you said. Whatever man.

And in the unlikely event one of his controllers is operating at x4 or x1 (some motherboards have PCIe slots that are physically x8 or x16 slots, but are only wired for x1 or x4) then moving all of his drives to a single controller operation at full speed certainly could increase his performance. (note: When I present an outlier case, I distinguish it as such)

Quote:
This is not the first time you have argued with me over what is largely and clearly semantics. You say yourself that you are not denying that a potential bottleneck exists. You take one motherboard, one SAS card, one expander, put 24 drives on it (or even drop to 20 with two SAS connections between the two), and then tell me that every drive is going to perform at its 100% peak, solely because the individual theoretical max throughput of a single port is fast enough for each individual mechanical drive, obviously you have not tested this theory.

I'm telling you with absolute certainty that the expander and/or controller and/or motherboard (it makes absolutely no difference which one) simply and absolutely can not handle a scenario where every port is occupied and all drives are simultaneously being read without *severe* bottlenecks. Frankly I don't care if you believe that or not, clearly you think you've got this all down and have all the facts. But you go ahead and keep telling people that they can get 100% performance from their mechanical drives during snapshot operations with a full expander, you're going to have some unhappy people when they realize how wrong that was. It doesn't matter how many times you tell me I'm "flat out wrong", it doesn't mean that these devices can suddenly perform miracles. I want you to provide some evidence to back up your claim that any of these involved components can handle the full load scenario required by snapshot RAID duties. It will still "work" no matter how many drives you hook up, and if using SnapRAID then you will quite possibly see a higher total MBps reading for each and every drive that you add (even once you're beyond the "max-drives" threshold). In FlexRAID the MBps reading will decrease as it essentially measures the weakest link whereas SnapRAID reports the total combined throughput. However, it's undeniable that the performance of each individual drive will be degraded once you pass a certain count and the hardware can no longer keep up. And that limited will be reached before the expander is full. Deny it all you want because you have the theoretical numbers to back it up, it's a fact. Test it yourself. Show the results. Prove me wrong.
SAS Overhead is approximately 12%
STP Overhead is approximately 20%

If we use one of the fastest consumer 4TB drives available (WD Black 4TB) it has a transfer rate of 168MB/s (the only benchmark relevant for snapshot operations) that gives us a "worst case scenario" in terms of bottlenecks.

168MB/s is well, well under the available real world 4.2Gb/s* (528MB/s*) between the drive and the SAS expander.
No bottleneck there. Tons of headroom.

With 16 (not 20 since you evidently can't figure out how a 2x4 configuration works) drives connected to the expander the total possible data transfer (16 * 168MB/s) is 2688MB/s
With 8 real world 4.2Gb/s* links between the expander and the HBA there is 33.8Gb/s* or 4224MB/s*
No bottleneck there. Plenty of headroom.

The HBA is connected to his motherboard via a PCIe 2.0 x8 link.
After PCIe protocol overhead each PCIe lane has ~400MB/s available of real world bandwidth. That's a total of 3200MB/s*
No bottleneck there. Still headroom to spare.

Now it's certainly possible if Axel's motherboard is only running PCIe 1.0 he could run into bottlenecks. But that's not the fault of his storage subsystem. If his motherboard is running his x8 card with only x4 (or heaven forbid) x1 PCIe lanes then he could experience some bottlenecks. As stated before, he could have a CPU bottleneck, something else misconfigured somewhere, or any number of other things that could prevent the pool from performing at 100% performance, but none of those things are relevant to the discussion. You told Axel not to go with Option #2 because of the expander/HBA combo. If his motherboard, CPU, or anything else is the problem, then it doesn't matter what expander, or HBA he's using, or how they're configured as that other bottleneck will still exist.

If he's dug up some 15,000 RPM Enterprise drives (unlikely since he said they were SATA II) he could run into some bottlenecks. But if he's using regular consumer grade hard drives on a 6Gb/s expander (in 2x4 configuration) connected to a 6Gb/s HBA that is connected to a motherboard via a PCIe 2.0 x8 link, there is no chance he has any performance bottlenecks in his storage subsystem. If a bottleneck exists in his system, it has nothing to do with the expander or HBA.

I do agree that this does resemble a previous conversation where you blurted out some vague claim that had little real world bearing on the question actually asked.

Your broad, vague statement is generally correct. There is a point where you will see decreased performance using a SAS expander. That's great information to put out there, but that point isn't anywhere close to what Axel has setup on his system. It might be true using other controllers, or other expanders, or other motherboards, but with Axel's HBA and Axel's expander, it'll work fine (close enough to full speed it makes no difference) assuming he has ordinary hard drives and a motherboard that is capable of PCIe 2.0 x8 connection.

You're taking one outlier case and trying to apply it to everything and it simply does not apply across the board as you seem to insist that it does. I don't doubt for a second that you had an HBA that was connected to multiple expanders, with dozens of drives. But just because you once had a setup that wasn't assembled with best performance in mind, doesn't mean that somebody else can't do it.

It's easy enough for Axel to try. If the case is easily accessible, he should be able to swap out his 2nd HBA for the expander in a matter of minutes.

Just out of curiosity, how many drives did you have connected to your expander when the performance went downhill? What types of drives were they, and what HBA and expander were you using?



*real world throughput available after protocol overhead is accounted for.
ajhieb is offline  
post #4255 of 4731 Old 02-26-2015, 11:32 PM
Member
 
Quaraxkad's Avatar
 
Join Date: May 2004
Location: Port Orange, FL
Posts: 125
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 63 Post(s)
Liked: 52
Quote:
Originally Posted by ajhieb View Post
With 16 (not 20 since you evidently can't figure out how a 2x4 configuration works) drives connected to the expander the total possible data transfer (16 * 168MB/s) is 2688MB/s
My mistake, I forgot for a second we were talking about the Intel expander and not the HP.

Again, you can not simply multiply a single ports theoretical bandwidth by the number of ports on a card and expect to get that level of performance. Those numbers tell you how fast ONE port can carry data. They can not all run at those maximum speeds at the same time.

As I tire of your nitpicking and failure to comprehend the very specific usage scenarios in which I am referring, just for ****s and giggles I will answer your final question. Who knows, maybe you'll wave your magic wand and solve all my performance problems! I have done relatively thorough testing using various controllers and expanders on different motherboards and CPUs. The tests are far from extensive, but they show a very specific and easily recognizable and reproducible pattern. In the best case scenario I saw noticeable degradation of per-drive performance while reading from merely 13 drives. Virtually no difference in performance from 1 to 11 drives. 12 showed a slight drop in each drive but not significant. Beyond 13 and the performance declines rapidly with each new drive.

Anybody who has these cards can test it themselves. I'd like to see the results. Until I do see some results, proof that these cards are capable of such feats, I am done here.

Last edited by Quaraxkad; 02-26-2015 at 11:37 PM.
Quaraxkad is offline  
post #4256 of 4731 Old 02-27-2015, 12:31 AM
thu
Member
 
thu's Avatar
 
Join Date: Mar 2011
Posts: 16
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 10 Post(s)
Liked: 11
I saw ajhieb mention "2x4 configuration" for the m1015 controller. I'm assuming this mean connecting it to the expander with 2 cables to increase throughput. Is that correct?

Does the m1015 automatically use both cables or does it have to be manually activated with a bios setting or something like that?

I have a m1015 connected to an HP SAS expander with two cables. I have Flexraid running with 18 drives. I'm only getting 22 MB/s with parity computation and I'm wondering if it is so slow because I have not actually activated the 2x4 configuration mode. I've checked the HP firmware and it is 2.06, so that should be running at full speed.

I see there is some debate over 2 controllers vs controller and expander. In my case with only 22 MB/s parity computation, I'm wondering if 2 controllers might make a difference. It is only going to be getting worse as I add more drives and I'm not really sure what else I can try to do to speed things up.
thu is offline  
post #4257 of 4731 Old 02-27-2015, 12:41 AM
 
ajhieb's Avatar
 
Join Date: Jul 2009
Location: The Commonwealth, not the Jelly.
Posts: 2,696
Mentioned: 15 Post(s)
Tagged: 0 Thread(s)
Quoted: 1037 Post(s)
Liked: 698
Quote:
Originally Posted by Quaraxkad View Post
My mistake, I forgot for a second we were talking about the Intel expander and not the HP.
A natural mistake to make since literally no one was discussing the HP expander.


Quote:
Again, you can not simply multiply a single ports theoretical bandwidth by the number of ports on a card and expect to get that level of performance. Those numbers tell you how fast ONE port can carry data. They can not all run at those maximum speeds at the same time.

Actually you can. That's what SAS Expanders do. It's called link aggregation.

But I'll let Intel explain it.

Quote:
Intel® RAID controller individual ports act as waterfall-type connections. If you are transferring less than 6 Gb (theoretical) from the controller to the Intel RAID Expander RES2SV240, you only use one port. Otherwise, if it is more than 6 Gb, it falls to other ports to enable the required throughput.
http://www.intel.com/support/motherb.../CS-033969.htm




Quote:
As I tire of your nitpicking and failure to comprehend the very specific usage scenarios in which I am referring, just for ****s and giggles I will answer your final question. Who knows, maybe you'll wave your magic wand and solve all my performance problems! I have done relatively thorough testing using various controllers and expanders on different motherboards and CPUs. The tests are far from extensive, but they show a very specific and easily recognizable and reproducible pattern. In the best case scenario I saw noticeable degradation of per-drive performance while reading from merely 13 drives. Virtually no difference in performance from 1 to 11 drives. 12 showed a slight drop in each drive but not significant. Beyond 13 and the performance declines rapidly with each new drive.

Anybody who has these cards can test it themselves. I'd like to see the results. Until I do see some results, proof that these cards are capable of such feats, I am done here.
If you have a specific usage scenario then share the specifics with us. If you've done the testing, then share the results. So far all you've done is give a guy questionable advice based in misconceptions about how these devices actually work.

For instance, if you were using a 3Gb/s or 1.5Gb/s expander, (like this HP for example) I wouldn't be surprised at all to find out that it was saturated when going beyond 12 drives, especially if you were only using a single connection to the HBA. That would be half the speed, and half the channels for a net of 1/4 of the Intel card setup in a 2x4 configuration. That's basically 1GB/s for more than a dozen drives to share. Of course that is going to start slowing down. That's actually right about the point I would expect it to hit it's limits. That's why I asked what equipment you were using, because the equipment used would have a huge bearing on the performance.

So again, I'll ask if you will share your test results (and methodology) with us.
ajhieb is offline  
post #4258 of 4731 Old 02-27-2015, 01:19 AM
 
ajhieb's Avatar
 
Join Date: Jul 2009
Location: The Commonwealth, not the Jelly.
Posts: 2,696
Mentioned: 15 Post(s)
Tagged: 0 Thread(s)
Quoted: 1037 Post(s)
Liked: 698
Quote:
Originally Posted by thu View Post
I saw ajhieb mention "2x4 configuration" for the m1015 controller. I'm assuming this mean connecting it to the expander with 2 cables to increase throughput. Is that correct?

Does the m1015 automatically use both cables or does it have to be manually activated with a bios setting or something like that?

I have a m1015 connected to an HP SAS expander with two cables. I have Flexraid running with 18 drives. I'm only getting 22 MB/s with parity computation and I'm wondering if it is so slow because I have not actually activated the 2x4 configuration mode. I've checked the HP firmware and it is 2.06, so that should be running at full speed.

I see there is some debate over 2 controllers vs controller and expander. In my case with only 22 MB/s parity computation, I'm wondering if 2 controllers might make a difference. It is only going to be getting worse as I add more drives and I'm not really sure what else I can try to do to speed things up.
I'm not sure if the HP supports multiple links to a non-HP controller. The Intel expander is based on the LSI chipset and generally plays nice with any LSI based HBA or RAID controller. The HP expanders that I've had my hands on were based on PMC chipsets. I wouldn't be at all surprised to find out that they have compatibility issues with Non-HP hardware.

That said, a myriad of things can affect how long it takes to calculate parity on a FlexRAID configuration. As you add more drives (and more parity drives) it becomes more and more CPU intensive. Depending on what sort of processor you have (and if it's using all cores) I wouldn't be surprised to find out it is CPU bound. FlexRAID is also file based, not block based, so file sizes can also make a difference. Evidently lots of tiny files are slower than fewer large file. Dissimilar file sizes can also impact performance.

I saw a thread on the FlexRAID forums where some poor sucker was only getting 6-7Mb/s on his parity calculations. He originally had a bunch of old SATA PCI-X controllers (directly connected to drives) then upgraded to an LSI card and Intel expander and the performance still didn't increase. As is the usual policy on the FlexRAID forums the "developer" indicated that it was being looked into and then dismissed it as a controller issue. (as he often does) To his credit the poor FlexRAId guy did seem to know what he was doing, and seemed like he was starting to figure out that Brahim's (FlexRAID "developer") testing methodology was somewhat flawed, and giving best case scenario numbers instead of real world number.

My experience with FlexRAID specifically is somewhat limited (I generally use hardware RAID) but I've read a great deal about it, and have followed several people through their FlexRAID journeys, and the pattern I've started to notice is that FlexRAID itself, regardless of the hardware used, doesn't seem to scale very well.

So while I'd say there is a good chance your expander is only operating with a single connection, I'm under the impression that your configuration is CPU bound which, if I'm correct, is the more pressing issue.
ajhieb is offline  
post #4259 of 4731 Old 02-27-2015, 02:01 AM
thu
Member
 
thu's Avatar
 
Join Date: Mar 2011
Posts: 16
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 10 Post(s)
Liked: 11
Thanks ajhieb. I know there are lots of variables, specially with Flexraid, and it is really hard to pin down. I can't really upgrade other parts right now and I was hoping to get lucky and get some speed increase with a forgotten setting with the controller card. Seems like it is not to be.

I'm not TOO bothered by the slow speed as I just use Flexraid for media server and am just adding to it in "small" chunks of maybe like 50-100 gigs at a time. So it is not that big of a deal to let it go for an hour for parity. It is just the times I do a verify where it is taking 35+ hours where it would be nice to get some more speed. Hopefully it won't ever get as bad as that guy with 7 MB/s speed!

The server cpu is a slower speed i3, so that very well may be a big factor. I'll definitely go for the higher end the next time I update the server.

I think I saw Brahim say a while back that he was going to be working on a new version of the Raid-F. Hope that is still in the plans. Hopefully he can optimize things a bit now after all these years of real world testing.
thu is offline  
post #4260 of 4731 Old 02-27-2015, 02:42 AM
AVS Forum Special Member
 
Defcon's Avatar
 
Join Date: Sep 2001
Posts: 3,032
Mentioned: 14 Post(s)
Tagged: 0 Thread(s)
Quoted: 1645 Post(s)
Liked: 921
I wonder what will happen in a few months when the 8TB/10TB drives become affordable!

Its pretty amazing how quickly storage has expanded. I mean today you can buy cheap 4-5TB externals, use them for full backups for less cost than the hardware needed for proper redundancy/parity, and have very few headaches. I guess its all about the speed with which your collection expands.

X = rate at which you buy+rip new blurays/acquire content
Y = rate at which you are willing to discard old content (Y may be 0 for many)
Z = rate at which hdd prices reduce (i.e. $/GB)

if X-Y <= Z, then you can continue to have full backups of media without extra expenditure. Or so it seems anyway with my weird math.
Mfusick likes this.
Defcon is offline  
Sponsored Links
Advertisement
 
Reply Home Theater Computers

Tags
Flexraid , Intel Core I5 4670k 3 4ghz Lga 1150 Quad Core Desktop Processor , Intel Core I3 4130 3 4 3 Fclga 1150 Processor Bx80646i34130 , Asrock , Asrock Z87m Extreme4 Lga1150 Intel Z87 Chipset Ddr3 Quad Crossfirex Quad Sli Sata3 Usb3 0 Microatx M , Asus , Asus Us , Asus Computer , Asus Components , Norco , Asus Computer International Direct , Flexraid Raid F , Seagate , Seagate Hard Drives , Seagate Freeagent Theater 1080p Hd Media Player Stcea201 Rk , Hitachi , Dell , Windows 7 Vista Xp Media Center Mce Pc Remote Control And Infrared Receiver , Hp Oem Window Media Center Mce Pc Remote Control And Infrared Receiver For Windows7 Vista Xp Home Pr , Intel , Intel Core I5 3570 3 4 Ghz Processor , Intel Pentium G2020 2 9ghz Lga 1155 Dual Core Desktop Processor , Intel Pentium G3220 3 0ghz Lga 1150 Dual Core Desktop Processor , Amd , Amd A10 6700 Richland 4 2ghz Socket Fm2 65w Quad Core Desktop Processor Amd Radeon Hd Ad6700okhlbox , Amd A6 5400k 3 6ghz Socket Fm2 Dual Core Desktop Processor , Amd A8 5600k 3 6ghz Socket Fm2 Quad Core Desk

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off