"Option 2 will be too slow for FlexRAID."
Originally Posted by Quaraxkad
I never suggested that he not use either the controller or the expander. I merely told him that he will not see increased performance by simply adding the expander, and if you argue against that then you're just a lunatic.
Yeah, I guess you didn't say that thing that you said. Whatever man.
And in the unlikely event one of his controllers is operating at x4 or x1 (some motherboards have PCIe slots that are physically x8 or x16 slots, but are only wired for x1 or x4) then moving all of his drives to a single controller operation at full speed certainly could increase his performance. (note: When I present an outlier case, I distinguish it as such)
This is not the first time you have argued with me over what is largely and clearly semantics. You say yourself that you are not denying that a potential bottleneck exists. You take one motherboard, one SAS card, one expander, put 24 drives on it (or even drop to 20 with two SAS connections between the two), and then tell me that every drive is going to perform at its 100% peak, solely because the individual theoretical max throughput of a single port is fast enough for each individual mechanical drive, obviously you have not tested this theory.
I'm telling you with absolute certainty that the expander and/or controller and/or motherboard (it makes absolutely no difference which one) simply and absolutely can not handle a scenario where every port is occupied and all drives are simultaneously being read without *severe* bottlenecks. Frankly I don't care if you believe that or not, clearly you think you've got this all down and have all the facts. But you go ahead and keep telling people that they can get 100% performance from their mechanical drives during snapshot operations with a full expander, you're going to have some unhappy people when they realize how wrong that was. It doesn't matter how many times you tell me I'm "flat out wrong", it doesn't mean that these devices can suddenly perform miracles. I want you to provide some evidence to back up your claim that any of these involved components can handle the full load scenario required by snapshot RAID duties. It will still "work" no matter how many drives you hook up, and if using SnapRAID then you will quite possibly see a higher total MBps reading for each and every drive that you add (even once you're beyond the "max-drives" threshold). In FlexRAID the MBps reading will decrease as it essentially measures the weakest link whereas SnapRAID reports the total combined throughput. However, it's undeniable that the performance of each individual drive will be degraded once you pass a certain count and the hardware can no longer keep up. And that limited will be reached before the expander is full. Deny it all you want because you have the theoretical numbers to back it up, it's a fact. Test it yourself. Show the results. Prove me wrong.
SAS Overhead is approximately 12%
STP Overhead is approximately 20%
If we use one of the fastest consumer 4TB drives available (WD Black 4TB) it has a transfer rate of 168MB/s
(the only benchmark relevant for snapshot operations) that gives us a "worst case scenario" in terms of bottlenecks.
168MB/s is well, well
under the available real world 4.2Gb/s*
) between the drive and the SAS expander.
No bottleneck there. Tons of headroom.
With 16 (not 20 since you evidently can't figure out how a 2x4 configuration works) drives connected to the expander the total possible data transfer (16 * 168MB/s) is 2688MB/s
With 8 real world 4.2Gb/s*
links between the expander and the HBA there is 33.8Gb/s*
No bottleneck there. Plenty of headroom.
The HBA is connected to his motherboard via a PCIe 2.0 x8 link.
After PCIe protocol
overhead each PCIe lane has ~400MB/s available of real world bandwidth. That's a total of 3200MB/s*
No bottleneck there. Still headroom to spare.
Now it's certainly possible if Axel's motherboard is only running PCIe 1.0 he could run into bottlenecks. But that's not the fault of his storage subsystem. If his motherboard is running his x8 card with only x4 (or heaven forbid) x1 PCIe lanes then he could experience some bottlenecks. As stated before, he could have a CPU bottleneck, something else misconfigured somewhere, or any number of other things that could prevent the pool from performing at 100% performance, but none of those things are relevant to the discussion. You told Axel not to go with Option #2 because of the expander/HBA combo.
If his motherboard, CPU, or anything else is the problem, then it doesn't matter what expander, or HBA he's using, or how they're configured as that other bottleneck will still exist.
If he's dug up some 15,000 RPM Enterprise drives (unlikely since he said they were SATA II) he could run into some bottlenecks. But if he's using regular consumer grade hard drives on a 6Gb/s expander (in 2x4 configuration) connected to a 6Gb/s HBA that is connected to a motherboard via a PCIe 2.0 x8 link, there is no chance he has any performance bottlenecks in his storage subsystem. If a bottleneck exists in his system, it has nothing to do with the expander or HBA.
I do agree that this does resemble a previous conversation where you blurted out some vague claim that had little real world bearing on the question actually asked.
Your broad, vague statement is generally correct. There is
a point where you will see decreased performance using a SAS expander. That's great information to put out there, but that point isn't anywhere close to what Axel has setup on his system. It might be true using other controllers, or other expanders, or other motherboards, but with Axel's HBA and Axel's expander, it'll work fine (close enough to full speed it makes no difference) assuming he has ordinary hard drives and a motherboard that is capable of PCIe 2.0 x8 connection.
You're taking one outlier case and trying to apply it to everything and it simply does not apply across the board as you seem to insist that it does. I don't doubt for a second that you had an HBA that was connected to multiple expanders, with dozens of drives. But just because you once had a setup that wasn't assembled with best performance in mind, doesn't mean that somebody else can't do it.
It's easy enough for Axel to try. If the case is easily accessible, he should be able to swap out his 2nd HBA for the expander in a matter of minutes.
Just out of curiosity, how many drives did you have connected to your expander when the performance went downhill? What types of drives were they, and what HBA and expander were you using?
real world throughput available after protocol overhead is accounted for.