"I'm in the camp of "bass is bass" the only thing that would really cause one driver to sound different then another is distortion."
yeah, but that's kind of like saying it sounds the same unless it doesn't. :-)
there is harmonic distortion, intermodulation distortion, frequency response distortion, power compression distortion, amp-clipping distortion, port compression and port resonance distortion, air spring distortion, just off the top of my head.
there are a variety of ways to address each of these depending on what one is trying to accomplish.
and, in most cases, the other part of the subwoofer-room system, the room itself, is frequently the worst offender.
except for the rear wall, i think your system does just about everything right. a limp-mass absorber tuned to first length mode might solve that one too.