hi - you should head over to the Denon xx12 model owner's thread as this isn't really a speaker forum question: http://www.avsforum.com/t/1334369/the-official-denon-avr-xx12-model-owners-thread
do your speakers even have a sub input (RCA style plug, not speaker wire)? If not, you CAN'T wire them up to the sub output on the Denon.
if you don't have a sub, the front speakers HAVE to be "large" because there is no sub to cross over to (the small/large setting is really just the on/off switch for bass management). You should bump up the center channel crossover to 60 or 80Hz (so all the bass goes to the powered subs in the towers).
second, you should NOT fiddle with the knobs on the subs after running Audyssey (btw it's MultEQ XT, there is no "EX" in Audyssey
) ... once the speakers are calibrated you should make adjustments in the receiver as needed.
There is a good chance the reason you are hearing boominess is a combination of Dynamic EQ (which boosts bass levels) and the sub volume being a bit "hot".
I would do the following:
1. first, find a good setting for the sub volume knobs that will be your baseline pre-Audyssey. Play some bass heavy music in DIRECT mode (which will bypass digital processing) and dial in the sub knobs to a point where they provide smooth response. Don't run them "hot" because Audyssey will just EQ them down to flat (that's why it is trying to do). Make sure the music is loud enough that you can really feel the bass when you do this; the human perception of bass drops a lot at lower volumes (which is why Dyn EQ exists).
2. after you get it dialed in, re-run Audyssey using good procedure (check out my Audyssey FAQ and read through it and the associated links here: http://batpigworld.com/wp/?page_id=37
3. after running Audyssey, change the center speaker crossover as noted above
4. finally, try listening to some of your reference material, but this time adjust the bass by going into the GUI and in the Audio Adjust area in Audyssey Settings set the Dynamic EQ Reference Level Offset to either 5, 10, or 15.... the higher numbers will diminish the impact of Dynamic EQ.