Hi, I'm not an expert as many here, so I can only talk about my limited experience and a lot of hours of research. My room (viewing area) is 13x15x9, in an open area, so i guess it is comparable to yours. Hope this helps.
I have struggled with this ever since I had my Onkyo HT-3300 all in one (among other problems when you go with a cheap all-in-one). What I discovered, even though speakers and room acoustics enhances the experience a great deal, the problem lies in original Bluray movie source. (ex. TV source may not sound as good overall, but you can hear the dialog better). The original mix in the bluray disc is mixed to be heard at "reference levels" which depending on where you live or who you live with, it may be too loud. That's what makes some dialogues in movies difficult to hear, specially in lower end systems. That is why you see many receivers using "gimmicks" to improve dialogue intelligibility.
Also, THX has DSPs on THX certified receivers to improve this, as they recognize the original mix on high definition audio movies is not optimal for small home theater at practical levels of listening. With that said, upgrading 1) speakers 2)room acoustics 3)receiver in that order will take you a long way.
My first upgrade was the speakers. Thanks to the help of many people here in AVS (and due to my tight budget) I ended buying the Pioneers BS22-LR for fronts and C22 for center. I kept my cheap onkyos speakers as surround and sub. The first impression was no more boominess, very clear and neutral sounding. Even with a wider dynamic range it let me increase listening sound level because of its clarity. (my cheap onkyos will actually make my room boomy, muddy and shaky) so it got unpleasant when increasing the volume. It improved dialogue intelligibility but I still had to apply a center gain to hear some. They are big (specially the center) and not the nicest looking one, but I am really happy with them.
Upgrade Cost: $200 for 3 speakers on Amazon.
Even though my 5.1 Onkyo was feature packed for its price (like all Onkyo are) I was wondering if a better receiver will greatly increase my experience. Like you, I wasn't concerned in second room, or hdmi availability over 4, or wireless features, etc. All I wanted is an improved sound with some dialogue enhancement feature and 7.1 to future proof it and last but not least, automatic audio calibration. The winner, Yamaha RX-V673 for $369 on Amazon. I also bought it with a Yamaha sub because my existing sub was passive.
Once I set it up, the sound spectrum increased. I heard sounds in movies I haven't heard previously, so I knew now the receiver was complimenting my newer speakers. They say receiver doesn't improve sound that much, and it may be true within they tiers, but upgrading from a cheap sub to a midrange did wonders for me. I think the key is in the quality of the amplifier. Yamaha uses on the 5 series and above, discrete amps per channel which help drive the speakers better. My particular sub wasn't rated high on CNEt, etc because at the price, it lacks some features like # of HDMI, built in wireless etc, except when it comes to sound, they said is along the top. Since all I cared was sound as long as it had my minimal feature requirements, I was sold.
I originally wanted Onkyo 616 because it was THX certified with THX DSPs, and many other features, but it seemed to have reliability issues. Also, I don't know if that series has improved HDMI input switch over my cheaper onkyo, the the Yamaha's is very fast and night and day with the Onkyo.
I won't go on since this is not a "receiver" review, but all I want to say is that overall, it improved dialogue intelligibility even though my center was automatic adjusted to -1db, which makes sense since it is a slightly greater distance and more powerful than the bookshelves. i have to say, I have set the dialogue enhancement feature to +2.
I initially used the dialogue lift, but it tends to echo in my fronts, so i didn't like it.
My last upgrade was 2 "presence" surrounds. I bough a $30 BIC America RTrv44-2; I haven't tried using dialogue lift with the presence yet. It helps with Yamaha's DSPs which I like for watching sports, etc.
Hope this helps.
Yamaha RX-V673 | HSU HC-1 MK2 | HSU HB-1 MK2 - LR | Pioneer SP-BS22-SLR | BIC RtR-V44-2 (Presence)