Since loudspeaker drivers produce sound on both sides of the cone it is necessary to design the enclosure to "deal" with these "back waves". If there is no dampening material (or poly-fill) in the enclosure then you will end up with standing waves. Plus some of the waves may reflect back onto the speaker cone causing distortion. Standing waves can also be combated by designing non-square sub enclosures. I assume that your sub enclosure is square shaped.
The density of the poly-fill is debatable. Some people like a lot, others like very little. Just make sure the poly-fill is not impeding the movement of the driverâ€™s magnetic assembly.
The "boomy" sound you describe could easily be caused by standing waves. Try the poly-fill. It may be that the problem lies in the enclosure volume. Some manufacturers design subs to sound "boomy" so they sound more "impressive" in the show room.