Damping material absorbs the radiation from the back side of the speaker cone. Some absorption is needed, but too much starts to reduce cone excursion.
An empty box will resonate at specific frequencies (see: standing waves), and those resonances will be audible through the cone (or out the port, if not sealed). A frequency response measurement will be ragged, with a lot of peaks corresponding to those resonances. It won't be anything specific, the speaker will just sound bad.
If you put in too much, you lose bass output. There's a happy medium inbetween, that's best found by ear.
Have fun,
Frank