You will have loose fibers that will become airborne, and blow out of the port, as well as falling on the back of the driver, and getting sucked into the voice coil gap. I would suggest using something other than fiberglass. If it was a sealed sonotube, and there was a way to prevent it from entering the gap, or sagging down / covering the vent on the back of the magnet, sure.
I did a bunch of measurements a few years back to see why an untreated sonotube sounded so 'ugly' when I fired the first one up. Turns out that the end to end dimension was long enough to create a big spike in the frequency response that rang way out to 300 msec after the initial signal.
Two polyfill pillows placed around the port slightly away from the endcap opposite the driver is enough to completely get rid of that nasty spike. It did however come with a slight penalty in efficiency above tuning, with a slight gain below tuning. Pillows are nice because the material is contained in the cover, they are non-shedding and easy to work with.
"You take the blue pill - the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill - you stay in Wonderland and I show you how deep the rabbit-hole goes."