To decide whether you want to use the plugs, you have to understand what the plugs do, and for that you have to understand what the port does. Basically a ported or vented speaker uses the air inside the enclosure as sort of a compression driver. The "plug" of air in the port acts as a resistance to the oscillation of the air inside the cabinet when excited by the woofer - this is commonly compared to an oscillating spring with a weight at the end. The length & width of the port defines the amount of air in the plug, and therefore the amount of "weight". The longer the port, the more resistance, and the _lower_ the resonant frequency of the air in the cabinet. The foam plugs add more resistance to the port, and therefore lower the resonant frequency of the cabinet.
So what does this mean? Well, if you are hearing "boomy" bass in your speaker location & room then most likely the cabinet is resonating too high, and you want to lower the resonance by putting a half or a full plug into the port. If you have less bass than you want, then remove the plugs and see what happens.
It's important to remember that putting in the plugs tends to reduce the bass mainly by moving the resonant frequency lower in the spectrum to a point where the driver loses power. There's no substitute for a large, dedicated driver (i.e. subwoofer).