Try in your mind to visualize a perfect point source soundwave. An infinitely small point in space creating the soundwave radiating out perfectly in all directions. It should sound beautiful.
Now place that point on an infinite vertical plane like a wall. The sound now only radiates in 180 degrees. A part of the wave follows the plane's surface but it doesn't ever really go to the listener.
OK so now visualize a raised object on this surface (such as a raised speaker rim). As the sound wave that was following the surface encounters this raised object a portion of the wave reflects off this object. It is almost like the point of deflection itself is generating sound. This sound also radiates outwards, but as it encounters other parts of the orginal soundwave they interfere, causing some cancellation, and some reenforcement and rippling somewhat up and down the frequency response.
Are you going to hear this, maybe yes and maybe no. It might be a little bit of distortion, some harshness, muddiness, or less precise imaging.
The protruding rubber surround of a cone driver causes about a 3 db dip in the driver's frequency response at approximately the frequency of 1 wavelength. For a 5" speaker this is about 2k. Some manufacturers try to account for this in the speaker design. I have even heard of some people placing absorbent material on the rubber surround.
Now we further limit our perfect soundwave environment by placing it on a speaker baffle instead of the wall. As the sound wave reaches the edge of the speaker you get diffraction from the edge, with similar negative effects as in hitting a raised object. So on a standard speaker we have each edge deflecting, the raised drivers, speaker covers, etc... etc. All of this will muddy our clarity and imaging.
The edge diffraction is is especially acute with line source speakers in a rectangular baffle, because they will uniformly encounter the edge for the entire length of the line. Whatever the type of speaker, rounded edges are better, the rounder the better.
Also higher frequencies will radiate at 180 degrees due to the baffle, but as the frequencies go lower they will be less affected by the surface of the baffle and tend to radiate more toward 360 degrees. This causes a significant loss of energy. From about 1000hz down to 100hz you gradually lose 6db of sound energy. This is called baffle step loss. Many builders modify the crossover to account for this.
So do what you wish, but it all has an influence on the sound of the end product.