First, let's consider a system that consists of an ideal voltage source (0 output impedance, infinite amperage, etc) an impedance device we'll call "source impedance", and a load impedance, wired in series. In a real system, the amplifier can be modeled as the voltage source and the source impedance, and the load is your speaker - so your speaker system sees the voltage source of the amp, and the source impedance.
Matching impedance (having the load and source impedance be the same) results in the maximum power being transferred in the system. However, efficiency is low - 50%! Half your power is wasted in the amp! Higher load impedances mean the efficiency goes up, but the total power transferred is lower.
Let's take some numbers - let's say the voltage source can output a max of 10V, while the output impedance is 0.5ohms. The most power that can be produced by this is 10 watts (into a 0.5 ohm load). If we increase the load impedance to 4 ohms, the amp produces just over 2 watts. However, the efficiency goes up from 50% to 89% so your amp isn't heating up as much.
Practically speaking no amp will be able to handle the theoretical maximum - the power supply limits the output voltage and current.
In fact, because speaker impedance varies with frequency, the load presented to the amp varies, which means the amount of power transferred can vary which results in the volume varying by frequency. A rule of thumb is that the output impedance of the amp should be 1/8th that of the load - which results in the output varying under 0.5db from 8 times source to infinity. Now, a 4 ohm amp you'd think would mean 0.5 ohms, but no, a 4 ohm speaker can have significant frequencies where it drops to 2 ohms or so, meaning you'd want an amp with 0.25ohms or lower. In a poorly matched system, this means the amp and speakers do some EQ on the output signal - at some frequencies, the amp is louder, and other frequencies, the amp is softer, instead of being the same volume throughout the entire frequency range (the ideal amp response - the amp should be flat, not doing EQ!)
In the tube days, this required work because tubes have a higher output impedance, so you'd need matching transformers to bring it down. But modern solid state amps can have arbitrarily low output impedance.
For all practical purposes, a modern amp doesn't need impedance matching because the designers know this and design it to have a very low output impedance.
The only reason you have "4 ohm" settings on amps is that it drives the amp more conservatively which limits the output power that can be produced so the finals don't overheat because the amp can produce more power into a lower impedance load. And because the power supply can be power limited, so a high powered signal can cause clipping which can destroy speakers as well.