If you operate a PSU at a small fraction of its rated max power, the efficiency will be lower. For example, an 80+ rated power supply does not have to meet the efficiency number at less than 20% of max rating.
So, while it is certainly true that a 500W PSU will NOT draw 500W of power when given a 20W load, it will probably draw more power than an equivalent 80+ rated 250W or 300W power supply.
But even if the efficiency were absolutely awful, say 50%, you would still only be drawing 40W for a 20W load.