I would echo what lovin' and braveheart have said and say that, unless your Yamaha cannot cleanly (ie. no clipping) power your Polks to your desired maximum
volume levels, an external amp will make no tangible difference.
With respect to lower level listening, realise that the rated power of an amp or AVR indicates the power it can produce in the given test scenario. In real word use on program material, the power supplied (well voltage actually) at any point in time will be dictated by the size/shape of input signal and the position of the volume knob (ie. desired level). In other words, it doesn't matter what power the amp can potentially
produce; if it's not required by the program and volume setting, it's not being generated and sent to the outputs and can make no difference to how the speakers sound. All else being the same, exactly the same amount of power will be supplied to your speakers whether it's from your Yamaha or an external amp.
You may be interested in this thread
where we worked out peak power requirements for a fella who was considering adding an Emotiva. You may be surprised at how little power you are actually using. The OP ended up repositioning his speakers and furniture slightly and paying attention to his sub/speaker crossovers and went away very happy. All without spending a cent! Bottom line: the real sonic gains are in upgraded speakers and subs, proper positioning and in paying attention to the room. An amp is the last place to look, and then only if your AVR is struggling to produce your desired maximum volume levels.