AJCxZ0 is totally right.
Indeed on Onkyo receivers there is no Small / Large speaker setting. It's either Full Range, or a choice of crossover frequencies. 80 Hz is nice, meaning that the AVR will send 80 Hz and above to your speakers and 80 Hz and below to your sub.
But on the sub, you should disable the low-pass filter as bass is managed by your AVR. If you add another low-pass filter at 80 Hz, you'll have a double effect and you'll create a deep dip around the crossover frequency.
It is totally normal to set the sub close to the max setting (0dB) -- the max setting is in fact the recommended setting on the sub if you AVR is able to attenuate the signal enough. However the fact that you are running at +12 on the AVR and -8 on the sub means you're driving it pretty hard. If your sub is not producing a satisfactory output there is probably something else wrong... But what, this is the open question.