Every piece of A/V gear has a "gotcha" hidden somewhere waiting to trip up users with mixed old and new hardware: this audio glitch is Pioneer's. There is no setting you can make: the coax audio is only active when HDMI is turned off, and the recorder is feeding a traditional receiver/TV system with analog video (component/S/composite) and digital audio over the coax. If you turn on HDMI, the recorder assumes you've moved on from 2001 to 2011 and now own an up-to-date, completely HDMI-connected system. HDMI is a pretty monolithic, all-or-nothing standard so this lockout of simultaneous coax digital audio is not unusual.
There are workarounds, there always are for those persistent enough, but it usually isn't cost-effective unless you REALLY love the old receiver and can't possibly afford to replace it with an updated similar-spec unit. Some combination of HDMI splitter and audio converter should be possible, a lot of people have very elaborate expensive systems that would require a similar solution. But the cost of these accessories might exceed the cost of a new mid-price all-HDMI receiver, which is the easiest answer if it suits your budget and you're not attached to the old receiver for any particular reason.
Aside to kymics: you found a Pioneer 460 "recently"? You are lucky indeed: these 2008 "ultimate" Pios are rare beasts now, and fetch sky-high prices even used on eBay. Enjoy it: I own two 460s and love 'em!