My bad. It was on the Onkyo manual not to connect the RCA and XLR at the same time.
Yamaha does not state that it is not possible, so you could drive the two front subs using the RCA and XLR 1 output and the RCA and XLR 2 for the rear and then assign as front + back in the sub setting menu.
However, I guess that volume matching will be a little difficult as xlr runs at 2V while the RCA runs at 1V not to mention that 0.06THD over XLR is achieved at 13V while through the RCA at 6.5V. Not sure how that would work.
If I were you, I would use a splitter. I use it for the two surround chanels to have speakers on both seating rows.