My suspicion is that (for cost cutting reasons) there aren't enough line amplifiers in the 8801 to be able to drive both at the same time. When RCA is selected, one amp drives the line. When XLR is selected, that amp plus a second one drive the XLR output.
Has anyone been able to obtain a maintenance manual for the 8801? It probably would resolve some of these questions.