Remapping the channels is a small problem and can be done with the MIDI tool.
Actually, XBMC and VLC can extract the core track just fine.
I think you'll find, the deeper you get into playing back blu ray rips in OS X, that both of these approaches will cause you problems. You really don't want to go into AudioMIDI very much and mess with re-assigning output channels because the files you're going to play can be very different, they've been ripped, extracted, down mixed and encoded differently...players approach audio differently...and even if you eventually settle on one player for all your viewing needs, the sources of your files can vary tremendously. Most of us here recommend that you let your player, say XBMC, interact with Core Audio and AudioMIDI unchanged, that way when you have an issue, you know to go into the settings for XBMC to try to solve it. Let XBMC interact with CoreAudio according to Apple's APIs, at least at first.
And as far as relying on the player to extract the core audio in real time, again, my advice is long term you may not be happy relying on this, the players don't yet do this consistently well: how they extract the core and downmix the core varies...it's why channels get mixed up, why 2 channel is passed through sometimes instead of 5.1 or why the volume seems way out of whack on certain channels. Plus, it'll cause your Mac to run hotter and work harder than it has to. Let's say you don't really care about the latter, well, in my experience the former--real time downmixing of HD audio--will be problematic as you move beyond test files and get into actual rips of blu rays.
So as far as I know right now there are three ways of re-encoding DTS to PCM
I don't re-encode to LPCM so I'm afraid I can't be much help to you. I'm perfectly happy with what MakeMKV extracts from the blu ray for me which I then let XBMC pass as encoded digital and/or then transcode with Handbrake for the aTV and iPads in the house. I don't even retain the HD audio when I rip with MakeMKV. You might want to poke around on the MakeMKV discussion forums because there now is an option to save HD audio to FLAC:
which you then might be able to do something with.
The strategy that's worked for me is for Handbrake and MakeMKV to put these tracks in our files ahead of time, your options are AC3 (ffmpeg), 6-channel discrete or AC3 Passthrough or DTS Passthrough depending on the source--you'll have to experiment to see which sounds best for you, yes it's extra work but it's more reliable in a home theater context if it is already there
when it comes to players like XBMC and Plex. Of course, the main reason we're using Handbrake is to create a file that's also playable on the iPad and aTV, which admittedly you're not that concerned about, but having that second encoded digital or discrete multi-channel audio track right there can make life a lot easier for Macs in the home theater as well.
That's why I suggested you re-rip a title with MakeMKV allowing it to extract the core for you--then you could compare what a given player does 1) down mixing from the HD audio in real time versus 2) what MakeMKV did ahead of time. I'd be curious to see if there was any difference...
As you start troubleshooting, you probably want to start being very specific about your hardware and software, just saying "7.1" and "Mac" and "XBMC" is not enough, there's a lot of road to cover from Snow Leopard to Lion to ML and Dharma to Eden to Frodo, etc. Every update can break something or introduce a new quirk, many of us have learned the hard way to become pretty loathe to update once we've gotten something set up that works. For instance, if you're on the latest Mountain Lion and Frodo I can't help you because I'm still really happy on older versions, none of my home theater Macs, even though they are 2011 Minis, are on ML or Frodo yet. One is still running Dharma because it has been so solid.