Does 7.1 surround audio work on Mac?

What files are you trying to play and what kind of "7.1 surround audio" do they have?

If you're talking sd video sources, they are 2.0 or 5.1, DTS ES or NEO 6 or ProLogicIIx are usually matrix functions of your AVR expanding sound out into all your speakers and don't really have anything to do with your Mac or its settings.

If you're talking about Dolby TrueHD or DTS HD from blu ray rips, most here already know it's best not to try to play the actual HD track in OS X...asking a software player to downmix that in real time to something supported in OS X can lead to problems with the software player not putting the channels in the right place. It's been a quirk that the developers of those players have always had to deal with. What most of us do is let MakeMKV extract the DD or DTS core track, which is almost always 5.1, and simply play that, which players like XBMC usually have no problem with. Your AVR should light up with DTS or DD 5.1 if it is properly passed through and if you then want to expand that to your remaining two speakers you usually can do that by changing the surround setting on your AVR.

If you've converted the HD audio to FLAC or LPCM those are different animals, there's no agreement among the players or Handbrake or MakeMKV about how to handle that as far as setting channel order or re-ordering channels is concerned, that's all still in flux so expect some inconsistencies. What doesn't work right in Plex, for instance, may work just fine in XBMC.

If you haven't tried it yet, re-rip a disc with MakeMKV and don't retain the actual 7.1 HD audio track, just extract the core, then see how that plays in XBMC and what your AVR can do with it. In an ideal world, you should be able to rip and retain the original Dolby TrueHD or DTS HD track and then your player would automagically know how to extract the core when needed. I still don't think we're there yet, hence why it might be more prudent letting MakeMKV or Handbrake do it for you first.

Which Mac and AVR you have, which OS and player versions, whether you connect over HDMI and/or optical, all can make a difference in what you're able to achieve.

If what you're trying to play is a 7.1 DTS HD-MA track as you indicated, I already answered your question, it can't be played on a Mac in OS X. What can be played is the 5.1 core DTS. Send that to your AVR and then let the AVR matrix sound to those other two speakers.

Your only other option in OS X is to convert that 7.1 DTS HD-MA track to 8 channel LCPM ahead of time, which then can be passed to your AVR over HDMI with either Plex or XBMC, but it might be tricky to get all 8 channels in the right place. There have been previous threads here about this:

http://www.avsforum.com/t/1387397/br-lpcm-tracks-working-in-osx-w-plex-and-hdmi-mac-mini/0_40

Also, for any blu ray rip with Dolby TrueHD, Plex should be able to transcode it in real time and output LPCM 7.1 to your AVR.

Hope that helps.

I think you'll find, the deeper you get into playing back blu ray rips in OS X, that both of these approaches will cause you problems. You really don't want to go into AudioMIDI very much and mess with re-assigning output channels because the files you're going to play can be very different, they've been ripped, extracted, down mixed and encoded differently...players approach audio differently...and even if you eventually settle on one player for all your viewing needs, the sources of your files can vary tremendously. Most of us here recommend that you let your player, say XBMC, interact with Core Audio and AudioMIDI unchanged, that way when you have an issue, you know to go into the settings for XBMC to try to solve it. Let XBMC interact with CoreAudio according to Apple's APIs, at least at first.

And as far as relying on the player to extract the core audio in real time, again, my advice is long term you may not be happy relying on this, the players don't yet do this consistently well: how they extract the core and downmix the core varies...it's why channels get mixed up, why 2 channel is passed through sometimes instead of 5.1 or why the volume seems way out of whack on certain channels. Plus, it'll cause your Mac to run hotter and work harder than it has to. Let's say you don't really care about the latter, well, in my experience the former--real time downmixing of HD audio--will be problematic as you move beyond test files and get into actual rips of blu rays.

I don't re-encode to LPCM so I'm afraid I can't be much help to you. I'm perfectly happy with what MakeMKV extracts from the blu ray for me which I then let XBMC pass as encoded digital and/or then transcode with Handbrake for the aTV and iPads in the house. I don't even retain the HD audio when I rip with MakeMKV. You might want to poke around on the MakeMKV discussion forums because there now is an option to save HD audio to FLAC:

http://www.makemkv.com/forum2/viewtopic.php?f=1&t=5710

which you then might be able to do something with.

The strategy that's worked for me is for Handbrake and MakeMKV to put these tracks in our files ahead of time, your options are AC3 (ffmpeg), 6-channel discrete or AC3 Passthrough or DTS Passthrough depending on the source--you'll have to experiment to see which sounds best for you, yes it's extra work but it's more reliable in a home theater context if it is already there when it comes to players like XBMC and Plex. Of course, the main reason we're using Handbrake is to create a file that's also playable on the iPad and aTV, which admittedly you're not that concerned about, but having that second encoded digital or discrete multi-channel audio track right there can make life a lot easier for Macs in the home theater as well.

That's why I suggested you re-rip a title with MakeMKV allowing it to extract the core for you--then you could compare what a given player does 1) down mixing from the HD audio in real time versus 2) what MakeMKV did ahead of time. I'd be curious to see if there was any difference...

As you start troubleshooting, you probably want to start being very specific about your hardware and software, just saying "7.1" and "Mac" and "XBMC" is not enough, there's a lot of road to cover from Snow Leopard to Lion to ML and Dharma to Eden to Frodo, etc. Every update can break something or introduce a new quirk, many of us have learned the hard way to become pretty loathe to update once we've gotten something set up that works. For instance, if you're on the latest Mountain Lion and Frodo I can't help you because I'm still really happy on older versions, none of my home theater Macs, even though they are 2011 Minis, are on ML or Frodo yet. One is still running Dharma because it has been so solid.