I had an entry level 2011 Mini (2.3 i5 w/ Intel HD 3000 Graphics) in the home theater for almost a year: it was plenty powerful enough to play full-size 1080p bluray rips back with XBMC and I never had any problems running HDMI directly to a 47" Vizio LED and optical to my AVR. That 2.3 i5 Mini was a very large leap forward for the price in terms of Mac home theater. I eventually swapped that Mini out for the mid-level 2011 Mini, the one with the Radeon HD 6630M graphics. Occasionally I came across a particularly challenging bluray that needed an acceleration boost from XBMC but most of the time I left that turned off in settings.
So, your Mac can
do the job, but it's not plug and play foolproof because so many aspects of home theater require cooperation between devices and manufacturers and there are lots of settings and menus to learn and pay attention to and tweak. You're doing so many things that could be causing little glitches along the way, among them:
1) VLC. I think it sucks. Try playing the same files back with a better front end like Plex or XBMC. If an alternative player does a better job with your files then you know the problem was with VLC;
2) your 1080p .mkvs themselves--do all files or just certain specific files exhibit this behavior? There are lots of different "1080p .mkvs" out there. Be more precise. Are you making these files yourself or downloading the work of others, which could vary in quality? If ALL files are affected, those you rip plus those you pirate, that tells you it's more likely a setup and settings issue with your software player and hardware rather than the files per se, if only certain files are affected but others play perfectly well, you'd have to take a different approach;
3) could be an issue with your particular TB to HDMI adaptor and cable, consider swapping in something else to test with...likewise, are you going through your Pioneer first and then running an HDMI cable out to your TV or bypassing the AVR??? There can be issues related to either approach...and lots of settings you have to first, understand, and second, tweak.
4) there could be audio passthrough issues related to you going to the TV first and then optical out from the TV to the AVR...most here would go directly to the AVR when possible and not rely on such a circuitous route...no reason you couldn't go optical directly to your Pioneer to see if it makes any difference;
5) the location of your files can matter, i.e. whether there are network or streaming issues involved.
That's a lot to take in, but what all that really means is that it can take a lot of time to figure out the right menu settings for your devices--which are unique to you and different from my devices--which means allow yourself some time to experiment to find just the right settings for you
with your software player, your AVR, and your TV. And don't short change your TV, HDTVs these days apply a lot of processing and oftentimes have confusing settings and menus. It took me quite a while to figure out what gave me the best performance with XBMC, and I had to tweak things in both the XBMC video menu settings and in the HDTV picture settings.
I don't think I'm getting true 5.1 audio either - even though the audio is supposed to be encoded with 5.1
Again, it's best to rip and transcode your own files because then you know what the audio is supposed to be because you've checked off what you want within the various apps like MakeMKV, Handbrake, etc.
There have been a tons of "not getting true 5.1 and choppy/stuttering video" threads like yours over the years. Almost all of them come down to human error, i.e. not reading the manuals thoroughly, not having the right settings, not fully understanding the software, so sanity check-wise, with a little more patience and more specific tests, we probably could help you figure out what's really going on. It's not unusual things just didn't magically work right from the start for you. Home theater is a veritable minefield of sorts.