what could be some of the possible problems?
My vote would be the one thing you haven't mentioned at all: your display. What model, how you're connected to it (HDMI?) and how you've set it up in its menu. HDTVs aren't dumb displays anymore, there's lots of little settings that can potentially interfere with a good picture if you're not careful.
I also suggest you read up on what stutter and "jutter" and artifacts actually mean, so you might be able to do a better job of describing what it is you think you're seeing.
There are different versions of the various players you mention, as in if XBMC-do you have Dharma or Eden, and each has different ways to turn on graphics assist or hardware acceleration for bluray content, so perhaps you might mention what you're using, what your settings are, etc. (I have the same 2011 Mini, the 2.3, and I find that XBMC can play back all of my MakeMKV rips without any hardware acceleration.)
Do you have 'Adjust display refresh rate to match video' turned on? Do you have 'Sync playback to display' turned on? With certain types of HDTV, these can make a difference.
Disconnect your "computer monitor" and just connect to your HDTV when you test this stuff.
Are these your own rips done with MakeMKV? Do they have the high def audio tracks included and are those what you're trying to play back?
As you can see, there are lots of little variables and it isn't easy to troubleshoot.