I really want to utilize the 360's ability to play video files and want to encode my dvd collection. The problem is that I love the quality I get with h.264 (in an MP4 container) but the 360 won't play anything encoded in 5.1 (dolby digital). AVI files on the other hand can play on the 360 with 5.1 sound but Handbrake doesn't encode AVI files with 5.1 (only options I get are DPLII, Dolby Surround, and stereo).

Is there something I'm missing? Will the 360 play h.264 movies with AC3 passthru? I basically want to use h.264 (for the superior video quality) and get at least DPLII audio...but I'm unclear if the 360 supports it.

I posted this topic in Handbrake's forums and got this lovely reply from their moderator:

RTFM (Read the @!#$ Manual)

Excellent customer service.

I tried a lot of different things to get the XBOX360 to output 5.1. The only thing that I found to work was DVR-MS files.

For playing DVDs through the Xbox 360 dashboard with 5.1 your best bet, in my experience, is with xvid encoded avi. I use the combination dvd decrypter + dvd shrink + autogk.

If you have the time you could try to encode an HD quality WMV. I never had a machine that made this practical.

The 360 doesnt support H.264 + AC3 5.1 [yet].

XviD + AC3 5.1 in AVI playable in dash
WMV + WMA 5.1 in WMV playable in dash and media center extender mode
MPEG-2 + AC3 5.1 in DVR-MS playable in media center extender mode
MPEG-2 + AC3 5.1 in MPG playable in media center extender mode but downmixes to 2 channel

Use VideoReDo if you want to rip a DVD into DVR-MS without re-encoding.

I'd recommend XviD4PSP as an XviD encoder it has some really good quality presets and XviD at a high enough bit-rate wont look that much different from x264.

With Windows 7 Microsoft added AVCHD [H.264 + AC3 5.1] support to WMP12 and the Xbox 360 can see these videos but not play them so its probably just a matter of time before it does.

Thanks for the info guys (it's nice to go to a forum where people treat you with some decency)....

Ok. So Xvid with AC3 5.1 will work...got that. My only hangup is the quality I get with AVI files. I encoded a couple of movies in both h.264 (mp4) and XviD (avi) and the differences between the two was pretty noticeable. All things being equal, the h.264 video was a lot richer and had less artifacts.

I'd be willing to do everything in XviD but maybe I'm missing some settings in Handbrake. I've tried it at 3000mps quality and 2-pass with both h.264 and XviD...it still doesn't help the XviD look any better.

I'm a Mac guy so my choice of encoders is a bit limited. Am I missing something in terms of a quality setting that could get me a better result with AVI?

Don't really know the OSX encoder scene but if Handbrake has an XviD encoder check to see if it has a constant quality mode preset for XviD as that gives you a level of quality without being concerned about file size (x264 also has a constant quality mode so make sure its not that mode its encoding to).

If you have access to someone else Windows machine install XviD4PSP then select AVI as the container & finally select XviD HQ Ultra as the profile (a constant quality mode).

See how the video quality is after that if its better than other solutions you've seen you could always get a virtual machine for your mac like parallels and a copy of winxp to run XviD4PSP in.

The only other option for H.264/AC3 is encode into AVCHD [.M2TS container] which all the Windows x264 encoders support so Handbrake probably does too and keep your fingers crossed that Microsoft update the Xbox 360 with AVCHD support during Win7 launch later this year.

So what's a good percentage when dealing with constant quality? I've heard anywhere from 60-80%...what do you think?

Another big advantage of AVI (if I can get the quality I want) is the speed vs. encoding with h.264. H.264 just takes too long (especially with 2-pass).

I've verified a passthru AC3 AVI with my xbox and it works. The sound is awesome...so if I can get the picture right then I'm set.

The only other drawback to AVI is the lack of a bult-in chapter option. But, in all honesty, I rarely use this feature on actual DVDs.

It's something really you'd have to try for yourself (because Handbrake may have its own XviD settings that are different from others), cut a piece out of a movie scene you know you weren't happy with when you first encoded with XviD.

Then use that mpeg-2 sample as your target and try encoding it at various settings till you see something you like, then run with that for all your movies and see if that's acceptable image quality wise.

So what's a good percentage when dealing with constant quality? I've heard anywhere from 60-80%...what do you think?

Another big advantage of AVI (if I can get the quality I want) is the speed vs. encoding with h.264. H.264 just takes too long (especially with 2-pass).

I've verified a passthru AC3 AVI with my xbox and it works. The sound is awesome...so if I can get the picture right then I'm set.

The only other drawback to AVI is the lack of a bult-in chapter option. But, in all honesty, I rarely use this feature on actual DVDs.

I'm trying to do the same exact thing you are doing. Have you found sufficient settings for picture quality yet?

I had too many issues with Handbrake and my 360. Half the time the audio would not work if the source was DTS. I used GotSent and it was much easier process. It's not as slick as Handbrake but always makes files the 360 can play.

http://www.videohelp.com/tools/GOTSent

Thanks for the link.

Anything new on this front? I foolishly updated handbrake recently and saw that the 360 is no longer supported. Went to the handbrake forums and sure enough, even though it is the most common set top box extender in use, none of the handbrake crew have one (really? yeah, that's what the community indicates) and they don't want to develop a profile for it.

I actually gave up on my 360 because it was too difficult to play mkv and other file formats on it. I built my own XBMC box (windows) and it's awesome. You can play any file you throw at it with no conversions. I do use handbrake to convert multiple file DVD rips to a single mkv file which is very handy.

I have five computers in the house. Don't want another! But I would love to watch videos I have on my server, in the room where we have a TV and 360 connected to it.

I agree, a PC is far more flexible.

I think it's funny that the handbrake forum community must be the only such group of techies on the planet that don't have a single xbox360 among them....

(They also contend that while the install base of 360s is large, using them to play media is a rare occurrence. I guess one could say the same of an iPhone. In fact, given the frequency of people using 360s to play, for example, Netflix movies, I'd guess that more people watch video on a 360 than on an iPhone. Yet there is still an iPhone profile and support..... of course, this gets into the whole religious war of Apple vs Microsoft vs Sony, which one suspects is really what is going on.)

But hey, it's open source community software, and they have no obligation to support any particular platform that they don't want to support. I guess I just find it disappointing that they dropped 360 support and use these strange reasons to explain it. I'd much rather they say "We dropped 360 support because we don't like 360s."

Thanks for the link.

Anything new on this front? I foolishly updated handbrake recently and saw that the 360 is no longer supported. Went to the handbrake forums and sure enough, even though it is the most common set top box extender in use, none of the handbrake crew have one (really? yeah, that's what the community indicates) and they don't want to develop a profile for it.

Just because they don't have a 360 profile anymore doesn't mean you can't encode videos for it. The "Normal" profile works just fine for 360 or PS3 from a H.264 video standpoint. Audio is the only slight hiccup. I was originally using 5.1 AAC, which works OK on the PS3.

But after discovering the awesomeness of WMC Extender on the 360, I'm using that as my front end. Problem with that was the 360 only plays my files with AAC at 2.0 stereo.

I believe the answer is with AVCHD (.m2ts extension h.264 + AC3 5.1). Using Handbrake all you need to do is use the Normal profile and select AC3 passthru for the audio track. You should output to mkv from Handbrake, but it doesn't really matter since we need to change the container to m2ts anyway.

Then use any of a number of programs, I use tsMuxer, to change the container from mkv to m2ts. The container change only takes 3-5 mins so not much added to your normal ripping process. At the end of the day you have a .m2ts file with h.264 and AC3 5.1 that I believe plays flawlessly on both the 360 and PS3. The only thing I need to go back and double-check is whether the 360 was playing the audio at 5.1 and not downmixing to stereo.

Thanks. Good to know it is possible. But it was so much easier before when there was a single setting to choose, instead of hot-rodding a pre-set and then using steps after the conversion to make it compatible.

Anyone have a lead on an all-in-one tool?

Haha, I didn't realize clicking a drop box for AC3 passthru and adding the track was "hot-rodding" a preset. I think you can save the preset, but I couldn't really imagine it being any easier. Granted, the conversion is an added step involving another program, but again, all it consists of is selecting an input file clicking output to m2ts and done.

You make it sound as though I gave you a convoluted 30-step 9-hour process to get what you want. This is so easy a caveman could do it.

I'm glad your solution works well for you. Because of my setup, it's all on a headless server, so unless I can use the batch function without user intervention, it means hooking up extra connections and gear. It's a drag that a feature was removed that worked fine, before, but from the discussion it does sound like I need to try some other approaches.

I believe I have some questions, and answers about this topic.
I have 3 videos that are AVI files containing 5.1 audio that the 360 will play. All of them are AVI files with Resolution of 1900x800 and Dolby AC3.

Video 1: Lord of the rings: The Two Towers
[AVI] Res: 1920x800 (1080P) (2.40:1) BR: 20002 Kbps Codec:Xvid MPEG-4 Codec
Audio: 6 Channels / Codec: ATSC/A-52 Dolby AC3 / Sample rate: 48000 hz
Bitrate: 448 Kbps

Video 2: Lord of the rings: fellowship of the ring
All parameters are the same except for the Video bitrate, which is 21615 Kbps.

Video 3: Lord of the rings: Lord of the rings: Return of the king
(split into 4 files)
Video:[AVI] Codec: Intel ITU H.264 (?) never seen this one! Br: 20779 Kbps
Audio Parameters are identical, except the Bitrate is 384 Kbps.

All 3 play perfectly on the 360 when I'm playing from a disc, but my network isn't strong enough to stream them flawlessly. So I know that of the 2 video codecs shown, either one would be perfect for HD videos that have 5.1 audio. These are examples of the perfect END-product. My question is this: What's the easiest way to achieve this starting from an MKV file with 5.1 AC3 audio and H.264 video codec? I'm not worried if I have to re-encode the video, as I have a very powerful machine. Any advice on what program(s) to use would be great, as most of mine are outdated. I imagine since one of the videos HAS H.264 codec and plays fine, I will simply have to re-encode the audio and remux it into an AVI container. Please post any advice on this process if you have information.

Just because they don't have a 360 profile anymore doesn't mean you can't encode videos for it. The "Normal" profile works just fine for 360 or PS3 from a H.264 video standpoint. Audio is the only slight hiccup. I was originally using 5.1 AAC, which works OK on the PS3.

But after discovering the awesomeness of WMC Extender on the 360, I'm using that as my front end. Problem with that was the 360 only plays my files with AAC at 2.0 stereo.

I believe the answer is with AVCHD (.m2ts extension h.264 + AC3 5.1). Using Handbrake all you need to do is use the Normal profile and select AC3 passthru for the audio track. You should output to mkv from Handbrake, but it doesn't really matter since we need to change the container to m2ts anyway.

Then use any of a number of programs, I use tsMuxer, to change the container from mkv to m2ts. The container change only takes 3-5 mins so not much added to your normal ripping process. At the end of the day you have a .m2ts file with h.264 and AC3 5.1 that I believe plays flawlessly on both the 360 and PS3. The only thing I need to go back and double-check is whether the 360 was playing the audio at 5.1 and not downmixing to stereo.

What software are you using for the 360? Tversity/WMC/Twonky?

I tried the above but nothing shows up in my folder for the small file I converted.

"All 3 play perfectly on the 360 when I'm playing from a disc, but my network isn't strong enough to stream them flawlessly."
whoops. The 3rd video with H.264 video actually deosn't play, but the first two do.

What software are you using for the 360? Tversity/WMC/Twonky?

I tried the above but nothing shows up in my folder for the small file I converted.

I'm using WMC Extender with the MediaBrowser plug-in. The files play fine out of there. I did notice the other day that when going through the traditional video folders on the regular xbox interface, no files show up. Kind of strange really.

Also I have found that .m2ts is not the end all be all (what is when it comes to streaming right?) It seems to have some kind of bandwidth restriction on the 360. Any higher bitrate videos I have that go over ~15Mbps at any point seem to stutter and flake out. It's not a network issue. Convert the exact same file to a .wtv container and no such issue.

The only real reason I see to use .m2ts like I do is if you have multiple streaming devices throughout the house, i.e. PS3s. Since I have a mix of both .m2ts is still the best option. If you're only using the 360 in WMC extender mode, .wtv or .dvr-ms is the way to go.

If you're not using the 360 in WMC extender mode, well then I don't know what the best options are.