I think the problem is your bitrate, I have no problem playing mpg files that are around 8-10mbps but HD content is still proving to be finicky, either it won't play or it stutters.
I find that if I compress down to 8000 or so the files play fine, haven't really spent a lot of time messing with it though.
Also it might help to copy the file to the ps3 first, you can do this when using DLNA just by hitting the triangle and selecting copy. Apparently some people have luck copying files that won't play over the network (ymmv).
edit: xvid works fine for me as well, generally, some older stuff doesn't