I've been disappointed with Plex and the Roku. I don't understand why it has to transcode all the time. Just freaking play the full bitrate file!
The ROKU has a strict conformance to an HLS streaming protocol. Even though it can support a 1080p MKV file via usb (R2 XS), PLEX has to do 'transcoding' to break it up into HLS packets just so the ROKU can receive them via a channel. There is also a mechanism by which the source transcodes at a lower quality if the connection cannot handle it. By limiting to that protocol, Roku helps ensure channels work by placing the burden on the channel provider. That is good, but it also means that there won't be a solution that does not require transcoding of some sort.
I use the term transcoding loosely above, as it still can be H264, just packaged differently.