If youe 760 & remote arrived clean & unscratched then I wouldn't bother returning it. It's really quite a good little unit - far cheaper than an HTPC and nice'n'quiet.
ISO DVD files generally play well when mastered correctly. I don't know what sort of encoding you're doing, but check the bit-rate and the MPEG-2/VOB streams you're encoding into your home DVDs - of the bitrate is too high then you may choke the 760HD (and most other players). For MPEG-2 DVD (720x480), a bitrate of around 4.0 - 5.5 is appropriate.
Another quick check is whether your home DVD's frame rate matches the 760HD's setup... if your TV/LCD supports both NTSC and PAL, then try switching the 760HD into NTSC (if currently in PAL) or PAL (if currently in NTSC) and then try playback again.
The black screen at the end of playback is a known issue with the current firmware. Generally pressing STOP, RETURN or POWER-OFF and the end of the movie will get you out of the blank screen the fastest. Like Burnerbum said - it usually takes 30 seconds or less.
Most cheap media players have their short-comings - the MVIX is no excpetion. It is, however, built on top of a really good hardware platform (the Sigma Designs chip inside is a real video power house), which means this: Hang on to your unit if you like it... because there are some of us who are looking to rewrite the firmware to take full advantage of that amazing hardware.