Worth asking, but are you using the HDMI output of the DirecTV receiver? If not, then have you considered going to a different DirecTV receiver?
The H12 receiver is a standard definition box which offers composite video output with analog audio (yellow/red/white RCA connections), and has a BUILT IN MODULATOR. So, you can feed other TVs from the modulator directly on the box itself. No external modulation is necessary and you don't have to split the video signal.
Considering the price, it may just be worth it to pick one up to use instead of dealing with all this other headache.
Worth saying, you should try the TV you want to use and connect it to the modulator locally with the modulator connected directly to the DirecTV receiver you currently have. No splitters, no long RF cables, just a short point to point connection with the modulator in between.
If the image looks great, then you can start adding things one-by-one until you lock down into what is causing the issue.
I mean, it's really unusual in a home for a RF cable to go from one room to another location, so I want you to be 100% that is all that it does - as you said "coax cable which goes under the house and into another room" - just want to be sure that's a point to point connection.
Now, keep in mind this is all standard definition TV and not HD if you are just using the composite video outputs. Also keep in mind that standard definition, modulated to RF, is about the worst possible video signal you can get. So, any HDTV really will miss out on potential quality with this stuff.
If you are set on all this modulated stuff, then you may want to consider a quality used agile modulator...
Blonder Tongue is commercial grade stuff used in major installations and any pricing under $50 on their modulators is ridiculously low with very high quality.