Looks like the booster is providing some cable equalization which is making up for some of the cable's deficiencies, but not enough with the receiver. Basically if this was a pipe, you are sending too much water through the pipe at one time and the pipe is leaking (best analogy I've got right now).
So, what you want to do is to cut down on the amount of data you are sending or fix the pipe. Every place you can find, disable "Deep Color". Make sure it is Off not Auto or On. That will reduce the amount of data and you won't notice any picture difference (actually it could improve the picture quality in some cases).
If that doesn't work try, temporarily, 1080i. If that doesn't work, I'd be surprised.
Once you get to that step, then we can discuss options.