You won't find 75 ohm "cat 5" since cat 5 is a specification, and that spec says it has to be 100 ohms. The theory is that connecting the shield with the signal grounds bring the characteristic impedance down closer to 75 ohms. It's not clear to me how this works when one end uses BNC connectors.
If you have a bit of length to spare for an experiment, you should try replacing the BNC end with another HD-15 and seeing how it works using the Computer 1 input of your D-ILA. If this looks good, then you'll know that the BNC connectors are the culprit. I know that you need to use Computer 2 for the HD, but this experiment will at least tell you if it's the BNCs that are giving you trouble. Keep in mind too that 40' is right about at the limit of where this solution works well. My first attempt at one of these cables was 50' long with a female HD-15 on one end, which I connected to my Computer 2 input via a HD15-to-5BNC breakout cable. I was quite disappointed with the ghosting. I then cut it down to 38.5' with a male HD-15 and connected it to Computer 1 and it looks great. I was never sure whether it was the length reduction or getting the breakout cable out that was responsible for most of the improvement. Fortunately for me, using Computer 2 is not a requirement.
The "color smearing" may be a different problem. If you mean by this that it appears that the R,G, and B don't appear to be perfectly converged, then this is probably caused by the "pair skew" that Dizzman brought up in an earlier post in this thread, and which has been discussed some more in another thread.
If you have Dilard, you can use the Pixel Alignment Wizard to check this and adjust for it. One thought I've had is that with the proper choice of which pair is used for which color, you might even be able to use the pair skew to your advantage to compensate for slight panel misalignment that most D-ILAs have.