The length is really not so much the cost of the cable, check my figures I posted above. Your length of 10' is only around $17-$20 in Canare or Belden cable, which you can't buy better. Most of the cost is in the good quality BNC connectors. I'm sure you could find cheaper BNCs, but you get what you pay for.
I agree, for shorter lengths of wire, especially with low bandwidth digital signals, almost anything, including a coat hanger will carry the signal just as well as a $600 cable. Problem is with digital, as you go to higher and higher clock speeds the cable becomes more important. Look at DVI, the cables can't even be made by a human because the clock speeds are so high that you cannot have the different channels varying in length by even the smallest amount, or else you get data errors.
Its best to match the proper impendence when you can, so I wouldn't use a 100 ohm cable for video, especially on long runs. I also wouldn't use 50 ohm BNC connectors (in trying to save money), as they can introduce ghosting, etc.
Many cable companies may actually spend less on the wire. That sheilded VGA cable you got from Best Buy most likely cost $1 to make, but has a huge profit margin in it for everyone in the retail chain. The other cable you got off the 'net most likely cost $2 to make, and has a much lower profit margin. That $1 difference in materials is mostly why you are seeing a better signal, its most likely better shielded (higher braid and foil coverage) and uses better wire.
I'm pretty sure BetterCables, Rivercables, etc. rebundle Canare or Belden wire (or both depending on what they can get at their target price this month). They charge higher prices for marketing and fancy cable wrappers. Some of the raw stock for the cable wrappers they use is more expensive than the wire itself on a per-foot basis... Rivercables was started by a group who is also a Canare/Belden reseller for the pro-av market... Makes you wonder just how much profit is in the cable market when you see them starting this venture.