You are in a similar situation to where I was on my main PC six months ago. I really wanted more than 6GB of ram with my Phenom II X4, but DDR2 prices are obnoxious. What I ended up doing was getting a new AM3 board for my Phenom II, transplanting the CPU, dumping a bunch of DDR3 in there, then upgrading the CPU to a FX-8350 (also AM3) later. It let me do it in stages and left me with parts to put together a Steam box.
Games still want three to four cores. The Phenom II X6 is a waste. Throwing any additional resources at a DDR2 motherboard is also a waste. Here's what I'd do:
If you want to spend the cash all at once, then get a $120 Intel board, a $215 i5 3570, and $80 of 16GB of DDR3. Drop that in to your HTPC and move the Phenom II X4 back to the desktop. Combine what is hopefully 2x1GB and 2x2GB for 6GB of DDR2 on the Phenom II, or dump the DDR2 board and grab an $80 AM3 board and another 16GB of ram for $80 to go with that Phenom II X4.
If you want to do this in stages, then dump each board when you can. Grab a pair of AM3 boards with 16GB of DDR3 each. Then upgrade the Athlon II X2 to an FX-8350 for use in the HTPC. Keep the Phenom II X4 in the desktop. Yes, I just said games only want four cores and a six-core Phenom II is a waste, but the eight-core FX-8xxx is the only way to get four FPUs on the new architecture. The FX-8xxx is the direct replacement for the Phenom II X4, not the FX-4xxx series.
In either case worry about the video cards later, and go Nvidia next time to get drivers that aren't crap.