The major problem with using the 3ohm AVR with 8 ohm speakers is NO POWER! If the system is 30w (3ohm) it will be like 12w (8ohm). I would hate to see you buy nice speakers and then destroy them by feeding them crap power.
Start with separates. The Denon 1610 or Onkyo 507 would be fine. Going with older models (1609 or 1908) Denons/ Onkyo 606 or 605) can save money if features don't matter much. These won't crap out when the power is turned up to a moderate level.
Speakers. Start with 2.0 or 2.1 for upgrades. Add the rest later. A good 2.0 or 2.1 system will destroy a crap 5.1 system. If you are using your TV speakers now it will be a huge improvement. Paradigm Atom or Mini speakers would be nice. Energy makes some good speakers for low money. Honestly there are a ton of choices that will destroy HTiB speakers.
Blu-Ray players are very cheap. I personally like the Panasonic BD-60
. amazon has a great price and it upscales DVD movies very well.
Subwoofer from Premier Acousics, eD, and a few others will be no comparison to the subs that come with HTiB.
If you use this forum to research you can build yourself a killer system for the same or little more than your budget. It will be a large improvement in performance for you. When you spend a lot on an HDTV, you should try to match the sound to complete the experience. My sound system cost WAY more than my HDTV. In fact I have more in my speakers than the three HDTVs I have bought the last 4 years.
To me it sounds like you don't want HTiB. Those systems have their place but they really aren't built to expand on. Good luck, and have fun!