1) In terms of the R8a's the price looks to be mostly due to the matching finish of the R series. I use the cheaper Q50a's personally in my setup which retain the same angled cabinet shape if you're wall mounting these for surrounds. If you're not wall mounting, you can easily go with cheaper bookshelves like the Q350 on stands. Also as an option, you can try and track down the now discontinued R50's which have been replaced by the R8a earlier this year, whether it be through your dealer or the used market.
2/3) Although I personally haven't tested them in home, I've researched the 12b & R400b as potential options in the past 6 months for my living room setup for the nice finish and smaller footprint. Going by reviews and measurements though, these subs just don't stack up to the other internet direct companies subs like most here will tell you when it comes to performance and pricing.
4) The need/usefulness of seperates will definitely depend on your volume and listening habits. If you're going to get one of the higher end Marantz/Denon units, they may be perfectly sufficient for your needs. I have always used bookshelves and never ran full towers before though, so in that application maybe someone with better knowledge can chime in. If seperates are the way to go - then it would allow you to just stick with the AVR that has the necessary processing channels that you need (7,9,11,etc.), and not worry so much about going for the highest powered unit. I will mention in regard to Denon models, I know they are currently rolling out the 2019 models, so you may be able to save some of your budget with a 2018 model on closeout pricing or something... again that would be dealer specific.
Like pase22 says, there's a really good selection of subs, especially if you're located in the US and don't have to worry about crazy shipping costs and customs fees. Most people will recommend ported subs for your application, and multiple of them. It will mainly come down to your budget, and if other things matter like the size/finish/footprint of the cabinet then you might want a sealed option. For example, because I have a living room setup, and want to move my sub placement to the front soundstage - I have been specifically looking for a smaller footprint sub with a nice finish to match my KEF R300/600c combo. I was pretty set on the SVS SB3000 (since I am in Canada all the other sub companies I contacted for final to the door pricing were crazy)... but with luck I came across someone with a used ad selling his 2 HSU ULS-15 mk2's, and I will be picking one of those up today! It allows me to move up to a 15" driver with a decent looking box that's a smaller footprint than my current ported model.