Question A - speaker size
How would you think about choosing between
— Book shelf vs Tower
— Types of tower, e.g. I’ve been told that the speakers like Kef R11 or Focal Electra 1038Be may overwhelm a room like this
Some of that is going to be a personal choice - a tower vs. a bookshelf. Space and/or placement limitations, aesthetic consideration, etc. If you can't run subs., you might lean towards a tower speaker that could be better for low end performance. But there are a lot of bookshelves that are very capable, and can compete with towers. It really depends on the speaker.
Sure, a really large tower might overwhelm a room, at least with its visual/physical presence, and could be considered overkill for the size of the space from an output standpoint. (Yes, you could do a lot less speaker than those two you mentioned, and fill the space plenty fine...)
I've been auditioning speakers but I'm confident they are going to sound entirely different in my room vs the listening room
They will perform differently in your space vs. the auditioning room, that's to be expected. It's a question of how much, and if you'll notice it. It's not often practical/possible to audition speakers of interest in your actual environment, but if it is - do so.
Question B - point of diminishing returns
— After a ton of reading and some listening I’ve narrowed down on something like: Kef R3 / R7 or 11s or equivalent Focals / B&Ws - still auditioning options + matching center & surrounds -- powered by a Rega Elicit R -> Denon x4500h: [Xbox / PS4 / Apple TV (steam) / Nvidia shield / Bluesound] -> LG 77C8 77inch OLED TV
Indeed, where is that point of diminishing returns? It's hard to say - a lot of factors that play into price. Of course sound quality, build quality, finish, name to a degree, etc. Only you can really determine that based on your preferences. If I had to put a dollar figure on it though, and this is clearly nothing more than an opinion, but the $3-5$K range is gonna get you damn nice out of a pair of speakers, at least it had better do so. And yeah, you can get excellent for less than that too.
Now to be blunt and I say this a little sheepishly I’m not super budget constrained - I could quite easily look at the Blade Two or Kef Reference 5 type budget and kick everything up a few notches - I'd be worried about room size so maybe the Kef Reference 1s (?) but you get the point. I know, even for audio, this is an insanely subjective question and you start getting into silly questions about what is the value of money, it’s all relative etc - and yes I will demo the Blades and Kef Reference 3/5s et al.
Sure, spend more if you'd like - it's your money - but don't spend it just for the sake of doing so and chasing a speaker line. Those are big jumps up from the R-Series. I'm going to recommend a brand for you to explore, IF you can audition them:
https://totemacoustic.com/
Especially with your mention of speakers that might seem overwhelming, have a look at the Tribe Tower, and don't let the small size fool you. But, if you want something larger and of course more expensive - have a look at the Element Metal towers, or the Element Fire bookshelves. I've demo'd all of these (but not the new Metal V2, which is an improvement on what I've heard) - and these are all exceptionally impressive. The Elements, while not inexpensive, outperform in my experiences at my former local dealer, speakers that out-price them substantially. So, just throwing that out to you as another option. Paradigm Persona line is also right up there.
That said, no one in my personal life is even remotely into audio or AV gear and garnering other enthusiasts opinions is really helpful as I think this through. I know myself and I know I’ll enjoy these toys more if I feel like I’ve tried to be thoughtful about it, e.g. the Denon would be a refurb buy. For folks that have been into this hobby for a long time I'm guessing people have upgraded and downgraded over their life and I'd love people's opinion on if they found a sweet spot or where people think diminishing returns start to kick in. e.g. in the PC gaming community there are some pretty well agreed upon price/performance ranges, (I know it's more measurable). Maybe I'm thinking about this the wrong way but it's been a thought that's been stopping me from pulling the trigger...
If you're spending big coin on speakers, I'd spend a bit more on a different receiver or pre-pro, rather than a refurb Denon. Not that Denon is bad, but you're going to a length it sounds like to treat the room (good!) to go along with those awesome speakers, then why not stretch that and go for something with arguably better room correction? Anthem with ARC, or something with Dirac will be a step up in my opinion. Ideally your room would be so awesome as to not need any kind of room correction, but you might want to consider this.
