I see three challenges. First is bookshelf speakers in niches. Most bookshelf speakers are rear ported and need a good 18-24" behind them and some room to the sides to get the full effect of the bass port. That leaves you looking for front ported or sealed speakers.
The second challenge will be the center. One of the things that you want to avoid like a plague is a timbre mismatch between the L&R fronts and the center. You want a seamless transition between the mains and the center and back. I just don't know of any ceiling speakers that are timbre matched to front ported bookshelves. Maybe somebody else will have an idea but my recommendation is that you rethink the center.
The third challenge is price. When dealing with a tight budget I usually recommend splurging on the front 3 and then adding the surrounds later.
Speakers are very subjective but I'll toss out some ideas to audition. I'm not a huge Klipsch fan but if you want a rock/party/movie speaker that's front ported and will fill a large room then I'd try to audition a pair of Klipsch RB-61 bookshelves ($500/pr
). Some find them a bit fatiguing after long loud sessions while others are head over heels in love with the sound. The correct center to match these is the RC-62.
I run a pair of Usher S-520 bookshelves (~$400/pr
) in my master bedroom. These are more of an audiophile speaker. They sound bloody awesome for their price - as long as you can live with their limitations. As a really small speaker they can't be expect to blow the windows out of a huge room and they need a sub to deal with everything below 80hz. If you don't provide a sub you'll bottom the woofer at high volume and damage the speaker. They do a great job with both movies and music in my 18x12x11' room. The matched center is the S-525 and that's the second limitation. The center is 4ohm and needs a 4ohm rated receiver.
A step down in price and sound are Infinity's Primus P162. A decent front ported bookshelf for less than $200/pr
. These are common enough that you should be able to find a pair to audition. The matched center is the PC350 for around $200. Alternately you could go with the P362 towers for the front L&R and avoid the niche issues and have (2x P362, 1x PC350, 2x P152) for about $750. I'd run these with a 4ohm capable receiver even though they are rated at 8ohms.
You might also look into Emotiva reference speakers. I haven't personally heard these and so can't vouch for them but reviews of their bookshelves have been good. The reason that I bring them up is that they also make ceiling speakers and may have a match for their sealed bookshelf speakers. The downside is that you'll need a 4ohm rated receiver.
Receivers: I'd look at Onkyo's TX-NR708. It's a good quality receiver with a ton of features and if memory serves is rated down to 4ohms. If you stick with 8ohm speakers then the TX-SR608 is a great choice. If you're in the US you can often find these refurbished.