First off, thanks to all for responding.
It's clear I need to provide more detailed information. I will attempt to do so here.
Here is how far I have gotten on my own before I got stuck.
The speakers will go in my mancave. Dimensions and listener placement are in the image below. The mancave is under the roof, hence the angled ceilings. I only modeled part of the ceiling, but you get the idea.
The speakers I designed are supposed to challenge my woodworking skills. I designed large cabinets in SketchUp, with curved sides for extra difficulty.
I went through a couple of designs, but the above one is the winner regarding looks and size. I also like the idea of dual 10" woofers. It seems like just the right amount of overkill.
To find out if what I want to do is at least possible to make, I built a feasibility study box, on the same dimensions, except the height. Images below (sorry about the messy workshop):
I am confident at this point that I can make the cabinets. It will take 14 days, and 3 bottles of glue, but it can be done.
Driver wise I have the following preliminary selection:
2x RSS265HF-4 10" Reference HF Subwoofer 4 Ohm
2x RS150-4 6" Reference Woofer 4 Ohm
1x RST28F-4 1-1/8" Reference Series Fabric Dome Tweeter 4 Ohm
1x 4" Precision flared port kit
Cabinet volume is roughly 150L, give or take a couple of L. I have not calculated the exact bracing volume yet. Dropping the box size to 140L in WinISD did not change the graphs much.
I modeled in WinISD:
In the above model, I sent the maximum power to the midranges (80 Watts) and matched the woofer SPL to that. I ended up with 160 Watts going to the woofers to model flat. In that box, max power for the woofers is +/- 420 Watts, so 160 should be no problem. Driving the midranges at max RMS power can be an issue, but it models to 107dB, and I don't want to go over 100 dB. I figure that sending 25% of rated RMS power to the midranges would still yield acceptable distortion numbers.
The speakers are designed to have the tweeter at ear level when I sit in my most comfy seat. I had my girlfriend measure.
Regarding the box volume and port tuning, I have the DATS measurement system from Dayton which I used in the past to measure the real tuning frequency of the speaker+cabinet for my subs. It seems WinISD is always a bit off with the real tuning...
I will be 2.4m away from the speakers.
Speakers will be angled 60 degrees apart and face the listener position directly.
I am looking for 100 dB.
Cabinets can be large.
I have only considered Dayton drivers at this point because I am happy with the ultimaxes I have from them. I can expand to anything available on the soundimports website, although it seems Dayton is the more reasonably priced brand.
Please educate me on where I went wrong. I appreciate all feedback, including "it sucks, start over" (in that case please tell me why).