which older Mini do you have?
My concern is the HD in the mini is not nearly larger enough for everything on the server (1TB of movies; 250G of Music; 250G of photos; and 200G of files).
depending on which model Mini you have, you may be able to add a second drive inside, but regardless it is very easy to add external storage to any Mini over firewire and USB. Simplest and cheapest connect a relatively inexpensive 3 TB external to your Mini, a Seagate or WD can often be had for $120 or so if you shop well. Buy two, keep one connected to the Mini and keep a full backup on the second, unplugged and disconnected from the Mac and power source.
yes I've done a search on the forums and while I've seen several people talk around this topic, no one has actually posted any results or performance issues.
Something simple and relatively inexpensive for you to consider...Apple supports something called AirDisk, plug any number of external USB drives into an Airport Extreme base station and those drives can be shared. Works great, I have 12 TB of movies, TV and MakeMKV rips stored on 3 enclosures connected to a USB hub, that hub in turn connected to the Extreme, and all of that is easily shared around the house. Any Mac in the house can be set to mount those volumes automatically at boot.
That means you can keep one copy of your media connected to the Mini over USB and a second copy of your library connected to your Extreme over USB. Double coverage if you will.
We have a 2011 Mini functioning as both player (with XBMC) and media server for the whole house, it boots off of a fast Samsung 830 series SSD and there's about 12 TB of media locally attached over firewire 800 and USB as well.
Will the USB connection be too slow to feed up BD MKV rips...?
No, plain old USB 2, even "AirDisk" USB, is perfectly fast enough for BD MKV rips, as long as what you are streaming to is also wired.
Hook up a 4-bay enclosure to the mini and host all of the files on there...Use OSX to run a SW raid over the 4 HDs in the enclosure (JBOD)
Not too long ago, the "in" thing around here was to get a 4 bay ReadyNAS or something similar and run RAID 5. There are plenty of threads discussing that approach over the years, and what happened when Infrant was bought by Netgear. Lots of threads about the Drobos, and also about something called unRAID. Look in the Windows HTPC forum and also in the content streamers forum, because those conversations tend to take place over and over again. Everyone eventually gets to the point where their content outgrows the few drives that they have and it's only natural to wonder hey, what's the best way to have all my content in one place that everyone can access. There's no right answer for everyone.
Anyone have any good recommendations on a decent 4-bay (or larger) enclosure that isn't $500? I've read up on Drobos and sounds like why they are nice they have serious performance issues during playback and as the drive space maxs out.
I don't have a 4 bay model, though I have had a lot of success over the years with 2 bay aluminum Stardom/Sans Digital models with FW, USB and hot-swappable drive trays...I have 4-5 of them and they've been rock solid.
There's a lot of love for Synology Diskstations here at AVS, perhaps look for threads on those, the 2 disc model is quite affordable and would easily meet your needs if you wanted a NAS rather than direct attached storage.
Inexpensive 4 bay USB enclosures, the kind you see on Amazon for a little more than $125 from MediaSonic or Sans Digital, which can hold any combination of 2 or 3TB drives, will work just fine for your needs in JBOD mode, and there's no need for software "RAID" to complicate things. Of course, you can choose to set things up for some sort of disc failure/redundancy, but even if you employed RAID 1 or RAID 5 or a ReadyNAS/Drobo you'd still need backups, you should have some form of insurance to be able to recover if something takes the whole enclosure down. One copy local and attached, a second copy local and disconnected, as much as you can have on iCloud or something like Backblaze, and another copy kept off-site.
Fans, heat, noise, all issues that are very important if you plan to attach an enclosure locally to a Mini in your living room, not so important if you can put the Mini out of sight in a closet somewhere, to run solely as a media server, as you are planning to do.
Can't offer you any advice about a Boxee box, I only have Macs and Apple TVs (some stock, some hacked) throughout our house. I'm probably not a good source of information for you, though, for several reasons--I'm happy to do things, largely, the way Apple wants me to do things...I keep a Mini running 24/7, that means my iTunes library (mainly 750GB of music) is always on and readily available for home sharing to our other Macs, Apple TVs and iOS devices. I probably have 25TB of movies and TV archived, less than 1% of it is "in" iTunes, because of the Mini and AirPlay and apps like Air Video we can watch just about anything we want wherever we want on whatever device we pick up or turn on. I haven't ever tried the "have everything stored in one place on one device" approach.