reedl said it, the real issue people accessing your files OR you accessing their files and or viruses. I used my Linksys Wireless Home Audio to network to my brother and father, it works great but I trust the "security" of their networks. I couldn't imagine doing anything but streaming content from someone else and I suspect that would cause some network lag if you had enough people accessing your files at once.
I had considered coming up with a system on my own but decided to go with a pre-built solution in the end. My choice came down to the Linksys or a SONOS system, the deciding factor was the Linksys was N. My brother has a SONOS music network and there was no issue in getting them to talk.
Both solutions let's you access Rhapsody, Pandora, lots of other internet radio (about any statio you can get on WinAmp) and access songs from your iPod. For me songs have not been an issue, I usually play "I bet I have that song" when friends come over and I've won every time.
I must admit the Linksys looks kind of pricey but for what it does it's more than worth it.