Its only insane if you do not enjoy it.
You've gotten some good advice here, but keep in mind it is from the angle of enthusiasts who are addicted themselves
Unless you are independantly wealthy (limitless cash and no need to work for a living) any resources (time, money etc.) you dedicate towards this hobby will come at the expense of something else. Here are some tips to make the most of your resources and hopefully make your hobby more paletable for the people in your life (assuming they don't have the same interest).
1) Don't be afraid to buy used, especially early on in your hobby as you figure out what you do like.
2) Be aware of "low lying fruit", a $500 speaker set properly positioned will sound better than a multiple times more expensive speaker set positioned incorrectly. Sometimes simple modifications can work wonders.
3) Beware of "snake oil", cables, power conditioners, isolation pads etc. that cost multiples of what perfectly capable components cost.
4) Reseach your purchases to narrow down your choices and then AUDITION every significant purchase. Especially speakers which can be subjective, some folks love horns, I can't stand them, figure out what you like by listening.
5) The laws of physics cannot be broken. Quality (as defined by most on this site) low & mid bass is not possible with a HTIAB.
6) Speaker technology does not advance as quickly as electronics so buy smart and dedicate an appropriate amount of your budget to speakers, you should keep them for a long time.
7) Find a piece of music you love that is dynamic, well recorded, and indicative of the genre you listen to most. Consistently use this piece of music to benchmark the systems you hear. Take it with you shopping, bring it with you when visiting another audio enthusiast.
8) Protect your hearing. If you love this hobby beware that tinitus is real and at present an irreversible condition.
9) The room you are in is the most important component of your system. Learn as much about acoustics as you can so that when you are in the position to build a room from scratch you can have great sound.
10) No matter how much you spend, there will be something available that is better. Figure out what would make you happy and if its within your means then go for it, if not, be wary of making short term comprises. Sometimes its better to wait.