Depends on how you are getting things around.
I use wireless headphones plugged right into the back of the cable box. I turn down the sound from my main speakers when I don't want to listen to them (after kids are in bed) and I turn on my wireless headphones.
Your listening scenario is a bit vague, so while there may very well be a solution, I'm not sure beyond what I described you are trying to achieve.
Depending on your distributed system, you could make the headphones their own source and run them through a headphone amp or use wireless headphones.
Most headphones don't accept HDMI, but you often run your sources direct to a audio switching using analog audio.