Here's a recent comparo of lots of phones including iPhone 5/4S/4 and Galaxy S III
Most performed well, with the iphone 4S winning out overall, particularly with higher impedance headphones. Biggest surprise was that the iphone 4S even beat the newer iphone 5. Another advantage of the 4S over the 5 is that the old dock connector includes a line out connection for bypassing the internal headphone amp of the iphone when connecting to an outboard amp. Only way to get true line out on the iphone 5 is to use a lightning to dock connector adapter and then a dock connector to line out cable. It works, but it's bulkier than it has to be. The Fiio E02i Rocky inline amp is a decent compromise that works with the internal amp to give it some extra headroom, and it's very portable. It's no substitute for a real portable amplifier like the E7, E17, E11, E12, though.
With lower impedance headphones, there is a large group of apple and android phones that all perform very well. If you want to use in-ears, spend your money on nice headphones and just get whatever phone you like.
My setup is an iphone 4S -> Klipsch S4i Rugged when I'm walking around, and an iphone 4S -> E02i -> Sennheiser HD 380 pro when I'm on a plane or amtrak. Most of my music is ALAC ripped from CD or higher res sources downsampled to 48/16 and saved as ALAC. I wouldn't mind nicer in-ears and a better amp down the road, but my current setup is giving me pretty good bang for the buck.