my first smartphone was a Samsung Galaxy S1 (Captivate), and I tried to transition it to my main portable player, coming from a first-gen iPod touch and a Creative Zen Vision:M DAP. It sounded decent but it seemed to have some sort of volume limiter on it or something -- no matter which IEMS I plugged into the thing it just didn't seem to get very loud compared to my iPod touch and creative player. Also, I used it up until it got Froyo, and I can say during that time at least, that Android's native music player was garbage -- poor navigation, downrezzed cover art, etc. The cool thing about Android is that you can replace the stock apps, which lead to me download Poweramp which was really great. Like I said, that was using Froyo, so the built-in Android music app may have improved since then. Loved that I could expand the storage via microSD, but due to poor usability/performance, I switched to an iPhone. I've had three different iPhones -- 16GB 3GS, 64GB 4S, and now a 64GB iPhone 5. If you can deal with iTunes then I think the usability/UI is best with Apple phones. If you're looking to customize everything or are a power user, it might not be for you, but I have found my iPhone to be great as a media player.
Most of the music on it is ripped in ALAC (e.g. most stuff I haven't purchased via iTunes), and with a pair of IEMs it sounds great -- the type of phones you're using will be the largest determinant in the sound you hear. I use a few different IEMs on a regular basis (Klipsch Image S4, Martinlogan Mikros 70, Ultimate Ears triple.fi 10), and they all sound great. It sounds pretty good with my Sennheiser HD598s but it's a bit noisy and the volume needs to be turned up high.
ht Panasonic 60" ZT60, Monitor Audio: Silver RX6, RX Centre, Radius 90HD; Martinlogan Dynamo 700, Marantz SR5006, PS3, Oppo BDP-103D
2ch Sony KDL-32W650A, Sony BDP-S1000ES, Marantz PM8004, JVC T-X3 tuner, Monitor Audio Silver RX1, REL T3, Apple TV, Peachtree Audio DAC•iT, Sennheiser HD598
lr Panasonic 50" ST60, Sony BDP-S5100, Apple TV
pc Energy CB-5, Audioengine D1, FiiO A1