Even that may not eliminate the difference entirely. Most CDs are overly "hot" and compressed, in that they are tweaked so that every sound is as loud as possible (as loud as can be encoded into a 16-bit digital sample). The song you're listening to from CD may include whispering or other portions that are intended to be soft, but they'll be made as loud as every other section of the song in a vain attempt to make the entire thing sound "loud".
This is referred to often as the "loudness wars". It results in an unnatural and fatiguing sound, and can actually cause peak sound levels to be lower because you'll be unable to turn up the volume as far. It's hated by most audiophiles, and is increasingly being recognized by others as damaging the quality of music (but not recognized as quickly as we'd like!). This overly-compressed, loud sound is also found in most television and radio broadcasts, so those will often seem 'louder' than LPs, too.
Most vinyl LPs pre-date the "loudness wars", and therefore don't suffer from this crazy push to make every note of every song sound as loud as possible. This, IMO, is the primary reason why the LP versions of classic rock albums often sound better than the CD versions -- especially when compared to recent "remastered" CD versions of those albums, which are often worse than the original CDs.
On your original topic -- when you tested the turntable with the switch in the "phono" position, I assume that you connected the turntable to a phono input on your receiver/pre-amp? (i.e. you didn't leave it connected to a CD, tape, or other aux input?). If you instead left it connected to a CD/tape/aux input with the turntable's built-in phono pre-amp disabled, then the overall volume will be extremely low and bass/treble response will also be off.