Not at all.
If you were to play you music files from raw .wav audio, you would be looking at 10 to 15 MBytes per Minute, which translates to 80 to 120 Mbits per minute. You .ogg files are considerably smaller than that.
You USB 2.0 (if indeed that is what you have) supports 480 Mbits per second, or 60 Mbytes per second throughput.
.wav audio 80 - 120 Mbits per sec
USB I/O 480 Mbits per sec
.ogg audio 20 - 30 Mbits per second (just guessing, assuming an ogg file is 7 to 12 MBytes for a 3 minute song).