I personally misdoubt you'll be able to drive any reasonable speaker to reasonable loudness with only 5 W (Paul Klipsch's famous quote aside). However, it may sound loud enough, especially since the single-ended tube amps distortion will likely be dominated by the bit smoother-sounding second harmonic distortion rather than the more typical (and worse-sounding) third... In fact, 1 W may be enough average power, but the ratio of peak to average in music is about 17 dB, a factor of 50 in power.
My concern over time is that clipping generates higher-frequency content that can damage your tweeters.
All that said, I love the sound of a good tube amp... - Don
"After silence, that which best expresses the inexpressible, is music" - Aldous Huxley