There are two levels of signal you're dealing with. Line level and speaker level. Line level signals are much, much lower in amplitude than speaker level signals. It's like trying to power a tablesaw with a nine volt battery. Sure, a bit of current will flow, but you can't get any useful work done.
There are many amplifiers available for your use, and you should be able to obtain something quite inexpensively either used or even new. Amazon for instance has the "Lepai TRIPATH TA2020 Class T Mini Amp Hi-Fi Stereo Audio Amplifier" for 20 bucks.
Part of what fools people is that cheap speakers (the kind found in Home Theater in a Box bundles and such) sometimes use RCA connectors on hard wired lines. You won't find such connectors on amplifiers or better speakers. In order to hook up your speakers you'll need to either cut off the RCA connectors and use the bare wire, or find or make a female RCA -> bare wire adapter.