Go to a place that sells camera equipment and buy yourself something called a Lens Pen. I got mine at Best Buy for US $10 or so. The directions on the package are self explanatory.
Alternatively you can use a very clean, lint free cloth. Moisten the lens/filter a bit with your breath and wipe.
The Lens Pen seems to do a slightly better job removing oily residues, and is more convenient to use.
If you are in a very
gritty environment you may want to used a few bursts of compressed air first.
Now, go and smudge no more!