It was the pics on this forum (I usually frequent Pulp and a few others) that got me interested. I picked up a penguin book - Call of Cthulhu, and other stories - and have been hooked.
I'd always enjoyed Stephen King, more for the writing style than the plots, but the thing that I liked about King is that he understood the necessity of exposition in horror. Scary isn't the thing that jumps out and tries to eat the protagonist, scary is the vaguely unsettling story that the old man at the corner store tells the protagonist. And Lovecraft? His stories are almost all expository, and therefore almost all frightening.
My favorites read thus far were the ubiquitous Call of Cthulhu and Shadow Over Innsmouth (both in the Penguin book). Both are very well written, exciting, and chilling. I just picked up At the Mountains of Madness yesterday and am thoroughly enjoying it so far.
If you don't mind a giant, heavy, fairly small print book, Barnes and Noble has a lovely "complete works" for 12.95 in Hardcover. I didn't get that one because I got their Conrad and James Fenimore Cooper volumes and they were a pain on the subway.