It looks like there have been a few options mentioned, but I wanted to offer a personal recommendation for the host I've patronized the last five years (and no, no affiliation; a web designer recommended them to me years ago and I am simply doing the same
http://www.asmallorange.com/
I use their small plan, which is $5/month. It is month to month, no contract, and can be paid via PayPal. They are also rather lenient on, uh, er, late payments. Not that I would know, of course.
I've experienced down-time once in the last five years, and they generally reply to support questions within an hour or two.
They offer a CPanel interface (
http://en.wikipedia.org/wiki/Cpanel), which is great if you don't want to use an FTP client or want remote access to upload files. It also offers an automatic back-up feature, which we should all, ahem, be doing regularly. I'm still working on learning that lesson, ha.
Oh, I guess got a bit long-winded there, eh? Now if only I could find a barebones cellphone plan that I was as confident about as my webhosting plan
Good luck,
Jenn