This guys cover gallery has a good listing of Cap. appearances:
http://www.medinnus.com/winghead/comics/index.html
Not much of the Golden Age stuff has been reprinted, however the later stuff (issue #100 and up) can be had cheap.
Marvel started reprinting old title in black and white a few years ago, calling it the "Essential" line. Currently, there are 2 vols. of Cap out that reprint approximately the first 50 issues of Silver Age stuff. You can get these books cheap. Check
http://www.instocktrades.com first, as they'll be the cheapest.
They run around $15 for 25 issues and they are great, as long as you don't mind the black and white.