Yup, your yellow finches that you see in the summer are these American Goldfinches in summer plumage. I doubt they stopped coming, just changed colors! I have finch feeders outside now, and they're covered in Goldfinches, so go ahead and put out your finch feeder if you want to attract these guys.
On another note, do you have a good bird identification guide? I was wondering, since you're new to birds, but having one can make bird feeding a lot more fun. It's a nice feeling when you have the means to identify all the birds that come to your feeders! If you don't have one, and you want one, I'd highly recommend the Sibley Guide to North American Birds - it's a really fantastic reference for bird identification, and you can get smaller versions of this guide for Eastern and Western North America. For just feeder visitors, though, this site has a good ID poster that means that you don't have to pick through all of the birds in Eastern North America to find a Downy Woodpecker:
Welcome to Whatbird!