It can take a while for birds to return to a feeding station, esp in times of abundant food supply in nature.
At our feeders, the house sparrows chase the smaller birds and defy every bit of advice that I've ever read about what house sparrows don't like.
The house sparrows and house finches do eat the safflower we put out. We haven't had the goldfinches, chickadees, or titmice return in quite some time.
You might want to add a dripping water source to the area. The sound also attracts birds. Also there are feeders which have a grid outside them, so only smaller birds can get through. Not sure if this would work for you. It's useless for discouraging house sparrows.