He might be busy. College is a lot of work, and he may simply not have time to talk to you as much as he'd like. This is especially true if he's taking a heavy course load or if he's involved in extracurricular activities.
He might not be interested in you anymore. It's also possible that he's simply not interested in you anymore. This could be because he's met someone else, or because he's realized that you're not a good match for each other.
If you're not sure why he's only talking to you when you go first, the best thing to do is to ask him. He may be willing to open up to you about what's going on. If he's not, then you'll at least know for sure that you don't have a chance with him.