Keep good care of your books during the semester. College bookstores and independent textbook stores will refuse books that look damaged. If the goal is to be rid of these books, you do not want to give the returns associate an excuse not to take them.
Separate out the books you know will not sell. If you have a book with a missing back cover, don't bother bringing it unless the store is willing to recycle damaged books, thereby taking it off your hands. If you have friends who took the same courses, you can often hear rumors about which books have been discontinued by the instructor.
Go early, right when the returns period opens. The longer you wait, the more likely the store will have met its quota on certain titles. Being at the front of the line on the first day lessens the likelihood of this.
Be prepared to receive a lot less money than you paid at the beginning of the semester. The pricing policy changes from college to college as to why you'll receive a fraction of what you paid; however, in nearly every case, you will receive less money.
Put the remaining unsold books on an Internet auction site. If they don't sell, you'll still be charged listing fees, however. Timing is everything. Typically, at the end of the fall semester, you should wait till the end of summer unless you know the text in question is being used in a summer course. However, if you hope to sell the books, make your asking price a lot lower than the bookstore's shelf price for used titles. Remember, you are competing to sell the volume in question.
Make fliers with pull tabs at the bottom. List the title of the book and the condition. Put some sort of contact info on the pull tabs, but since this is very public, use an email address where you don't mind having to routinely clean out the spam filters. Post these fliers everywhere it is legally accepted to do so. If your college has a policy against posting textbook fliers on certain bulletin boards, follow it. The flier does have your contact info, after all.