The correct way to phrase it is:
"The book was written by me."
"I" is the nominative case pronoun, used for the subject of a sentence. "Me" is the objective case pronoun, used for the object of a verb or preposition.
In the sentence "The book was written by me," "me" is the object of the preposition "by."