Without a lesson plan, you essentially have to start over each quarter or semester when it comes to devising a curriculum. Granted, you may have a general idea of what you did previously and what you wish to do the next time, but if it's not written down, you create more work for yourself. In addition, you won't have the ability to make notes on the plan in regards to what works and what doesn't. Lastly, there's the issue that you may teach it differently every time, which can become time-consuming for you.
You're only human, and life will inevitably pull you in all sorts of different directions, be it in the form of illness or family emergencies. When you're not able to be in the classroom, the absence of a lesson plan can create disorganization since your substitute will at best only have a general idea of how to carry out the objectives of the day.
A lesson plan keeps you on schedule and helps to foster order in the classroom. Not having one leaves you vulnerable to getting sidetracked. Sometimes, it may be that you have a particularly deep, discussion-rich type of topic that causes you to lose track of time and miss covering other important aspects of the material. Other times, without a lesson plan to keep you honest, you yourself may feel compelled to add extra points or veer off on tangents.
The students themselves are also affected by not having a lesson plan. It's one thing for disorganization to make your job harder, but it's entirely another when you compromise the value of your class' education. For starters, your credibility can be undermined if you appear to not be on top of things. It's even worse when you create extra work for students based on an in-class error on your part. For instance, if you make students take a quiz that they have already taken, it can discourage their learning under your tutelage.