People who are learning English as a second language often have a hard time pronouncing words, especially if the bulk of their education has been by nonnative speakers. This creates problems for an ESOL teacher, as high-level English students may have impeccable grammar and a large vocabulary, but still have a hard time making themselves understood. Teaching becomes difficult because the teacher then needs to teach pronunciation at a profoundly different level then when teaching reading and writing.
Another common frustration is the fact that many ESOL students have better speaking skills than they have listening skills. This means that students will give the appearance of proficiency --- as evident by the fact that they can talk with high-quality pronunciation and a large vocabulary --- but it is only superficial. Speaking English is often easier than listening to and comprehending English, so one difficulty with communicating with ESOL students is the fact that you will often understand far more of what they say than they will of what you say.
People who have just started to speak English as a second language have no English skills. This means that you need to completely change your teaching style with these (often young) learners. Since they cannot understand a single word you say, you need to make all of your instructions extremely simple and make sure that there is a physical component to essentially everything you do.
If you speak the learners' native language, it is tempting to lapse into it to give instructions and explain things. You should not do this, though. Instead, give instructions in simple English; over time, the students will start to understand you based solely on immersion.
A final difficulty in teaching ESOL learners is only really applicable to people teaching in the ESOL learners' home country. Education is a pervasive part of everyone's life in most cultures, and it is therefore an area where cultural differences are highlighted. These differences can be frustrating at times for the ESOL teacher, as the differences can seem wrong, where in reality they are just approaching things from a different angle.