Relax. If you put too much pressure on yourself to speak English perfectly, you create excess tension, and this will affect how you speak. Calm down so the words flow more easily.
Prepare. Allocate a specific conversation topic to review each week. Focus on building up a set of key phrases and vocabulary, so you can enter into the conversation more prepared. For example, take the subject of music. Learn how to describe your favorite bands, explain a gig or festival, and ask about other people's music preferences.
Practice what you know already so it becomes even more familiar to you. Your confidence will grow as the English you know becomes second nature. Set aside at least 30 minutes a day to repeat verb conjugations, difficult vocabulary or numbers, for example.
Watch English-speaking television shows or films. Surround yourself in the English language so you can recognize how words are pronounced and how sentences are formed. Repeat back words you hear to develop your speaking skills without the fear of people listening to you.
Get a English-speaking conversation partner and regularly meet up. Slowly build your confidence with the same person so you can learn to relax when speaking. Correct each other's mistakes, and exchange tips on improving your language skills.
Get an English teacher or take a course to get expert help in training your English skills. He will be able to help you with the areas you struggle in, thereby building your confidence.