- Born on March 20, 1934, in Mineola, Texas
- Raised by his mother, Willie Mae Brown, and stepfather, Lee Roy Smith
- Attended segregated schools in Mineola and began showing an aptitude for public speaking and leadership
- Worked various odd jobs to help support his family, including shining shoes and delivering newspapers
- Graduated from Jarvis Christian College with a degree in political science in 1957