The Mudville fans are worried at the beginning of the poem because their team is losing badly, and their chances of winning are slim. The poem describes the score as "four to two, with but an inning to go," meaning the Mudville team is behind by two runs with only one inning left to play. This situation is especially dire because the Mudville team is already known for being "a rotten team" and notorious for losing.
Their only hope is their star player, Casey, who is expected to be the hero and "clout the ball clear out of sight" to win the game. However, the fans are still apprehensive, as they've been disappointed by their team's performance all season. This anticipation and nervousness sets the stage for the climactic moment when Casey steps up to bat.