Here's what he says:
"Don't you go downtown with Calpurnia. It's not right, it's not right at all, you know. There's enough to do here. You've got your books, and you've got each other. And you've got Dill, Scout. Don't you worry your head about Calpurnia, she's all right. She'll be back."
Atticus's words imply that he believes it's inappropriate for his children to go downtown with Calpurnia, likely due to the racial segregation of the time and the potential for discomfort or prejudice they might face. He also suggests that they have enough to keep them entertained at home, emphasizing their books, their company, and Dill's presence.
This scene highlights the complexities of race and social norms in Maycomb, Alabama, as well as Atticus's own understanding and sensitivity to these issues.