You're likely thinking of the book and TV series "The Secret Garden" by Frances Hodgson Burnett. In the story, the children avoid
Misselthwaite Manor, which is the large, gloomy house where Mary Lennox, Colin Craven, and Dickon live.
While Radley Place is from "To Kill a Mockingbird," it's understandable to get the houses mixed up! Both are mysterious and somewhat frightening places that the children initially avoid.