Roderick Usher, the last living member of the Usher family, suffers from extreme hypochondria and nervous anxiety, which cause him to withdraw from society. He invites his friend, the unnamed narrator, to his house to alleviate his melancholia. During the narrator's stay at the House of Usher, he observes many strange occurrences and becomes concerned for Roderick's mental health.
Towards the end of the story, Roderick's twin sister, Madeline Usher, dies and is temporarily entombed in the family vault. However, Roderick becomes increasingly convinced that Madeline is alive and attempting to escape the tomb. In his madness, he accuses the narrator of being in on a conspiracy against him and declares that the house itself is haunted.
As the climax reaches its peak, Roderick frantically chases after the narrator, thinking he is trying to escape the house with his sister. During this pursuit, Roderick cries out, "Madman! I tell you that she now stands without the door!". This accusation suggests that Roderick believes the narrator is in league with Madeline and is trying to help her escape from her tomb. The line highlights the depth of Roderick's mental decline and his inability to distinguish between reality and his delusions.
In conclusion, it is Roderick, not the narrator, who calls the narrator a "madman" in "The Fall of the House of Usher." This accusation arises from Roderick's deepening insanity and his irrational fears and delusions.