He wrote about his dislike for them in his autobiography, "Boy," describing them as "vile, revolting, and utterly disgusting." He even went as far as to say that he wouldn't touch them with a barge pole!
While his reasons aren't explicitly stated, it's likely he found the sweet and sickly combination of chocolate and fondant unappealing.