Think of it this way: CADD is a tool. Architecture is the art and science of building design. You need to know how to use the tool (CADD), but you also need to learn the art and science of architecture.
To get a degree in architecture, you'll need to apply to and be accepted into a Bachelor of Architecture (B.Arch) program. Some B.Arch programs might give you credit for some of the CADD coursework from your Associate's degree, shortening the overall time to completion. However, you will still have to complete all the required architecture coursework, studios, and design projects.
In short: Your CADD associate's degree could be beneficial and potentially save you some time, but you still need to complete a separate, full B.Arch program to earn an architecture degree.