Let $n$ be the number of students in the adult school before the new students join.
The total age of the students before the new students join is $An$.
The average age of the students is given as 40 years, so $A = 40$.
One hundred and twenty new students join the school, whose average age is 32 years.
Let $n_{new} = 120$ be the number of new students.
The total age of the new students is $n_{new} \times 32 = 120 \times 32 = 3840$.
The total number of students after the new students join is $n + n_{new} = n + 120$.
The total age of all students after the new students join is $An + 3840 = 40n + 3840$.
The new average age is $A_{new} = \frac{40n + 3840}{n + 120}$.
The average age is reduced by 4 years, so the new average age is $40 - 4 = 36$.
Therefore, we have:
$36 = \frac{40n + 3840}{n + 120}$
$36(n + 120) = 40n + 3840$
$36n + 4320 = 40n + 3840$
$4320 - 3840 = 40n - 36n$
$480 = 4n$
$n = \frac{480}{4} = 120$
So there were initially 120 students in the adult school.
The total age of the initial students is $120 \times 40 = 4800$.
After 120 new students join, the total number of students is $120 + 120 = 240$.
The total age of all students is $4800 + 3840 = 8640$.
The new average age is $\frac{8640}{240} = 36$.
The average age is reduced by $40 - 36 = 4$ years.
Final Answer: The final answer is $\boxed{120}$