Do you have to fill out the whole census form?

No, there are parts of the census that you can leave blank.

For example, the census form asks for your citizenship, but if you prefer you don't have to answer this question. You can leave this box blank and will not face any consequences.