Write down what you want to measure with your test. This can be as simple as determining the number of objects the average person can hold in his short-term memory, to an individual's aptitude for piloting a fighter plane.
Research your topic to see how expert psychologists set up tests on the same topic. Use this as a basis for setting up your own test.
Organize your test based on construct validity: the effectiveness of the test's construction at measuring the intended goal. For example, a multiple-choice construct is probably a better way of determining whether a person possesses suicidal tendencies than a Rorschach blot test. Consider constructs that experts have used and design a plan for your test's construct.
Examine the test for content validity. Content validity refers to the content of the test's questions or problems and their scope and relevance to the test's goal. For example, when designing an intelligence test, questions about family background probably lack relevance, but questions regarding logical reasoning play a vital role in establishing content validity.
Administer the test and check for reliability. There are two categories to psychological test reliability: test-retest reliability and internal consistency reliability. Test-retest reliability refers to the ability to reproduce similar results when a test subject retakes the test. If the same subject or types of subjects get wildly different results, your test has a test-retest reliability problem. Internal consistency reliability is the relationship between the test items. If your test flows well from one section to the next and appears to have a continuity, then it is internally consistent.
Adjust your test based on your reliability trial. Identify questions that are vague and reconstruct any questions with validity problems. Interpret the results and determine if your test was valid as constructed. Administer it again if needed.