How to Report Split Sample Results

Researchers often use split samples in tests of assessment validation. A split sample is formed when a researcher randomly divides a full sample into two subsamples. When statistical analyses conform on both the subsamples as well as the full sample, the generalizability of results is more likely. Although the three samples should be similar, there will be variability between them. The point is to see how much variability exists. Researchers generally hope that there is low variability, since this supports the validity of the assessment.

Instructions

    • 1

      Report the independent and dependent variables. The dependent variable is the variable that the assessment evaluates. The independent variables are those that are suspected to affect the dependent variable's value for a given subject. For example, if you are assessing how a person's personality influences the amount of time he spends on the Internet, the independent variables are those related to his personality (extroversion, neuroticism, conscientiousness, etc.) while the dependent variable is the amount of time he spends on the Internet.

    • 2

      Report the statistical significance of the relationship between each independent variable and the dependent variable. The significance is shown in the form of the p-values you receive from running a statistical test of your study. Report the statistical significances for the full sample, the first subsample and the second subsample separately.

    • 3

      Interpret the variation in statistical significance for your samples. For this you need to choose an alpha-value. The alpha-value shows the probability of a false positive finding. The standard alpha-value is 0.05. If there is a single sample in which the p-value is greater than 0.05, then the split-sample validation fails. If this is the case, state this. Otherwise, report that statistical significance analyses are in favor of validation.

    • 4

      State the R-squared values separately for all three samples. The R-squared values are automatically given when you perform regression for the assessment data.

    • 5

      Interpret the R-squared values. Compare each subsample to the full sample in terms of R-squared. For validation, you want each subsample's R-squared value to be within 0.05 of the full sample's R-squared value. If this criterion is not met, state this. Otherwise, report support for validation.

Learnify Hub © www.0685.com All Rights Reserved