Analyzing data from a pretest-posttest control group design: The importance of statistical assumptions

Among the gold standards in human resource development (HRD) research are studies that test theoretically developed hypotheses and use experimental designs. A somewhat typical experimental design would involve collecting pretest and posttest data on individuals assigned to a control or experimental group. Data from such a design that considered if training made a difference in knowledge, skills or attitudes, for example, could help advance practice. Using simulated datasets, situated in the example of a scenario-planning intervention, this paper aims to show that choosing a data analysis path that does not consider the associated assumptions can misrepresent findings and resulting conclusions. A review of HRD articles in a select set of journals indicated that some researchers reporting on pretest-posttest designs with two groups were not reporting associated statistical assumptions and reported results from repeated-measures analysis of variance that are considered of minimal utility.

Using heuristic datasets, situated in the example of a scenario-planning intervention, this paper will show that choosing a data analysis path that does not consider the associated assumptions can misrepresent findings and resulting conclusions. Journals in the HRD field that conducted pretest-posttest control group designs were coded.

The authors' illustrations provide evidence for the importance of testing assumptions and the need for researchers to consider alternate analyses when assumptions fail, particularly the homogeneity of regression slopes assumption.

This paper provides guidance to researchers faced with analyzing data from a pretest-posttest control group experimental design, so that they may select the most parsimonious solution that honors the ecological validity of the data.