Is the empirical validity of prior experience for performance affected by rating approach?

In the public sector, Training and Experience (T & E) exams assess prior experience and are one of the most often used methods for selecting job applicants. This study uses a KSA approach, where raters judge the quality of job relevant prior experience, not its duration or quantity. It was hypothesized that an additional rater and a consensus meeting between raters would increase reliability and validity.

T & E and supervisory ratings were obtained over a 12-year period for 166 candidates seeking promotion to a budget analyst position. Validity was measured by the correlation between T & E scores and supervisory ratings. Consensus was required only for T & E scores differing by a specific amount (hybrid consensus).

Intraclass reliability was 0.73, 0.84, and 0.95 in the one-rater, two-rater, and hybrid consensus conditions with each coefficient greater than the next (p < 0.05) showing the benefit of multiple raters and consensus for reliability. Validity was significant at 0.21, 0.26, and 0.251 for each rating condition, respectively (two-tail test; p < 0.01). Validity was greater in the two-rater condition than in the one-rater condition (one-tail test; p < 0.05). Consensus did not improve validity beyond that of two raters. For consensus T & Es (n=76), two raters improved validity (one-tail test; p < 0.05), moving from 0.112 to 0.231 but not reliability; consensus improved reliability (two-tail test; p < 0.05) but not validity.

There has been a vacuum in T & E research for close to 20 years. Validity data are difficult to obtain but critical for meta-analysis. T & Es showed validity. Use of two raters improved validity but consensus did not increase the gain.