Assessing the reliability of the Outcomes Star in research and practice

Abstract

Purpose

The purpose of this paper is to describe a pilot to test an approach to measuring inter-rater reliability of the Outcomes Star suite of tools. The intention, in publishing this account, is to show transparency in on-going development of the tool, and to invite further co-operative development.

Design/methodology/approach

In total, 24 workers, trained to use the first edition Family Star, scored a tested case study. Scoring was analysed using two metrics on the ten-point scale and the underlying five-point Journey of Change. The case study approach and metrics were evaluated for validity and accessibility.

Findings

This initial evaluation suggests this edition of the Family Star has good inter-rater reliability for the five-point Journey of Change, reaching the accepted threshold of 0.8 for the inter-rater reliability coefficient when three outlying workers are excluded. The reliability for the full ten point scale was moderate.

Research limitations/implications

The sample size of 24 raters is small, though sufficient for an initial test of the approach, which will now be applied to larger samples, using other versions of the Outcomes Star.

Practical implications

The findings indicate that it is important that service providers test worker understanding of the scales to ensure consistency of use. The second edition of the Family Star incorporates more precise definitions of the ten-point scales to help improve the reliability.

Originality/value

The case study method and metrics provide an accessible measure of reliability, both for Star development and to enable managers to assess the reliability of an organisation's client data for internal and external purposes.

Citation

Mackeith, J. (2014), "Assessing the reliability of the Outcomes Star in research and practice", Housing, Care and Support, Vol. 17 No. 4, pp. 188-197. https://doi.org/10.1108/HCS-11-2014-0027