Abstract
Purpose
The quality of voluntary sector-led community health programmes is an important concern for service users, providers and commissioners. Research on the fidelity of programme implementation offers a basis for assessing and further enhancing practice. The purpose of this paper is to report on the fidelity assessment of Living Well Taking Control (LWTC) – a voluntary sector-led, community-based education programme in England focussing on the prevention and management of type 2 diabetes.
Design/methodology/approach
This fidelity of implementation (FoI) study was conducted with the Devon-based LWTC programme. A fidelity checklist was developed to analyse audio records of group-based lifestyle education sessions – implementation was rated in terms of adherence to protocol and competence in delivery; the influence of wider contextual factors was also assessed. Kappa statistics (κ) were used to test for inter-rater agreement. Course satisfaction data were used as a supplementary indicator of facilitator competence.
Findings
Analysis of 28 sessions, from five diabetes prevention and two diabetes management groups (total participants, n=49), yielded an overall implementation fidelity score of 77.3 per cent for adherence (moderate inter-rater agreement, κ=0.60) and 95.1 per cent for competence (good inter-rater agreement, κ=0.71). The diabetes prevention groups consistently achieved higher adherence scores than the diabetes management groups. Facilitator competence was supported by high participant satisfaction ratings.
Originality/value
An appropriate level of implementation fidelity was delivered for the LWTC group-based education programme, which provides some confidence that outcomes from the programme reflected intervention effectiveness. This study demonstrates the viability of assessing the FoI in a voluntary sector-led public health initiative and the potential of this method for assuring quality and informing service development.
Keywords
Citation
Kok, M.S.Y., Jones, M., Solomon-Moore, E. and Smith, J.R. (2018), "Implementation fidelity of a voluntary sector-led diabetes education programme", Health Education, Vol. 118 No. 1, pp. 62-81. https://doi.org/10.1108/HE-04-2017-0019
Publisher
:Emerald Publishing Limited
Copyright © 2018, Michele S.Y. Kok, Mat Jones, Emma Solomon-Moore and Jane R. Smith
License
Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
Background
Voluntary sector agencies play an increasingly important role in the delivery of community-based health promotion and disease management programmes. Such agencies may be well placed to engage socially disadvantaged groups, respond to local health needs, and create innovative and value-for-money interventions (Curry et al., 2011; South, 2015). However, the quality of these initiatives is a key concern for potential service users, commissioners, partner agencies, and other stakeholders. This is partly because voluntary sector agencies are not necessarily subject to the same procedures for quality assurance or the employment of registered professionals as mainstream health service organisations (Baggott, 2013). In this context, research focussing on the fidelity of programme implementation offers a basis for assessing and further enhancing voluntary sector-led practice. The present study reports on one such assessment of Living Well Taking Control (LWTC) – a voluntary sector-led, community-based type 2 diabetes education programme that focusses on the prevention and management of the condition.
Fidelity of implementation (FoI) refers to the degree to which an intervention is delivered as intended, and is critical to successful translation of evidence-based interventions into practice. Literature reviews have demonstrated that higher FoI is associated with greater intervention effects (Durlak and DuPre, 2008; McIntyre et al., 2007). Attaining and demonstrating fidelity enables researchers to identify the active ingredients of an intervention, contrast it with a control or standard treatment, and replicate findings (Hildebrand et al., 2012). Failure to demonstrate fidelity can undermine the internal and external validity of evaluation studies, and makes development of new interventions difficult (Bellg et al., 2004).
FoI is recognised as an important issue for type 2 diabetes prevention programmes. Dunkley et al.’s (2014) meta-analysis of pragmatic lifestyle interventions found evidence suggesting that diabetes prevention programmes are effective, but that effectiveness varied substantially between programmes. They concluded that “adherence to international guidelines on intervention content and delivery explained much of the variance in effectiveness” (Dunkley et al., 2014, p. 931). Therefore, the challenge in ensuring effectiveness of such programmes is twofold: not only does their design need to maximise adherence to guidelines, but also their implementation needs to adhere to that specified design, there needs to be evidence of implementation fidelity.
There are several barriers to maintaining FoI in community settings, which include adaptations of interventions to the local context, limited pre-implementation specification and training, individual variations in facilitator adherence and competence, lack of technical support and ongoing monitoring, limited resources for supporting the intervention at the site level, the learning effect of facilitators in line with programme development over time, and competing demands for the facilitators’ time that can diminish their commitment or effectiveness (Cross and West, 2011; Breitenstein et al., 2010; Andersen et al., 2014). It is essential to outline clear and feasible strategies for monitoring, measuring, and ensuring FoI to avoid potentially useful interventions appearing to be ineffective (Breitenstein et al., 2010). Furthermore, non-systematic assessments of FoI can decrease the quality and usefulness of fidelity data (Nelson et al., 2012).
According to Borrelli (2011), there are five domains of fidelity: study design; training; intervention delivery; intervention receipt; and intervention enactment (defined as the extent to which participants apply the skills learnt). The heart of fidelity is often considered to be intervention delivery, the core components of which are adherence and competence (Mars et al., 2013; Gearing et al., 2011). Whilst adherence is concerned with the extent to which the intervention protocol is implemented, competence refers to how well the protocol is implemented and delivered (Breitenstein et al., 2010). Adherence measures, therefore, evaluate the presence or absence of components that are considered to be specific, essential and optional to the defined intervention. If facilitators are found to be drifting from the protocol, then remedies such as feedback, individualised coaching, and group discussion may be applied to identify and remove obstacles to fidelity (Whitmer et al., 2005). Competence measures include facilitator qualities related to communication, technical abilities, and skills in responding to the needs of intervention participants (Breitenstein et al., 2010). Due in part to the methodological difficulties surrounding the monitoring and measurement of competence, this aspect is less often assessed and reported in literature than adherence (Cross and West, 2011; Mars et al., 2013).
In addition to adherence and competence, FoI studies of intervention delivery need to take into account the programme context (Breitenstein et al., 2010). Contextual factors such as group dynamics, unplanned interruptions, or the point of assessment in the programme’s delivery, can influence the ability of facilitators to deliver an intervention (Durlak and DuPre, 2008; Fixsen et al., 2005). Assessment of the role of contextual factors is key to gaining a broader understanding of the programme’s implementation and of environmental settings that can enhance or impede delivery, which subsequently impacts the interpretation and generalisation of the evaluation outcomes (Moore et al., 2015).
This study aimed to assess the FoI of the LWTC programme, with a focus on intervention delivery in terms of both adherence and competence.
Methods
Study setting: LWTC – programme structure and delivery
LWTC is a lifestyle behaviour change programme developed by two UK voluntary sector agencies – Westbank Community Health and Care, Devon, and Health Exchange, Birmingham. LWTC focusses on the prevention and management of type 2 diabetes, in non-clinical, community settings. In 2013, funding obtained as part of the Big Lottery Fund’s well-being initiative provided developers with the opportunity to roll out the LWTC programme across four sites in the UK (Devon, Birmingham, Newcastle and Darlington), and align its delivery to national best practice guidance (National Institute for Health and Care Excellence (NICE), 2016; NICE, 2015; NICE, 2012; DOH, 2012; DOH, 2001). The present FoI study focussed on the Devon-based delivery of the programme by Westbank.
The core component of the LWTC programme is a group-based, structured lifestyle education intervention to promote sustainable healthy lifestyle changes for people with Impaired Glucose Regulation (IGR, also known as pre-diabetes or non-diabetic hyperglycaemia) and those newly diagnosed with type 2 diabetes (T2D). Participants are allowed to have a carer, partner, or other family member attend the education sessions with them to provide support. The intervention is based around the use of behaviour change processes and techniques derived from self-regulation theories, such as the social cognitive theory (Bandura, 1985) and control theory (Carver and Scheier, 1982). These include goal setting, action-planning, self-monitoring, progress feedback, problem-solving, and reviewing goals. Group-based interventions have been shown to be effective for diabetes prevention, self-management and support, providing an environment that facilitates peer discussions about problems and personal experiences, resulting in significant benefits, including weight loss, improved fasting glucose levels, improved energy, and improved emotional state (Katula et al., 2011; Trento et al., 2010; Holma et al., 2008).
Programme structure and delivery
In order to standardise the intervention structure, content and delivery, LWTC programme partners developed protocol manuals for the diabetes prevention and management education sessions, to train and guide programme facilitators to provide comprehensive lifestyle advice. The facilitators were also provided two days of training in the use of a person-centred, empathy-building approach to delivering the intervention, based on the use of motivational interviewing techniques to explore and enhance motivation, exchange information, and deliver other elements of the programme, such as reviewing progress and problem-solving (Miller and Rollnick, 2002). Programme facilitators were recruited from backgrounds that included undergraduate education in health promotion, physical activity and nutrition. All facilitators had been employed since the inception of the programme, 15 months prior to the present study being undertaken.
As per NICE recommendations, the programme aims to give participants at least 16 hours of contact time, through a combination of group and one-to-one sessions. At the start of the programme, participants receive a one-hour individual introduction session where they meet the facilitator, have the opportunity to ask questions, and complete baseline measures (see below). Participants then attend four group-based education sessions over four weeks, each covering a different component: pre-diabetes/T2D and a healthy lifestyle, healthy eating, physical activity, and positive mental health and well-being. The first session is scheduled for an hour, and subsequent sessions are two hours each.
At the end of the four-week group sessions, participants are offered one-to-one or group follow-up contacts at 2, 3, 6, 9, and 12 months to review goals, reflect on changes made, and identify the need for any additional contacts focussing in more detail on certain aspects of the programme (e.g. nutrition, physical activity, well-being) or support through local community services (e.g. smoking cessation, alcohol reduction, health trainer). Biometric measures – weight, height, body mass index (BMI), and blood pressure – are obtained at baseline, on week 4 of the group sessions, and during follow-up contacts at 2, 3, 6, 9 and 12 months. Glycated haemoglobin (HbA1C) levels are assessed at baseline, 6 months and 12 months.
Data collection
Data for the FoI study were collected primarily through audio recordings of programme group sessions delivered by each of the two facilitators at Westbank. In order to prevent selection bias and ensure an adequate sized and representative sample, all sessions delivered from 20 January to 5 March 2015 were audio recorded. There were five diabetes prevention and two diabetes management groups during this period. Verbal and written consent were sought from participants and any accompanying persons (i.e. partners or family members) at the start of the first session. None of the participants declined to provide consent for recording, giving a total of 49 hours of recording from 28 group sessions. Audio recorded data were supplemented by a feedback questionnaire, which participants completed at the end of the four-week group sessions to rate their satisfaction with the programme and the facilitators.
Data coding and analysis
In line with the Medical Research Council’s recommendations for process evaluation (Moore et al., 2015), the FoI analysis and initial reporting were conducted prior to knowledge of outcomes effects to guard against interpretation bias. Information from the questionnaire that participants completed as part of the wider service evaluation was used to conduct t-tests to assess if there were any significant differences between the participant characteristics of the FoI study sample compared to the overall Westbank population of programme participants. The FoI study sample is a subset of the overall Westbank population.
Audio recordings were assessed by a principal rater (MK) and reviewed by a second member of the research team (RA or ON), using a fidelity checklist. The fidelity criteria were developed to reflect key elements of the standardised programme protocols, with input from the facilitators and managers. The initial checklist was piloted with recordings from the first group that underwent each of the programme sessions, after which minor revisions were made.
The final checklist comprised a total of 62 compulsory and 14 optional items. The overall structure of the checklist is shown in Table I and full details are in the appendix.
The two core components of intervention delivery were defined as follows:
adherence – the extent to which facilitators followed the programme protocols, e.g. providing essential information, presenting benefits of and barriers to change, and undertaking goal setting activities; and
competence – the facilitators’ skills in delivering the programme, including communication skills, e.g. through creating opportunities for participants to ask questions or lead group discussions, and reminding participants of earlier commitments.
The FoI for each item was rated as “low/not observed” (scored 1), “observed to a small degree” (scored 2), “observed to a medium degree” (scored 3), or “high implementation” (scored 4). FoI scores for adherence and competence were obtained as mean scores across the compulsory items. An overall FoI score for each group was the mean score from across the four sessions converted into a percentage. Although there is no agreement from existing literature as to what constitutes an acceptable level of implementation fidelity (Breitenstein et al., 2010), the goal for this assessment was set at a minimum of 80 per cent.
In addition to adherence and competence ratings, any notable observations on group dynamics and contextual variations from the audio recordings were documented separately on, for example, verbal comments about the group made by the facilitator, interruptions and deviations. Supplemented by data from the course satisfaction feedback forms, a list of factors that appeared to either enhance or impede programme delivery was developed through open coding and iterative clustering of the observational notes, to provide a broad understanding of facilitators’ competence.
Reliability and validity
To ensure reliability and internal consistency, 10 per cent of the audio recorded intervention sessions were tested for inter-rater agreement (Mars et al., 2013). Two independent raters (RA and ON) were specifically recruited to analyse five hours of recordings each. An online random sequence generator was utilised to randomly select groups and sessions for analysis. Kappa statistics (κ) were calculated using SPSS Statistics for Windows, Version 22.0 (Armonk, NY: IBM Corp.), as an index of inter-rater agreement for compulsory adherence and competence items, and interpreted using benchmarks published in Peat (2001), with <0.4 indicating poor agreement; 0.41-0.60 fair; 0.61-0.80 good; and 0.81-1.00 very good agreement. p-values were also calculated to test whether the estimated kappa was not due to chance, with significance set at p<0.05 (Viera and Garrett, 2005).
Results
Group and participant characteristics
There were 49 participants across the seven groups that received the intervention during the study period, a median of seven participants per group (range 3-9). Table II shows a breakdown of the number of participants per group, and any partners or family members who were consistently present to provide support during sessions.
Results of t-tests (see Table III) found no difference between the participants in the groups used for fidelity assessment (n=46) compared to the overall Westbank programme participants (n=322) in terms of gender, ethnicity, employment status, BMI, waist circumference, general health, life satisfaction, level of physical activity and smoking and disability status. However:
participants in the FoI study were significantly heavier (p<0.05) on entry to the programme, but had a significantly lower HbA1c (p<0.05) than the programme participants overall; and
the programme participants overall had a significantly higher education level than participants in the FoI study (p<0.05).
The study sample was considered to be generally representative of the wider Westbank participant population; differences were not expected to affect fidelity.
Adherence to intervention protocol
Data on adherence to the LWTC intervention protocol are shown in Table IV. Inter-rater agreement for adherence criteria was moderate (κ=0.60, p<0.001). The pre-diabetes groups scored higher than the diabetes groups in all sessions.
Sessions 1 and 3 overall exceeded the 80 per cent minimum target for acceptable mean implementation fidelity score. Items that were rated as “high implementation” for all groups are summarised in Table V, for example, assessing the importance and confidence in making healthy lifestyle changes, introducing the “Eatwell Plate”, and discussing the benefits of physical activity. Session 4, which focussed on positive mental health and well-being, consistently scored the lowest out of the four sessions for all groups. However, every session had adherence criteria that were often either not mentioned by the facilitators or only very briefly mentioned, for example, recapping and assessing retention from the previous session, discussing the importance and confidence to maintain healthy emotional well-being, and setting new goals for future – these are also summarised in Table V.
Competence in intervention delivery
Table VI shows that the overall implementation fidelity scores for each session ranged from 3.71 (92.8 per cent) to 3.93 (98.3 per cent), which indicates a high level of competence in intervention delivery across all groups. Inter-rater agreement for competence criteria was good (κ=0.71, p<0.001). Facilitators did not achieve full implementation scores for Session 1, most commonly due to omission of the “confidentiality agreement” criterion, which is a measure of how competent the facilitators were in creating an environment where participants could openly and comfortably express themselves and share opinions, with the assurance that all information would be kept private and confidential (all participants had already provided verbal and written consent at the start of the session).
Course satisfaction data were available for 31 out of the 49 participants (63.3 per cent). The other participants had either not attended Session 4 and hence did not receive the course satisfaction questionnaire, or they had not returned the questionnaire after completion. Table VII shows the data alongside that of the wider LWTC programme population at Westbank. Overall, the FoI sub-sample of participants gave high satisfaction ratings for the LWTC programme, which is similar to ratings of the wider group. This not only affirms facilitator competence in intervention delivery, but indicates potential generalisability of the FoI findings across the whole programme.
Implementation of optional items
The overall level of implementation of optional items ranged from 45.5 to 63.6 per cent across all groups. “Offering refreshments” and “repeating clinical metrics” were always implemented. The optional walk or seated exercise in Session 3 was only provided for one group, while the relaxation exercise in Session 4 was never delivered. Mental health concerns were not raised in Session 4; therefore, there was no signposting. Other forms of signposting, i.e. to healthcare professionals, local services (e.g. smoking cessation, alcohol reduction) or additional support were carried out as required. There were three additional optional items for participants with diabetes: expectations from healthcare professional, information about annual reviews, and the 15 Healthcare Essentials – none of these were implemented for either diabetes group.
Group dynamics and contextual factors
Analysis of the recordings and course satisfaction feedback forms identified no unplanned disruptions or curtailed sessions due to an external interruption. Participants did not report problems with the venues and facilities and there were no instances of disruptive behaviour by individuals.
Group verbal interactions changed over the duration of the course, but not necessarily in a consistent way for all groups – in four groups there was high level verbal interaction from the outset of the course; by contrast, three other groups started off with very few verbal contributions from participants, but participants engaged in dialogue more fully after the end of the first session. Group size also influenced verbal interactions. Participants in the smallest group (D21) had more opportunities to participate in discussion than those in the other groups. One D21 participant expressed that it was “so much easier and more comfortable to ask questions at this session, compared to the other diabetes session [conducted by another organisation]”.
In six out of the seven groups, at least one participant brought a family member, who was usually their partner. Family members asked questions, raised points of clarification and, in most cases, actively contributed to the group discussion items. There was no evidence from the course feedback that participants objected to the presence of the family members of others in the group. The presence of these individuals appeared to facilitate group interactions and personal support. Overall, the frequency, breadth, and content of verbal interactions indicated good group dynamics across the groups. For each session, all participants made at least one verbal contribution, and many offered verbal support to one another in response to personal disclosures.
Data from audio recordings indicate that participant characteristics may have affected group dynamics and delivery, particularly in group D20 where during the sessions, the facilitator described the group as “well-informed” and “well-read”, and expressed that “time always ran away” with this group. Analysis of the delivery of session items to this group recorded facilitator communication that was diverted by the direction of participant-led conversations. Although implementation ratings for adherence items imply that these factors impeded delivery in group D20, it may also have demonstrated the ability of the facilitator to adapt in response to the specific needs of the group. Two groups had participants who expressed sceptical views and were less receptive to the idea of behaviour change than others in the group. Some concerns were expressed by participants with diabetes about symptoms, medications, and other topics that were not covered within the scope of the programme.
Discussion
This study has designed a fidelity assessment tool and successfully applied it to a voluntary sector-led public health initiative to provide insights into its implementation. The FoI for the overall LWTC programme was satisfactory in terms of adherence to the protocol, with a high level of competence in delivery. The p-values obtained for inter-rater agreement for both adherence and competence criteria, show that the level of agreement was not due to chance (p<0.001). The moderate inter-rater agreement for the adherence criteria may have been influenced by the raters having varying interpretations of some of the criteria, and applying different levels of rigour. Overall, it appears that fidelity assessment tools, such as the type developed in the present study, can have a role in assessing and assuring the quality of initiatives delivered through voluntary sector agencies.
There were variations across the groups. The mean implementation fidelity score for adherence criteria for the pre-diabetes groups was 80.6 per cent, which is considered as an acceptable level of adherence according to precedents set by other studies (Breitenstein et al., 2010). However, the mean adherence score for the diabetes groups was 69.3 per cent – participants in these two groups came across as more knowledgeable, opinionated, inquisitive, and were more actively engaged in discussions. Given the nature of the diabetes groups, it would have been challenging for the facilitator to address as many of the participants’ questions within the time allocated for each session, while still adhering to the protocol. This reflects some of the pedagogical challenges faced by educators in other diabetes group-based programmes (Andersen et al., 2014). Nevertheless, intentional and/or unintentional adaptations of interventions, including changes to programme delivery and not completing core elements, are common and natural in community-based settings for reasons such as to generate/maintain participant engagement or to cope with barriers like time constraints (Carvalho et al., 2013). The balance between fidelity and adaptation needs to be continuously monitored and evaluated, to ensure that the intervention remains relevant to participants’ needs and still leads to the desired outcomes (Carvalho et al., 2013).
The optional walk or seated exercise was only offered to one pre-diabetes group, and carried out by the facilitator whose expertise was in physical activity. Lack of suitable equipment on location, lack of facilitator expertise, or insufficient time, are several possible reasons why the optional activities or items were not implemented more often. The consent process for recording of the sessions took place for all groups at the start of Session 1. Since the recorder might not have been switched on by then, the “confidentiality agreement” criterion was not captured in four out of the seven groups, and was therefore scored as “not observed”. This highlights that effective data capture through audio recording is dependent on facilitators remembering to switch on the recorder at the right time.
There were also variations across the sessions, with omissions occurring most notably in Sessions 3 and 4. Physical activity diaries were not reviewed during Session 3, and in four out of the seven groups, there was no mention of it by the facilitator; participants in the other three groups did not use the diaries, and therefore they could not be discussed. This may be expected since in the prior session, facilitators did not remind participants to complete the activity diary. Nevertheless, the facilitators still assessed whether the participants managed to make any changes in their level of activity since the last session, or had any motivation or plans to increase it.
The facilitators had not received training in the area of mental health and well-being, which might have affected their confidence in delivering this particular component in Session 4. Since evidence suggests that mental health and well-being are crucial factors in enabling participants to make positive and lasting lifestyle changes (NHS Confederation, 2011), it would be interesting to see how its omission/low level of implementation would affect intervention effectiveness. Goal setting and reviewing goals play important roles in the programme to translate motivations into action, and to support longer term maintenance of behaviour change (Bandura, 1985; Carver and Scheier, 1982). While goal setting was well implemented in the first three sessions, especially for the pre-diabetes groups, reviewing of goals and setting new goals were mostly omitted or implemented at a low level in Session 4. It is possible that this might have been due to time constraints towards the end of the session.
Limitations
The supplementary qualitative analysis was useful to provide a general insight into group dynamics and contextual factors. The FoI study sample was largely similar to the wider LWTC programme participant population and reflected a typical context for delivery. However, there were a number of limitations to the present study. Audio recordings could not capture non-verbal aspects and environmental factors (e.g. cramped intervention venue, extraneous events that distract participants), therefore, it could not be determined if these issues occurred or how they might have influenced facilitators’ adherence and competence. Although results show a high overall level of implementation and competence, the accuracy of these ratings might have been undermined by the method of assessment used.
Future directions
The FoI findings point towards several areas for improvement. The programme protocol needs to give clearer guidance and enhance facilitator training in the area of mental health and well-being, in order to improve delivery of that intervention component. This form of guidance is likely to be particularly important where initiatives, such as those often found in the voluntary sector, are delivered by facilitators with less advanced professional healthcare training. Further research is needed to establish fidelity measures with optimal validity and reliability. A six-point Likert scale that incorporates the Dreyfus system (Dreyfus, 1989) for denoting competence may be a more robust method of assessing both adherence to intervention protocol and skill of the facilitator in intervention delivery. Such scaling could have a role in setting higher assessment point thresholds, which in turn, can help drive up standards of implementation when used over time. Interviews and focus groups would provide more robust and in-depth qualitative data for a more comprehensive understanding of the intervention delivery. It would also be useful for future research to further assess the impact that group size, group dynamics and age differences between participants within a group have on the FoI of group-based interventions.
Conclusion
In conclusion, the study suggests that an appropriate level of implementation fidelity was generally achieved for the LWTC group-based education intervention with a satisfactory level of adherence to the protocol and a high level of competence in delivery. The development of standardised protocol manuals for intervention design and training were critical to ensuring FoI. However, a higher level of implementation fidelity would have been more desirable to increase confidence in the interpretation of findings around intervention outcomes and effectiveness. The eventual outcomes of the associated evaluation studies will help to determine if components that were omitted or implemented at a low level were crucial to ensuring effectiveness of the intervention, and whether adjustments are needed to the intervention and training protocols. This study has shown that it is feasible and valuable to evaluate the FoI of a voluntary sector-led public health initiative for quality assurance and practice enhancement.
Structure of the LWTC fidelity checklist
| Session: programme component | No. of compulsory items | No. of optional items |
|---|---|---|
| 1. Pre-diabetes/T2D and a healthy lifestyle | Adherence items: 11 | 2 |
| Competence items: 5 | ||
| Total: 16 | ||
| 2. Healthy eating | Adherence items: 13 | 1 |
| Competence items: 3 | ||
| Total: 16 | ||
| 3. Physical activity | Adherence items: 11 | 6 |
| Competence items: 3 | (3 of these applied to diabetes patients only) | |
| Total : 14 | ||
| 4. Positive mental health and well-being | Adherence items: 13 | 5 |
| Competence items: 3 | ||
| Total: 16 |
Group sizes and composition in the fidelity assessment
| Group ID (P: pre-diabetes; D: diabetes) |
No. of participants | No. of partners/family members | Total |
|---|---|---|---|
| P31 | 8 | 0 | 8 |
| P32 | 7 | 2 | 9 |
| P33 | 7 | 1 | 8 |
| P34 | 9 | 3 | 12 |
| P35 | 8 | 1 | 9 |
| “Pre-diabetes” sub-total | 39 | 7 | 46 |
| D20 | 7 | 1 | 8 |
| D21 | 3 | 1 | 4 |
| “Diabetes” sub-total | 10 | 2 | 12 |
| Total | 49 | 9 | 58 |
Comparison of participant characteristics between fidelity groups and the overall Westbank group
| Participant characteristics | Fidelity groups (n=46a) | Overall Westbank group (n=322) | ||
|---|---|---|---|---|
| Age range | 39-80 years | 28-91 years | p-value | |
| Diagnosis | Pre-diabetes | 80.4% | 68.2% | p=0.49 |
| Diabetes | 19.6% | 31.8% | ||
| Gender | Male | 43.5% | 40.2% | p=0.31 |
| Female | 56.5% | 59.8% | ||
| Ethnicity | White | 82.2% | 87.8% | p=0.16 |
| Body mass index (BMI) | Normal weight | 7% | 12% | p=0.25 |
| Overweight | 37.2% | 37.1% | ||
| Obese | 55.8% | 50.8% | ||
| Weight | Mean | 91.3 kg (SD 18.3) | 86.1 kg (SD 18.3) | p=0.02 |
| Range | 57.8-130.8 kg | 56.7-152.2 kg | ||
| Waist circumference | Mean | 106.8 cm (SD 14.1) | 103.5 cm (SD 15.1) | p=0.07 |
| Range | 82-140 cm | 69-174.5 cm | ||
| HbA1c | Mean | 42.4 mmol/mol (SD 4.4) | 45.4 mmol/mol (SD 9.0) | p=0.04 |
| Range | 36-53 mmol/mol | 32-109 mmol/mol | ||
| Health | Mean | 68.1 (SD 23.9) | 71.1 (SD 20.8) | p=0.34 |
| Range | 14-100 | 7-100 | ||
| Life satisfaction | Mean | 7.1 (SD 2.2) | 7.4 (SD 2.0) | p=0.25 |
| Range | 2-10 | 0-10 | ||
| Physical activity | Met guidelines | 66.7% | 59.8% | p=0.50 |
| Employment | Retired | 62.2% | 48.7% | p=0.95 |
| Employed | 8.9% | 24.8% | ||
| Self-employed | 6.7% | 7.9% | ||
| Unemployed | 4.4% | 1.9% | ||
| Carer | 0% | 2.2% | ||
| Student | 2.2% | 0.9% | ||
| Long-term sickness/disabled | 2.2% | 3.1% | ||
| Education | Completed school up to 16 years | 40.9% | 36.9% | p=0.004 |
| Did some extra training or A levels | 34.1% | 41.2% | ||
| Did an undergraduate or postgraduate degree | 15.9% | 21.9% | ||
| Smoking status | Non-smokers | 83.7% | 92.5% | p=0.37 |
| Disability | No disability | 69.2% | 81.3% | p=0.32 |
Notes: aData were available for 46 out of the 49 participants included in the FoI analysis, either due to refusal to answer the questionnaire or failure to return it. The small amount of missing data is unlikely to have any significant effect on the t-test results
Implementation fidelity scores for adherence criteria
| Implementation fidelity scores for adherence items, from 1 (low) to 4 (high) | |||||
|---|---|---|---|---|---|
| Group ID (P: pre-diabetes; D: diabetes) |
Session 1 | Session 2 | Session 3 | Session 4 | Overall |
| P31 | 3.55 | 3.31 | 3.55 | 2.54 | 3.21 |
| P32 | 3.56 | 2.92 | 3.36 | 3.00 | 3.17 |
| P33 | 3.73 | 3.00 | 3.64 | 2.62 | 3.21 |
| P34 | 3.73 | 3.08 | 3.55 | 2.77 | 3.25 |
| P35 | 3.64 | 3.00 | 3.73 | 2.85 | 3.27 |
| Mean for “pre-diabetes” | 3.64 | 3.06 | 3.57 | 2.76 | 3.22 |
| D20 | 3.45 | 2.62 | 3.09 | 2.31 | 2.83 |
| D21 | 2.82 | 2.77 | 2.82 | 2.46 | 2.71 |
| Mean for “diabetes” | 3.14 | 2.70 | 2.96 | 2.39 | 2.77 |
| Overall mean | 3.50 | 2.96 | 3.39 | 2.65 | 3.09 |
| 87.43% | 73.93% | 84.79% | 66.25% | 77.32% | |
Summary of adherence criteria that were highly implemented or commonly omitted/implemented at a low level, across all sessions
| Highly implemented adherence criteria across all sessions (i.e. scored 4) | |
| Providing information | Overview of what diabetes/pre-diabetes is Introduction to the 5 key healthy lifestyle messages Explaining clinical metrics Introduction to the “Eatwell Plate” and food labelling Introduction to physical activity guidelines |
| Assessing motivation | Assessing importance and confidence in making healthy lifestyle changes, healthy dietary changes, and/or to increase physical activity |
| Discussing key elements | Discussing the importance of low fat and high fibre diets, and about each of the food groups Discussing the benefits of physical activity, and the different types of activity |
| Adherence criteria commonly omitted/implemented at a low level across all sessions (i.e. scored 1) | |
| Comprehension and retention of information | Recap and assess retention from previous week’s session Review all items at the end of Session 4 |
| Activity diaries | Remind participants to complete activity diary in time for Session 3 Monitor awareness and reflection of physical activity undertaken, and assess motivation to increase activity levels |
| Positive thinking | Introduction to the concept of positive thinking Importance and confidence to maintain healthy emotional well-being Barriers to positive thinking |
| Goal setting | Review goals set, and set new goals for future |
Note: There were no adherence criteria highly implemented in Session 4
Implementation fidelity scores for competence criteria
| Implementation fidelity scores for competence items, from 1 (low) to 4 (high) | |||||
|---|---|---|---|---|---|
| Group ID (P: pre-diabetes; D: diabetes) | Session 1 | Session 2 | Session 3 | Session 4 | Overall |
| P31 | 3.40 | 4.00 | 4.00 | 4.00 | 3.79 |
| P32 | 3.25 | 4.00 | 4.00 | 4.00 | 3.77 |
| P33 | 3.80 | 4.00 | 4.00 | 4.00 | 3.93 |
| P34 | 3.60 | 4.00 | 4.00 | 4.00 | 3.86 |
| P35 | 3.20 | 4.00 | 4.00 | 4.00 | 3.71 |
| D20 | 3.40 | 4.00 | 4.00 | 4.00 | 3.79 |
| D21 | 3.40 | 4.00 | 4.00 | 4.00 | 3.79 |
| Overall mean | 3.44 | 4.00 | 4.00 | 4.00 | 3.81 |
| 85.89% | 100.00% | 100.00% | 100.00% | 95.14% | |
Course satisfaction data
| Fidelity groups | Overall Westbank group | |
|---|---|---|
| Did the course benefit you? No. (%) of participants who responded “yes” |
30 (96.8%) Total n=31 |
232 (98.7%) Total n=235 |
| On a scale from 1-10, how much did you enjoy the course? (1: Not at all – 10: Very) | Mean =9.13 Range 7-10 Total n=31 |
Mean =8.84 Range 1-10 Total n=237 |
| Did the course meet your specific needs?a No. (%) of participants who responded “yes” |
30 (96.8%) Total n=31 |
213 (97.7%) Total n=218 |
| On a scale from 1-10, how much would you recommend this course to friends or family? (1: Extremely unlikely – 10: Extremely likely) |
Mean =9.23 Range 5-10 Total n=31 |
Mean =9.13 Range 2-10 Total n=184 |
Note: aOne participant did not answer this question because it was “difficult to say due to [my] disability”
LWTC audio recording fidelity checklist
 |
Appendix
References
Andersen, T.H., Hempler, N.F. and Willaing, I. (2014), “Educator challenges using participatory methods in group-based patient education”, Health Education, Vol. 114 No. 2, pp. 152-165, doi: 10.1108/HE-07-2013-0032.
Baggott, R. (2013), Partnerships for Public Health and Well-Being: Policy and Practice, Palgrave Macmillan, Hampshire.
Bandura, A. (1985), Social Foundations of Thought and Action, Prentice-Hall, NJ.
Bellg, A.J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D.S., Ory, M., Ogedegbe, G., Orwig, D., Ernst, D. and Czajkowski, S. and the Treatment Fidelity Workgroup of the NIH Behavior Change Consortium (2004), “Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium”, Health Psychology, Vol. 23 No. 5, pp. 443-451.
Borrelli, B. (2011), “The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials”, Journal of Public Health Dentistry, Vol. 71 No. S1, pp. S52-S63.
Breitenstein, S.M., Gross, D., Garvey, C.A., Hill, C., Fogg, L. and Resnick, B. (2010), “Implementation fidelity in community-based interventions”, Research in Nursing and Health, Vol. 33 No. 2, pp. 164-173.
Carvalho, M.L., Honeycutt, S., Escoffery, C., Glanz, K., Sabbs, D. and Kegler, M.C. (2013), “Balancing fidelity and adaptation: implementing evidence-based chronic disease prevention programs”, Journal of Public Health Management and Practice, Vol. 19 No. 4, pp. 348-356.
Carver, C.S. and Scheier, M.F. (1982), “Control theory: a useful conceptual framework for personality – social, clinical, and health psychology”, Psychology Bulletin, Vol. 92 No. 1, pp. 111-135.
Cross, W. and West, J. (2011), “Examining implementer fidelity: conceptualising and measuring adherence and competence”, Journal of Children’s Services, Vol. 6 No. 1, pp. 18-33.
Curry, N., Mundle, C., Sheil, F. and Weaks, L. (2011), The Voluntary and Community Sector in Health, The King’s Fund, London.
Department of Health (2001), National Service Framework for Diabetes: Standards, Department of Health, London, available at: www.doh.gov.uk/nsf/diabetes.htm (accessed 19 February 2016).
Department of Health (2012), Public Health Outcomes Framework 2013 to 2016, Department of Health, London, available at: www.gov.uk/government/publications/healthy-lives-healthy-people-improving-outcomes-and-supporting-transparency (accessed 19 February 2016).
Dreyfus, H.L. (1989), “The Dreyfus model of skill acquisition”, in Burke, J. (Ed.), Competency Based Education and Training, Falmer Press, London, pp. 181-183.
Dunkley, A.J., Bodicoat, D.H., Greaves, C.J., Russell, C., Yates, T., Davies, M.J. and Khunti, K. (2014), “Diabetes prevention in the real world: effectiveness of pragmatic lifestyle interventions for the prevention of type 2 diabetes and of the impact of adherence to guideline recommendations”, Diabetes Care, Vol. 37 No. 4, pp. 922-933.
Durlak, J.A. and DuPre, E.P. (2008), “Implementation matters: a review of research on the research on the influence of implementation on program outcomes and the factors affecting implementation”, American Journal of Community Psychology, Vol. 41 No. 3, pp. 327-350.
Fixsen, D.L., Naoom, S.F., Blasé, K.A., Friedman, R.M. and Wallace, F. (2005), Implementation research: a synthesis of the literature, FMHI Publication No. 231, National Implementation Research Network, FL.
Gearing, R.E., El-Bassel, N., Ghesquiere, A., Baldwin, S., Gillies, J. and Ngeow, E. (2011), “Major ingredients of fidelity: a review and scientific guide to improving quality of intervention research implementation”, Clinical Psychology Review, Vol. 31 No. 1, pp. 79-88.
Hildebrand, M.W., Host, H.H., Binder, E.F., Carpenter, B., Freedland, K.E., Morrow-Howell, N., Baum, C.M., Dore, P. and Lenze, E.J. (2012), “Measuring treatment fidelity in a rehabilitation intervention study”, American Journal of Physical Medicine and Rehabilitation, Vol. 91 No. 8, pp. 715-724.
Holma, R.R., Paul, S.K., Bethel, M.A., Matthews, D.R. and Neil, H.A.W. (2008), “10-year follow-up of intensive blood glucose control in type 2 diabetes”, New England Journal of Medicine, Vol. 359 No. 15, pp. 1577-1589.
Katula, J.A., Vitolins, M.Z., Rosenberger, E.L., Blackwell, C.S., Morgan, T.M., Lawlor, M.S. and Goff, D.C. Jr (2011), “One-year results of a community-based translation of the diabetes prevention program: Healthy Living Partnerships to Prevent Diabetes (HELP PD) Project”, Diabetes Care, Vol. 34 No. 7, pp. 1451-1457.
McIntyre, L.L., Gresham, F.M., DiGennaro, F.D. and Reed, D.D. (2007), “Treatment integrity of school-based interventions with children in the journal of applied behavior analysis 1991-2005”, Journal of Applied Behavior Analysis, Vol. 40 No. 4, pp. 659-672.
Mars, T., Ellard, D., Carnes, D., Homer, K., Underwood, M. and Taylor, S.J.C. (2013), “Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity”, BMJ Open, Vol. 3 No. 11, p. 7, doi: 10.1136/bmjopen-2013-003555.
Miller, W.R. and Rollnick, S. (2002), Motivational Interviewing: Preparing People for Change, 2nd ed., Guildford Press, New York, NY.
Moore, G.F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O’Cathain, A., Tinati, T., Wight, D. and Baird, J. (2015), “Process evaluation of complex interventions: Medical Research Council guidance”, British Medical Jounrnal, Vol. 350, p. 7, available at: https://doi.org/10.1136/bmj.h1258
National Institute for Health and Care Excellence (2012), “Type 2 diabetes: prevention in people at high risk (PH38)”, National Institute for Health and Care Excellence, London, available at: www.nice.org.uk/guidance/PH38 (accessed 19 February 2016).
National Institute for Health and Care Excellence (2015), “Type 2 Diabetes in Adults: Management (NG28)”, National Institute for Health and Care Excellence, London, available at: www.nice.org.uk/guidance/ng28 (accessed 19 February 2016).
National Institute for Health and Care Excellence (2016), “NICE Quality and Outcomes Framework Indicator”, National Institute for Health and Care Excellence, London, available at: www.nice.org.uk/Standards-and-Indicators/QOFIndicators?categories=3896&page=1 (accessed 19 February 2016).
Nelson, M.C., Cordray, D.S., Hulleman, C.S., Darrow, C.L. and Sommer, E.C. (2012), “A procedure for assessing intervention fidelity in experiments testing educational and behavioural interventions”, Journal of Behavioral Health Services and Research, Vol. 39 No. 4, pp. 374-396.
NHS Confederation (2011), “From illness to wellness: achieving efficiencies improving outcomes”, Briefing No. 224, NHS Confederation, London, available at: www.nhsconfed.org/~/media/Confederation/Files/Publications/Documents/illness_to_wellness_241011.pdf (accessed 19 February 2016).
Peat, J. (2001), Health Science Research: A Handbook of Quantitative Methods, Allen & Unwin, Sydney.
South, J. (2015), A Guide to Community-Centred Approaches for Health and Wellbeing, Public Health England, London.
Trento, M., Gamba, S., Gentile, L., Grassi, G., Miselli, V., Morone, G., Passera, P., Tonutti, L., Tomalino, M., Bondonio, P., Cavallo, F. and Porta, M., and ROMEO Investigators (2010), “Rethink organization to iMprove education and outcomes (ROMEO): a multicenter randomized trial of lifestyle intervention by group care to manage type 2 diabetes”, Diabetes Care, Vol. 33 No. 4, pp. 745-747.
Viera, A.J. and Garrett, J.M. (2005), “Understanding interobserver agreement: the kappa statistic”, Family Medicine, Vol. 37 No. 5, pp. 360-363.
Whitmer, K., Sweeney, C., Slivjak, A., Sumner, C. and Barsevick, A. (2005), “Strategies for maintaining integrity of a behavioural intervention”, Western Journal of Nursing Research, Vol. 27 No. 3, pp. 338-345.
Acknowledgements
This work was supported by the Big Lottery Fund, Wellbeing Phase 2 Programme 2013-2016. This study was given ethics approval by the University of the West of England’s “University Ethics Committee” (Ref: HAS/14/03/49). All potential participants were provided with written and verbal information about the study, opportunities for withdrawal of data, and asked to complete a consent form. The authors declare that they have no conflict of interest.