Implementing and assessing a performance framework for the innovation measurement in a European manufacturer

Abstract

Purpose

Performance framework (PF) is a well-established practice to measure innovation performance and identify improvement opportunities. However, whether PFs academic research are applicable to companies remains unclear, as well as their support in the definition of improvement actions. This study aims to present the implementation and assessment of a new and updated PF proposed in previous research in a real industrial context.

Design/methodology/approach

The PF was implemented through an in-depth case study carried out in a European machinery manufacturer and further assessed by practitioners.

Findings

The results indicate that the PF enabled the creation of a multidimensional view of the innovation performance and the definition of improvement projects in the company. Additionally, the findings also reveal an overall positive assessment of the PF by senior managers who work with the innovation process.

Research limitations/implications

As a case study, this research is inherently limited in the extent to which results can be generalised. Thus, the analyses are reductive and rationalising. Future research is needed to assess the replicability of the PF.

Practical implications

The study's practical contribution is based on the combination of insights and steps that provide a straightforward and actionable approach for the company to improve performance.

Originality/value

This study aims to advance the importance of implementing the new and updated PF after its proposition, which is often overlooked in preceding research. Furthermore, the assessment of the PF also enables to infer its value to the company's employees.

Citation

Nappi, V. and Kelly, K. (2024), "Implementing and assessing a performance framework for the innovation measurement in a European manufacturer", International Journal of Productivity and Performance Management, Vol. 73 No. 11, pp. 69-95. https://doi.org/10.1108/IJPPM-07-2022-0356

1. Introduction

Performance measurement is vital for effective management in organisations. It is well established in the academic literature that performance measurement practices enable companies to understand how the results produced are contributing to the achievement of strategic goals; to track the effectiveness of actions, projects, or programmes; to chart the progress that is being made and implement the necessary adjustments; and, ultimately, to support informed decisions (Adams et al., 2006; Chiesa et al., 2009; Cooper and Kleinschmidt, 1995; Crossan and Apaydin, 2010; Lakiza and Deschamps, 2019). To do so, previous research informs valuable learnings for managers and researchers, such as the importance of including relevant performance dimensions to address the process being measured and the need for support in identifying opportunities to tackle the performance gaps (Dziallas and Blind, 2018; Frishammar et al., 2019; Lopes et al., 2022).

For innovation performance measurement, it is no different. In essence, measuring the performance of the innovation process requires the support of performance frameworks (PFs) which provide relevant information considering appropriate performance dimensions to assess the company's current position against its innovation goals on many fronts, enabling managers to develop and implement better strategies to achieve them (Lakiza and Deschamps, 2019; Richtnér et al., 2017). Nevertheless, research on PFs should account for how companies pursue innovation nowadays (Becheikh et al., 2006; Dewangan and Godse, 2014; Dziallas and Blind, 2018; Lopes et al., 2022; Saunila, 2017).

Firstly, there is an underlying demand for relevant performance dimensions up to date with the company's practices (Nappi and Kelly, 2022a). This is driven by the need for dimensions such as knowledge management (Dziallas and Blind, 2018; Frishammar et al., 2019; Nappi and Kelly, 2021) as well as related to new trends like openness (Kazemargi et al., 2022), servitisation (Gaiardelli et al., 2021) and sustainability (Almeida and Wasim, 2023; Hristov et al., 2022) within innovation environment that is often overlooked in the past (Nappi and Kelly, 2022a). Secondly, several researchers, Frishammar et al. (2019), Lakiza and Deschamps (2019) and Nappi and Kelly (2022b), acknowledge that PFs, besides providing pertinent dimensions, also need to be actionable and go beyond measuring performance to support the definition of improvement actions after the measurements. However, whether PFs with some performance dimensions and their measurement approaches theoretically proposed are actionable to companies remains unclear (e.g. PFs from Adams et al., 2006; Brattström et al., 2018; Dziallas and Blind, 2018). Consequently, without empirical development or testing of PFs in practice, companies may face the problem of measuring too little (by not considering relevant and comprehensive performance dimensions) or even refraining from identifying and implementing improvement actions at all once performance is measured (Lakiza and Deschamps, 2019; Lopes et al., 2022; Turulja and Bajgoric, 2019).

Therefore, existing PFs overlook important performance dimensions or lack support for developing action plans to improve innovation process performance in practice (Brattström et al., 2018; Lakiza and Deschamps, 2019; Lopes et al., 2022; Turulja and Bajgoric, 2019). To address this research gap, this study aims to implement and assess an action-oriented PF from previous research (Nappi and Kelly, 2022b) that enables the measurement of the innovation process performance across current and relevant dimensions as well as the definition of suitable improvement actions. This exploratory research is based on an in-depth case study to test the theory, which focuses on understanding the PF within specific settings of a European machinery manufacturer, studying phenomena in its environment rather than independent of context. In this way, this study helps to advance the importance of implementing the new and updated PF after its proposition and determining its value to practice beyond the companies where the proposal was developed, which is typically missed in previous research (already highlighted since Dziallas and Blind, 2018; Richtnér et al., 2017). The fresh arrangement of actionable steps delivers a valuable approach for managers in the company to make informed decisions regarding the improvement of innovation performance.

The remainder of this paper is organised as follows. First, the related theoretical background is discussed in Section 2, and the research method employed is presented in Section 3. Following this, the findings regarding the implementation of the PF are presented in Section 4, whereas its assessment is discussed in Section 5. Section 6, in turn, discusses the PF's applicability. Finally, Section 7 presents the research and practical implications, limitations and ideas for future research.

2. Background literature

The innovation process can be defined as iterative cycles of concurrent and sequential activities intertwined with decision gates aiming to develop ideas into marketable solutions: products and services (Crawford and Di Benedetto, 2011; Lee and Markham, 2016), and nowadays, the product–service systems (PSS), in which the material component is inseparable from the service, allowing new streams of revenue and lower environmental impacts than the products and services offered separately (Manzini and Vezzoli, 2003; Mourtzis et al., 2017).

Measuring the performance of the innovation process entails the definition of relevant performance dimensions and the application of performance indicators (PIs) to benchmark best practices to evaluate antecedents, activities and outcomes, thus ensuring that innovation is sufficiently supported and efficiently managed (Adams et al., 2006; Becheikh et al., 2006; Crossan and Apaydin, 2010; Dziallas and Blind, 2018). In this context, a PF enables managers to define what is essential to the company in terms of appropriate dimensions and PIs and how this information should be reviewed to identify performance gaps and define improvement actions accordingly (Chiesa et al., 2009; Crossan and Apaydin, 2010; Lakiza and Deschamps, 2019).

Several PFs overlook performance dimensions already proven critical to innovation, e.g. knowledge management (Adams et al., 2006; Crossan and Apaydin, 2010; Mishra et al., 2022). Others pay little attention to emerging dimensions from the current innovation landscape, mostly related to the innovation environment, such as openness, sustainability and servitisation, as identified by (Dziallas and Blind, 2018; Guimarães et al., 2016; Lee and Markham, 2016). Hence, new research must address relevant dimensions to provide a multidimensional view of the innovation process (Brattström et al., 2018; Frishammar et al., 2019; Nappi and Kelly, 2022b). Table 1 indicates the performance dimensions introduced in the PF proposed by (Nappi and Kelly, 2022b), noted as significant in the literature for the innovation process measurement from several studies.

According to Becheikh et al. (2006), Dziallas and Blind (2018), and more recently, Nappi and Kelly (2022a, b), two categories can be set up as internal and external aspects that affect the performance of the innovation process. The company-specific dimensions refer to those particular to a company's internal capability: innovation strategy; organisation and culture; knowledge management; portfolio management; project management; and team management. Secondly, contextual dimensions relate to a company's capability to deal with its surrounding environment: innovation environment (which should include openness, servitisation and sustainability), technology management and market.

By considering a wide range of performance dimensions, a PF can provide a comprehensive take on the measurement of the innovation process performance and further definition of actions to improve performance (Adams et al., 2006; Chiesa et al., 2009; Dziallas and Blind, 2018).

For this endeavour, PIs are indispensable (Lakiza and Deschamps, 2019; Nappi and Kelly, 2022b). PIs can be defined as metrics used by managers to track performance, determine the degree to which strategic objectives have been met and provide a standard basis to understand performance throughout time (Neely, 2005). In this sense, dimensions must be populated with related PIs to support performance measurement (Dziallas and Blind, 2018). The mentioned PF also provide examples of PIs to address the dimensions, e.g. “level of awareness and clarity of innovation goals” for the innovation strategy dimension see (Nappi and Kelly, 2022b). A sample of PIs applied for each dimension in the PF proposed by (Nappi and Kelly, 2022b) is provided in Table 2.

Another issue relating to PFs refers to the lack of procedures or steps indicating what to do after measurements (Lakiza and Deschamps, 2019; Lopes et al., 2022). Action must always follow measurement; otherwise, there is no point in wasting efforts in the process of measuring (Neely, 2005). Therefore, there is a latent need to extend PFs further than just the measurement of the PIs within the dimensions to support analysis of performance and the consequent definition of actions to improve performance steps, activities or procedures (Brattström et al., 2018; Crossan and Apaydin, 2010; Henttonen et al., 2016; Lopes et al., 2022). A deeper look into (Nappi and Kelly, 2022b) reveals a two-stage procedure first to measure innovation performance and then to define improvement actions. Nonetheless, the need to implement and assess the new PF remains in a real-world situation beyond the companies where the proposal was developed (Dziallas and Blind, 2018; Richtnér et al., 2017).

3. Research method

This study fits within the qualitative research paradigm as it applies a case study. A case study is the research method in which a detailed investigation of an object of study in its real-life context is carried out, drawn from multiple sources of evidence (Voss et al., 2016). This research design was employed for two main reasons: to implement and test the PF proposed from previous research in a real-world setting and gain a deeper understanding of actual innovation process performance measurement that enables the enrichment of existing theory.

The case study focuses on in-depth rather than large-scale research covering an entire population of interest. The aim is not to exhaust every possibility but to add relevant understanding in a strategically defined sample that may involve a small number or a single in-depth case (Voss et al., 2016). For this research, the selection of the unit of analysis was based on the same criteria from (Nappi and Kelly, 2022b): the formalisation of the innovation process, the existence of strategic objectives relating to innovation and the expressed concerns about the current innovation performance measurement process as well as the relevance of the technology-intensive company in the innovation landscape.

The single case study was conducted for 18 months, with nearly 630 h of empirical work in a medium-sized European machinery manufacturer in 2020 and 2021. The company has over 40 years of experience in high-technology farming solutions (products, services and PSSs), with approximately 165 employees. During the study, evidence was collected consciously and deliberately through several data-gathering methods following Nappi and Kelly's (2022b) research method aiming to test the PF and gain insights with key employees. Seven senior and middle management staff from research and development (R&D), operations, servicing, innovation management, sales and commercial, finance and front-end application participated consistently throughout the implementation, with additional employees participating on occasion. Through conducting research in the actual company and being exposed to real situations, new insights can be identified and developed, not by distant academics but by those working in close contact with the case company (Voss et al., 2016).

Following Nappi and Kelly (2022b), the data-gathering methods applied in the case study encompassed: document analysis, semi-structured interviews, focus group workshops and assessment questionnaires (Table 3). All the evidence gathered was captured in journals and analysed collectively. The document analysis involved the study of process models, roles and responsibilities, research project documentation and technical and financial reports. Seven semi-structured interviews were conducted to capture 136 data points per interview to measure the PIs within the dimensions to build up the innovation process performance (Appendix I: Interview). Each interviewee was asked to provide additional evidence to justify the answers, naming procedures, process outputs, and so on. The interviewee's responses were also triangulated with the other actors as well as with published reports and internal documents. Four focus group workshops were also held to shape the implementation of the new PF steps. Lastly, assessment questionnaires were applied to the company's employees against measurable success criteria for new PFs (Braz et al., 2011; Issa et al., 2015; Pigosso et al., 2013). The responses analysis was based on the average scores achieved using a quali-quantitative scale from (1) unsatisfactory to (4) very satisfactory, and the within-group interrater reliability rwg(i) to determine the level agreement among the employees' responses [1].

Even though the PF was proposed in previous research, it is not possible to generalise the findings from one single case study. Nonetheless, it is essential to highlight that the main purpose of this study is not to generalise the results but to increase understanding of the implementation of the PF and advance the existing theory a bit further to build a sound foundation for future analytical generalisation, as well as assess the value of the PF to practice.

4. Implementation of the PF

Based on the findings from the Nappi and Kelly (2022b) proposal, the PF design resulted in a two-stage procedure: Stage 1 aims to measure the current innovation process performance across a range of dimensions, and Stage 2 aims to evaluate and interpret performance to identify opportunities and define improvement actions.

Figure 1 displays the steps implemented at Stage 1 in the case study company. The kick-off started with the researchers outlining the importance of using relevant performance dimensions as well as establishing a systematic procedure for the endeavour of measuring and defining improvement actions to the company's head of innovation and head of operations (see step 1.1 in Figure 1). A step was also created to set up the core team to lead this and future cycles of implementation (step 1.2). Following the team's setup, a lesson learnt was to hold discussions to balance the core team's expectations and ensure their access to the information, documentation and systems, the so-called pre-requirements (step 1.3).

Once these early steps were carried out, the core team initiated the identification of the actual innovation process activities performed in the company (step 1.4). It was based on document analysis and observations of key employees to understand the day-to-day innovation process characteristics, e.g. the process formalisation, main drivers, closed/open innovation activities, and the surrounding environment. This characterisation equipped the core team, in step 1.5, to adapt the PIs covering the performance dimensions and PIs (shown in Table 2 Section 2) to the company's vernacular, ensuring employees' understanding of their terms.

Then, in step 1.6, the core team and representatives of the company's – R&D, sales and commercial, finance, operations, servicing, innovation management and front-end application defined key employees to be interviewed (ideally 6–25) [2] to quantify the PIs to measure the innovation process performance. In total, seven employees were selected who worked directly in the innovation process and product development. Face-to-face interviews were held to capture the data points to measure the PIs to determine the innovation process performance within the dimensions (step 1.7). It is important to highlight that the interviewees were asked to provide further evidence to support their answers, such as process outputs, documentation and reports. For this, the researchers also triangulated each interviewee's responses from the other participants as well as published reports and internal documents.

Based on the evidence collected from the interviews, the values for each PI were assigned, creating the view illustrated in Figure 2 (step 1.8). It presents a radar diagram to convey the diagnosis of the current innovation process performance across the nine dimensions radiating outward on spokes from a central hub. The visual arrangement of the PIs into 4 distinct levels designates an evolutionary line of action, in which these levels were sequentially ordered, from an initial level up to an ending level, considered the level of “excellence” across every dimension. As a starting point, the PF adopted the four-level range, one to four, to indicate the progression of performance that a company might present. The characterisation of each performance level was categorised with quantitative information with benchmark values of the PIs, depicting gradually increasing performance. In addition, the characterisation of each performance level was qualitatively complemented with the description of practices of increasing sophistication as the levels of performance increase.

The diagnosis shows that the company can be mainly characterised by level 1, with nine measurements still at this level, located in the following dimensions: innovation strategy, innovation environment, knowledge management, technology management and team management (see Figure 2). The remaining measurements are located at levels 2 (20 PIs) and 3 (5 PIs). The core team established that the company's level would be defined by the lowest level, with at least eight measurements since 34 PIs cover the four levels. The current innovation performance of the company reveals that there are limited resources, so the focus is on obtaining those necessary to maintain the main (or most profitable) products and PSSs in the market. The company is oriented to its internal setting and daily operations, paying little attention to the external environment in terms of prospecting potential partners or building cooperation networks to open innovation. Innovating is not a priority, but interest in the topic has been awakened due to competitors' analysis and customer demands, even though senior management does not entirely understand what innovation implies for the company. For these reasons, the core team level labelled level 1 “innovation revealed” insofar as innovation is perceived (“revealed”) as an alternative (“salvation”) for the company's long-term survival, even if management only knows they should do something about it.

Once the company's current innovation performance was consolidated, the diagnosis was validated by not only the heads of innovation and operations but also a wider crowd of employees in the company during presentations. With the “go ahead”, Stage 2 steps began to establish steps to identify improvement opportunities and suitable actions. The resulting step-by-step approach is exemplified in Figure 3.

Stage 2 started with a step for defining which level of performance the company wishes to pursue and achieve, creating a vision of the desired performance level (levels 2 to 4). For this, the core team proposed a pathway that allows a degree of flexibility to the company according to the current performance shown in the diagnosis (step 2.1). There are two possibilities:

1. Staged approach (step 2.1’): an orderly way to define the vision of the desired performance level for the company with a low-performance level (levels 1 and 2). It is based on progressing one level at a time, targeting the gaps at the lowest performance level and then moving up.

2. Continuous approach (step 2.1’’): a flexible approach recommended only for the company with a higher performance level (3 or 4). This way, the company can choose to focus on different levels related to one or several dimensions according to its drivers and strategic objectives.

In essence, the company's current innovation performance dictates the improvement pathway to follow. The company was characterised in the previous stage as level 1, so the staged approach was the one to be employed. This meant that the nine measurements still at level 1 are the performance gaps that must be addressed for the company to achieve level 2. They refer to the following dimensions: innovation strategy, innovation environment, knowledge management, technology management and team management (see Figure 2).

The next step aims to identify suitable innovation practices to help define improvement actions for addressing the gaps identified earlier by either a staged or continuous approach (step 2.2). Each gap was analysed with the help of a collection of innovation practices relating to each PI (Appendix II: Improvement actions). This collection of practices allowed the core team to benchmark and define the most beneficial innovation practices to attend to the gaps at performance level 1 to levelling up the company to level 2. In total, the core team identified seven improvement actions, as summarised in Table 4.

These improvement actions were further detailed in improvement projects to facilitate their implementation (step 2.3). These improvement project charters contained: the projects' goals, short descriptions, main deliverables, implementation requirements, risks, estimated time and resources. Besides, additional PIs were selected for each project to help track the implementation efforts (step 2.4).

With seven projects in hand, the core team set out to prioritise them (step 2.5). For this, the core team developed an electronic spreadsheet applying the analytical hierarchy process (AHP) [3] to help prioritise. One or more criteria can be used in the prioritisation spreadsheet according to the stakeholders' preference: implementation time, strategic alignment, top management support, resources availability, cost, competitive advantage, legal compliance and return on investment, with implementation time as the default criterion. Based on the discussion between the core team and the heads of innovation and operations, projects 1 and 2 were prioritised.

From this point onwards, senior management at the company was responsible for defining the projects' schedule, work packages and people involved (step 2.6) to produce a roadmap for implementing projects 1 and 2 and planning for future implementations (projects 3–7). The total duration of the improvement cycle of the PF can vary according to top management support and the company's resources.

During the implementation of the improvement projects, special attention should be directed at people change management, as people are the gatekeepers of change. It is a critical factor for the success of improvement projects (Jeston and Nelis, 2006). Resistance to change, leadership roles, change planning and communication, employees' motivation and staff training should be taken into account. Having a core team to lead the implementation of the PF in the company could facilitate the consideration of people change management issues.

5. Assessment of the PF

As a new PF, it needs to be assessed by its users to demonstrate value to the practice. That is the reason why the PF was assessed in an individual questionnaire by the company's employees. Using the same measurable success criteria from previous research (Braz et al., 2011; Issa et al., 2015; Pigosso et al., 2013), the following were included: the PF's utility, consistency, scope, precision, broadness, objectivity, clarity, depth, coherence, instrumentality, simplicity and forecast. To analyse the level of agreement among the employees' responses ranging from (1) unsatisfactory, (2) needs improvement, (3) satisfactory to (4) very satisfactory, the within-group interrater reliability (rwg) was applied. The interrater varies between 0 and 1; the closer to 1, the stronger the agreement, i.e. the more consistent the responses are. Values ≥ 0.70 are considered an indication of a sufficient agreement (Farris et al., 2007).

Four senior managers related to either R&D or innovation management, who participated in the case study, answered the questionnaire. This provides a more robust assessment dataset than similar studies proposing new PFs (e.g. Issa et al., 2015; Pigosso et al., 2013) that are normally based on a single senior manager's responses. Figure 4 illustrates the average score achieved among the employees' responses (μ(i)) from the assessment questionnaires, while Table 5 complements it with the standard deviation (S(i)), and level of agreement (rwg(i)) for each criterion.

The data analysis reveals a positive assessment of the new PF, with satisfactory average scores (μ(1→7)>3.00) and sufficient levels of agreement (rwg(1→7)>0.70) were obtained for all criteria. One of the highest scores given by the company's employees relates to the depth of the diagnosis. In contrast to one of the issues from preceding research referring to the lack of current performance dimensions discussed Section 2, the new PF offers a comprehensive view incorporating the relevant dimensions and PIs into the diagnosis. This overview enables managers to not only get a comprehensive picture of the innovation process but also focus on particular dimensions of interest. To corroborate this point, practitioners from the company also stated in the assessment that “[the PF] contributes to raising awareness among employees to measure relevant dimensions of the innovation process systematically from time to time”.

The second highest assessment score refers to the PF's simplicity. This assessment can reflect the understanding achieved in the company as the management level was able to see the improvement opportunities in a single vision (in the diagnosis) and to make decisions on which ones to address based on relevant information (improvement projects 1 and 2). In fact, the actionable steps of the PF could be a resourceful aid for innovation managers to apply further cycles to create a continuous improvement basis, with more replications of the study. Therefore, managers may use the steps as a roadmap for supporting their improvement practices.

In sum, the assessment findings confirm that the new PF is seen by the case company as a valuable instrument that supports the innovation management measurement and further identification of improvement actions, thus, supporting the proposition advocated in this research.

6. Discussion

The findings of this research emphasise the importance of establishing a PF to support management in the measurement of innovation process performance for the case company. By applying a case-oriented research method, the developed PF has the potential for further applicability. To discuss applicability, a comparative analysis was carried out between the new PF and existing ones from the literature based on the works of (Henttonen et al., 2016; Nappi and Kelly, 2022b).

Table 6 presents the synthesis of the comparative analysis. The following features discussed in previous research were used for the comparison: (1) display of the current innovation process performance, preferably in a single vision, via a diagnosis, an audit or equivalent assessment (Alegre et al., 2006; Chiesa and Frattini, 2007); (2) a broad set of performance dimensions (Boly et al., 2014; Markham and Lee, 2013); (3) supply of a procedure throughout the application (Medori and Steeple, 2000; Neely et al., 2002); and, (4) support in the definition of actions plans to improve performance (Niven, 2006; Tangen, 2004).

Firstly, from the existing PFs shown in Table 6, not all display the current innovation process performance in a single vision in a graphic format to update managers with performance information. Nevertheless, researchers such as Pigosso et al. (2013) show that managers are more willing to pursue improvements in the process of interest when a diagnosis (or equivalent) is provided. This study also shares this finding, as, after the measurements, the company's managers shared with the researchers that the PF increased employee awareness to measure relevant dimensions of the innovation process. Furthermore, though each company has its own specificity, including the case company, this finding could support policymakers in their efforts to better foster innovation. For instance, the single vision could provide a visualisation aid to complement the European innovation scorecard in an actionable channel at the company level. By doing so, it would be possible to compare the innovation performance (Figure 2) of different companies in the divisions of the manufacturing sector, later in distinct industries and then to inform recommendations based on insights for policymakers to develop and communicate clear policies to promote the sectors where they want to foster innovation.

Secondly, even though most PFs apply performance dimensions, they do not provide a complete overview dimension-wise. Compared to preceding PFs (Table 1 in Section 2), it is possible to observe that none of the studies applies all cited dimensions at once. In contrast, the new PF applied in this study provides a holistic view incorporating the performance dimensions most commonly demonstrated in the literature. This enables innovation managers not only to get an overall picture of the performance but also to focus on particular dimensions of interest, considering the strengths and weaknesses particular to the company. Additionally, providing the dimensions can help managers who may need to define relevant dimensions and know more about recent trends in the innovation landscape, e.g. innovation environment with openness, servitisation and sustainability. This identification of dimensions can present benefits for researchers too. Dimensions that are reliable and valid enable the accumulation of research in a scientific field and free further researchers from redeveloping them. Furthermore, the comprehensive take on dimensions can also help policymakers pinpoint critical dimensions, further establish objectives to be reached and, particularly, develop adequate financial support in the form of subsidies for companies which wish to undertake innovation-related activities. However, it should be noted that too much focus on only one may lead innovation managers or even policymakers to miss valuable opportunities.

Thirdly, providing a procedure in the PF is an aspect typically related to practice but also acknowledged by researchers (Medori and Steeple, 2000; Neely et al., 2002). Still, a little more than half of the PFs either explicitly present a procedure or implicitly demonstrate it through case studies. On the other hand, the proposed PF clearly provides a step-by-step procedure in a two-stage approach. This also enables innovation managers to spend less time “reinventing the wheel” all over again and more time on adjustments and tailoring the steps. This detailed procedure could also help managers aiming to implement the International Standard Organisation ISO 56002: 2019 newest standard focused on the establishment, implementation, maintenance and continual improvement of a system to manage the innovation process. Even though the standard aims to provide guidance, it is limited in terms of actionable instructions (Lopes et al., 2022). Thus, the procedure provided in this study could support companies in preparing and successfully passing the innovation audits. In addition, it is worth mentioning the possibility of using one of the two approaches (staged or continuous) proposed in the procedure for defining the desired performance has the potential to facilitate its adaptation to other companies similar in size.

Fourthly, only three previous PFs hint at what to do after measurements are taken (e.g. Chiesa et al., 1996; Frishammar et al., 2019; Lakiza and Deschamps, 2019). Nonetheless, they do not provide explicit support with detailed steps for identifying improvement gaps and the subsequent definition of action plans. In this sense, the PF adds to the literature detailed steps for managers to systematically identify the current gaps and deploy action plans but also provides further examples of a collection of improvement actions that can be further specified into improvement projects. For policymakers, this collection of improvement actions could help disseminate best practices within the setting of meeting places and occasions maintained by policymakers where various economic entities (i.e. companies, financial institutions, research institutes, etc.) belonging to the same or related sectors can meet and exchange ideas to encourage innovation.

Regarding the transferability of the research results, the case company can be considered a representative of the European industry, as medium-sized enterprises correspond to most manufacturing companies (Eurostat, 2020). Furthermore, high-technology farming solutions can be regarded as a good proxy of the manufacturing innovation industry because the technologies and the market are prominent. Nevertheless, before transferring the results to another environment, the contextual differences should be carefully considered.

To summarise, when compared with previous PFs, the novelty of this study resides in the fact that the PF consolidates distinct elements from the literature, e.g. the performance dimensions populated with PIs, but combined in a new actionable way that goes beyond only the measurement, including a comprehensive definition of improvement actions, particularly for the company participating in the study.

7. Conclusion

Performance measurement has a central role in supporting the management of the innovation process. Nonetheless, until now, research does not advise a PF providing performance dimensions relevant to the current innovation landscape nor support the identification and definition of improvement actions after the measurements (Dziallas and Blind, 2018; Frishammar et al., 2019; Nappi and Kelly, 2022b).

In turn, this study applied and assessed a new and updated PF that enabled the case-study company to measure the innovation process performance across relevant nine dimensions captured in a multidimensional diagnosis, identify improvement opportunities after this measurement and define suitable actions as two improvement projects were selected to be implemented. As the PF was tested in a manufacturing company through a case study, it allowed real-world insights to be considered empirically into the development of the tool. Furthermore, the PF was also assessed by the practitioners in the company, the real users, with positive results, providing legitimacy to the instrument.

This study has contributed to theoretical and empirical knowledge in the field of innovation process performance measurement. Firstly, the consideration of nine relevant performance dimensions and related PIs arranged in a multidimensional overview allows the creation of a novel and comprehensive diagnosis of the company's innovation process new to the literature. Secondly, the systematisation of a new step-by-step PF enables the deployment of performance measurement ‘results’ into action-oriented plans to improve performance and sheds light on parts often overlooked in PFs of preceding studies. Moreover, combining these two contributions helps establish a continuous improvement basis in the innovation measurement in an actionable manner for the company providing a fresh perspective to research on performance measurement of the innovation process.

This study also presents practical implications. To begin with, it makes available useful advice in the new PF, helping managers in the case study company to measure and evaluate performance to make informed decisions regarding their innovation process. This is substantiated by the results attained in the assessment and the practitioners' anecdotal evidence in the final questionnaire. Additionally, the study reveals the PF's potential to establish a common language across the company concerning the innovation process and its performance measurement, which is especially critical in times of remote and on-site working, a trend more common these days. On the other hand, implementing a PF in a company is by no means straightforward. To avoid common mistakes, managers should take a holistic perspective on their company's innovation process and consider its specificities. Finally, for policymakers, the findings also provide avenues to better foster innovation, e.g. by delivering a single vision that could complement the European innovation scorecard.

Some limitations of this study should be mentioned. Because of the adopted research method (single case study), results cannot be statistically generalised. They may only be analytically extended to other medium-sized European machinery manufacturer operating in high-technology farming solutions. Even if the construct and internal validity of the empirical results are ensured by using multiple data-gathering methods, establishing a chain of evidence and triangulation of multiple sources, and by having the case study journals reviewed by key participants, the study does not present generalisability of the research findings beyond the immediate study.

The aforementioned limitations, as well as the increased understanding of the researchers on the topic, allow several avenues of further research presented here as questions, as follows:

1. What is the landscape of the performance level of manufacturing companies operating in other industries in specific geographical regions? Are there significant differences to be identified in the distinct divisions of the manufacturing sector?

2. How might the PF function as an online tool in the context of companies enabling more remote work for parts of their staff due to the pandemic? Are the results the same as for face-to-face interactions?

3. How to support the use of PF in the context of innovation ecosystems (a network of companies and other entities)? What are the changes to be made in the supporting elements of the PF?

4. What is the role of big data once a more extensive base of case studies is achieved? Could big data also play a role in identifying further indicators to feed the PF (e.g. service performance data for the innovation environment)?

5. How can companies be supported in applying the PF in the context of eco-innovation? What are the changes to be made in the database elements?

6. Is it possible to apply the PF in companies with a low innovation process formalisation? What adjustments would need to be made?

7. How to keep the PF up to date considering the increasing proposition and development of innovation practices?

8. How to spread the application of the PF across the supply chain? How to deploy the application of the PF among suppliers?

Figures

Figure 1

Diagram depicting the steps of Stage 1

Figure 2

Company's innovation performance

Figure 3

Diagram depicting the steps of Stage 2

Figure 4

Graph illustrating the assessment scores achieved in the company

Table 1

Performance dimensions from Nappi and Kelly (2022a, b) performance framework (PF)

	Performance dimensions
Company-specific	Innovation strategy (IS)	Adams et al. (2006), Becheikh et al. (2006), Chiesa et al. (1996, 2009), Crossan and Apaydin (2010), Dziallas and Blind (2018), Lee and Markham (2016), Mishra et al. (2022)
	Organisation and culture (OC)	Adams et al. (2006), Becheikh et al. (2006), Crossan and Apaydin (2010), Dziallas and Blind (2018), Lee and Markham (2016), Mishra et al. (2022)
	Knowledge management (KM)	Adams et al. (2006), Becheikh et al. (2006), Chiesa et al. (2009), Crossan and Apaydin (2010), Dziallas and Blind (2018), Lee and Markham (2016), Mishra et al. (2022)
	Portfolio management (PFM)	Adams et al. (2006), Crossan and Apaydin (2010), Lee and Markham (2016)
	Project management (PM)	Adams et al. (2006), Becheikh et al. (2006), Chiesa et al. (1996, 2009), Crossan and Apaydin (2010), Dziallas and Blind (2018), Lee and Markham (2016)
	Team management (TEAM)	Adams et al. (2006), Chiesa et al. (1996), Crossan and Apaydin (2010)
Contextual	Innovation environment (IE) (Openness, servitisation and sustainability)	Becheikh et al. (2006), Chiesa et al. (1996), Dziallas and Blind (2018), Lee and Markham (2016)
	Technology management (TM)	Becheikh et al. (2006), Chiesa et al. (2009), Chiesa and Masella (1996), Lee and Markham (2016)
	Market (MA)	Adams et al. (2006), Becheikh et al. (2006), Chiesa et al. (2009), Crossan and Apaydin (2010), Dziallas and Blind (2018), Lee and Markham (2016)

Source(s): Prepared by the authors

Table 2

Sample performance indicators (PIs) from the performance dimensions (Nappi and Kelly, 2022a, b)

Performance dimensions	[Id] PI
IS	[IS1] Level of awareness and clarity of innovation goals
	[IS2] Corporate goals for the new product development program
	[IS3] Product planning horizon (years, product generations)
	[IS16] Top management support for innovative ideas
OC	[OC1] Organisational climate for innovation projects
	[OC9] Work environment support for innovation projects
KM	[KM1] Rate of generated ideas according to formal versus informal innovation activities
	[KM8] Knowledge acquisition vs knowledge absorptive capacity
	[KM16] Importance of diversity of knowledge sources
	[KM27] Time off for creative things
PFM	[PFM1] Level of formalised portfolio management
	[PFM10] Portfolio decision-making effectiveness
	[PFM11] Innovation project portfolio alignment
PM	[PM1] Level of commitment of resources for innovation/new product projects
	[PM23] Percentage of use of project management tools
	[PM24] Frequency of post-launch evaluation procedures
	[PM31] Internal and external communication quality
	[PM32] Time-to-market management
TEAM	[TEAM1] Level of cross-functionality in teams
	[TEAM2] Identifiable project leader
	[TEAM3] Frequency of cross-functional training
	[TEAM5] Dedicated group assigned to innovation tasks
	[TEAM17] Innovative team behaviour
IE	[IE1] Recognition that key problems must be solved with skills outside the organisation
	[IE2] Collaborative projects through an externally vs internal focused open innovation system
	[IE12] New product diversification as a strategy: goods, services, or inseparable mix of both
	[IE21] Utilisation of sustainability criteria for innovation projects
TM	[TM2] Level of monitoring new technologies
	[TM12] Intellectual property protection strategy effectiveness
	[TM13] Degree technology tools used
	[TM22] R&D intensity
MA	[MA1] Percentage of use of market research tools
	[MA11] Product customer testing proficiency
	[MA15] Market launch proficiency

Source(s): Prepared by the authors

Table 3

Data gathering methods applied

Data gathering methods	Main activities	Hours spent	Total hours
Document analysis (656 pages)	Collection Analysis	42 77.5	630
Key employees' interviews (7)	Planning Interviews Transcription and coding Analysis	63.5 14.6 78.4 103.4
Focus-groups workshops (4)	Planning Execution Analysis	65.4 15 116.7
Evaluation questionnaires (7)	Planning and application Analysis	18 35.5

Source(s): Prepared by the authors

Table 4

Summary of the defined improvement actions for gaps in level 1

Performance dimension	Improvement actions	Charters
Innovation strategy	1. Implement the “Delphi Method” for innovation planning	Project 1
Innovation environment	2. Roadmap innovation partnerships	Project 2
	3. Screen for PSS	Project 3
Knowledge management	4. Implement an idea management system	Project 4
Portfolio management	5. Establish innovation portfolio management	Project 5
Technology management	6. Monitor in-house R&D	Project 6
	7. Train “ambidextrous teams”	Project 7

Source(s): Prepared by the authors

Table 5

Analysis of the performance framework (PF) assessment

	Criteria	μ(i)	S(i)	rwg(i)
1	Utility: How do you evaluate the utility of the PF in supporting the company's innovation performance measurement and definition of improvement actions?	3.50	0.50	0.80
2	Consistency: How do you evaluate the consistency of the nine performance dimensions and the PIs used in the diagnosis of the PF?	3.25	0.43	0.85
3	Scope: How do you evaluate the PF in relation to the adequacy of the scope in the proposition of the improvement actions?	3.50	0.50	0.80
4	Precision: How do you evaluate the PF in relation to the precision of the diagnosis with the dimensions and PIs provided?	3.50	0.50	0.84
5	Broadness: How do you evaluate the PF in relation to its applicability in manufacturing companies from different sectors?	3.00	0.00	1.00
6	Objectivity: How do you evaluate the objectivity of the PF in performing the company's diagnosis and proposing the improvement projects?	3.00	0.00	1.00
7	Clarity: How do you evaluate the PF concerning the clarity in which the results are presented, e.g. the diagnosis?	3.50	0.50	0.80
8	Depth: How do you evaluate the PF in relation to the depth of the diagnosis and the performance dimensions?	3.75	0.43	0.85
9	Coherence: How do you evaluate the coherence of the diagnosis and the improvement projects proposed using the PF?	3.25	0.43	0.85
10	Instrumentality: How do you evaluate the PF in relation to its instrumentality in the diagnosis (e.g. interviews, workshop) and the proposition of the improvement actions (improvement projects)?	3.25	0.43	0.85
11	Simplicity: How do you evaluate the PF in relation to the simplicity of the resulting procedure?	3.75	0.43	0.80
12	Forecast: How do you evaluate the procedure in relation to the definition of the next steps to be taken after the proposition of the improvement actions?	3.25	0.43	0.80

Source(s): Adapted from Braz et al. (2011), Issa et al. (2015), Pigosso et al. (2013)

Table 6

Comparison of the performance frameworks (PFs) found in the literature

PFs	Features
	i) Display current performance	ii) Performance dimensions	iii) Supply of a procedure	iv) Support for action plans
Brown and Gobeli (1992)		X	X
Chiesa et al. (1996)	X	X	X	X
Werner and Souder (1997)		X
Loch and Tapper (2002)	X	X	X
Barczak et al. (2006)	X	X
Berg. et al. (2009)	X	X	X
Chiesa et al. (2009)
Crossan and Apaydin (2010)
Lakiza et al. (2018)		X	X	X
Frishammar et al. (2019)	X	X	X	X
New PF	X	X	X	X

Note(s): Empty cells indicate the feature (presented in the corresponding column) is absent from the specified PF

Source(s): Prepared by the authors

Notes

1. The within-group interrater reliability is calculated as follows: rwg(i)=1−(Si2i/σi2), where S is the standard deviation of the given scores by the respondents, and σ2 is the expected variance due to random. The variance is calculated assuming that the scores have a uniform distribution, i.e. the scores have the same probability of occurrence. Variance is: σi2=(A2−1)/12, where A is the number of possible answers (James et al., 1984). The assessment results are discussed in Section 5.

2. Although little practical guidance is available for defining sample sizes of interviews, Guest et al. (2006) and Morse (2000) indicate that six can be sufficient to capture meaningful results and achieve theoretical saturation (with a maximum of 25).

3. AHP is a decision-making method that involves decomposing a decision into pairwise comparisons so people can make value judgements about alternatives that are arranged into a ranking (Saaty, 1990).

Appendices