Understanding business analytics continuance in agile information system development projects: an expectation-confirmation perspective

2.1 Business analytics

BA refers to the analytical component of business intelligence (BI), that is, the analytics and reporting processes and technologies used in BI (Chen et al., 2012). BI is the process of combining insights from data with business knowledge to support decision-making processes (Larson and Chang, 2016). Today, BA spans several technical areas including (big) data analytics, text analytics, web analytics, network analytics, social media analytics and mobile analytics. All these areas rely on data mining, statistics and artificial intelligence (AI) techniques like machine learning (ML) and natural language processing (NLP) (Chen et al., 2012; Larson and Chang, 2016). BA investments are very resource-intensive and risky (Wang et al., 2019). Thus, value is the most important of all the Vs used to characterize (big) data used for BA (Acito and Khatri, 2014; Ashrafi et al., 2019; Fosso Wamba et al., 2015). Business organizations need to capitalize on the value generated from speedy actionable insights and the pervasive use of BA by employees (Wixom et al., 2013) to gain competitive advantages, enhance firm performance and create strategic value. This value is created by uncovering hidden knowledge, improving decision making and supporting strategic planning based on data (Chae, 2014; Chiang et al., 2018). Furthermore, maximizing value from BA investments requires aligning strategy and desirable behaviours to business performance management in conjunction with analytic tasks and capabilities (Acito and Khatri, 2014, p. 566). Analytical tasks include producing, consuming or enabling the creation of insights while capabilities refer to the tools, methods and technologies that support decision making, data analysis and information management.

Recent studies have mostly focused on how BA adoption affects competitiveness, agility and firm performance. Firm agility refers to the ability of a firm to quickly sense changes in an environment (opportunities or threats) and adapt to such changes quickly enough to take advantage of the situation (Ashrafi et al., 2019; Kitchens et al., 2018). BA improves firm agility by enhancing information quality and innovative capabilities (Ashrafi et al., 2019). High-quality information will enable firms to better sense changes in their environments and enable them to make timely decisions needed to adapt to the changes. BA provides such timely information, allowing firms to innovatively adapt to changes and outsmart their competitors. Firm agility enhances firm performance, especially in technologically turbulent environments. The adoption of BA has no direct effect on firm performance but it helps improve business process performance which in turn influences firm performance (Aydiner et al., 2019). However, analytics resources (analytics technology assets and BA capabilities) have a direct effect on business performance (Krishnamoorthi and Mathew, 2018). The relationship between BA and firm performance is mediated by business process change (Torres et al., 2018).

Nevertheless, the adoption of BA depends on technological, organizational or environmental factors related to individual perceptions within the organization and could vary with the stage of adoption. Technological factors (data infrastructure, data quality management) influence all stages of the innovation diffusion process, organizational factors (managerial obstacles, analytics centralization) influence the adoption and assimilation stage, while environmental factors (competition intensity) influence the initial stage (Nam et al., 2019a). IT competence (IT infrastructure, IT business spanning, IT proactive stance) influences data management capabilities (data quality, data integration) which then affects the use of BA for customer relationship management (CRM) and eventually the CRM performance (Nam et al., 2019b). Meanwhile, BA competency and organizational absorptive capacity influence BA assimilation which in turn helps firms improve competitive advantage (Wang et al., 2019). This confirms that firms can only benefit from their BA investments if their users adopt and use them effectively (Jaklič et al., 2018).

However, aligning BA processes and software with the work styles, changing needs and expectations of users is a challenge that affects user perceptions of how well BA fits their jobs. Compatibility directly affects the intention to use BA and mediates the effect of performance perceptions on intention to use. However, the issue of how well BA fits with user expectations has not been thoroughly researched (Jaklič et al., 2018). This line of thought has led to several calls for studies that can help researchers and practitioners understand the behavioural decisions of BA users regarding the (continuous) use of BA in organizational contexts (Dennehy et al., 2020; Elhoseny et al., 2020). This is the research gap this study seeks to fill by examining what employees in agile ISD projects expect from BA and how this affects their intentions to continue using the BA technologies adopted by their organization.

2.2 Agile information system development projects

The IS discipline has accumulated a significant body of knowledge on ISD that significantly complements ISD literature in software engineering and project management disciplines (Hassan and Mathiassen, 2018). A major organizational concern today is the failure of ISD projects. However, the extant literature pays little attention to the conditions and situations that occur during ISD projects that lead to their failure (Baghizadeh et al., 2020). Understanding such conditions and situations could lead to early detection and timely responses towards avoiding project failures. A mutual understanding between key project stakeholder groups regarding project planning and control mechanisms is critical for project success (Jenkin et al., 2019). Furthermore, control type, control degree, control style and control execution affect the development of mutual understanding between the stakeholders (Gregory et al., 2013). ISD projects are inherently complex because they involve dealing with organizational and technological issues (Benbya and McKelvey, 2006; Xia and Lee, 2005) like increasing dependence on a particular IS or limited IT infrastructure to accommodate evolving technologies. Furthermore, there are considerable inherent risks in ISD projects regarding differences between expectations and perceptions which are specific to each actor's knowledge, goals and values (Öbrand et al., 2019). Coping with diversity, knowledge and structure are the most persistent problems and challenges faced by developers in ISD projects at the organizational, environmental and individual levels (Bergvall-Kåreborn and Howcroft, 2014).

A growing methodological approach adopted in ISD projects today is the agile methodology. ISD agility is the continual readiness of an ISD method to rapidly or inherently create change, proactively or reactively embrace change and learn from change while contributing to perceived customer value (economy, quality and simplicity), through its collective components and relationships with its environment (Conboy, 2009, p. 340). This methodology was developed to overcome challenges related to change management, project visibility, transparency, productivity, time-to-market and cost reduction in fast-paced project environments (Alaa and Fitzgerald, 2013; Baham et al., 2017; Kautz, 2011). In ISD projects, agile methodologies can be used for project management and software development practices. In either case, this methodology positively affects employees' perceptions of job satisfaction and job characteristics like job autonomy and task significance (Tripp et al., 2016; Werder and Maedche, 2018). Team diversity, team capabilities, team perceptions, project communication, project technology and project setting are categories of knowledge sharing risks across agile software development teams (Ghobadi and Mathiassen, 2017). Within these teams, project managers are more concerned about project setting barriers while other team members like developers, testers and user representatives are more concerned about project organization and team capability barriers (Ghobadi and Mathiassen, 2016; Taylor, 2016).

Regarding BA usage in ISD projects, current literature mostly focuses on the role of agile methodologies in BA projects. It mostly examines how organizations have successfully adopted agile principles and practices in their BI/BA projects, and how this has altered agile methodologies (Larson and Chang, 2016). The agile principles and practices used for ISD align with the nature of BA projects and significantly decrease the chances of project failure (Larson, 2019; Vidgen et al., 2017). This has led to the creation of a development style called agile analytics, which describes how agile principles and practices can lead to agility in BI projects (Collier, 2012). To successfully implement agile analytics, a strong customer involvement, a methodical project definition process and effective agile implementation methods are required (Batra, 2018; Kisielnicki and Misiak, 2016; Tsoy and Staples, 2020). Nevertheless, BA and agile methodologies when used together improve the productivity of ISD teams. It can support every step of the ISD life cycle, including code repository analytics, project management analytics and application usage analytics (Biesialska et al., 2020). It also helps ISD projects achieve greater agility through improved speed and flexibility (Stodder, 2013). For example, it makes translating business requirements into BA products and services much easier and faster for agile ISD teams than for teams that do not use agile methodologies (Wixom et al., 2013). Nevertheless, no research examines the reasons why ISD project members would continue using.

4.1 Data collection

To test the proposed research model, an online questionnaire-based survey was used to collect data from employees working in agile ISD projects. This approach was chosen because it is easy to replicate, facilitates the investigation of multiple constructs simultaneously and easy to generalize the results during exploratory studies based on predictive models (Boudreau et al., 2001; Pinsonneault and Kraemer, 1993). Well-established measurement items were used to measure each construct used in the model. All constructs were measured using a 7-point Likert scale. A pre-test was conducted on 20 employees working on ISD projects to assess the face and content validity of the items used in the study as well as the clarity of the questionnaire.

The study was sent to 400 employees working on agile ISD projects in companies in Ireland and Sweden, two of the most globally competitive countries worldwide in terms of productivity (Klaus, 2019). Respondents were contacted either directly through phone calls and emails or via LinkedIn groups. At the end of the questionnaire, participants were also offered the opportunity to comment on the questionnaire and to provide additional comments on their perceptions of BA use, especially regarding their intention to continue/discontinue using BA software or techniques in their projects. The data collection process lasted three months (June 2020–August 2020) and the average completion time of the questionnaire was 10 min. From the 161 responses collected, 153 responses were retained for further analysis.

Table 1 presents a description of the sample. Most of the respondents were from Europe (74%) from technology-oriented companies including software and services development (56%). The remaining respondents work in various business sectors such as IT consulting (13%), e-commerce (9%) and others.

Summarily, most respondents had at most 4 years of experience with BA (60%). The respondent has worked in small teams (1–9 (31%) or 10 to 49 (28%)) and occupied mostly team lead (25%), developer (23%) or business analyst (18%) positions (see Table 2).

4.2 Data analysis

To avoid measurement errors due to common method bias (CMB), the following measures were taken (Podsakoff et al., 2003): (1) questions in the survey were juxtaposed to create a psychological separation of measurements; (2) to ensure that respondents provided honest answers, they were informed that the study was for academic research purposes, their participation will remain anonymous and there were no wrong or right answers; (3) scale items were improved after the pre-test by rewording some terms to make the questionnaire easy to understand thereby reducing biases related to the ambiguity of items; (4) Harman's single-factor test was performed to determine whether most of the variance in the model can be accounted for by a single factor. The test showed that there is no threat of CMB in this study since the maximum variance explained by a single factor was 28.8% which is below the 50% threshold value.

To evaluate the model proposed in this study, partial least squares structural equation modelling (PLS-SEM) approach was used with the help of the SmartPLS 3.2.9 software package. The PLS-SEM approach is a well-established approach used in quantitative research to understand relationships between variables and the predictivity of models in exploratory studies (Hair et al., 2014; Hair et al., 2016; Leguina, 2015). It is a variance-based SEM technique which is more appropriate for exploratory studies where the aim is to examine relationships between variables based on theory (Reinartz et al., 2009). This approach involves two main phases: (1) evaluating the measurement model and (2) evaluating the structural model (Hair et al., 2017).

5.1 Measurement model

Evaluating the measurement model involves assessing the reliability and validity of the constructs and their corresponding items. Based on Hair et al. (2016), items are reliable if the item loadings are greater than 0.70. For constructs to be reliable, they should have composite reliability (CR) and Cronbach's alpha (α) values above 0.60. To ensure that the constructs significantly explain the variance of its items, the average variance extracted (AVE) should be greater than 0.50. Finally, the Fornell–Larcker criterion can be used to assess the discriminant validity of the constructs. Discriminant validity is verified if the square root of the AVE for each construct is greater than the correlation between the constructs (Fornell and Larcker, 1981).

The items used to measure each construct were obtained from existing measurement scales that have been empirically tested and validated in previous studies (Bhattacherjee, 2001; Chen et al., 2015). Table 3 presents the constructs, their operational definitions, measurement items and the results of the measurement model evaluations. All item loadings, Cronbach's alpha and composite reliability values are greater than the 0.70 thresholds, indicating the reliability of the constructs and items used in the measurement model. Also, all AVE values are greater than the 0.50 threshold value indicating convergent validity.

According to Fornell and Larcker (1981), convergent validity is satisfactory when constructs have an AVE of at least 0.5. Table 4 presents the results of the discriminant validity, which is based on the Fornell–Larcker criterion with correlations, and the square root of AVE values on the diagonal.

5.2 Structural model

Evaluating the structural model involves assessing the predictability of the model and the significance of the relationships that exist between variables in the model. According to Hair et al. (2016), there should be no collinearity among the constructs to avoid biasing the regression results. Thus, the variance inflator factor should be less than five to show the absence of collinearity. p-values of less than 5% indicate significant path coefficients between variables. Finally, R² values of 0.75, 0.50 and 0.25 indicate substantial, moderate and weak predictive accuracy of the model respectively. Figure 2 summarizes the results of the PLS structural model analysis.

The structural model was evaluated by running 5,000 bootstrap subsamples. There was no collinearity biasing the regression results of our model since the maximum VIF value among all items used in the model was 3.818, which is below the 5.0 threshold value. Based on R² values, the structural model explains 48.4% of the variance for BA continuance intention, 50.6% of the variance in satisfaction, 36.7% of the variance in perceived usefulness and 31.9% of the variance in technological compatibility. These coefficients of determination indicate the moderate to substantial predictive power of the model. By analysing the significance of the path coefficients (β) using p-values (p), the model shows that perceived usefulness (β = 0.366, p = 0.000) and technological compatibility (β = 0.403, p = 0.000) significantly affect BA continuance. Also, confirmation significantly affects perceived usefulness (β = 0.606, p = 0.000), technological compatibility (β = 0.565, p = 0.000) and satisfaction (β = 0.362, p = 0.001). Additionally, perceived usefulness significantly affects satisfaction (β = 0.365, p = 0.000). Contrary to what was expected, technological compatibility had no significant effect on satisfaction (β = 0.089, p = 0.263), and satisfaction had no significant effect on BA continuance (β = 0.020, p = 0.835). The analysis also shows that perceived usefulness had a mediating effect between confirmation and BA continuance intention (β = 0.0.222, p = 0.000) and between confirmation and satisfaction (β = 0.221, p = 0.001). Technological compatibility also mediated the relationship between confirmation and BA continuance (β = 0.228, p = 0.000). Table 5 summarizes the findings of the structural model evaluation.

6.1 Theoretical implications

With the rising adoption of BA software in ISD projects as well as the increase in project failure (Shah et al., 2019), it is important to investigate the factors that affect BA continuance in ISD projects. This study has two major implications for research. First, it uses a theoretical perspective to empirically examine the antecedents of BA continuance. Prior studies have mostly focused on BA adoption and not on its continuous use, even less in agile ISD project contexts (Daradkeh, 2019). This study extends the existing body of knowledge on BA adoption at the organizational level. Specifically, the research model, which is grounded in ECM, is readily applicable and generalizable to future studies on BA continuance. This study also highlights confirmation of expectations regarding the perceived usefulness and technological compatibility of BA as key determinants of BA continuance. However, the methodology used does not explain the nature of the relationship between these factors. Thus, future research should examine the nature of the relationship between the factors and how this affects organizational performance. Also, future research should replicate this study in other ISD environments to determine whether new contexts could reveal new factors that affect BA continuance at the organizational level. Furthermore, this study suggests that technological factors have a direct effect on BA continuance. Thus, future research should integrate complementary theoretical perspectives to further identify factors and explain this phenomenon.

Second, this study contributes to big data research on the potential outcomes of big data initiatives for relevant stakeholders (Abbasi et al., 2016; Mandal, 2019). Specifically, this study assesses the outcomes of BA initiatives in agile ISD projects and its outcomes for project team members. It provides a more holistic understanding of BA continuance in agile ISD projects and the salient factors to be considered at the project team level by explaining how BA contributes to the management of agile ISD projects (Dennehy et al., 2020). By extending the ECM with technological compatibility, this study also provides a more contextualized understanding of the motivations behind the continuous use of BA in agile ISD projects. Thus, it contributes to the understanding of factors in agile ISD projects that lead to project success or failure (Baghizadeh et al., 2020) and how well BA fits with user expectations (Jaklič et al., 2018).

6.2 Practical implications

The results of this study also have important managerial implications. Since maximizing the value of BA requires its pervasive use by employees, understanding their expectations is crucial for BA continuance in organizations (Wixom et al., 2013). Furthermore, understanding BA continuance in this context is becoming indispensable as agile processes and methodologies are arguably becoming the norm in ISD projects (McAvoy et al., 2013; Tripp et al., 2016). Thus, this study could serve as evidence of the importance of managing user expectations in BA projects to ensure BA continuance. The results of this study show that BA project managers need to focus on managing user expectations regarding the usefulness and compatibility of the BA implemented to ensure continuance. Based on existing literature reviewed in this study, these managers should prioritize BA solutions that favour change management, project visibility, transparency, productivity, time-to-market and cost reduction because agile teams work in fast-paced project environments. Furthermore, the findings of this study provide insights for managers on how to assess and ensure BA continuance. In agile ISD projects, for example, agile principles and practices would have to evolve to meet new BA trends like fast analytics and data science (Larson and Chang, 2016). Thus, managers in charge of choosing the BA to be adopted by their organizations should ensure that the BA chosen is compatible with their organizational values and work practices of the users. Project managers also need to focus on the expectations of BA users regarding the usefulness of the BA for individual tasks, especially how BA would affect their operational efficiency and job performance. Through this approach, managers would be able to identify, resolve, understand and anticipate BA discontinuance, thereby prevent BA project failure. This includes avoiding misfits between the BA adopted, organizational values and work practices.

6.3 Limitations and future direction

This study has some limitations which are also opportunities for future research. First, some respondents were LinkedIn contacts and members of groups specialized in BA. Thus, there was limited control over the type of participants. However, such limitations could be alleviated by the fact that respondents worked in different projects, thereby showing significant diversity in our dataset. Second, it is important to consider that respondents were individuals who represented different organizations. Consequently, a single person cannot answer on behalf of the whole business unit/organization. Further, getting respondents from companies was quite challenging especially given the strict criteria of this study that required participants to be part of an agile ISD project and current BA users in the project. Since respondents were current and continuing users of BA, they may also have been biased in their perceptions. Third, this study collected data only at one point in time (cross-sectional study). It would have been ideal to perform a longitudinal study to compare the pre-acceptance and post-acceptance perceptions of respondents. However, this was too challenging for this study especially given the ongoing coronavirus disease (COVID) pandemic that made data collection very challenging as employees were usually inaccessible and some projects had to close down temporarily. Additionally, a majority of the respondents are from European-based companies. In the future, it would be worth having an in-depth investigation in various companies, countries or business sectors. This would enable a more in-depth analysis of changes in perceived usefulness, technological compatibility and satisfaction over time. These limitations present potential ways of extending the current research and validating our model in other settings.