Editorial: Information systems evaluation: what does it mean?

The Authors

Zahir Irani, Brunel Business School, Brunel University, London, UK

Abstract

Purpose – The purpose of this paper is to outline the purposes and challenges facing those seeking to evaluate their information systems (IS).

Design/methodology/approach – Looks at the many problems associated with the evaluation of IS systems.

Findings – Many of the problems originally identified over 15 years ago are still prevalent, exacerbated even further because of the complexity associated with linking intra- and inter-organisational IS.

Originality/value – The paper is of value in providing the reader with several touch points that serve to outline the purposes and challenges facing those seeking to evaluate their IS.

Article Type:

Research paper

Keyword(s):

Information systems; Function evaluation.

Journal:

Construction Innovation

Volume:

Number:

Year:

2008

pp:

88-91

Copyright ©

Emerald Group Publishing Limited

ISSN:

1471-4175

The importance of investing in new information systems (IS) architectures and infrastructure has become a topical issue within organisations. This is predominantly motivated by the need to deliver better value products and service through robust and responsive supply chains. With this in mind, business managers are seeking to use appropriate methods and techniques to appraise and justify the financial contribution of IS at strategic, operational and tactical levels. Yet, managers often express concern regarding their ability to appraise IS investments prior to committing financial and emotional resources (Raymond et al., 1995; Irani et al., 1997, 2006, 2007).

The problems associated with the evaluation of IS are ubiquitous, and as a result, a wealth of research and discussion has taken place. The design and development of IS systems can be a lengthy process, time consuming, complex, and costly to implement. In addition, IS have to be maintained, which again requires capital expenditure and management. As a result, issues associated with their evaluation would presumably assume great importance however; this is often not always the case.

Many of the problems originally identified by Farbey et al. (1993) are still prevalent today, albeit exacerbated even further because of the complexity associated with linking intra- and inter-organisational IS. What is noteworthy however, is the proliferation of IS has often coincided with lower macroeconomic figures of productivity and profitability in sectors such as agriculture, construction, manufacturing and service delivery (Baily and Chakrabarti, 1988; Roach, 1991). The term productivity paradox has thus been used to describe the alleged inability of IS to deliver in practice the benefits they promise in theory (Brynjolfsson, 1993). Indeed, according to Stratopoulos and Dehning (2000), it is more important how organisations manage and utilise their IS assets than how much they invest in technology. This is now considered fundamental, with Irani and Love (2001) demonstrating through their research that it is often the soft; human and organisational factors associated with the adoption and implementation process, which can lead to organisational learning and improved readiness, which in turn, supports the efficient and effective utilisation of the IS resources. However, the experiences of businesses reported in normative literature should provide the foundations for learning and improvement, and thereby assist with the evaluation process; but, unfortunately this has not occurred. Indeed, organisational learning is rarely seen as a “spin-off” benefit from a robust evaluation process.

According to Willcocks (1992), “Information systems evaluation is not only an under-developed but also an under-managed area which Organisations can increasingly ill-afford to neglect.” The increased complexity of IS combined with the uncertainty and unpredictability associated with IS benefits and costs, clearly point to the need for improved evaluation procedures.

There remains a clear need to justify investments in IS; empirical evidence from Serafeimidis and Smithson (2000), and more recently, Irani et al. (2007) points to the lack of widespread evaluation processes (financial or otherwise). The increased complexity of IS combined with the uncertainty and unpredictability associated with assessing the system's benefits and costs, clearly point to the need for appropriate evaluation procedures. Farbey et al. (1993) suggest that the search for a single “best” approach is fruitless due to the wide variety of complex interacting variables. Yet, in a hope to find the panacea for the “evaluation paradox” which organisations face, researchers are constantly proposing new evaluation methods. One reason for the wide range of different approaches to evaluation is that it is not an end in itself. The purpose of evaluation is to provide management and stakeholders with concise information, enabling them to make informed decisions about the future direction or prospects for the organisation in question. The nature of the impending decision is a key factor in choosing an appropriate and effective evaluation strategy. IS evaluation is inevitably context dependant. The need for such evaluations arises in a variety of circumstances, ranging from internal post implementation reviews of a single system, to external reviews addressing the “heath” of an organisation in the light of its complete portfolio.

As a consequence, of the inherent problems consistently identified with IS evaluation, it would appear that there is a “crisis of understanding” confronting the private and public sector regarding its importance, role and relevance throughout a projects' life cycle. A lack of understanding as to why, how, and when to evaluate IS appears to be the central issue facing managers, with little consensus amongst the academic community, represented by Irani and Love (2002) in their presentation of taxonomies of IS evaluation approaches. Fundamentally, it appears that managers need to have a better understanding about the impact of IS on organisational performance, and a better understanding of the benefits, costs and risks associated with financial and social capital investments in developing such infrastructures. Such understanding can help an organisation better utilise resources and improve its position vis-à-vis its competitors. Failure of such understanding can have disastrous consequences, such as inappropriate resource allocation and competitive disadvantage.

IS investment often differ in nature from other capital investments as there is a substantial human and organisational interface (Irani et al., 2001). In addition, they are characterised by being high risk, having erratic cash-flows, timing several, and often have significant intangible costs. Regardless of these challenges that place IS investments in a space that is quite different from other capital investments, for example in plant and equipment, IS projects are often evaluated using the same “traditional” appraisal techniques. In here lies, much of the dilemma facing IS evaluation. Appraisal techniques that form part of a broader capital budgeting process are typically used by decision makers to support their evaluation of investments. The reasons as to why organisations appraise their IS investments are explained by Irani and Love (2002) as being to:

compare between different projects;
rank projects in terms of organisational priorities;
justify investment requests by management;
control expenditure, benefits, risk, development and implementation of projects;
provide a framework that facilitates organisational learning; and
facilitate mechanisms to decide whether to fund, postpone or reject investment requests.

Although these reasons demonstrate the importance of an investment appraisal process, and add weight to the use of such techniques, Primrose (1991) noted that many managers view project appraisal as a financial hurdle that has to be overcome and not as a technique for evaluating a project's worth.

IS projects are renowned for their high-failure rate although the authors are mindful that defining success and failure is subjective and nebulous. However, it is still common to see articles claiming that the vast majority of IS fail. This, of course, begs the question of what constitutes IS failure. Although definitions vary within the literature, the common thread seems to be that a project fails if, in some way, it does not meet the expectations of its users or sponsors (Smith and Keil, 2003). These expectations may be “hard” measurable targets, like implementation costs, delivery deadlines or functional requirements. Alternatively, they may be “soft” targets like increased business volumes, improved product quality or more efficient working practices.

The whole point of IS evaluation is to predict (ex ante) or assess (ex post) how well an IS project meets the various expectations of stakeholders. To view evaluation in simple cash or cost-benefit terms only addresses one particular area of expectation when engaging in an IS project. Although, a cash return on investment is necessary for a business operation to remain healthy a positive result cannot be taken as an indication that the project will succeed to meet its stakeholders' expectations. The purpose of this editorial has been to provide the reader with several touch-points that serve to outline the purposes and challenges facing those seeking to evaluate their IS.

References

Baily, M.N., Chakrabarti, A. (1988), Innovation and the Productivity Crisis, Brookings Institution, Washington, DC, .