Heuristic evaluation applied to library web services

The Authors

Marjo-Riitta Aitta, Department of Finnish, Information Studies and Logopedics, University of Oulu, Oulu, Finland

Saana Kaleva, Nokia, Oulu, Finland

Terttu Kortelainen, Department of Finnish, Information Studies and Logopedics, University of Oulu, Oulu, Finland

Abstract

Purpose – The purpose of the paper is to present usability heuristics for the evaluation of public library web services.

Design/methodology/approach – Heuristics for library services are based on Nielsen's classical list of heuristics and results of previous usability research of library web services. A total of 15 public library web sites were evaluated on the basis of these applied heuristics. One part of the study was supported through usability tests. The results of these studies were utilized to evaluate the applied heuristics.

Findings – The applied heuristics are divided into three categories: heuristics critical from the usability viewpoint; heuristics concerning major problems; and heuristics connected to minor usability problems but still important and concerning conventions of web design. The use of the heuristics and the results they give are evaluated to provide a basis for their use in future.

Research limitations/implications – The applied heuristics lists have been tested in two different studies, and the combined list based on them has so far been utilized in practical evaluation, but has not been formally tested.

Practical implications – The heuristics for library web services presented in this paper can be applied to usability evaluation of public library web services.

Originality/value – Library web sites should be user friendly, because the services are meant for all citizens. The applied heuristics for public library web services provide a starting point for usability evaluation that can be continued with other methods.

Article Type:

Research paper

Keyword(s):

Public libraries; Customer satisfaction; Worldwide web; Information services; Finland.

Journal:

New Library World

Volume:

109

Number:

1/2

Year:

2008

pp:

25-45

Copyright ©

Emerald Group Publishing Limited

ISSN:

0307-4803

Introduction

Public libraries offer through the internet services and collections that are aimed at a heterogeneous group of library users with varying needs for information (Battleson et al., 2001, p. 198) and computer-usage skills (Glosiene and Manzhukh, 2005, p. 304). Therefore, high requirements are set to the usability and accessibility of the internet services particularly of public libraries. Yet, according to Ebenezer (2003) and Manzari and Trinidad-Christensen (2006, p. 164), among others, evaluation of library internet services is often ignored.

Usability is a crucial factor for the internet and the Worldwide internet. For organizations offering their services through the internet it is a competitive factor and a quality measure of a product or a service (Sinkkonen, 2006). Studies show that it has an impact on user satisfaction (Top of the Web … , 2003). In designing products, usability factors direct the attention of the designers on the users (SFS-EN ISO 9241-11, 1998). Good usability makes it possible to offer services equally to all users, which is especially important for public services (Mielonen and Hintikka, 1998).

Usability questions concerning library sites have been addressed by several authors during recent years. In her evidence-based proposal for redesigning and modifying a internet site, Ebenezer (2003, p. 136) addresses the design and navigation aspects of usability, but also the usefulness, value and appropriateness of the site's content in relation to the perceived role of the library. Novljan and Zumer (2004) give guidelines for the content of library internet pages. Manzari and Trinidad-Christensen (2006) report a usability test applied to the redesign of a library internet site. Battleson et al. (2001) report a case study on the usability of an academic library web site, while Chowdhury et al. (2006) make a comprehensive review on usability and impact of digital libraries.

Numerous usability evaluation methods have been developed (see Nielsen, 2007; Pace, 2003), but, according to Glosiene and Manzhukh (2005, p. 317), the available spectrum of usability methods and tools should be tailored to the real-life conditions of memory institutions. In addition to this, Manzani and Trinidad-Christensen (2006, p. 164) point out that Nielsen's heuristics often are not included in library internet site usability studies. This paper seeks to address this point; in order to promote the usability viewpoint in the provision and evaluation of library internet services, it presents classic usability heuristics applied to the special requirements of public libraries that serve diverse user populations.

Heuristic evaluation gives a good starting point for evaluation as part of the process of constructing internet services, and may also point out the most urgent needs for further evaluation, e.g. with users. Nielsen calls it a discount usability evaluation method, and it may therefore be applicable even in small units. It contributes best in combination with other evaluation methods (see Nielsen, 2002a). However, this paper does not discuss any other inspection methods apart from heuristic evaluation. Nevertheless, it emphasizes that a combination of several methods achieves the best results in the evaluation of usability.

This study is focused on the following questions:

How can heuristic evaluations be applied to the study of the usability of public library internet services?
What results do heuristics applied to public library internet services provide?
What special factors should be taken into account when studying public library internet services through heuristic evaluation?

Library internet services in this article refer to a wide range of collections and services offered by libraries through the internet. Such services include, for example, access to library online catalogues, the possibility to make reservations, possibilities for searching or, for example, inter-lending. The library web services are available to users inside or outside the physical library building. However, in most cases the actual collections are not available through the pages. Therefore, the cases studied in this paper do not include digital libraries.

This paper is based on two usability studies, one of which applied heuristic evaluation as the only assessing tool, and the other applied it in combination with usability testing. The applied heuristics, presented in this paper, are derived from both of these studies and presented in a combined form. The following sections first discuss the concept of usability, then heuristic evaluation, followed by an empirical study and discussion. In this article the combined heuristics lists are referred to as heuristics for library internet services.

The emphasis in the analysis is on the heuristics applied to the evaluation of public libraries' internet services, their use, strengths and weaknesses, on the basis of the empirical studies conducted, and in addition, to a lesser degree on the results of the evaluations.

Elements of usability

The concept of usability has many definitions. The two most commonly used definitions, presented by ISO 9142-11 standard and by Nielsen (1993), are discussed briefly as follows.

The ISO standard 9241-11, of 1998, defines usability as the extent to which a product can be used by specified users to achieve specific goals with effectiveness, efficiency and satisfaction in a specified context of use. The components of usability in this standard are the user, his or her task, equipment and environment that form the context of use, the product and the goals that represent the intended outcome of the use. In order to specify or measure usability, it is necessary to identify the goals and to decompose effectiveness, efficiency and satisfaction, which are the measures of usability. The measures of effectiveness (productivity) depict how exactly and completely the user can achieve his goals. From the viewpoint of efficiency, productivity is studied in relation to resources. Satisfaction measures the attitude of the users and to what degree they do not perceive uncomfortableness.

Nielsen (1993, p. 25) divides usability into five factors:

Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
Efficiency: Once users have learned the design, how quickly can they perform tasks?
Memorability: When users return to the design after a period of not using it, how easily can they re-establish proficiency?
Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
Satisfaction: How pleasant is it to use the design?

The method of evaluation and the focus of evaluation depend on the users, their needs and ways of using the product, and their heterogeneous skills in using it. This is especially important in the case of internet services, as internet competence differs greatly between groups and individuals (see, e.g. Chisnell et al., 2006, p. 43).

Accessibility is also an important concept in examination of public library internet services. Accessibility means that all services, equipments and information are designed so that they are easy to utilize despite functional limitations due to context of use, device in use or disabilities (Korpela, 2002).

Based on earlier studies, it seems that the use of library jargon or terminology is the most problematic factor for users of library internet services. Even standard terminology commonly used in the library field has been perceived as problematic to users (Dickstein and Mills, 2000; McGillis and Toms, 2001). The use of library terminology is also a contributing factor behind other usability problems in library internet services, e.g. navigation (e.g. Battleson et al., 2001, p. 196; Ebenezer, 2003; Crowley et al., 2002; Spivey, 2000; Puputti, 2003). Other usability problems noticed in earlier studies of library web sites have been, for example, the complexity of the interface, difficult learnability, lack of information (Battleson et al., 2001, p. 192); emphasizing services in the site structure that are not focal to the users (Ebenezer, 2003); hierarchical bias: confusion of the organization's structure with customer services, expert proficiency: design of the layout that presumes comfort in the internet environment, preferred and novel services: developers often use certain services and over-emphasize them in their design. Battleson et al. (2001, p. 198) also mention ineffectiveness in the presentation of information. Methods of usability evaluation are discussed in the following section.

Usability evaluation methods

A usability evaluation method is a process through which a measurement or an assessment of the usability of a user interface is obtained. An evaluation usually results in a list of usability problems identified in an interface. This, however, should not be the outcome of an evaluation, but, instead the target should be an improved user interface that meets the usability objectives set for it (Karat, 1997; Riihiaho, 2000). An idea of the usability of a system can be gained through several methods, which can be classified in numerous ways, for example into automatic, empirical, formal and informal methods (Mack and Nielsen, 1994), inquiry, inspection and formal usability testing (Battelson et al., 2001, p. 192). The most basic way to classify usability evaluation methods (Riihiaho, 2000) is to divide them into:

Usability inspection (or expert evaluation, expert-based) methods: a set of informal, analytical methods that can be applied without user involvement, for example heuristic evaluation. The evaluation of a user interface is based on the considered judgement of the inspectors (Mack and Nielsen, 1994).
User testing methods (or user-based methods): a set of empirical methods employing representative users, for example usability testing. The evaluation is based on testing the system with, observing, surveying, etc., real users or test users that represent them.

Usability inspection

Usability inspection methods include, among others, heuristic evaluation, cognitive walkthrough, pluralistic walkthrough, and, for example, perspective-based usability inspection (Nielsen, 1993, p. 155; Battleson et al., 2001, p. 192; Glosiene and Manzhukh, 2005, p. 315). With these methods, the site's designers and information specialists serve as testers and subjects, putting themselves in the place of the user to perform various tasks using the site. Each of these and other usability evaluation methods are introduced in Nielsen's (1993) Usability Engineering.

Heuristic evaluation is a usability engineering method in which a small set of expert evaluators examine a user interface for design problems by judging its compliance with a set of recognized usability principles or heuristics (Nielsen, 1993; Manzani and Trinidad-Christensen, 2006, p. 164). According to Nielsen (1993, p. 156), one evaluator uncovers about 35 percent of usability problems through heuristic evaluation. Therefore, he recommends at least three experts per study. However, the addition of more evaluators does not relatively produce more results (Nielsen, 1993), but it increases the reliability of the results. Test users are not needed, and, therefore, the whole procedure is considered cost-effective.

Nielsen's (2007) frequently applied heuristics include the following areas:

visibility of system status;
match between the system and the real world;
user control and freedom;
consistency and standards;
error prevention;
recognition rather than recall;
flexibility and efficiency of use;
esthetic and minimalist design;
help users recognize, diagnose, and recover from errors; and
help and documentation.

All evaluators go through the site independently at least twice, first to get a feeling of the site, then to compare its characteristics with usability principles. In addition to the checklist of general heuristics, the evaluator obviously is also allowed to consider any additional usability principles or results that come to mind that may be relevant for any specific dialogue element in the interface (Nielsen, 1993, p. 158). The severity rating of the discovered usability problems is carried out by usability experts, either on the basis of the textual description of the usability problems or when accessing the evaluated system. Each of the recommended 3-4 evaluators gives an individual severity rating, and the combined results produce the final severity rating of the usability problems. A single rating scale (Nielsen, 1993, p.103) is:

0 = not a usability problem;

1 = cosmetic problem, which need not be fixed unless extra time is available in the project;

2 = minor usability problem; fixing this should be given low priority;

3 = major usability problem, important to fix, should be given high priority;

4 = usability catastrophe, imperative to fix this before product can be released.

The output of heuristic evaluation is a list of usability problems in the interface. Each usability problem is reported with reference to the heuristic it violates, and how it violates it. It is good to include illustrations in the report to clarify the results.

User testing

In formal usability testing, users are observed using a target application or a prototype of it to perform given tasks or to achieve a set of defined goals (Nielsen, 1993). Each test session is captured for further (video) analysis, which is an essential part of the usability method. It may also include a pre-test or post-test interview or questionnaires. A typical usability test includes 6-12 participants in two or three subgroups with three to five people in each (Dumas and Redish, 1999). It has been suggested (Virzi, 1992; Dumas and Redish, 1999) that 80 percent of usability problems are detected with between four and five participants and 90 percent with ten test participants. The number of participants in a test is dependent on time and money and on how many subgroups are needed to satisfy the goals and targets of the study (Dumas and Redish, 1999).

Both quantitative and qualitative results can be derived from usability test data:, e.g. how many attempts the user needs to complete tasks or how long it takes to accomplish a certain task. Qualitative data may describe users' behaviour when interacting with the target application. However, the nature of the method is always artificial, as tests are usually carried out in a test laboratory and users perform predefined test tasks. Also, the reliability of the results is dependent on how well the test participants represent actual end users of the target application, or how successfully different target groups have been defined (Nielsen, 1993). Still, usability testing is the most objective way to evaluate the usability of a target application (Sinkkonen et al., 2002).

Discussion on the methods

Studies have been conducted to examine the efficiency of different usability evaluation methods, usually to study the efficiency of inspection methods compared with usability testing (Grey and Salzman, 1998; Dumas and Redish, 1999). These studies show, although not conclusively (see: Jeffries et al., 1991), that usability testing is more efficient than usability evaluation methods in finding usability problems in user interfaces. Heuristic evaluation conducted by usability experts is more efficient than other inspection methods. It uncovers the most severe usability problems of a user interface, but also the least severe ones. In the results of an inspection, so called “false positive” means that a finding violates a heuristic rule, but is not detected as a problem in a usability test with actual end users (Dumas and Redish, 1999; Manzani and Trinidad-Christensen, 2006, p. 164). A question can be raised, whether they can be considered as real usability problems at all. To prevent such cases, evaluation should be based on real end users' tasks instead of mere browsing of the target system. Also, elements derived from a cognitive walkthrough will further help distinguish valid usability problems from false positives (Sears, 1997).

User inspection and usability testing methods concentrate on different aspects of human characteristics as users of a system. The former focuses on general characteristics that more or less similarly affect all individuals, whereas the latter more takes into account individual differences in facing the interface (Sinkkonen et al., 2006; Sinkkonen, 2006). Because these methods uncover different kinds of usability problems, they should be used complementarily to gain a broader picture of the usability of library internet services.

Applying heuristic evaluation to library web services

A classic way to utilize Nielsen's heuristics is in their original form, as has been reported by, for example, Manzani and Trinidad-Christensen (2006), whereas Chisnell et al. (2006, p. 44) have elaborated the general heuristics with specific questions to suit an audience with diverse skills and abilities. In their case the audience consisted of older adults and the questions point to the specific issues that the research shows older adults will have. One reason for this is that most heuristics and guidelines are too broad or too general (Chisnell et al., 2006, p. 44) and, therefore, may require more expertise of the evaluators.

In the present study, Nielsen's heuristics were applied in another way, not to suit a specific audience, but, instead, to suit the study of public library internet services. This elaboration was based on the usability problems found in previous usability studies concerning library internet sites, and applied in two different studies, the first of which was focused on 14 Finnish public libraries (Aitta, 2004), while the second focused on IntroAktiivi, an interactive internet-based library service for enhancing patrons' possibilities to utilize the library (Kaleva, 2004). The first study was based solely on heuristic evaluation, while the latter also utilized usability testing. The 14 library sites in the study represent all the geographic areas of Finland and varying library sizes, as estimated on the basis of the number of inhabitants in the municipality. They did not include much functionality, and they were studied by only one researcher through heuristics defined on the basis of earlier literature on internet design. IntroAktiivi was evaluated by four evaluators with the help of applied heuristics (Kaleva), and the results were compared with the results of usability testing with eight users.

The applied heuristics for library services, presented in this paper is listed below (see a more detailed version in the appendix), are a combination of the heuristics developed in these two studies, which partly overlapped and partly presented unique solutions. The library heuristics are based on earlier studies focused on factors affecting the usability of library web sites, and studies concerning usability in general (Instone, 1997; Nielsen, 1997b; Schneiderman, 1997; Hatva, 1998; Mielonen and Hintikka, 1998b; Rosenfeld and Morville, 1998; Spool et al., 1999; Tuomela, 1999; Chisholm et al., 1999; Krug, 2000; Pearrow, 2000; Hohhman, 2001; Juntunen et al., 2002; Korpela, 2002; Kuutti, 2002; Nielsen, 2002a; Korpela and Linjama, 2003). In addition to these publications, the heuristics are also based on experiences gained in empirical studies by Aitta (2004) and Kaleva (2004). The structure of the library heuristics partly reflects that presented by Nielsen. However, due to specialization on a certain type of organization, they are more detailed. The heuristics for library web services are as follows:

No. 1. Use language familiar to the user, not library jargon, and a logical and natural sequence of information[1]:
- The terminology of the system should be based on terms that are familiar to the actual user. Symbols and icons should also be clear to him. Dialogs should be in the user's native language, if possible[2].
- The users of internet-based systems may have very different backgrounds, and therefore finding a common language may be problematic. The terms used on the site should be based on the user's expressions of the task, not on the structure of the system or its technical realization. The information on the site should be situated in a natural and logical sequence so that adjacent matters would be close to each other or in the order of their use[2].
- If the use of library terminology is unavoidable, provide clarification of the terms[3].
No. 2. Support the user's feeling of freedom, free movement and control[2] by using explicit and understandable navigation mechanisms[3]. Provide accelerators (shortcuts) to enable efficient use of the system[2]:
- A good navigation system indicates the location of the user in relation to both the whole Worldwide web and the structure of a single site. It does not require the user to have any knowledge of the library field.
- Experienced users should be provided with a possibility to perform routine tasks quickly, e.g. with accelerators (shortcuts) on the keyboard or the mouse, abbreviations, automatic complementing of a command and function buttons.
- The user should be constantly aware of what the system is doing and how it has interpreted his commands or input. The user must also be aware of his or her location. Response times should be as fast as possible. Navigation bars and location indication should facilitate moving in and identification of the site on the internet.
- The user should be provided with an easy escape from different situations. This can be realized through, e.g. interrupt or back commands (which lead to the previous situation), a cancel command for long operations, an end program command (escape from all situations), and defaults for returning to the original situation.
- The user must feel in control of navigation on the site and he or she should feel free to choose the functions he wants on the site. The user should not be forced to use certain fonts, colors, frames, browser windows or browser versions.
No. 3. Consistency: obey the conventions of Web design and make the pages of the site uniform[1]:
- Effects should be consistent, so that the same commands, words and functions effect in the same way in similar situations, allowing the user to predict effects. The same information and control should be in the same place on each screen and in each choice. The terms used on the site should be used consistently, both in text, links and buttons[2].
- Conventions contribute to the intuitiveness of the site. Uniformity is gained through e.g. uniform layout identifying single pages to a particular site.
No. 4. Make the text easy to scan and read and take into account the special requirements of reading from a screen[3]:
- Note that users don't read in the Worldwide web – they scan the text. Follow the general rules of good web writing: use subtitles, lists and highlighting. The credibility of the site is affected by misspelling and factual errors: edit the text and check the pages to see that they don't contain obsolete or inaccurate information.
No. 5. Clearly distinguish normal texts from links, visited links from not visited links, and make it easy to conclude, where a link leads to[3]:
- Link texts should contain the most important words that clearly indicate where the link leads to and they should differ from other links on the same page. They should be understandable even when separated from their context. Check the links on a site at regular intervals.
No. 6. Esthetic nature and simplicity of the user interface[1] and minimizing the need to remember[2]:
- The user should not be forced to remember things when moving from one dialog to another. Instead of, e.g. exact commands, it would be better to offer the user an opportunity to recognize the needed matter from different alternatives.
- Objects, functions and alternatives should be visible in the user interface. Good titles and links describing their target facilitate identification, and consequently the risk of getting lost in the site decreases. Instructions for using the application should be easily reached, and different functions and objects should be readily visible.
- The Worldwide web is a very visual media, and this may cause problems for the visually impaired. When designing web sites that contain graphics or animation, also note the users that are unable to see them by providing alternative ways of presenting the same information, e.g. writing alt texts for all graphics. Also pay attention to the graphic design of a web site, use of colors, etc.
- It would be ideal to show the user only the information he/she needs at one time. The principle “less is more” crystallizes the first heuristic: every additional character means more to learn for the user[2].
- It is worth condensing the information on the page, and information irrelevant to the user should be removed or hidden behind links[2].
No. 7. Prevention, identification and processing of errors[2]:
- Error messages should be given in natural language, because otherwise their content is not clear to the user. The messages should not place blame. Instead, they should reveal what happened and why. The system should help the user to recover from an error.
- Attention must be paid to preventing errors already in the planning phase of the system. For example, too similar commands are known to expose the user to errors.
No. 8. Take special groups into account[3]:
- Pay attention to the needs of special groups, e.g. children, the elderly and people with disabilities, when designing the whole site, and take into account the needs of special groups in the pages or sub-sites directed to them. This is highly connected to the accessibility of the site, which should also be checked.
No. 9. Instructions and documentation[2]:
- Use of the system should be so easy that instructions for use are not needed. In practice, user instructions generally are necessary. All instructions contained in the system should be easy to find and connected to tasks that the user performs with the application. The instructions should contain directions and steps towards a certain goal, but they should not be too long.

Compared with Nielsen's heuristics, the library heuristics are more detailed and present a library viewpoint, in this respect corresponding to Glosiene's and Manzhukh's (2005, p. 317) wish. The reason for introducing a combination of two different heuristic lists is that they complement each other and overlap only slightly. By leaving out either of the studies from this paper, the result would have been less comprehensive. Moreover, both lists have been designed for evaluation of library internet services. To some degree, the library heuristics resemble Chisnell's et al.'s (2006) 20 heuristics including 88 entries that, however, concern a different topic, design for older adults. Therefore, the contents differ, although there are common heuristics, such as: let the user stay in control and retrieval rather than recall.

The compiled results of the evaluations based on the library heuristics are presented in the following section, which concentrates especially on the usability problems revealed by each heuristic. The library heuristics are evaluated on the basis of the empiric research and their strengths and weaknesses are considered. Whenever possible, the results of user evaluation are compared with those of heuristic evaluation.

Results of evaluations using the applied heuristics

This chapter presents the combined results of the two heuristic evaluations of library internet services. The first of them was a heuristic evaluation, conducted by one evaluator and is marked by superscript 1 (Aitta, 2004). The second one consisted of heuristic evaluation by four evaluators, and a usability testing with eight test users, and is marked by superscript 2 (Kaleva, 2004). In the latter case, the heuristic evaluation procedure was conducted in a group of four people, although the classic heuristic evaluation is suggested, to be conducted independently by all evaluators. In this evaluation, also tasks essential for end users were used as a basis of evaluation. (Kaleva, 2004.) Also in the former study (Aitta, 2004), heuristic evaluation procedure was in many ways adapted. Instead of one user interface being evaluated by a group of three to four evaluators, 14 library internet sites were evaluated by one single evaluator. This causes certain limitations to this evaluation: some possibility of subjectivity in choosing and applying heuristics exists, and possibly not all usability problems are found in an interface (Nielsen, 1993; Nielsen, 1994).

In the following, the results are divided into three parts: heuristics critical from viewpoint of the library web services, heuristics concerning major problems and heuristic connected to minor usability problems but still important and concerning conventions of internet design. The evaluation of the heuristics is based on experiences gained from their use. They illustrate what issues they revealed in the evaluation, in what respects they proved utility and what should be taken into account when applying them in usability evaluation of an interface. (A more detailed version of the heuristics is in the Appendix.)

Heuristics that reveal critical problems

The most critical problems revealed in the studies concerned terminology, navigation and documentation, and violated mostly the following heuristics:

No. 1. Use language familiar to the user, not library jargon, and a logical and natural sequence of information[1].
No. 2. Support the user's feeling of freedom, free movement and control[2] by using explicit and understandable navigation mechanisms[3]. Provide accelerators (shortcuts) to enable efficient use of the system[2].
No. 5. Clearly distinguish normal texts from links, visited links from not visited links, and make it easy to conclude where a link leads to[3].
No. 9. Instructions and documentation[2].

Use of library terminology. The most frequent and serious usability problems faced in both studies and through both heuristic evaluation and usability testing, were connected to terminology and violated heuristic No. 1. The use of confusing library terminology made it difficult to navigate, e.g. to find the library database of the site. The use of the same terms (e.g. search) in different contexts meaning different things was considered to cause confusion on the basis of the heuristic evaluation. These results were confirmed by usability testing. Applying Spivey's (2000, p. 153) idea, problematic terms can be divided into three categories:

nouns for library services (such as union catalogue, register of materials, collection of materials, browsing collection or support of acquisitions);
library acronyms (such as opac); and
names of databases or search services (e.g. ERIC).

Such terms made it difficult to predict where different links would lead. These problems were present at all the sites in the study. The severity of this usability problem was in most cases rated as 3, i.e. usability catastrophes were not present, but several cases were in need of improvement. These results confirm several earlier studies (e.g. Battleson et al., 2001, p. 196; Ebenezer, 2003; Crowley et al., 2002; Chisnell et al., 2006, p. 55). The terminology problems influence several other heuristics, as noted below.

Heuristic No. 1 can be considered highly important, and according to both these and earlier studies it does not produce false positive findings. It is moreover important to take library terminology issues into account, when the evaluator is a librarian, and the problems caused by professional jargon may be difficult to detect. For example, test users did not understand the difference between the terms “browsing” and “searching”. The terminology should be suitable for the intended audience, for example, texts in children's pages should be suitable for children.

Navigation. The logical and natural sequence of information (No. 1) refers to the structure of the pages, which should correspond to users' needs and ways of using the site, i.e. the structure should be user-oriented instead of being organization-oriented, reflecting the hierarchy of the organization Five usability problems were found to violate this heuristic. Logical sequence of information also means that all relevant links are located coherently, without exclusion of some links, e.g. a link to children's pages that in one case was placed in a totally different place and was, therefore, difficult to find. Structural problems were also caused by placing the pages of a library into the frames of the respective municipality. Usability problems caused by the structural features of the pages have also been noted by Battleson et al. (2001).

Violation of heuristic No. 2 means the user cannot necessarily locate himself or herself in the site. Because the sites were not very large, this did not cause major problems in the study. However, a missing link to the home page complicates navigation of an inexperienced searcher. Heuristic No. 2 is connected to most of the other heuristics, especially No. 5. Inappropriate or unhelpful labels on links, together with non-functioning links are also mentioned by Chisnell et al. (2006, p. 57). Using understandable, consistent terminology, minimal design and placing information logically are key factors in preventing navigational failures. The severity of this problem depends on what it is connected to, and was estimated between 0-3.

Heuristic No. 2 was also violated in cases, where the system did not give feedback, whether an operation had been performed successfully or whether it was still going on, e.g. when removing books from a virtual book shelf. An easy escape from problematic situations can be a link back to the home page, a back arrow, or an opportunity to interrupt long procedures. Links to the home page were noticed to be missing in several cases, but violation of this heuristic did not cause major problems in the evaluations, perhaps due to the rather limited scope of the sites studied. However, in pages with long procedures, this is an important concern and may complicate navigation. The severity rating of this heuristic was in most cases estimated as 2 or 3. Navigational problems concerning getting back to home page were also present in Manzani's and Trinidad-Christensen's (2006, p. 167) study.

Instructions and documentation. Problems concerning instructions and documentation (No. 9) were cases, where this information was needed but insufficient or totally missing. An example was how to use some function, e.g. how to fill virtual bookshelves. Three usability problems concerning this heuristic were noticed. In the use of IntroAktiivi, missing instructions presented a major obstacle to their use for several test users.

Another problem was the placing of instructions in different parts of the system, which made finding them difficult, when not impossible, and at least, illogical placing does not promote the use of the functions. The severity of this problem was rated between 1 and 2. The problems concerning instructions and documentation are probably not easily noticed if the evaluator is a library professional.

Heuristics that reveal major problems

No. 4. Make the text easy to scan and read and take into account the special requirements of reading from a screen[3].
No. 7. Prevention, identification and processing of errors[2].

Long texts may not influence usability itself, but they reduce subjective satisfaction by making the texts difficult to read (No. 4) and can also make it difficult to find specific information on the page if a user is scanning the text. Other factors leading to the same effect are non-functioning links and too small-sized texts, findings made also by Chisnell et al. (2006). Misspelling may reduce the reliability of the pages. The severity of these problems was estimated 0-2.

Not many problems were connected to errors in these studies (No. 7). Such situations are critical above all from the viewpoint of a novice user. On the other hand, misleading terminology violating heuristic 1 causes error situations for users, as well as violations of some other heuristics. If processing of errors is unsatisfactory, the visitor may not return to the site or use the feature in question again.

Heuristics that reveal minor problems

No. 3. Consistency: obey the conventions of internet design and make the pages of the site uniform[1].
No. 6. Esthetic nature and simplicity of the user interface[1] and minimizing the need to remember[2].

The results of violating the consistency heuristic (No. 3) depend on what conventions of internet design were not obeyed. In the case of colours the resulting usability problem is not difficult, but if it concerns, e.g. inconsistent use of terminology, it is equally difficult for the user as is violation of heuristic No. 1. The same can be said about heuristics No. 4 and No. 5: confusing texts in the links cause similar problems, and non-functioning links complicate the free movement of the user. This, as also violation of No. 6 (esthetics of the pages) did not cause severe usability problems (most often rated as 2), but may burden the user's memory.

Consistency covers both the terminology and layout of the pages. It was violated, for example, by using the same terms with different meanings in different contexts. It may also be difficult to distinguish between several different terms that all contain the word “search”. This problem is closely connected to violation of No. 1, and in some cases in fact it cannot be considered as “minor”.

Inconsistency could also mean that a link to the home page is missing from several pages. Inconsistent colours on the pages may be confusing, although they did not cause major usability problems. The severity rating was between 1 and 2. The inconsistency of page layout has also been mentioned by Manzari and Trinidad-Christensen (2006, p. 164).

The heuristic concerning the simplicity and esthetics of the user interface was violated mostly by information that was irrelevant to the user and might lead his or her attention to less important issues. Also, some texts were not clearly distinguishable from their backgrounds. Severity of these usability problems were between 0 and 2. Similar problems were present also in Chisnell et al.'s (2006, p. 57) paper.

Special groups

No. 8. Take special groups into account[3].

On some children's pages, navigation was not simple enough. Children prefer bright colours and entertaining contents, which were not taken into account on children's pages. Children also like multimedia. These preferences contradict several other heuristics. Despite them, special groups' needs should be taken into account if they are to be kept as internet clients. Severity 0-2.

To be able to take No. 8 into account, the special requirements of different groups must be identified. Nielsen (2002b, c) has studied both children and senior citizens as internet users. Chisnell et al. (2006) have presented a detailed list of heuristics from the viewpoint of older adults. For checking of accessibility, there are several guidelines, for example Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0.

Experiences on the use of applied heuristics

In the comparison of the results of the heuristic evaluation and usability testing of the IntroAktiivi system, 37 percent of the noted usability problems were overlapping. Usability testing was able to reveal more different kinds of usability problems. However, heuristic evaluation was able to predict usability problems that would be found through testing with users. This result is in accordance with several earlier respective studies (e.g. Sinkkonen, 2006; Manzani and Trinidad-Christensen, 2006). To obtain the opinions of the users and the usability problems they really face, a user test is necessary. However, heuristic evaluation gives a useful base for this.

Heuristic evaluation may not be sufficient to determine which terms are problematic when using library professionals as the only evaluators. Because library internet services are directed to people without special library knowledge, other types of evaluators are also needed to provide a non-professional perspective on the terminology used. This also provides more reliable results in assessing problematic terminology. To some degree this finding contradicts Nielsen's (1994) view that “double experts” are the most efficient evaluations.

The problems noted only through usability testing were those connected to users' individual abilities to observe and remember things. Heuristic evaluation failed to reveal problems caused by non-intuitiveness of some functions connected to test users' characteristics. In order to find these problems through heuristic evaluation, special emphasis should be put on this factor.

Discussion

Heuristic evaluation is a usability evaluation method that is applied in an early phase of internet site design, but, nevertheless, it can also be applied to a system already in use. It is a discount evaluation method as expressed by Nielsen (1993). To get optimal results, it requires three to six evaluators, but no test users. Therefore, its implementation is lighter than usability testing. It reveals usability problems and gives evaluators a better basis for conclusion than does an intuitive determination of user-friendliness. For these reasons it is applicable even in small units, where the construction of internet pages is a relatively large effort, and useful as a starting point for evaluation and for persuading decision-makers on the possible needs of further evaluation.

Battleson et al. (2001, p. 198) reiterate Nielsen's statement that the importance of usability testing of library internet sites cannot be understated. Usability evaluation should also be an iterative process and it serves as both catalysts for design changes and as a tool for evaluating those changes, especially as library sites strive to meet the increasing information demands of users.

The heuristics for public library internet services can also be used as a support for designing library internet services. They have proved their utility in practical work of one of the authors. They direct the attention of the designers to the factors that affect the usability of internet services and the needs of users. This would mean a shift from the library viewpoint to user-centered design.

It should be noted that when the evaluator is a librarian, problems caused by library terminology or jargon may be difficult to notice. For example, the test users did not understand the difference between the terms browsing and searching, which may be clear to library professionals.

The library heuristics presented in this paper contain a lot of details and a clear library bias. This may be considered a limitation from the viewpoint of applying them to sites with other contents. From the viewpoint of especially small libraries, a detailed heuristic list may be a strength, giving clear advice on what to consider and how to enhance usability. Their applier in a library may not have previous experience or competence in usability evaluation. This may, of course, cause heavy dependence on the heuristics, and, as a consequence, lead to biased results on one hand. On the other hand, heuristics that are tailored for evaluating library systems may reduce the amount of so-called false positive findings and could thus provide more reliable results. The user of the library heuristics should take this into account and hopefully continue the evaluation of usability through either test users or assessment methods. In any case, the heuristics for public library internet services provide a starting point in cases where argumentation is needed for improving the service. Even small units should have the opportunity to evaluate their sites in order to develop them further.

References

Aitta, M.-R. (2004), "Suomalaisten yleisten kirjastojen internet-sivujen sisältöjen ja käytettävyyden tarkastelu (“A study of the contents and usability of the internet pages of Finnish public libraries”)", Department of Finnish, Information Studies and Logopedics, University of Oulu, Oulu, Master's thesis, .

Heuristic evaluation applied to library web services

The Authors

Marjo-Riitta Aitta, Department of Finnish, Information Studies and Logopedics, University of Oulu, Oulu, Finland

Saana Kaleva, Nokia, Oulu, Finland

Terttu Kortelainen, Department of Finnish, Information Studies and Logopedics, University of Oulu, Oulu, Finland

Abstract

Article Type:

Keyword(s):

Journal:

Volume:

Number:

Year:

pp:

Copyright ©

ISSN:

Introduction

Elements of usability

Usability evaluation methods

Usability inspection

User testing

Discussion on the methods

Applying heuristic evaluation to library web services

Results of evaluations using the applied heuristics

Heuristics that reveal critical problems

Heuristics that reveal major problems

Heuristics that reveal minor problems

Special groups

Experiences on the use of applied heuristics

Discussion

References

Further Reading

Appendix

No. 1 Use language familiar to the user, not library jargon, and a logical and natural sequence of information

[1]

[2].

[2]

[3]

No. 2 Support the user's feeling of freedom, free movement and control

[2]

[2]

No. 3 Consistency: obey the conventions of web design and make the pages of the site uniform

[1]

[2]

[3]

[3]

[3]

[3]

[3]

[3]

No. 4 Make the text easy to scan and read and take into account the special requirements of reading from a screen

[3]

No. 5 Clearly distinguish normal texts from links, visited links from not visited links, and make it easy to conclude where a link leads to

[3]

No. 6 Esthetic nature and simplicity of the user interface

[1]

[2]

[3]

e[3]

[3]

[3]

[3]

[2]

[2]

No. 7 Prevention, identification and processing of errors

[2]

No. 8 Take special groups into account

[3]

No. 9 Instructions and documentation

[2]

About the authors