ASIS Mid-year 1999: evaluating and using networked information resources and services

Citation

Hendricks, A. (1999), "ASIS Mid-year 1999: evaluating and using networked information resources and services", Library Hi Tech News, Vol. 16 No. 11. https://doi.org/10.1108/lhtn.1999.23916kac.001

ASIS Mid-year 1999: evaluating and using networked information resources and services

Arthur Hendricks

Introduction

If one peruses the Journal of the American Society for Information Science (ASIS), one usually finds a 30-page article filled with differential calculus to prove that node size in a hypermedia system is important. Such articles are not very helpful to a systems librarian when a co-worker calls up to complain about that strange noise her hard drive is making. However, the mid-year ASIS meeting, this author's first, was a pleasant surprise. Most of the papers did not involve complex math, and a few had practical information. The conference was held in Pasadena, California, from 24 to 26 May. Over 325 people attended this conference where it was hard to tell who was what. The guy in the T-shirt and ragged jeans might be a famous information scientist. There was a fairly equal amount of men and women, and the crowd was mostly middle-aged.

The primary goals of this conference were to identify what is known about use and evaluation of networked resources and services and to propose strategies for improving our knowledge of their use and evaluation. The conference did offer an opportunity to find out what progress the digital library initiative (DLI) is making as well as ways to effectively teach Web literacy. Whether anyone left the conference with an idea on how to better evaluate the Internet is another matter. No one has found a way to catalogue it, much less evaluate what continues to change and grow.

Monday 24 May

Distance education: contributed papers

Samantha Kelly Hastings, University of North Texas, moderated this session. James Carey and Vicki L. Gregory, both from the University of South Florida, presented "Web-based course delivery: the effect of the medium on student perceptions of learning and their use of library-based resources". Traditionally, the University of South Florida approached distance education through televised one-way video, two-way audio, and adjunct and traveling faculty. When the World Wide Web became viable four years ago, they began using it as a substitute for videoconferencing and sending faculty off-site. Carey and Gregory's study was designed to measure Web students' performances, their access to information resources, and their attitudes toward their education experience. The study, conducted from 1997 to 1998 via e-mail, involved four faculty and five classes and had a 50 to 60 percent response rate. An overwhelming majority of the students (89 percent) felt that the depth of new information acquired in the Web course was the same as or more than that acquired in the face-to-face classes they had taken. While the majority felt that they were "less" or "much less" a member of a group, 60 percent of the students indicated that they had learned "more" or "much more" from their Web-based classmates. If students had to commute more than 60 minutes, they preferred to take Web-based courses. Otherwise, they preferred face-to-face interaction.

Sam G. Oh, Sung Kyun Kwan University, presented "Evaluating a gateway to faculty syllabi (GFS) on the Internet". His project involved collecting a database of syllabi on the Internet from around the world and making a searchable database using metadata tags that are derived from Dublin Core (DC). He produced a Web-based front-end to his search engine. The system currently has only syllabi metadata related to library and information science but will be expanded to other areas in the future. The active server pages technology is employed to provide browsing and searching capabilities to those metadata. The URL for the project is http://lis.skku.ac.kr/gfs/.

His collection of syllabi raised some concerns from the audience about intellectual property rights. Someone asked him whether he asked permission from the professors and how he knew a professor's specialization from just the syllabus. He responded that he had not asked permission ­ the idea had not occurred to him. Oh said he had not thought in terms of intellectual property and that he thought sharing syllabi was a good idea. Charles McClure made the point that someone had "borrowed" his syllabi and he felt his intellectual property rights were violated. Then Howard Besser, from University of California at Berkeley, asked if the syllabus was the same as McClure speaking. McClure did not really answer the question, just made a joke. He said, "Howard and I will be discussing this over beer until 4:30 a.m. You're all welcome to join us". They agreed to disagree.

Measurement and evaluation of federated digital libraries

Larry Lannom, from the Corporation for National Research Initiatives (CNRI), moderated a panel dealing with evaluating a federated digital library. ("Federated" means a seamless collection.) In order to receive funding for the digital library initiative (DLI) from the Defense Advanced Research Projects Agency (DARPA), there was a need for a quantifiable evaluation of work of the DLI. Such evaluation is difficult with something as nebulous as a digital library, which is distributed, comprised of different systems, and has different users and collections. One attempt to tackle this problem led to the creation of the D-Lib Working Group on Digital Library Metrics. William Arms, from the CNRI, discussed his involvement with D-LIB. This group's objective is to develop a consensus on usable and useful metrics to evaluate and compare the effectiveness of digital libraries and component technologies in a distributed environment. The D-Lib Test Suite is a group of digital library testbeds for research in digital libraries, information management, collaboration, visualization, and related disciplines. Three of the six universities engaged in the DLI which were involved with image recognition decided to try their various search algorithms on each other's systems. The group's emphasis was on information searching with the person in the loop and the retrieval of that information. They categorized the systems to be measured by the number and variety of corpora, users, and services.

Carl Lagoze, Cornell University, spoke specifically about the difficulty of Web metrics. Mirrors and caches interfere with true measurement. How does one know which files are significant? How many times does a single user read a file? How often is a downloaded file read? Web statistics cannot answer these questions.

The test suite for D-LIB resides at http://www.dlib.org/test-suite. By making the test suite freely available on the Internet, they hope to lower the barrier for new research projects and to provide a standard dataset for quantitative research as well as provide a platform for experiments in interoperability and distributed systems.

The audience did not have much to say or ask after these presentations. Someone in the audience announced the formation of the SIG Metrics group.

Plenary: the importance of evaluating networked information services and resources

This plenary session, moderated by Charles McClure, dealt with issues and strategies for improving evaluation of networked services and resources. McClure began the session with an overhead of a "Calvin and Hobbes" cartoon which garnered some levity from the audience. Peter Brophy, from the Centre for Research in Library and Information Management, The Manchester Metropolitan University, presented "The 'yes' factor: delivering value to customers of information services". The yes factor is when users say, "Yes, that's the service I was looking for!" This is important to us because we are professionals who want to deliver the best services, we have to persuade stakeholders to support us, and we are facing competition (some users are drifting away, lured by services such as amazon.com).

Ann Bishop, University of Illinois at Urbana-Champaign, presented "Socially grounded evaluation of networked information services and resources". Normally, researchers focus on the quantitative and assume social impacts are beyond the control of scientific research. However Bishop asserted that evaluation approaches should emphasize use and impact in relation to people's goals; seek to identify and address the ways in which technology forestalls equity and gain the participation of traditionally underserved people. We should include socially grounded approaches in all Library Information Science evaluation criteria and in every evaluation we perform to help create an engaged university.

Ron Larsen, Information Technology Office at DARPA, stated that we sometimes manage resources we do not understand and that it is possible to measure the wrong aspects of them correctly. He gave an example of how Pacific Command, which monitors two-thirds of the Earth, wanted a translingual server. He said we assume metrics helps, that better search engines yield better searches, that there is an answer in the search, and that the information we want is in English. At the end of his presentation, someone in the audience asked him how evaluations apply to DARPA. He responded that it's easy to form closed communities and evaluate what we think is important.

Community networks: contributed papers

Roy Tally moderated this session, which consisted of two contributed papers. The first was "Community Networking Initiative research and evaluation activities: methodological issues related to evaluating user needs and outcomes related to community information systems" by Ann Bishop, Tonyia Tidline, Susan Shoemaker, and Pamela Salela. The Community Networking Initiative (CNI) is a collaboration between the Urban League of Champaign County (Illinois), Prairienet (a computer-based network providing access to local information and Internet services), and the University of Illinois. (The URL for the project is http://www.prairienet.org/cni) This project is funded by the US Telecommunications and Information Infrastructure Assistance Program (TIIAP) in the Department of Commerce; its aim is to build the participation of low-income residents in community networking by distributing computers to low-income households and offering training and support in information technology. During the first year of the CNI project, its primary research goal was to conduct a community analysis in order to uncover problems facing low-income residents of the Champaign-Urbana area, learn what information is useful in addressing these problems and how it is currently obtained, and explore attitudes related to computer use. By the second year, 700 applications were processed, 200 interviews were conducted with 35 follow-ups, and 300 adults had been trained for 12 hours each and 350 donated computers had been placed in homes.

Bishop fielded questions after her presentation. One person wanted to know if they had analyzed expectations. Health and parenting information was the most wanted. Another asked whether literacy was a problem. It sometimes was. Why all women? Because it was through the Urban League coordinator, a woman, who contacted those she knew. This program was based on the principle of first come, first served.

The second paper, "Winners and survivors: evolution of digital community networks", by Linda Schamber and Terry Sullivan, presented the results of a follow-up survey of community network managers to assess changes in their situations and networks since they responded to an identical survey more than two years ago. At that time, many networks were grassroots operations, instigated and implemented by volunteers who were mainly fascinated by the technologies. Out of the 26 sites that replied to the survey she sent, 16 were left and only eight returned usable surveys. Out of the eight, she could only call five winners. The problem is that it is easy to start a home page but hard to maintain one. The top concerns in 1996 were funding, staffing, community support, shared goals, and content. The focus in 1999 is less on money and more on partnership. At question time, someone asked her what the difference was between a community network and a cork bulletin board circa 1970. She said the Web pages were new and that they all share some identity with the community that they represent.

Tuesday 25 May

Methods: contributed papers

Philip Sallis, University of Otago, New Zealand, and Linda Hill, University of California at Santa Barbara, presented "Alexandria digital library user and use evaluation: experiments with a neural network method for log data analysis". The Alexandria Digital Library (ADL) Project at the University of California, Santa Barbara, is a research project to develop a digital library of georeferenced information where a key descriptive and retrieval attribute is the spatial, map-based longitude and latitude. The emphasis of the presentation was on user and use evaluation experiments with log data analysis. Their evaluation goal was to combine registration data and session log data to characterize groups of users. They recorded time stamps, actions, what collections were used, and what search buckets were chosen. They defined a search bucket as high-level indexes with multiple underlying sources. The two groups studied were researchers with PhDs and Library Information Specialists with Master's degrees. The data were collected and mounted on a Web page so that the participants could see the results and give the researchers comments on the ADL system. A graduate student wrote PERL and SQL statements to make vectors for data mining with neural networks so that Sallis could build a system of profiles employing neural network techniques, self-organizing maps. SOMs are proposed in the research literature as alternative supplemental data mining tools that generate self-modifying illustrations of data relationship when new data is input over time. They provide a multidimensional mental map compared with standard statistical two-dimensional tables.

Someone asked whether when collecting registration data, one can use the same technique if the users are not known. Sallis answered positively, but said one must rely on what is known about the users. The Internet protocol (IP) address of the users can be acquired; they can still get session frequency.

John Fieber, a doctoral candidate at Indiana University School of Library and Information Science, presented "Transaction log analysis in Web-based information systems". Fieber stated that this study is more exploratory than providing results. When all components of an information system are under administrative control, transaction logging can produce an accurate record of use analysis. However, with the World Wide Web, the provider relinquishes control over the client and the communications protocol used between the client and server. The client can no longer be instrumented directly; data collection from the client is limited to what the protocol will carry; http is a stateless request response protocol.

Log analysis works if all the users have their own, static IP addresses. However, many users share the same Web address. WebTV uses proxies differently; each request is sent through a different proxy. In addition, some data are missing in the logs as most browsers cache data locally. Browsers do not fetch the original page twice but rather get it off the disk cache.

Fieber used five subjects browsing the bureau of labor statistics Web site to answer seven questions. He captured their video displays and network transactions. Within the first 15 minutes, the cache rate varied widely between the five users. It correlated to whether the subject was surfing the site or specifically looking for something. His conclusion was to be skeptical of server log reports. Reporting quirks, time analysis, time stamps, and log entries may not reflect reality as Web servers record when a transaction ends, not begins. Someone asked him if there is any benefit of setting header to force everything to cache. It doesn't necessarily reduce uncertainties; one can gather more statistics, but it makes the Web site very slow.

Jonathan Lazar gave a paper on "Sampling methodologies for using electronic surveys to evaluate networked information resources", which he co-authored with Jennifer Preece. He stated that electronic surveys are becoming more prevalent in evaluation. The downside to electronic surveys is that one individual can respond to a survey 50 times. The upside is that respondents receive the survey quickly, one saves costs of copying and mailing the paper surveys, and one can directly enter survey responses into a database program, saving time and eliminating errors on data entry. However, sampling is an important issue to ensure that the survey respondents are representative of the population of interest. If access to resources is restricted to a well-defined population, demographic data may be available. If not, one can include demographic questions in the survey. One can enforce response diversity by accepting only one Web-based survey per IP address. Lazar concluded that it is important to use sampling methodologies with electronic surveys to ensure that surveys implemented over the Internet have valid results.

A member of the audience asked him how to make pop-up surveys more enticing. He suggested doing what Georgia Tech did, which was to offer prizes for completing a survey. Someone else asked him if there was not a question of ethics in collecting survey responses online.

Evaluating the Web: a look at Web pages, databases and evaluation theory

Evaluation of Web-based information resources is an important part of a librarian's work. However, what are the criteria for evaluating the accuracy and authority of Web pages? What are the criteria for evaluating proprietary Web-interface databases? What is being taught in ILS/LIS today that will equip librarians for this important task? Laura Cousineau, Information Access Librarian, Perkins Library, Duke University, moderated a session in which all these questions and more were answered by the panel participants.

Keith H. Stirling, Electronic Access Librarian, Harold B. Lee Library, Brigham Young University, presented "On evaluating/implementing vendor electronic products", which detailed his involvement with his university's Electronic Resources Committee (ERC). This committee approves purchase of electronic resources and assists subject specialist in selecting them and sustaining their purchase beyond the first year. The committee attempts to ensure that the infrastructure exists within the library to mount/load, access and maintain the resource. It also monitors the licensing, acquisition, cataloging, menuing, mounting/loading, and review processes and attempts to assure that patron access functions properly.

Gregory B. Newby, School of Information and Library Science, University of North Carolina at Chapel Hill, presented a paper on "Beyond glitz and into contents". Newby stressed that, for Internet design and evaluation of use, one must first have a target audience in mind; the target audience is often missed. Second, one should work from content to implementation and design. He defined glitz as using techniques that look great: JavaScript, pulldown menus, buttons that light up when the mouse passes over them; Microsoft Internet Explorer or Netscape enhancements. He suggested that to move beyond glitz into content, one should start with content and work towards organization and design. One should assume multiple access points, so extensive navigation methods should be provided. To evaluate content, one should ask who is creating it. The great answer: the right people with the right knowledge, aided by Web page designers and implementers. The not so great answer: the Web page designers, with little input or feedback from the actual content experts.

Finally, Jeff Rosen, a librarian at the University of Arizona, presented "Evaluating Web pages", co-authored with Ann Eagan, Science-Engineering Librarian at Arizona. This paper focused on the role of librarians in evaluating Web sites. From 1993 to 1995, the general attitude was that the Web is not scholarly, hence not worthy of librarians' attention. Rosen claimed that librarians' attitudes changed for several reasons, including the Internet's lack of editors or other filters that already exist in the print publishing world and the realization that many students don't have the skills to evaluate Web information.

He said there is a conflict in the way librarians evaluate Web resources. In the past they could merely present all that was available. For the present/future, librarians must become comfortable recommending one site over another and therefore they must work to create a uniform review process on which users can rely.

Network access to visual information: a study of costs and uses

Howard Besser, University of California, Berkeley, moderated this session devoted to the findings of the Mellon-sponsored study of digital image distribution focusing on the Museum Educational Site Licensing Project (MESL). In the MESL project seven repositories supplied an identical set of 10,000 images and accompanying descriptive metadata to seven universities, and each university mounted this information within its own customized delivery system. Besser provided an overview of the Mellon study, which evaluated the costs, infrastructure, and efforts involved in implementing the MESL project as well as user reaction to functionality. The study also examined costs of running analog slide libraries and compared these to costs and functionality associated with digital image distribution. Besser said that the study found people got wildly different results when the same query was submitted to each repository.

Robert Yamashita, from California State University, San Marcos, talked about assessing the underlying cost implications of digital and analog distribution. The most costly part is the interface. Rosalie Lack, a graduate student from the University of California, Berkeley, stated that the Mellon study found that digital images were more beneficial to the individual than the group. It is easier for one person to sit in front of a computer than it is to project digital images in the classroom. Besser concluded that it will be a long time before digital image repositories will be able to deliver the critical mass of images needed for instruction and research. Analog slide libraries and digital image repositories will necessarily coexist for many years. While the higher education community is enthusiastic about providing access to digital images, many impediments to widespread adoption must be dealt with ­ ranging from lack of comprehensive content and the absence of necessary tools to facilitate use, to inadequate recognition and support for faculty who adopt new technology in their teaching. The full study final report is available at http://sunsite.berkeley.edu/Imaging/Databases/1998mellon/.

Evaluation of community networks

Community networks (CNs) was the subject of a panel moderated by Ann Bishop. CNs are computer-based services designed to provide online information resources and communication tools to residents of a particular geographic region. CNs are a relatively new type of information system, which raises special problems in assessing user needs and outcomes. Of special significance is the role that CNs play in reducing the "digital divide" that currently separates haves and have nots in the information age. In addition, a natural alliance exists between the goals and services of CNs and public libraries; a number of public libraries are, in fact, major partners in their local CN initiatives.

Philip E. Agre, Graduate School of Education and Information Studies, University of California, Los Angeles, noted that network and community seem complementary. People in modern society live independently of one another; however, people make connections. He predicted that content will become stratified and that goods and services will be made locally.

Douglas Schuler, Seattle Community Network and the Evergreen State College, sees community institutes as attempting to institutionalize public space in cyberspace and advance social goals. He feels it is impractical to evaluate a CN because its goals are ambiguous, broad, utopian, and/or non-existent; its effect would show up only after several years and only in conjunction with other trends; it is hard to show true cause and effect.

Joan Durrance and Karen Pettigrew, School of Information, University of Michigan, spoke of the need to study specific incidents of citizens' information behavior in context of their social worlds and, more specifically, the perceptions and expectations of other key players (e.g. service providers, search intermediaries).

Gregory B. Newby, University of North Carolina, Chapel Hill, closed the session. His main point was that when it comes to evaluating a CN, the cheapest and easiest element to analyze is Web server logs. The problem is that one is then looking at aggregate data. The analysis is counting hits per hour, not getting info about sessions or what paths were taken.

Wednesday 26 May

Accessing knowledge through the window: database providers' methods for evaluating network interfaces

Commercial database access vendors recently have shifted to graphical user interfaces and Web page interfaces. The traditional text-based interface, while still an alternative, soon may fall to the wayside as the appeal of Web/graphical interfaces continues to increase. However, users of these database products are not all in agreement about the actual appeal and usefulness of the new interfaces. To gain insight into the evaluation of network interfaces and the decision criteria employed by the vendors, this open forum aired specific questions poised to the vendors prior to the meeting.

Trudi Bellardo Hahn presented a brief historical background on interface design decisions of the 1960s and 1970s and how users reacted to them. She showed the earliest instance of a search request form: a paper form that asked the user to supply keywords and authors of interest, and which keywords to exclude. Next, Daniel Pliske of LEXIS-NEXIS spoke about his company's efforts to evaluate Web usability. Purely technology-centered approaches often lead to problems such as user-system mismatches and result in inflexible, hard-to-use systems. He said that the Web is forcing a rethinking of the way they evaluate their information access products. Three major aspects of usability deserve close attention in designing interfaces for the Web: content organization, site navigation, and user satisfaction.

Joe Pryor, from DIALOG, discussed three of his company's Web-based products: DIALOG select, DIALOG Web, and intranet toolkit. DIALOG select, their first Web-based product, utilizes a flexible search option and Boolean capabilities; search forms translate user input into dialog search commands. DIALOG Web, version 2.0, is a Web-interface to command DIALOG. It includes built-in tools, bluesheets, pricing database selection, and quick functions as well as target search where a search query is formulated for the user. The intranet toolkit features a custom search forms generator so that companies can develop their own interfaces.

Authentication and authorization: how is it being implemented?

Using networked information resources and services today increasingly involves controlling access to licensed databases or student coursework and course materials. This control requires some method of first identifying positively who the requesters are (authentication) and ensuring that they are allowed to enter that resource (authorization). What are the options for accomplishing this control? How are libraries, instructors, and the university community handling this challenge? Clifford Lynch, Executive Director of the Center for Networked Information, outlined the issues involved and described several national initiatives studying this issue, including the Coalition for Network Information's White Paper on Authentication and Authorization. Sal Gurnani from the California Digital Library spoke about their project working with OCLC and JSTOR to pilot the use of X.509 client certificates in conjunction with a directory interface to authenticate individuals and authorize access to licensed content on publisher Web sites.

Lynch stated that authentication is easy to view as a technological and implementation problem. However, a lot of authentication is really about complex and serious interactions with fundamental and touchy policy issues; privacy, and user accountability.

Lynch made the point that it is worth understanding that authentication is a process whereby you assert your rights to use a name. That right is granted or attested to by some organization that authenticates you. A university might issue employee numbers for e-mail accounts. Alternatively, they might establish a proxy server from the university to verify session by session that you have the right to be known by that name and act under that name.

There are two kinds of approaches: proxies, based on credentials. In the first, the credentials approach, a user goes to the site and asks for digital credentials. The publisher can take those and validate the credentials with the issuing organization. The simple way to do this is through a secure socket layer (SSL) connection with your browser without prior agreement. It is quite secure; one passes a userid and password to a publisher. The publisher, through a second channel, verifies with the university.

Lynch concluded by stating that, within the next couple of years, we will see more complex authentication and authorization issues around ability to make statements. What kind of person wrote this? What kind of organization vouches for this? The use of metadata will increase. The Coalition for Networked Information is pursuing these questions. A white paper may be found at http://www.cni.org

Merri Beth Lavagnino, Director for Information Technology, Committee on Institutional Cooperation (CIC), spoke of working to implement interinstitutional authentication and authorization. The institutions are the big ten institutions plus the University of Chicago with 600,000 students, 200,000 faculty and staff, and 3.3 million living alumni. The charge of the committee was to create a CIC information technology environment that enables faculty, staff, and students from any CIC institution to routinely and seamlessly gain access to information and to interact with resources at any other CIC school for which access rights have been given. However, this was not totally received by all institutions. The committee agreed that it would be easier if one school were involved.

The technology they decided to use is known as distributed computing environment (DCE), which provides a many-to-one mapping to manage inter-institutional data via group definition. A server known as a cell is devised for each institution. Each cell provides many-to-one mapping. The cost is $25,000 per institution (includes hardware, software, and staff). The cells have intercell trust.

Sal Gurnani, Advanced Technology Group, California Digital Library, spoke about certificate-based authentication and directory services. The Digital Library Foundation (DLF) was looking for new authentication to support work across the University of California. They had problems with Kerboros and they needed technology that could be deployed quickly across many campuses. Gurnani stated that authorization assumes you start with ability to issue a certificate. An authority database with entries for all members with unique identities is the hardest to build. He gave details regarding the tests of an authentication protocol and operational model for using digital certificates for authentication.

Summary

The last session was the conference wrap-up, moderated by John Bertot, State University of New York at Albany, and Carol Hert, Syracuse University. One point made at the wrap-up was that information scientists must seek evaluation strategies that explicitly recognize and incorporate the social/political context. Also, the complexity of networked resources and their usage indicate that evaluators should use multiple methods in evaluations. A mechanism is needed to share information as to what methods of evaluation work, so that investigators do not reinvent the wheel with each evaluation. This effort should be international. Finally, each of us should be a forceful advocate of effective evaluation in our workplace.

The URL for the ASIS Mid-year 1999 meeting is http://www.asis.org/Conferences/MY99/.

Arthur Hendricks is Assistant Systems Librarian, Branford Price Millar Library, Portland State University, Portland, Oregon. bvah@odin.cc.pdx.edu