Design, Evaluation, and Analysis of Questionnaires for Survey Research

Citation

Borelli, M. (2008), "Design, Evaluation, and Analysis of Questionnaires for Survey Research", Journal of Workplace Learning, Vol. 20 No. 6, pp. 453-456. https://doi.org/10.1108/13665620810892111

“Designing a survey involves many more decisions than most researchers realize. Survey specialists, therefore, speak of the art of designing survey questions (Payne 1951). However, this book introduces methods and procedures that can make questionnaire design a scientific activity. This requires knowledge of the consequences of the many decisions that researchers take in survey design and how these decisions affect the quality of the questions” (p. v). Following this incipit, the book by Willem Saris and Irmtraud Nora Gallhofer accompanies the reader interested in the survey design methodology, analysing in great detail the survey design process, focusing on statistical knowledge and on an extensive two‐decade literature based meta‐analysis. The book is also an effective theoretical support to the Survey Quality Prediction (SQP) programme, software co‐developed by the same authors together with William van der Veld, the beta version of which is freely downloadable at www.sqp.nl

The book starts with an introductory section, in which the rather complex procedure for obtaining data for research is briefly outlined, focusing on a number of decisions that researchers are required to take. In the first part (Chapters 1‐3), a three‐step procedure to design requests for an answer with high certainty is considered, while part two (Chapters 4‐8) concerns the choices involved in questionnaire design. Part three (Chapters 9‐12) discusses the effects of survey characteristics on data quality, and the fourth part (Chapters 13‐16) provides an applicative section devoted to social sciences research.

In the introductory section, highlighting the increasing importance during recent decades in the survey research field in social and behavioural sciences, and stressing the complexity of the procedures for obtaining research data, Saris and Gallhofer briefly summarize the steps that a questionnaire design requires. The first step is represented by the correct choice of a topic, in descriptive or in explanatory studies, both in experimental and non‐experimental research. The second choice that a researcher is called to take in designing a survey concerns the variables to be measured, taking into account their direct and indirect effects in the structural equation model framework. The choice of a suitable data collection method is needed as a third step, also taking into account costs balance versus the quality of data collected and question formulation, which, according to the authors, naturally leads to the fourth decision to make while designing a survey – the choice of the operationalisation (i.e. the translation of concepts into questions). The fifth step is linked to the quality testing of the questionnaire in terms of checks on face validity, routing control, the prediction of questions' quality and questionnaire testing by means of a pilot study: at this level, quality can be usually explored by involving an expert panel, or on the basis of a coding scheme, or more recently by means of dedicated software, like the authors' free software, SQP. Further steps involve the final formulation of the questionnaire (taking into account recent results on the influence of layout on survey quality), the necessity of a rigorous sampling procedure to assure the correct inference on population, and design of the fieldwork in order to manage both interviews and interviewers.

The first part of the book fully describes a three‐step procedure to design requests for an answer. In Chapter 1, the authors provide a number of literature studies involving the effect on responses of the wording of survey questions, also recalling the partial lack of research into the problem of translating concepts into questions. In particular, attention is focused on the difference between concept‐by‐intuition and concept‐by‐postulation, their distance and the possibility of measuring them, confirming by examples that concepts‐by‐intuition can be easily transformed into questions, while concepts‐by‐postulation cannot be operationalised directly in survey questions. Chapter 2 is devoted to emphasising how assertions can be formulated for the most common social sciences concepts‐by‐intuition. The authors provide a definition of a survey item, discussing in great detail the linguistic components of sentences with a view to analysing the way to set assertions and requests for an answer, identifying three basic assertion structures that can be used in representing most concepts‐by‐intuition, and providing a simple way to detect which kind of concept has been used in assertions applied in practice. In Chapter 3, exploiting a large body of research literature in linguistics, the discussion about how assertions can be transformed into requests for an answer is concluded, providing a simple classification of different types of both direct and indirect requests which ask the respondent to make a choice from a set of possible answers, and focusing on the aspects that one can formulate very different requests for an answer while the concept, the topic of research and the set of possible responses remain the same.

Part two adopts a more “handbook” structure, dealing in great detail with other choices which have to be made in designing a questionnaire, or the different ways in which requests for an answer can be formulated, which response alternatives can be chosen and which are the main structure differences between open‐ended and closed survey items. Moreover, structures of batteries of survey items and the importance of question order, question layout and data collection methods are examined. In Chapter 4, features connected with the research goal, like time reference, social desirability and saliency are allotted, and the authors also provide advice to enhance respondent comprehension, i.e. to avoid so‐called “double‐barrelled requests” and, more generally, assertions with more than one component. The advantages and disadvantages of using “batteries of stimuli” is discussed, along with other topics like the formulation of comparative or absolute requests for answers, the use of conditional clauses, and balanced or unbalanced requests for answers in which both negative and positive answers are analysed. In Chapter 5 the classical methodological aspects of open and closed answer categories is dealt with. In particular, the authors report detailed examples of closed categorical requests, categorising them by means of nominal, ordinal and continuous scales, while in Chapter 6 the question is deepened by analysing the basic components of a survey item, distinguishing between parts embedded in the request for an answer and parts that can be juxtaposed before or after it. The authors also provide some survey item complexity measures that have appeared in the literature, and give some recommendations about what form of survey item should be used, assuming that a trade‐off exists between precision and complexity in sentence formulation. In Chapter 7 some detailed examples show the differences between batteries with respect to the way they are operationalised in their respective modes of data collection, in oral, mail and CASI surveys. Lastly, in Chapter 8 the importance of considering the presence or the absence of an interviewer is assessed, while the oral or visual mode of presentation and the role played by computers in selecting the proper data collection method – also recognising the role played by question ordering and by questionnaire layout – is examined.

The third part of the book is devoted to the effects of survey characteristics on data quality. The authors explain that designing a survey questionnaire can be a scientific activity only if some quality criteria are followed, if reliability and validity estimates are provided and if the effects of the measurement characteristics on the quality of the survey questions are taken in account. Chapter 9 starts with a literature‐based discussion on multitrait‐multimethod design (MTMM) and provides indications of how to detect the quality of measures by means of a correlation matrix, giving definitions of reliability, validity, systematic method effect, total quality of measure and the effect of the method on the correlation inside the structural equation model framework. Chapter 10 provides general procedures and examples of reliability, validity and method effect estimates obtained by the LISREL software, and the advantages and disadvantages of MTMM are discussed. The more technical Chapter 11 introduces the split‐ballot MTMM technique, while Chapter 12 represents a summary of the state of the art in investigating the effects of questionnaire design choices regarding reliability and validity. The authors report a meta‐analysis performed on a number of survey items and, despite its work‐in‐progress nature, they provide a summary of choices that could have a positive or negative effect on reliability and on validity in questionnaire item survey design.

The last part of the book deals with some applications of the prior theoretical framework. Chapter 13 describes the features and potentiality of the SQP software, which can be understood as an expert system allowing the survey designer to evaluate the quality of the proposed survey items of a questionnaire on the basis of previously reported meta‐analysis knowledge. SQP is able to read and code questionnaire survey items, to predict the quality of proposed survey items on a number of criteria, to provide information on the effects of different choices, and to provide suggestions for improving those items. Its user‐friendly interface helps users to apply information in designing the questionnaire at a point where there is still time to improve it. Chapter 14 discuss the modelling approach in the latent variable framework by which the techniques learned that are used to assess quality of measures for concept‐by‐intuition can also be used in concept‐by‐postulation, the latter possibly being seen as an aggregate of several measures of concept‐by‐intuition. In Chapter 15, the authors describe an approach involving the estimation of simple substantive models that include the composite score of the variables, which can overcome difficulties related to coping with measurement error in survey research, directly linked to oversimplified models constructed in the structural equation framework, or to simple models where the variables are the composite scores of concept‐by‐postulation. In particular, information about the quality of single questions can be used to estimate the effect that variables corrected for measurement error have on each other. Lastly, Chapter 16 is devoted to the problem of measurement errors in cross‐cultural research over countries, and in particular to the fact that those errors can affect comparisons, leading researchers to attribute erroneous differences in countries to effective differences in the quality of measurement instruments.

Design, Evaluation, and Analysis of Questionnaires for Survey Research is written for a broad of range of readers and, although it assumes a basic background in statistics and in latent variable models, it could represent a sound text for both undergraduate or graduate‐level survey research courses in social sciences. A number of thought‐provoking exercises help readers to grasp the methodology and to deepen their understanding of the techniques presented. The book can also be considered as a reference text for practitioners in business and social research fields. In particular, a lot of papers published in Journal of Workplace Learning use questionnaires to detect and to analyse how organisational environment influences learning and how the learning process works. Very often papers lack a detailed description of the questionnaire, even though this would be very important in order to give readers some more indications about the quality of the questionnaire. This book suggests effective methods to detect questionnaire quality that could be adopted by researchers in workplace learning.