The Impact of Instructions on Survey Translation

RESEARCH REPORT SERIES (Survey Methodology #2007-18) The Impact of Instructions on Survey Translation: An Experimental Study Yuling Pan1 Brian Kleiner2 Jerelyn Bouic2 Statistical Research Division1 U.S. Census Bureau Washington, DC 20233 Westat2 1650 Research Boulevard Rockville, Maryland 20850 Report Issued: June 15, 2007 Disclaimer: This report is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau. The Impact of Instructions on Survey Translation: An Experimental Study Final Report Authors: Brian Kleiner Jerelyn Bouic Westat Yuling Pan U.S. Census Bureau January 2007 Prepared for: U.S. Census Bureau Prepared by: WESTAT 1650 Research Boulevard Rockville, Maryland 20850 (301) 251-1500 TABLE OF CONTENTS Page Introduction ......................................................................................................................... 1 Background and Purpose of the Study.................................................................................. 1 Study Approach .................................................................................................................... 4 Source Instrument ............................................................................................ 4 Recruitment of Translators .............................................................................. 5 Instructions Provided to Translators ................................................................ 5 Evaluation of Translations by Survey Researchers.......................................... 7 Interviews with Evaluators .............................................................................. 10 Analysis of Data............................................................................................... 10 Study Results ........................................................................................................................ 11 Quantitative Findings....................................................................................... 11 Qualitative Findings......................................................................................... 15 Evaluator Background ........................................................................ 15 Level of Difficulty of the Task ........................................................... 15 Assignment of Ratings........................................................................ 16 The Role of QxQs ............................................................................... 18 Discussion and Implications ................................................................................................. 19 Future Research ............................................................................................... 21 References ......................................................................................................................... 23 Appendix A: Source Instrument .......................................................................................... A-1 Appendix B: QxQs .............................................................................................................. B-1 LIST OF TABLES Table Page 1 Distribution of 27 translators into nine subgroups..................................................... 4 LIST OF EXHIBITS Exhibit 1 Translator recruitment criteria ................................................................................... 5 2 Core instructions provided to translators ................................................................... 6 3 Instructions received by the three instructional subgroups........................................ 7 4 Instructions given to evaluators ................................................................................. 8 5 Example page from Spanish evaluation form............................................................ 9 6 Protocol for interviews with evaluators ..................................................................... 10 LIST OF FIGURES Figure 1 Mean ratings for overall quality, faithfulness to intended meaning, and cultural appropriateness, by instructional subgroup ............................................................... 11 2 Mean ratings for French translations for overall quality, faithfulness to intended meaning, and cultural appropriateness, by instructional subgroup............................ 12 3 Mean ratings for Spanish translations for overall quality, faithfulness to intended meaning, and cultural appropriateness, by instructional subgroup............................ 13 4 Mean ratings for Chinese translations for overall quality, faithfulness to intended meaning, and cultural appropriateness, by instructional subgroup ............. 14 If a translation is to meet the four basic requirements of (1) making sense, (2) conveying the spirit and manner of the original, (3) having a natural and easy form of expression, and (4) producing a similar response, it is obvious that at certain points the conflict between content and form (or meaning and manner) will be acute, and that one or the other must give way. —Eugene Nida, 1964 Introduction Minimally directed translations of survey instruments, even by experienced translators, risk being of poor quality, whether it is because translators are not informed about the intended meaning of survey items (and therefore have to bring their own interpretations of meaning), they do not take into consideration cultural and communicative norms, or they are not given guidance about an acceptable degree of adaptation. For any of these reasons, translations may misrepresent the intent of survey items or may sound unnatural or even offensive to survey respondents. Nonetheless, minimally directed survey translation work is still the norm, rather than the exception. The experimental study described in this report aimed to assess the impact of different types of instructions given to translators for their translation of survey instruments from English into three target languages (Mandarin Chinese, Spanish, and Canadian French). The instructions were meant to guide translators toward a better understanding of what the survey items were intended to mean as well as how to make items sound more natural and culturally appropriate. Study findings indicate that while the provision of such instructions to translators had a significant impact on their translations, the direction of the impact (positive or negative) differed across the target languages, according to ratings of professional survey researchers who were also native speakers of those languages. The differences in ratings are attributed to the beliefs of the survey researchers and their level of commitment to two conflicting general types of equivalence in survey translation: equivalence of stimulus and equivalence of effect. Background and Purpose of Study Eugene Nida’s distinction between “formal” and “dynamic” equivalence in literary translation and the assertion that the two are inevitably in conflict bear strong relevance to current issues in survey translation theory. Within Nida’s framework, formal equivalence focuses on correspondence of both form and content between a source and a target language. Such correspondences may be grammatical, lexical, and/or semantic, such that “the message in the receptor language should match as closely as possible the different elements in the source language” (p161). In contrast, dynamic equivalence focuses less on formal and semantic correspondences and more on establishing an equivalent effect on the receiver of the message, such that “the relationship between receptor and message should be substantially the same as that which existed between the original receptors and the message” p156. Rather than attempting to impose the cultural patterns and formal properties of the source-language context, a dynamic translation aims for naturalness of expression and recreates the message with respect to modes of behavior that are meaningful within the context of the receptor’s own culture. While the translation of surveys is a far cry from translation of literature or poetry, Nida’s formal/dynamic equivalence distinction provides important background to a current debate within the field of comparative survey research. A guiding principle of survey research is that of standardization, which dictates that survey respondents receive exactly the same stimulus in the same manner: When respondents are all asked the same questions in exactly the same way, this reduces the chance of bias and 1 measurement error. For the same reason, survey researchers who conduct surveys that cross linguistic and cultural borders generally place a high premium on formal equivalence (in Nida’s terms) and aim for standardization and equivalence of stimulus within the translations of source instruments. Unfortunately, in practice the aim of equivalence of stimulus and of one-to-one formal and semantic correspondence quickly runs into difficulties in the translation of survey items. Van Ommeren et al. (1999) provide several examples that illustrate the need for adaptation from an instrument developed for use with Nepali-speaking Bhutanese refugees in living in Nepal.1 In one example, an English verb (exert) has no corresponding word in Nepali and had to be translated as a larger phrase: Source: “Have you ever had shortness of breath when you had not been exerting yourself?” Translation (into village Nepali): “Have your ever had shortness of breath when you had not been working, running, or climbing up a steep path?” In another example, rather than adding text not directly corresponding to the source, text was omitted in the translation, because it was not relevant to the ordinary experiences of target respondents: Source: “Did drinking ever cause you to give up or greatly reduce important activities – like participating in sports, going to school or work, or keeping up with friends or relatives?” Translation: “Did drinking ever cause you to give up or greatly reduce important activities – like going to school or work, or keeping up with friends or relatives?” In this case, participation in sports is not an important value for Nepali villagers and so this phrase was removed from the question. Here the adaptation was not strictly necessary for purposes of comprehension, but did assist in making the translation more relevant and meaningful within the cultural context of respondents. Sometimes cultural and linguistic norms require adaptation in a translation and departure from formal equivalence. In a third example, the translation of a sensitive question adds a great deal of text not found in the source item and modifies the question itself to ensure acceptability and cooperation among potential respondents (e.g., women for whom public discussion about sexual activities and interests is strictly taboo): Source: “During one of those periods [of feeling depressed] was your interest in sex a lot less than usual?” Translation: “Now I am going to ask you a private question. Please do not feel bad about answering as it will remain confidential. During one of those periods [of feeling depressed] was your desire of sleeping with your spouse a lot less than usual?” These examples show that maintaining equivalence of stimulus in translations of survey items can be a vexing challenge. The adaptation required to promote comprehension, relevance, and cultural appropriateness is often at odds with the researcher’s goal of standardization and formal and semantic correspondence between a source and a target survey item. On the other hand, strict adherence to formal 1 The authors, following Manson (1997), focus on four main ways in which translations must depart from a formal equivalence approach or risk failing to be meaningful to respondents: 1) comprehensibility, 2) acceptability, 3) relevance, and 4) completeness. 2 equivalence may lead to incomprehensibility, irrelevance, offensiveness, or awkwardness that may critically undermine the translation. It is in response to these sorts of challenges that some researchers and translation theorists have come to regard formal equivalence as an impractical and undesirable aim, opting instead for an approach based on dynamic equivalence and a greater tolerance for adaptation. Harkness and Schoua-Glusberg (1998) argue that a translation should adequately maintain the measurement properties of the source item provided that it faithfully conveys the intended meaning of the source. Such a pragmatics-based approach to translation situates the creation of meaning firmly within the communicative process and the linguistic and culture-specific norms and maxims that guide the conveying and interpretation of intent (Gutt 1991). Harkness (2003) points out that the “Achilles heel” of survey translation is that translators do not ordinarily take into consideration “pragmatic equivalence,” which involves ways of employing linguistic forms in particular contexts to communicate meaning and intent. Kleiner and Pan (2006) demonstrate that cross-cultural survey researchers and survey translators not only do not sufficiently consider sentence- level pragmatic equivalence, but further do not ordinarily take into consideration cross-cultural differences in discourse-level norms of language use. In this light, translation of survey items should primarily involve the communication of an intended interpretation by way of exploitation of the appropriate linguistic and cultural norms of the target community of respondents. This adherence to dynamic equivalence in translation thus requires a shift from equivalence of stimulus to equivalence of effect. Nida recognized a continuum of standards and practice between the extremes of full devotion to formal or dynamic equivalence in literary translation. In a similar way, survey researchers who work with translations of source instruments fall on different points of a stimulus/effect continuum. One finds among survey researchers, therefore, differing levels of fidelity to strict formal equivalence on the one hand and tolerance to adaptation on the other. In calling for faithfulness to intended meaning as a guiding principle of survey translation, Harkness and Schoua-Glusberg (1998) also argue for the need to provide translators with documentation so that they can better understand the intended readings and research aims of survey items: “…given that meaning is not fixed and finite, one of the goals of translation must be to convey the intended and most salient reading of a well-written question. The intended meaning of an item should therefore be documented for translators in the source materials they receive for their task.” (p95) The authors also point out that translators should be given detailed guidelines and examples regarding an acceptable degree of freedom in adapting a target item. In practice, however, translators are rarely provided with such documentation and guidelines and so are normally left on their own to divine the intended interpretation of survey items and the extent to which they can adapt them. While providing translators with materials that clarify the intended meaning of survey items seems reasonable on the surface (at least to adherents of this approach to survey translation), there is currently little empirical work that lends support to the utility of this practice. The exploratory study described in this report examines the impact of providing such documentation and detailed guidelines to translators. Specifically, the experimental study that was conducted addressed whether translators given explanatory material and instructions are able to produce translations that are more faithful to the intended meaning of source survey items and more culturally appropriate and natural sounding than translators who receive no such material. The next sections describe the study design. These are followed by sections on study findings and discussion. 3 Study Approach The study involved an experimental design to examine the extent to which particular instructions provided to translators had a significant effect on the translation of survey items. In order to assess the impact of different types of instructions, 27 professional translators translated an English source instrument into one of three target languages—Mandarin Chinese, Spanish, and Canadian French, following one of three sets of instructions (see below for details about the instructions). Table 1 shows the distribution of the 27 translators into nine different subgroups. Translators who fulfilled our selection criteria (see below) were randomly assigned to one of three sets of instructional subgroups. Table 1.—Distribution of 27 translators into nine subgroups Chinese Spanish French Instruction set 1 (Group A) 3 3 3 Instruction set 2 (Group B) 3 3 3 Instruction set 3 (Group C) 3 3 3 Once the translations were completed, 15 professional survey researchers who were native speakers of the target languages conducted blind evaluations, with each evaluator examining three translated versions of each survey item, one version for each set of instructions. The evaluation involved rating each translated item on Likert scales along several dimensions (see below). Thus each survey item received three ratings along each dimension from each evaluator. Analyses involved comparison of ratings along the three dimensions overall and for individual items, following the three sets of instructions. It was presumed that, if significant differences were found in the ratings, this would suggest that certain instructions provided to translators may result in higher quality translations, which in practice would require less followup quality control and revision. The remainder of this section spells out the details of the study design, including discussion of the source instrument, instructions to translators, how translators and evaluators were recruited, how evaluators rated the various translations, and how the ratings were analyzed. Source Instrument With the permission of the National Center for Health Statistics (NCHS) of the Centers for Disease Control (CDC), we adopted items from the National Survey of Children with Special Health Care Needs for our source instrument. This “donor” survey was selected because it satisfied several criteria—this was a household telephone survey with sensitive and varied types of questions. The specific items that were adopted were selected based on the need for questions with various structures (e.g., yes/no questions, scales, multiple response categories, questions with prefaces, questions with topic shift indicators, discourse markers) and varying content (e.g., factual questions, opinion questions, sensitive questions about the health conditions of children, demographic questions). The 18 adopted items were maintained without changes to wording and were placed in a logical order.2 The National Survey of Children with Special Health Care Needs survey has been thoroughly 2 Sixteen of the items were actual survey questions, while the first two were drawn from introductory text where the purpose of the survey and telephone call are explain