Cross-Cultural Measurement of Protective Factors · Cross-Cultural Measurement of Protective...
Transcript of Cross-Cultural Measurement of Protective Factors · Cross-Cultural Measurement of Protective...
Cross-Cultural Measurement of Protective Factors
Fons J. R. van de Vijver
www.fonsvandevijver.org
Background
• How can we assess protective factors from three domains (Zautra, Hall, & Murray, 2010): (1) individual characteristics, such as intelligence and personality; (2) characteristics of families, such as family support; (3) the larger social context, such as cultural norms
• Special foci– Mixed methods
• Combination of qualitative and quantitative methods
– Combination of emic and etic approach– Linking conceptualization, methods, and analyses as
procedure to enhance quality management– Test adaptations
Contents
1. Emic? Etic? Emic or etic? Emic and etic?
– There may be light at the end of the tunnel…
– SAPI as an example of combining the two
2. Options in instrument choice in cross-cultural studies
3. Quality enhancement in cross-cultural studies using mixed methods
4. Conclusions
Etic Approaches to Personality
•Five-Factor Model (Costa, McCrae)
–Evidence for invariance of five factors across many
countries
–Neuroticism, Extroversion, (Openness),
Agreeableness, Conscientiousness
Emic Approaches to Personality
• Chinese concept of interpersonal relatedness
• South-African concept of Ubuntu
– "Humanity towards others"
– "I am because we are"
– "A person is a person because of other persons"
Interim Conclusion
• Evidence for universality of Five-Factor Model
• Evidence for emic perspective
• Yet, only case studies, not accumulating
• “Emic Etic War” disappearing
• Need to integrate etic and emic approaches
SAPI and Cross-Cultural Psychology
• Goals of Cross-Cultural Psychology:
1. To transport and test (hypotheses and findings)
2. To explore other cultures in order to discover psychological variation
3. To integrate findings in order to generate a more inclusive, globally applicable psychology
• Goals ≈ History of cross-cultural psychology
SAPI Project • Aim
– No test available with demonstrated validity in all groups
– Develop a personality instrument for South Africa that can accommodate its diversity
• Stages1. Qualitative Stage
– Identifying personality structure in an indigenous, comparative study
– Generating and piloting items/scales2. Quantitative Stage
Method
•Participants •Adults from all eleven language groups•Sample guidelines: Socio-economic Status, Education (low, medium, High), Males/Females, Age (18-35 yrs & 35 older)
•Interview done in participant’s first language•Instrument
•Person descriptions of familiar persons•Your own personality – Describe yourself •A parent •….
Stages in Data Analysis
• Phase 1– Preparing Data
• Translations
• Removing synonyms, …
• Phase 2 – Iterations of
• Condensing data
• Consulting language and cultural experts
• Phase 3:
– Conceptual Cluster Analysis
14
Condensing the Data
Facets Categorising Cleaning Response in English Response in Ndebele
abusive
(34) abusive abused He is abused Uhlukulumezekile epilweni
abusive abused him
he abused him even when abuse
was not there kade ambuser lokha ingekho
abusive abuses me He abuses me Uyangi hlukumeza
abusive Abusive person An abusive person Umuntu ohlukumezako
Meddlesomeness Conflict SeekingInterpersonal Relatedness
Amiability
Egoism
Empathy
Gratefulness
Approachability
FairnessAchievement Orientation
Thoughtlessness
DedicationSociability
Integrity
Dominance
Epistemic Curiosity
MaterialismEmotional Control
Emotional Sensitivity
Ego Strength
Self-discipline
Active support
EXTRAVERSION
INTEGRITYExpressiveness
Positive Emotionality
Orderliness
Neuroticism
Encouraging others
Guidance
FACILITATING
Broad-mindedness
Openness to Experience
Skillfulness
Aesthetics Reasoning
Social Intellect
SOFT-HEARTEDNESS
Hostility
RELATIONSHIP HARMONY
Courage
Balance
CONSCIENTIOUSNESS
INTELLECT
OPENNESS
EMOTIONAL STABILITY
Subcluster Facet Example Response (Language)
Approachability Accommodating Addressed us in English so we could understand (Xhosa)
ApproachableShe is approachable, I could speak to her about anything (S Sotho)
Arrogant He thinks he is better than all the other people (N Sotho)
Flexible Flexible to situation (Tswana)
Humble She is a humble and down to earth person (Ndebele)
Open for Others Accepts people for who and what they are (English)
Proud Is proud and thinks of herself better than others (Swati)
Stubborn Was stubborn, did not listen to anybody (Tswana)
Tolerant Tolerant (Afrikaans)
Welcoming Welcoming – to everyone (Venda)
Conflict-Seeking Argumentative Likes to quarrel (Xhosa)
Provoking Provocative and calls people names (Swati)
Troublesome Creates tension for nothing (Zulu)
Cluster: Relationship Harmony
Active Support Community Involvement (11/143)There is one person who is always looking after the community (Zulu)
Heedful She listens when you talk to her (S Sotho)
Helpful Is helpful when you are in need (Swati)
Protective Protective (Xhosa)
Solving Problems of Others If I have a problem, she knows how to solve it (Ndebele)
Supportive I like to give people my support (Tswana)
Amiability Friendly She is a friendly person (Tsonga)
Irritating He is annoying and irritating (S Sotho)
Kind Kind (Venda)
Likeable He is loved by everyone (S Sotho)
Pleasant He was a nice person to live with (Zulu)
Stern Always serious, not smiling (Xhosa)
Egoism Generous One who is generous and gives food when asked (Swati)
Greedy Greedy (Afrikaans)
Jealous A person who is jealous of other people’s possessions (Zulu)
Self-Centered All revolves around her, she thinks (English)
Selfish Wants everything for himself (Xhosa)
Cluster: Soft-Heartedness
Subcluster Facet Example Response (Language)
Approachability Accommodating Addressed us in English so we could understand (Xhosa)
ApproachableShe is approachable, I could speak to her about anything (S Sotho)
Arrogant He thinks he is better than all the other people (N Sotho)
Flexible Flexible to situation (Tswana)
Humble She is a humble and down to earth person (Ndebele)
Open for Others Accepts people for who and what they are (English)
Proud Is proud and thinks of herself better than others (Swati)
Stubborn Was stubborn, did not listen to anybody (Tswana)
Tolerant Tolerant (Afrikaans)
Welcoming Welcoming – to everyone (Venda)
Conflict-Seeking Argumentative Likes to quarrel (Xhosa)
Provoking Provocative and calls people names (Swati)
Troublesome Creates tension for nothing (Zulu)
Cluster: Relationship Harmony
Active Support Community Involvement (11/143)There is one person who is always looking after the community (Zulu)
Heedful She listens when you talk to her (S Sotho)
Helpful Is helpful when you are in need (Swati)
Protective Protective (Xhosa)
Solving Problems of Others If I have a problem, she knows how to solve it (Ndebele)
Supportive I like to give people my support (Tswana)
Amiability Friendly She is a friendly person (Tsonga)
Irritating He is annoying and irritating (S Sotho)
Kind Kind (Venda)
Likeable He is loved by everyone (S Sotho)
Pleasant He was a nice person to live with (Zulu)
Stern Always serious, not smiling (Xhosa)
Egoism Generous One who is generous and gives food when asked (Swati)
Greedy Greedy (Afrikaans)
Jealous A person who is jealous of other people’s possessions (Zulu)
Self-Centered All revolves around her, she thinks (English)
Selfish Wants everything for himself (Xhosa)
Cluster: Soft-Heartedness
Conclusion
• Initial analyses suggest the presence of both emic and etic factors
– Emic notably in the social-relational domain
• Five Factor Model well replicated for N, E, O, C
• Agreeableness to social-relational cluster
• Integrity also found
Links with Other Instruments
• History of (claimed) culture-specific concepts: amae(Japan), filotimo (Greece)…
• CPAI-2: Expanding the Big Five with Interpersonal Relatedness (IR)
– Relational Orientation
– Social Sensitivity
– Discipline
– Harmony
– Thrift vs. Extravagance
– Traditionalism vs. Modernity
• First identified in a collectivistic context (China), later replicated in other contexts (Asian countries, US)
SAPI, Big Five, CPAI-2
• SAPI items: Input from original responses
• Item criteria– Short, simple, and clear
– Written in 1st person, starting with “I” followed by concrete behaviors, object, and context
– No negations in item stems (e.g. “I do not express my opinion” vs. “I do what others expect without expressing my opinion”)
– Single activity, habit, or preference (terms such as like/dislike were avoided)
– Temporal qualifiers were excluded, e.g., often, always, sometimes, etc.
• Hierarchical factor analysis replicated subclusters
Scales
• SAPI SOCREL: 10 scales, 91 items (mean α = .81 in both groups;
all items paraphrased)
• Facilitating (10 items, e.g., “I give guidance to people in their life decisions”)
• Integrity (11 items, e.g., “I acknowledge my mistakes”)
• Relationship Harmony (10 items, e.g., “I help people live in peace”)
• Active Support (13 items, e.g., “I support others when they need it”)
• Empathy (6 items, e.g., “I consider how others feel”)
• Unreliability (7 items, e.g., “I fail to meet others’ expectations”)
• Harmony Breach (8 items, e.g., “I cause fights”);
• Arrogance (6 items, e.g., “I show that I am better than others”)
• Hostility (10 items, e.g., “I make people feel vulnerable”)
• Egoism (10 items, e.g., “I only think about my own interests”).
• Big Five: Basic Traits Inventory (BTI) developed in SA; 24 facets, 193 items (mean α = .71 B, .80 W)
• CPAI-2 IR: 6 scales, 74 items (mean α = .58 B, .59 W)
– “Usually when I talk with people, I take great care not to offend them”
– “I strongly support the principle that if a family lives in harmony all things will prosper”
Blacks Whites
Males / Females 429 / 613 126 / 311
Mean age in years (SD) 24.33 (4.79) 19.55 (2.23)
Sample
What Does “Equivalent” Mean?
• Eusebius Hieronymus (St. Jerome, famous bible
translator from Greek and Hebrew to Latin; ±347—
419/420):
– 2 types of translations: “words” and “meanings” (he
favored the latter)
Options• Adoption (Close “literal” translation)
– Advantage: maintains metric equivalence– Disadvantage: adequacy (too) readily assumed, should be
demonstrated
• Adaptation (changing contents of one or more items so as to increase cultural appropriateness)– Advantage: more flexible, more tailored to the context– Disadvantage: fewer statistical techniques available to
compare scores across cultures
• Assembly (composing a new instrument)– Advantage: very flexible– Disadvantage: almost no comparability maintained
What is the Best Option?
• One type is not intrinsically better or worse than another
• Main question is
NOT
• What is globally the best choice?
BUT
• What is the best choice in a specific case?
Construct equivalence
• Similarity of construct in source and target culture
Cultural equivalence
• Norms about interaction (modes of address)
• “Cultural fact sheet”
Linguistic equivalence
• Translation accuracy: Retention of denotation and connotation
Measurement equivalence
• Retention of psychometric features (response styles)
• Similarity of factors measured by a test and comparability of scores
Four Important Perspectives (Harkness &
Van de Vijver, in preparation):
• A good translation/adaptation combines equivalence perspectives
• What is a good translation/ adaptation?
–A translation or adaptation is good when it combines high levels of construct, cultural, linguistic, and measurement equivalence.
Is There a Best Way to Translate an Instrument?
• Simple items often straightforward to translate – Close translations will do well, various kinds of
equivalence jointly maximized
• More complex items often require choices about which equivalence will be maximized:– Maximizing comparability or cultural
appropriateness ?
• Different perspectives on equivalence often, but not always compatible
• Example: cross-cultural differences in modes of address – Maximizing linguistic equivalence may challenge
cultural appropriateness (e.g., requests may be too direct)
– Maximizing cultural appropriateness may challenge statistical equivalence (e.g., rephrasing may threaten comparability of scores)
Taxonomy of Adaptations
Need for adaptation can be
• Construct-driven
• Culture-driven (communication style)
• Language-driven
• Measurement-driven (familiarity/recognizability)
A Sample of Possible Procedures (after
Harkness, 2003)
• Translation back translation
• Committee approach (forward translations)
• Mixed approaches (e.g., independent forwards)
Translation stage
•Think alouds, focus groups•Feedback from mono- and bilinguals•Comprehension and readability checks
Pretesting stage (qualitative)
• Equivalence and bias analyses (DIF, structural equivalence)
Pretesting or actual administration (quantitative)
Strength and Weakness of Translations Back Translations
Main strengths
• Well accepted quality check; standard procedures well known in scientific community (incl. researchers, grant institutions and journal boards)
• No knowledge of target language required
Main weaknesses
• Capitalizes on linguistic, cultural, and item-writing skills of (usually) a single person
• Can produce stilted language
• Readability and comprehensibility in target language may be problematic
• Study of intelligence among urban slum children in India
• Influence of supplementation of micronutrients
• Source: Malda, Van de Vijver, Srinivasan, Transler (2008): Adapting a Western Cognitive Test for a Non-Western Context: The KABC-II in Bangalore, India
Subtests:
• Atlantis
• Number Recall
• Rover
• Triangles
• Word Order
• Pattern Reasoning
• Story Completion replaced by adaptation of
WISC(/-R/-III) Picture Arrangement
1. Example Construct-Driven
• Problems with the behaviors or attitudes associated with the construct or with communication norms pertaining to these behaviors or attitudes
• Usage of somatic and psychological symptoms in depression inventories
• Differential norms in allowance to express psychological symptoms across cultures
• Patel, Abas, Broadhead, Todd, & Reeler (2001) – In Zimbabwe, multiple somatic complaints such as headaches and
fatigue are the most common presentations of depression. On inquiry, however, most patients freely admit to cognitive and emotionalsymptoms. Many somatic symptoms, especially those related to the heart and the head, are cultural metaphors for fear or grief. Most depressed individuals attribute their symptoms to “thinking too much” (kufungisisa), to a supernatural cause, and to social stressors. Our data confirm the view that although depression in developing countriesoften presents with somatic symptoms, most patients do not attributetheir symptoms to a somatic illness and cannot be said to have “pure” somatisation. This means that it is vital to understand the culture specific terminology used by patients and to assess mood in thosewith multiple somatic complaints.
• Consequence – Common western measures of depression will under-diagnose
depression in Shona speakers.
47
2. Example Culture-Driven
Example: ‘Burglar’ (Picture Arrangement; adapted for use
in low-SES children in Bangalore, India )
Problems: 1. Unclear whether the burglar was getting in
or getting out;
2. Man not recognized as burglar;
3. Window was not recognized (vertically moving windows are uncommon in India)
Malda, Van de Vijver, Srinivasan, Transler (2008): Adapting a Western Cognitive Test for a Non-Western Context: The KABC-II in Bangalore, India
• Example: Do you often feel distressed? • Translation to Dutch:
– “Distressed” does not have an equivalent word in Dutch – Possible solutions
• Composite of different emotions in Dutch; ask for frequency of composite (“how often do you feel X and Y?”). Problem: composite may not be recognizable
• Choose a single emotion that is as close as possible; problem: change of item content if no close match can be found
• Describe the emotion in the item (e.g., vignette); problem: may require a similar description in English original
– Need to check adequacy of chosen solution in statistical analysis – Combination of judgmental and statistical evidence crucial in
instruments that are more difficult to translate/adapt
3. Example of Language-Driven Adaptation
• Language and test content:
– Adaptation of words in subtest Atlantis:
• Kannada nonsense words
(e.g., English ‘Dablee’ Kannada ‘Ribu’)
• Important: number of syllables
– Adaptation of digits in subtest Number Recall
• based on number of syllables (1 in English version; first 2 and then 3 in Kannada version)
50
4. Example of Measurement-Driven Adaptation (Unfamiliarity)
• Kaufman ABC used in Bangalore (Kannada-speaking children)
• Adaptation of words in subtest Word Order based on:
• Unfamiliarity and ambiguity of objects and words
• Number of syllables
Original version
Kannada version
Malda, Van de Vijver, Srinivasan, Transler (in review): Adapting a Western Cognitive Test for a Non-Western Context: The KABC-II in Bangalore, India
Original version
Kannada version
Problem: word for star in Kannada is too long, English word “star” is well known but too short (monosyllabic)
Original version
Problems:
(1) Key was often called ice cream;
(2) English word “key” was often used, which is too short (monosyllabic)
Kannada version
Original version
Problem: original drawing was not easily recognized as house, distinguishing features added
Kannada version
Rover
Test content:• Additional instructions in subtest Rover
• One additional instruction in subtest Pattern Reasoning
• Slight change of subtest composition and item order in subtest Triangles
Sample item Original version
Sample item Indian version
Problem: original sample item was too difficult; this item has been added as actual test item
Mixed Methods: The “newest kid on the block”
• Recent interest
• Attempt to overcome QUAL—QUAN dichotomy
Examples of books
• Brewer, J., & Hunter, A. 1989. Multimethod Research: A Synthesis of Styles. Newbury Park: Sage.
• Cook, T. D., & Reichardt, C. S. (Eds.). 1979. Qualitative and quantitative methods in evaluation research. Beverly Hills, CA: Sage.
• Creswell, J. W. 2003. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (2nd Ed).Thousand Oaks, CA: Sage.
• Frechtling, J., & Sharp, L. (Eds.). 1997. User-Friendly Handbook for Mixed Method Evaluations. Washington, DC: National Science Foundation.
• Greene, J. C., & Caracelli, V. J. (Eds.). 1997. Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms (New Directions for Evaluation, No. 74). San Francisco: Jossey-Bass.
• Newman, I., & Benz, C. R. 1998. Qualitative-Quantitative Research Methodology: Exploring the Interactive Continuum. Carbondale, IL: Southern Illinois University Press.
• Tashakkori, A., & Teddlie, C. 1998. Mixed Methodology: Combining the Qualitative and Quantitative Approaches (Applied Social Research Methods, No. 46). Thousand Oaks, CA: Sage.
• Tashakkori, A., & Teddlie, C. (Eds.). 2003. Handbook of Mixed Methods in Social & Behavioral Research.Thousand Oaks, CA: Sage.
From http://personal.bgsu.edu/~earleym/MIXEDMETHODS/resources.htm
More than Just Two Methods…
• Qual and quan have shown their own independent developments– Have paradigmatic properties
• The debate often has ideological undertones – The quantitative paradigm
• Associated with positivism: there is an objective truth that can be known
• Theories and methods are more adequate when they give a better representation of this truth
– The qualitative paradigm • Multiple realities or truths exist • These realities are constructed
Conceptual backgroundQUAL methods
• Background QUAL is heterogeneous, rangingfrom methods that are very close to QUAN topostmodernism and constructivism/socialconstructionism
• Examples: – Critical ethnography, Critical theory, Dialectical research, Discourse
analysis, Ethnomethodology, Ethnography, Focus group, Grounded theory, Hermeneutics, Participatory action research, Phenomenology (science), Semiotics
QUAL in this workshop
• QUAL not as a rebellion, a method for criticizing empiricial research or positivistic approach
• QUAL as a method for collecting information about the world that is replicable, accessible for others
Most common QUAL methods
• Participant observation – appropriate for collecting data on naturally occurring behaviors in
their usual contexts.
• In-depth interviews – optimal for collecting data on individuals’ personal histories,
perspectives, and experiences, particularly when sensitive topics are being explored.
• Focus groups – effective in eliciting data on the cultural norms of a group and in
generating broad overviews of issues of concern to the cultural groups or subgroups represented.
(from Overview Family Health International)
Some Highlights
• QUAL– Various approaches available
– Popular approach: Grounded Theory (Glaser & Strauss)
• is a systematic methodology in the social sciences emphasizing generation of theory from data in the process of conducting research. (Wikipedia)
• Rather than beginning by researching and developing a hypothesis, the first step is data collection, through a variety of methods.
• Theory as output (rather than input)
Grounded Theory: Four Stages of Analysis
Stage Purpose
CodesIdentifying anchors that allow the key points of the data to be gathered
ConceptsCollections of codes of similar content that allows the data to be grouped
CategoriesBroad groups of similar conceptsthat are used to generate a theory
TheoryA collection of explanations that explain the subject of the research
• QUAN
– Well established quality criteria
– Focus on reliability and validity
– Often aimed at hypothesis testing
However…
• Many differences between QUAL and QUANmethods are overrated
– Most QUAN studies have QUAL stages (e.g., instrument design)
• What is needed?
– Discussion of which method is more adequate in which conditions
– How to combine methods (Tashakkori & Teddlie, 2003)?
Strengths and Weaknesses of Qualitative and Quantitative Methods
• Qual methods display their main strength in the context of discovery – E.g., helpful to get information about various
cultural characteristics of an ethnic group we are dealing with for the first time, to build models, and to generate hypotheses.
• Quan methods are particularly strong in context of justification, testing procedures/hypotheses.
• So, there is complementarity
2. Mixed Methods
• Involves the use of both qual and quan in a single study– (also qual—qual and quan—quan possible)
• Definition– the collection or analysis of both quantitative
and/or qualitative data in a single study in which the data are collected concurrently or sequentially, are given a priority, and involve the integration of the data at one or more stages in the process of research
(Creswell, Plano Clark, Gutmann, & Hanson, 2003, p. 212)
Example• Poverty survey in KwaZulu-Natal (Adato, Lund, &
Mhlongo, 2006) – First stage: a longitudinal quantitative survey – Second stage: a qualitative study (using focus group
discussions and key informant interviews) building on quantitative part
• Covering various aspects of poverty that were not covered by the survey
– the phenomenology of poverty– identifying mechanisms how members of communities cope
with adverse conditions– finding useful information that could be brought back to the
community so that the community would profit from the study.
Taxonomy of Mixed Methods
• Three underlying dichotomies Leech and Onwuegbuzie (2008):
– Level of mixing • Partially mixed versus fully mixed
– Time orientation• Concurrent versus sequential
– Emphasis of approaches• Equal status versus dominant status
3. Some Myths Regarding the Relation between Qualitative and Quantitative
Methods
Myth 1: • Structured data collection methods such as
observations, psychological tests, questionnaires, and (structured) interviews, belong to the realm of quantitative methods, whereas unstructured data collection methods such as archival analysis and semi-structured or unstructured interviews can only be studied using qualitative methods
Myth 2: • There is a close link between the study of
cultural specifics and qualitative methods on the one hand and the study of cross-cultural universals and quantitative methods on the other hand.
Myth 3:
• The use of mixed methods will resolve all problems of qual and quan methods• Not a “third movement” that solves all problems
but “best of both worlds”
Myth 4:• All qualitative and quantitative methods can
be combined• In postmodern tradition the criterion of validity is
abandoned and is replaced by the extent to which other researchers are convinced by the arguments proposed by an author
• Incommensurable with methods proposed here (both qual and quan)
4. Challenges
• Challenge 1: Establishing Quality Criteria
• Sale and Brazil (2004)– A meta-analysis of mixed-method studies – No specific criterion to evaluate such studies
reported– Needed?
• Probably not. – Lincoln and Guba (1986):
• Set of criteria that apply to both qual and quan
Concept Description Quan Qual
Truth value Adequacy of study inferences
Internal validity Credibility
Applicability Specification of the context in which the information was obtained and new contexts in which the same information would hold
External validity Transferability or fittingness
Consistency Extent other researchers or procedures would yield similar outcomes
Reliability Dependability
Neutrality The influence of researcher and his or her ideas on study outcomes
Objectivity Confirmability
• Challenge 2. Triangulation
– Usually combining qual and quan information
• Yet, the concept of triangulation is broader and could also refer to combining qualitative information (or quantitative information, for that matter)
– More work on integration of information needed
Kinds of Combinations
• QUAL—QUAN:– Often transformed to qual—qual or quan—quan
combination
• QUAN—QUAN: – monotrait—multimethod matrices
• QUAL-QUAL:– Procedures less formalized.
– Triangulation of qualitative evidence is a fairly common procedure in ethnographic research.
• Common issue:– Accounts of the same events provided by various
informants or sources can have three relations with each other
– Possible outcomes: Information can be• 1. converging
– usually interpreted as strengthening the quality of the observations and inferences based on them
• 2. complementary when different sources address different aspects of past events
• 3. incompatible– Suggesting inadequacy of at least one of the
sources and makes it impossible to draw any firm conclusions (memory error, dependability of informant, etc. )
Example of Mixed Methods:QUAN + QUAL
• Benítez Baena, Van de Vijver, & Padilla García (2014): A study of adequacy of adaptations/translations
• Well-being instruments in large-scale surveysinvolving Spain and the Netherlands (English versions as reference version)
Study design
EVS EQLS WVS ESS SHARE
All these studies• Assessed well-being• Used large probability samples• Likert-type of items
Study 1: Quantitative Phase
• Confirmatory Factor Analysis used to compare factor models across the two countries
• Bias found in several items
– More bias in agreement than in frequency format
– Uniform bias related to topic
– Non-uniform bias related to other structural item characteristics, such as use of extreme qualifiers or terms (such as “never” or “strongly”)
Study 2: Experts Information
• We asked experts to compare translations (all combinations were examined)
– English – Spanish
– Spanish – Dutch
– English – Dutch
• Check of translation accuracy
Study 2: Experts Information (independent assessment)
Spanish
Podría decirme cómo de satisfecho está con su
vida familiar?
English
Could you tell me how
satisfied you are with your
family life?
Dutch
Kunt u aangeven hoe
tevreden u bent met uw familieleven?
“The reference to family is problematic because in Dutch there are two words for referring to family members: Gezin (nuclear family) and familie (extended family)… So, Dutch version refers to more
people than the Spanish version”
Study 3: Cognitive Interviews
• General and follow-up probes were included in the protocol for all the items assessed.
• Interview protocols were developed taking expert appraisal evidence
Materials
• Retrospective design: first participants responded to all the scales, and then the general and follow-up probes were applied.
Procedure
• “Pyramid model”: multi-stage approach (Miller et al., 2014)
• Q -Notes: Data Collection and Analysis software for cognitive interviews developed by National Center for Health Statistics (https://wwwn.cdc.gov/qnotes/Login.aspx)
Analyses
Study 3: Cognitive Interviews, InvolvingMonolingual Respondents
• Check of interpretation of questions, key concepts
Integration of Results
• 1. Some participants more inclined to useextreme categories and verbalizations
extremity as personality characteristic
Integration of results: 2. Problems with specific topics
Items Item Stem
ESS_A_3 How satisfied are you with your present standard of living?
EQLS_1 How satisfied you are with your education?
WVS_3 How important it is in your life: Leisure time
WVS_5 How important it is in your life: Work
EconomicIssues
Economic and social aspects
Manners and educational
level
Educationallevel
Hobbies and activitiesoutside
Householdactivities
NecessityFinancialresource
Overall Conclusions• Increasing interest in mixed methods • Beyond the stage of “believers”• There seem to be two main reasons for their less than
enthusiastic reception in the social and behavioral sciences: – Paradigm clash between qualitative and quantitative studies – Mixed methods are not widely known among editorial boards
and reviewers• No reason for reluctance in adopting methods
– Most cross-cultural survey research includes both qual and quan stages (e.g., cognitive pretesting usually qual, survey usually quan)
• We need to move from “if “ to “how”.
Taxonomy of Bias
Type Source
Construct bias Theoretical construct
Method bias Measurement aspects (e.g., sample, test, administration)
Item bias Specific item aspects (e.g., poor translation)
– Definition of happiness in individualistic and collectivistic countries?
• Example: Uchida, Norasakkunkit and Kitayama (2004):
Types and Sources of Method Bias
TTyyppee SSoouurrccee
SSaammppllee bbiiaass CCoonnffoouunnddiinngg ssaammppllee ddiiffffeerreenncceess ((ee..gg..,,
eedduuccaattiioonn))
IInnssttrruummeenntt bbiiaass TTeesstt cchhaarraacctteerriissttiiccss ((ee..gg..,, ssccoorriinngg ooff
ooppeenn eenndd rreessppoonnsseess,, rreessppoonnssee sseettss))
AAddmmiinniissttrraattiioonn
bbiiaass PPrroocceedduurraall aassppeeccttss ((ee..gg..,, iinntteerrvviieewweerr
eeffffeeccttss,, llaacckk ooff ssttaannddaarrddiizzaattiioonn ooff
aaddmmiinniissttrraattiioonn))
Method bias tends to have a global influence on cross-
cultural score differences (e.g., increment due to social
desirability)
Item Bias(also known as differential item functioning, DIF)
• Informal description Differences in psychological meaning of stimuli, due to anomalies at item level
• More formal definition:An item of a scale (e.g., measuring anxiety) is said to be biased if persons with the same trait anxiety, but coming from different cultures, are not equally likely to endorse the item.
Types of (un)biased items
(a) Unbiased item
0
1
2
3
4
5
Very low Low Medium High Very high
Score level
Mean
sco
re
Culture A Culture B
(b) Item with uniform bias
0
1
2
3
4
5
Very low Low Medium High Very high
Score level
Mean
sco
re
Culture A Culture B
(c) Item with non-uniform bias
0
1
2
3
4
5
Very low Low Medium High Very high
Score level
Mean
sco
re
Culture A Culture B
(d) Item with both uniform and non-
uniform bias
0
1
2
3
4
5
Very low Low Medium High Very high
Score level
Me
an
sc
ore
Culture A Culture B
Analysis of Variance and Item Bias
• Item behavior examined per item
• We do not test for cultural differences, but we test whether scores are identical for persons from different groups with an equal proficiency
• Note: regression approach quite similar (illustrated later)
Taxonomy of Equivalence
• Refers to level of comparability
• More flexible than identity– Identity as ultimate type of equivalence
• Is related to bias:Highest level of equivalence obtained for bias-free measurement
• Hierarchical concept
Types of EquivalenceThree types:
–1. “Structural” or “functional equivalence”
–2. “Metric equivalence” or “measurement unit equivalence”
–3. “Scalar equivalence” or “full score equivalence”
(a) “Structural” or “Functional Equivalence”
• Measurement of the same traits
• Various statistical tools available, e.g.,– exploratory factor analysis (with target rotation)
– confirmatory factor analysis
– nomological networks (particularly relevant when items/questions are not identical across cultures)
• Qualitative equivalence can be firmly established
(b) “Metric Equivalence”, “Measurement Unit Equivalence”
• Difference in offset of scales of cultural groups, equal measurement units
• Individual differences have a different meaning within and across cultures:
no problems with offset in intra-cultural comparison, offset has to be added in cross-cultural comparison
• Statistical tool: structural equation modeling (confirmatory factor analysis)
(c) “Scalar Equivalence” or “Full Score Equivalence”
• Complete comparability of scores, both within and across cultures; seamless transfer of scores across cultures
• Frequently taken as the aim of cross-cultural research
Comparability and Equivalence Levels
Equivalence Comparability
Structural Underlying construct
Metric Same plus score metric
Scalar Same plus origin of scale
• Many statistical procedures available for testing structural equivalence
• Common approach:
– Apply dimensionality-reduction technique
– Compare underlying dimensions across cultures
– Similarity of underlying dimensions is criterion for similarity of meaning
Most commonly used
• Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA)
• CFA gradually replaces EFA
EFA
• Conduct factor analysis in each group
• Compare factor loadings
– Target rotations
– Compute factorial agreement
– If sufficiently high agreement: factors identical (underlying constructs invariant)
Regress Item (Observed) on Test Scores (Latent)
0
2
4
6
Low High
Ite
m S
core
No BiasGroup A
Group B
0
2
4
6
Low High
Ite
m S
core
Nonuniform Bias: SlopeGroup A
Group B
0
2
4
6
Low High
Ite
m S
core
Uniform Bias: InterceptGroup A
Group B
0
2
4
6
Low High
Ite
m S
core
CombinedGroup A
Group B
Main Differences
• CFA: structure defined at beforehand
• CFA: no rotation
• CFA: Fit statistics available
Difference with Exploratory Factor Analyses
• Starts from covariance matrices– Use metric information
• More parameters tested for cross-cultural similarity; examples– Factor loadings
– Factor correlations/covariances
– Error component of latent variables
– Error component of observed variables
• Enables the testing of a hierarchy of models
• Configural invariance: same constellation (pattern), both uniform and non-uniform bias could be present
• Measurement weights: regression weights in the measurement part of the model. In the case of a factor analysis model, these are the "factor loadings".– Metric invariance (no non-uniform bias)
• Measurement intercepts: intercepts in the measurement part of the model– Scalar invariance (no bias at all)
• (Structural residuals: variances and covariances of residual (error) variables in the structural part of the model.
• Measurement residuals: variances and covariances of residual (error) variables in the measurement part of the model)
• EFA less and less popular
• CFA works well when number of cultures and items is small
– Parceling
– Partial measurement invariance
• DIF frequently tested in educational and cognitive tests, much less so in attitude and personality measurement
Comparing Approaches
• Note that these approaches cannot identify construct and method bias
• Important issue in DIF
– In huge samples most or even all items may be biased
– Shift from significance to effect size (e.g., only medium and high flagged as biased)
• The CYRM-28 was administered to a purposive sample of 497 youth – concurrent users of multiple services (child welfare,
mental health, juvenile justice, special educational supports and community programs)
– from rural and urban communities of Atlantic Canada participating in the Pathways to Resilience study
– mean age = 16.85 years (SD = 1.87)
– Subsamples• 281 (56.5%) of the participants were male and 220 (44.3%)
participants self-identified as visible minorities.
Conclusion
• Tool kit of cross-cultural research has expanded and will continue to expand
• Cross-cultural studies as balancing between standardization (with the aim of comparability) and contextualization (with the aim of cultural appropriateness)– Many QUAN tools available
• Focus on standardized procedures; yet, more flexible procedures available
– QUAL procedures less standardized, yet important• Documentation of quality assurance in manuscript