FRAMEWORKIndigenous Evaluation

Telling Our Storyin Our Place and Time

Joan LaFranceRichard Nichols

Dedication

This book is dedicated to our ancestors from our various tribes whose values,

traditions and spirits have guided our work, as well as to those individuals at the

regional focus groups who through their generous giving of insights and

information made this book a document that may be used for generations.

Also, this book is dedicated to future generations of Indigenous evaluators,

educators and others who will strengthen the vision of Indigenous evaluation. We

hope the challenge represented in that vision will help share and celebrate the

stories of our Indigenous communities.

Mural by Fort Berthold Community CollegeStudents

AIHEC • 121 Oronoco Street • Alexandria, VA 22314 • 703.838.0400

Writ ten for the

American Indian Higher Educat ion Consor t ium

by

Joan LaFrance and Richard Nichols

Telling Our Story in Our Place and Time

Indigenous Evaluation Framework

FRAMEWORKAcknowledgements

The project staff would like to thank the many individuals who contributedto this initiative and who are continuing to contribute in meaningful ways. Thismaterial was developed by the American Indian Higher Education Consortium(AIHEC) under the leadership of Joan LaFrance (Turtle Mountain Chippewa),Richard Nichols (Santa Clara Pueblo), and Carrie Billy (Navajo). AIHEC’sIndigenous Science, Technology, Engineering and Mathematics (STEM) EvaluationFramework Initiative is supported with funding from the National ScienceFoundation.

Project staff are Joan LaFrance, President, Mekinak Consulting; RichardNichols, President, Colyer Nichols, Inc.; Carrie Billy, President & CEO, and KayHeatley, Creative Director and Web Developer, AIHEC.

Members of the AIHEC Advisory Board for the Indigenous STEM EvaluationFramework have helped guide this initiative since its inception. They continue toprovide advice, feedback, and input on the Framework and materials. Membersof the Advisory Board are: Rosemary Ackley Christensen (Ojibwe Sakaogon),University of Wisconsin-Green Bay; Gregory Cajete (Tewa), University of NewMexico; Carol Davis (Turtle Mountain Chippewa), North Dakota StateUniversity; Stafford Hood, University of Illinois at Urbana-Champaign; Eric Jolly(Cherokee), Science Museum of Minnesota; Iris PrettyPaint (Blackfeet),University of Montana-Missoula; Sharon Nelson-Barber (Rappahannock), WestEd;and Mike Timms, WestEd.

Special thanks go to the many tribal college presidents, faculty, and staff; K-12educators; tribal leaders; and most importantly, tribal elders and cultural expertswho attended our focus group forums, pilot testing sessions, and other meetingsand shared with us their stories, their advice, and their wisdom. Without them,this initiative would have been impossible.

We are grateful for the contributions that Dr. Karen E. Kirkhart, SyracuseUniversity, John Haas (Lakota), Red Stone Group, and Mary Jiron Belgarde(Ohkay Owingeh/Isleta Pueblo) have made to this effort. Karen’s expertise inevaluation theory, multicultural validity, and evaluation standards has beenvaluable to our effort to integrate and coordinate evaluation theories. Likewise,John’s strong background in Indian education at all levels, his expertise as aNative evaluator, and his abiding commitment to improving Indian educationhave brought new depth and understanding to this initiative.

We particularly want to acknowledge the tremendous contributions of KayHeatley, whose creativity, expertise, and keen attention to detail have

American Indian Higher Education Consortium© AIHEC 2009

Acknowledgements

FRAMEWORK

transformed our thoughts and words into the beautifully illustrated documentbefore you. We are grateful, also, to the many photographers and artists whocontributed their work, especially Jason Kim, Systemic Research, and C. J.Heatley III, for their photographs and to Amy Bower, a student at SyracuseUniversity, for her conceptual model illustration.

Finally, we thank Dr. Elmima Johnson, Program Director, National ScienceFoundation, for her critical role in making this initiative possible and forproviding ongoing national leadership in the field of program evaluation. We aretremendously grateful for her efforts to call attention to the influence of cultureand other contextual factors on evaluation design and outcomes.

This material is based upon work supported by the National ScienceFoundation under Grant No. 0438720. Any opinions, findings, and conclusions orrecommendations expressed in this material are those of the authors and donot necessarily reflect the views of the National Science Foundation.

Acknowledgements

AIHEC Indigenous Evaluation Framework © AIHEC 2009

With the exception of personal stories and dialogue and articles from Web sites and thosereprinted from copyrighted sources, the materials contained within this workbook andaccompanying presentations are the intellectual property of the American Indian Higher EducationConsortium (AIHEC). Any use or reproduction thereof must be with the permission of AIHEC,and must contain the following citation:

“Used with the permission of the American Indian Higher Education Consortium.” All other rights are reserved. © AIHEC 2009.

The following copyrighted articles are reprinted by permission:

LaFrance, J. L., Crazy Bull, C. “Researching Ourselves Back to Life: Taking Control of theResearch Agenda in Indian Country,” D. Mertens & P. Ginsberg, eds., Handbook of Social ScienceResearch Ethics, Thousand Oaks, CA. © 2009 by SAGE Publications, Inc.

LaFrance, J. L., “Culturally Competent Evaluation in Indian Country,” New Directions forEvaluation, no. 102, Summer 2004, p. 39–50. © 2004 by Jossey-Bass.

Nelson-Barber, S. & Trumbull, E., “Making Assessment Practices Valid for Native AmericanStudents,” Journal of American Indian Education, 46(3), p. 132–147. © 2007 by Journal ofAmerican Indian Education.

Robertson, P., Jorgenson, M. & Garrow, C. “Indigenizing Evaluation Research: How LakotaMethodologies Are Helping ‘Raise the Tipi’ in the Oglala Sioux Nation,” in the American IndianQuarterly, vol. 28, nos. 3 & 4, Summer/Fall 2004, University of Nebraska Press, Lincoln, NE,2004, p. 499, 506–507, 519. © University of Nebraska Press.

Schwandt, T. A., “The Centrality of Practice to Evaluation,” in American Journal of Evaluation, vol.26, no. 1, 2005, p. 95–105. © 2005 by American Evaluation Association.

Weiss, C. H., “Which Links in Which Theories Shall We Evaluate,” New Directions for Evaluation,no. 87, Fall 2000, p. 35–45. © 2000 by Wiley Periodicals, Inc.

FRAMEWORKContents

FRAMING EVALUATION IN OUR COMMUNITIES . . . . . . . . . . . . . . . . . . . . . . . 1 Building the Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Guiding Principles of the Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Training Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Model for the Indigenous Evaluation Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 How to Use These Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

WEAVING THE BASKET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 The Power of Metaphor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

CREATING KNOWLEDGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Testing Our Understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Indigenous Ways of Knowing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Implications for Framing Indigenous Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

CORE CULTURAL VALUES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 People of a Place . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Centrality of Community and Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Recognizing our Gifts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Sovereignty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Cultural Values and Evaluation Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Reframing Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

ENGAGING COMMUNITY IN EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Examples of Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

CREATING OUR STORY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 How to Create the Story . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Grounding the Story in Core Cultural Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Examining the Story . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Telling the Story . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

BUILDING THE SCAFFOLDING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Major Approaches to Designing Evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Gathering Information Over Time and Measuring Change . . . . . . . . . . . . . . . . . . . . 70 Looking to Our Own Ways of Knowing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

American Indian Higher Education Consortium© AIHEC 2009

11

19

31

43

47Creating Our Story

Engaging Communityin Evaluation

Creating Knowledge

Weaving the Basket

Core Cultural Values

1Framing Evaluation inOur Communities Contents

67Building the Scaffolding

FRAMEWORK

RESPONSIVE INFORMATION GATHERING . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Using Culturally Responsive Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Looking to Our Strengths—Appreciative Inquiry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Building Capacity—Empowerment Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Recognizing Our Gifts—Performance-based Student Assessment . . . . . . . . . . . . . 88 Linking Assessment and Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Setting a Proper Tone and Respecting Culture Protocols. . . . . . . . . . . . . . . . . . . . . . 94 Considerations in Conducting Surveys, Interviews, and Focus Groups . . . . . . . . 99

PLANNING, IMPLEMENTING AND CELEBRATING . . . . . . . . . . . . . . . . . . . . 107Reflecting Cultural Values and Beliefs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107Shaping the Evaluation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Interpreting the Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Working in Partnership with External Evaluators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Reflecting, Learning, Celebrating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Grounding the Evaluation in Core Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Putting It Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Continuing the Dialogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125

EXERCISES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127

RESOURCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

READINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Contents

AIHEC Indigenous Evaluation Framework © AIHEC 2009

81

125Bibliography

Responsive InformationGathering

107Planning, Implementingand Celebrating

141Resources

127Exercises

173Readings

FRAMEWORKAcknowledgements

American Indian Higher Education Consortium© AIHEC 2009