A Complex-Systems Orientation to Evaluation that Supports...
Transcript of A Complex-Systems Orientation to Evaluation that Supports...
![Page 1: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/1.jpg)
AComplex-SystemsOrientation
toEvaluationthatSupportsa
CultureofHealth
![Page 2: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/2.jpg)
Welcome!
Thisconceptpaperisaboutthesimplicityontheothersideofcomplexity.Itmaynotbeenoughtogiveyourlifefor,buthopefullyit’senoughtoinspireyoutothinkanddialogueabouthowyouframeyourapproachtoevaluationandtoconsiderwhatreframingmightbehelpful.
OnewayevaluatorsmaycontributemoreusefullytocreatingaCultureofHealthisbyunderstandingfundamentaldistinctionsbetweenthehierarchicalandnetworkedstructuresofsystemsandhowtheyareintertwinedincomplexsystems.Distinguishingbetweenthesetwostructureshelpsevaluatorsgroundtheirevaluationsinthefundamentalstructuresthatareofimportanceataparticulartimeandplaceastheysupportevaluationusersintheirintentiontotransformsystemstoservethepublicgood.
Everynowandthena[person’s]mindisstretchedbyanewideaorsensation,andnevershrinksbacktoitsformerdimensions.—OliverWendellHolmes,Sr.
Iwouldnotgiveafigforthesimplicitythissideofcomplexity,butIwouldgivemylifeforthesimplicityontheothersideofcomplexity.—OliverWendellHolmes,Sr.
![Page 3: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/3.jpg)
vThePaper’sOrientation 2
vThePaper’sPurpose3
vKeyPointsofthePaper4 vAComplex-Systems
OrientationtoEvaluation27
vEvaluationGroundedinHierarchicalStructures29
vEvaluationGroundedinNetworkedStructures30
vContrastofEvaluativeFocusinHierarchical&NetworkedStructures33
v Culture35vACultureofHealth36vACultureofHealthActionFramework37
v ExampleofanEvaluationSituationandApproach38
TableofContents
vWhatWeMeanandWhyItMatters 6
vEvaluation&EvaluativeThinking 7
vSystemic&SystematicInquiry 9
vKnowledge&KnowledgeManagement10
vSystems 11vComplexSystems13vSystemsChange 21vSystemConditions22v ImplicitBias23vWhyItMatters25
01 02 03INTRODUCTION
Pages 34–41
Pages 26–33Pages 5–25
Pages 1–4 WhatWeMeanandWhyItMatters
AComplex-SystemsOrientationtoEvaluation
AComplex-SystemsOrientationtoEvaluationthatSupportsaCultureofHealth
RESOURCES
Page 42-43
v ImplicationsforEvaluationandEvaluators43
vReferences 45
vAcknowledgements46
Pages 44–46
CalltoAction
04
RW.17.rf.SystemsWhitePaper.2-14
![Page 4: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/4.jpg)
Introduction
v ThePaper’sOrientationv ThePaper’sPurposev KeyPointsofthePaper
|1
![Page 5: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/5.jpg)
ThePaper’sOrientation
Sinceevaluation’sinceptionasafieldinthe1960s,thefieldhasbeendominatedintheU.S.byan
implicitbiastowardhierarchicalstructuresthattendtobestable,predictable,andcontrollable.Such
controlleddynamicsarethebasisofhierarchiesandfactorymodelsoforganization.Inaboutthe
1980s,systemstheoristsrangingfrommathematiciansandphysiciststobiologists,sociologists,
psychologists,andecologistsbegantounderstandcomplexsystemsashavingnetworkedstructures
withemergentdynamicsthatinternallychange,formingflexible,complexnon-linearpatterns
(sometimescalled“self-organizing”dynamics).Thecapacityofcomputerswasbecomingsufficientto
modeltheseemergentpatternsinareasonableamountoftime.
Withtheexpandedcapacityofcomputersandsocialnetworks,evaluatorscanmorepowerfully
contributetounderstandinghowtoinfluencesocialconditionsforthepublicgoodbygroundingtheir
evaluationpracticeinthefundamentalsofcomplexsystems.Influencingsocialconditionsthrough
influencingcomplexsocialsystemsisattheheartofcreatingaCultureofHealth.
Introd
uctio
n
|4
![Page 6: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/6.jpg)
ThePaper’sPurposeThisconceptpaperprovidesawaytothinkaboutcomplexsystems.Thepaper
countersevaluation’sbiastowardassumingthatanevaluationisoccurring
withinhierarchicalstructureswiththeircontrolledandstablepatternsof
movement(“dynamics”).Thisdominantapproachpaysinsufficientattention
totheemergentdynamicsofnetworkedstructuresthatcharacterizemany
systemsthatalsocontributetoacultureofhealth.
Theevaluationfield’stendencytofocuslargelyonhierarchicalstructuresis
evidenced,forexample,inprogramlogicmodelsthatfocusondesired
measurableoutcomesandtheprogram’sefficiencywithintheconfinesofa
hierarchicallystructuredsystem.Suchevaluationstypicallypaylimited
attentiontootherresultsoftheinterventionthatarenotwithinthecontrolof
theprogram.
Acomplex-systemsorientationshiftsevaluators’attentiontomultiplepatterns
overtimebasedontheintertwinednatureofhierarchicalandnetworked
structureswithincomplexsystems.Webeginbylookingatcomplexsystems
andthenreturntoacultureofhealthandsuggestsomeimplicationsfor
evaluatorsandevaluationusers.
Thespecificsofhowtoapplysuchanorientationtoevaluationareinneedof
conversationanddevelopment.Thispaperisintendedtostimulate
conversationsratherthanprovidefinalanswers.
Purpose:v Toencourageevaluativeinquiryofcomplexsystems
v Togroundevaluativeinquiryofcomplexsystemsin:~ understandinghowhierarchicalandnetworkedstructurescreatecomplexsystems~ recognizingevaluation’simplicitbiastowardhierarchicalsystems
Introd
uctio
n
|3
![Page 7: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/7.jpg)
KeyPointsofthePaper
Acultureofhealthisacomplexsystem
Evaluationtosupportthecreationofacultureofhealthneedstobecongruentwiththe
fundamentalnatureofcomplexsystems
Dominantevaluationapproacheshaveanimplicitbiascountertoacomplexsystemsorientation
TheRobertWoodsJohnson
Foundation’sCultureofHealth
involvesfourresearch-basedaction
areas.Theyinteracttocreate
improvedpopulationhealth,well-
being,andequity.
Thefouractionareasare:
• makinghealthasharedvalue
• fosteringcross-sector
collaborationtoimprovewell-
being;
• creatinghealthier,moreequitable
communities;and
• strengtheningintegrationof
healthservices,andsystems.
Asevaluators,wecanassist
evaluationusersinseeing,
understanding,andinfluencingthe
developmentofacultureofhealth
byusingorientationsand
consequentmethodologiesthatare
congruentwiththenatureofthe
entangledstructuresanddynamics
ofcomplexsystems.
Thispaperilluminateswhyand
howtoleveragetwofundamentally
differentsystemstructures—
hierarchicalandnetworked—when
usingacomplex-systems
orientationinevaluation.
Today’sdominantevaluation
methodologicalorientationhas
animplicitbiastowardsystems
withhierarchicalstructures.This
dominantorientationfailsto
recognizetheimportantand
fundamentaldifferencesbetween
thehierarchicalandnetworked
structuresthatintertwinetoform
complexsystems.
Introd
uctio
n
|2
![Page 8: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/8.jpg)
WhatWeMeanby:v Evaluation&EvaluativeThinking
v Systematic&SystematicInquiryv Knowledge&Knowledge
Managementv Systems
v ComplexSystems
v SystemsChangev SystemConditionsv ImplicitBias
01andWhyitMatters|5
![Page 9: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/9.jpg)
WhatWeMeanandWhyItMattersWhatWeMeanby:v Evaluation&EvaluativeThinkingv Systemic&SystematicInquiryv Knowledge&Knowledge
Managementv Systemsv ComplexSystemsv SystemsChangev SystemConditionsv ImplicitBias
Oursocietyisdeeplyrootedinthinkingintermsof“stable
systems,”bywhichmostpeoplemeanhierarchicalstructures
withcontrolleddynamics.Tothinkaboutcomplexsystems,we
needtoaddnewtermsandconceptsandredefinesome
existingterms.
Thissectionexplainshowtothinkdifferentlyaboutconcepts
thatseemfamiliarbuthaveadifferentmeaninginthecontext
ofcomplexsystems.Itisespeciallyimportanttograspthe
differencesbetweenstructuresthattendtohavecontrolled,
stablepatternsofdynamics(hierarchicalstructures)andthose
thattendtohaveflexible,emergentdynamics(networked
structures). ??…[E]veryinstanceoflanguageusemakesitsownsmallcontributiontoreproducingand/ortransformingsocietyandculture,includingpowerrelations.(Fairclough&Wodak,1998).
01.W
hatW
eMeanandWhyItM
atters
|6
![Page 10: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/10.jpg)
01.W
hatW
eMeanandWhyItM
atters
??Evaluation&EvaluativeThinking
Evaluation issystematic
inquiry to determinemerit
(quality),worth (value),and
significance (importance)
aboutsomeentityorpurpose.
AccordingtotheEncyclopediaofEvaluation:
Evaluation isanappliedinquiryprocessforcollectingandsynthesizing
evidencethatculminatesinconclusionsaboutthestateofaffairs,value,
merit,worth,significance,orqualityofaprogram,product,person,policy,
proposal,orplan.
Conclusionsmadeinevaluationsencompassbothanempiricalaspect(that
somethingisthecase)andanormativeaspect(judgementaboutthevalue
ofsomething).Itisthevaluefeaturethatdistinguishesevaluationfrom
othertypesofinquiry,suchasbasicscienceresearch,clinicalepidemiology,
investigativejournalism,orpublicpolling.” (Mathison,2005)
Ofparticularimportanceisthelaststatementthatemphasizesthat“itisthe
valuefeaturethatdistinguishesevaluationfromothertypesofinquiry.”
Thefield ofevaluationas practicedintheU.S.andmostothercountriesis
groundedinanassumptionthathumanshaveequalrights.Alsoevaluationis
donewithinrealworldsettings,notlaboratorysettingsdisconnectedfrom
thecontextinwhichtheentityistofunction.
Thusthedefinitionofevaluationdoesnotendwithsystematicinquiry,but
usesthisinquirytodeterminemerit,worth,and/orsignificance.
|7
![Page 11: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/11.jpg)
01.W
hatW
eMeanandWhyItM
atters
EvaluativeThinkingExample
Evaluativethinkinginvolves
balancingmerit,worth,and
significance.
Forexample,assumeyouarelookingforanewcar.
Youfindanelectriccarthatisofveryhighquality(merit).Itisofhighvalue
(worth)toyoubecauseitalignswellwithyourcommitmentto
environmentalresponsibility.However,ifyourdaughtersuddenlybecomes
seriouslyillandyouneedtospendyourmoneyonhealthcareforher,the
cardropsinitsimportance(significance)becauseyourdaughterismore
importanttoyouthanacar.
Traditionally,evaluatorshavemadeevaluativedeterminationsbasedon
theirsystematicinquiryandpredeterminedcriteria.Asthecomplexityof
societyhasbecomemoreevident,evaluatorshavebeguntoengageafull
rangeofstakeholderswithmultipleperspectivesindeterminingmerit,
worth,andsignificance.Additionally,theyarelearningtohelppeople
balancemerit,worth,andsignificanceincomplexsystems.
??
|8
![Page 12: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/12.jpg)
01.W
hatW
eMeanandWhyItM
atters
Systemic&SystematicInquiry
• Engageinsystematicand
systematic inquiry.
• Considercontent,context,conditions, andconnectivity.
• Useinterdisciplinary knowledge.
Therearemanydifferentmethodologiesforengaginginsystematic
inquiryinwaysthatarebothethicalandofhighquality.Wearenot
addressingtherangeofmethodologiesinthispaper;ratherthe
focusisondoinganinquiryfromasystemicperspective.
Threekeypointsforaninquiryare:
• Theevaluatorengagesinsystemic aswellassystematicinquiry.
• Inthinkingsystemicallyandsystematically,theevaluatorlooks
broadlyanddeeplyintothecontent,context,conditions,and
connectivityofthesystemsinvolved.
• Thetheoryofchangeguidingasystems-orientedevaluation
typicallyneedstorelyonastronginterdisciplinarybodyof
knowledge.
??
|9
![Page 13: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/13.jpg)
01.W
hatW
eMeanandWhyItM
atters
Knowledge&KnowledgeManagement
Knowledgemanagement (KM)isthe
processofcreating,sharing,using
andmanagingtheknowledgeand
informationofanorganization.(Girard&Girard,2015)
Complexsystemsinvolvemulti-disciplinaryknowledge.Accessing
andsynthesizingabroadrangeofknowledgetoassistevaluation
usersindevelopinganappropriatetheoryofchangebecomesan
increasinglyimportantroleforevaluatorswhenworkingwith
complexsystems.Evaluatorsandevaluationuserstypicallyare
familiarwithamorelimitedrangeofdisciplinaryknowledgethan
whatisrelevantwhenworkingincomplexsystems.
Thispaperdoesnotaddressknowledgemanagementbutitisa
criticallyimportantaspectofengaginginsystems-oriented
evaluation.Fortunately,moretechnologically-supportedmeansfor
knowledgesynthesisandmanagementareemergingbutmuchmore
developmentisneededforefficientmanagementoftheknowledge
relevanttoseeing,understanding,andinfluencingcomplexsystems.
??
|10
![Page 14: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/14.jpg)
01.W
hatW
eMeanandWhyItM
atters
SystemsAsystemisacollectionofentitiesthatare“seen”bysomeoneasinteractingtogethertodosomething.(OpenUniversity,2011)
Severalaspectsofthisdefinitionareimportant:
• Itisacollectionofentities;multipleelementsareinvolved.
• Differentpeoplemaybeseeingadifferentcollectionofentitiesinteractingtogether,thus,seeingadifferentsystem.
• Thequotationmarksaround“seen”remindusthatthesesystemsmaybeseeninourmind’seyeratherthannecessarilybeingseenphysically.Systemsaremostoftenconceptualmodelsratherthanphysicalentities.
• Theentitiesareinteracting.Therearebothconnectionsanddynamicsinthoseinteractions.Noticemovement andenergy.
• Theinteractingentitiesaredoingsomething.Theycreatearesultthatisdifferentthantheresultsofthesumoftheparts.
• Inevaluation,desiredoutcomesaretypicallythoughtofastheresultthatasystemproduces.However,thisistoonarrowofadefinitionofwhatasystemdoes.Asystemislikelytodomorethanproducetheoutcomessomeoneishopingtosee.Forexample,theresultcanbechangesinpatternswithinthesystems,notonlyspecificproducts.
??
System: Asystemisa
collectionofentitiesthatare
“seen”bysomeoneas
interacting togethertodo
something. (OpenUniversity,UK)
Systemsthinking:Thinkingin
termsofsystemsratherthan
theparts;thinkingabout
movement,dynamics,and
patterns acrosstimeand
locations;thinkinginsystems
|11
![Page 15: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/15.jpg)
01.W
hatW
eMeanandWhyItM
atters
SystemsThinking
System: Asystemisacollection
ofentitiesthatare“seen”by
someoneasinteracting
togethertodosomething.(OpenUniversity,UK)
Systemsthinking:Thinkingin
termsofsystemsratherthan
theparts;thinkingabout
structures,dynamics,and
patterns acrosstimeand
locations;thinkinginsystems
Regardinghumanandotherlivingsystems,systemsthinkinginvolvesthinkingaboutstructures,dynamics,andpatternsacrosstimeandlocations.Sometimesitisreferredtoas“thinkinginsystems”.
Lookatthepicturesonthispage.Whatsystemsdoyou“see”?(Thepicturesmayshowonlyapartofasystem.)Forexample,thepictureoffoodinamarketmayleadyoutothinkofasystemof:
• transportationfromfarmtomarket
• theflowofmoneyrelatedtoagriculture
• theinteractionofpeopleatalocalfarmers’market
Whatsystemsdoyouseeinthispictureandtheotherpicturesbelow?
??
|12
![Page 16: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/16.jpg)
01.W
hatW
eMeanandWhyItM
atters
ComplexSystems
Complexsystems:Complexsystems
aremassivelyentangledsystems that
includebothhierarchicaland
networkedstructures.
Complexsocialsystemsarecomposedofmassivelyentangledstructures.Forexample,theyinvolveinterconnected families; hierarchical,bureaucraticorganizations; andnetworksofsmallformalandinformalgroups. Notonlyaretheyentangled,theyarealsoinmotionwithcomplexpatterns.
Fromatheoreticalperspective,complexsystemscanbethoughtofasanentangledmixofhierarchicalandnetworkedstructures(seefollowingpages).Hierarchicalandnetworkedstructuresareassociatedwithdifferencesintheirdynamics,i.e.,theirpatternsofmovementovertime;theirpredictability;andtheirknowability.
Complexsystemshavebeenstudiedextensivelyindifferentdisciplineswithanaccumulationofknowledgeaboutthedifferencesinhowtheyfunction.TheReferencelistincludesafewsourcesthatwehavefoundhelpfulwhenapplyingsystemsconceptstoevaluation.
Understandingacomplexsystembearssimilaritiestocomprehendingthisopticalillusion,whichcontainsimagesofbothayoungwomanandanoldwoman.Bothimagesarewithinthepicturebutifoneisoverlydominantinourview,wecan’tseetheother.
Wewanttogettothepointwherewecanmoveourfocusbackandforthbetweenhierarchicalandnetworkedstructurestoseethecomplexsystemasawhole.
??
|13
![Page 17: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/17.jpg)
C
ThinkinginComplexSystems
Lookagainatthepicturesyoulookedatonanearlierpage.Thinkabouttheentanglementsofmultiplesystemsthat
youcanseeinthepictures.
??
01.W
hatW
eMeanandWhyItM
atters
|14
![Page 18: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/18.jpg)
HierarchicalStructuresAmajoradvancewiththeIndustrialRevolutionwasthedevelopmentoffactoriesandassemblylines.Thesewereoftenaccompaniedbyhierarchicalorganizationalstructurestomanagepeople.Theunderlyingassumptionisthatplanningisusefulbecausethereisapredictablerelationshipbetweenactivitiesandresults.Suchstructuresaredesignedtogiveconsiderablecontroltoanorganizationalleaderormanager.Thesestructuresarefurthercontrolledbyrulesandpolicies;theytendtohavepartsthatcanbereplaced.
Whenaskedtoidentifyasystem, wehavefoundthatpeopleusuallynamesometypeofhierarchicalstructure(e.g.,theeducation“system”).Hierarchicalstructuresarepervasiveingovernment,business,andthenonprofitorganizationworlds.Theyhavemanysubsystemswithinthemthataresimilarlystructured(e.g.,themanagementsystem, theaccountingsystem).
Whenconditionsarestableandpredictableandthefunctionofthestructureistoproducewell-definedproductsoroutcomes,thistypeofstructurecanworkwell.Forexample,wewantcontrolandstabilityin anorganization’s payrolldepartmenttoassuretimelyandaccuratepaychecks.
Theassumptionsofstabilityandpredictabilityserveasthebasisforthinkingthatoneprogramcanbereplacedbyanotherwithinasocialsystemandthat,ifeachpartisworkingwell,theoverallsystemwillalsobeworkingwell.
Hierarchicalstructurestendtocreate controlleddynamicswithstable,predictablepatterns.Theyhaveconstraintswithinoraroundthemthatcreatetheirpredictability.
??
Manytoolsandmethodsexistforunderstandingandinfluencinghierarchicalstructures.Thesetoolsandmethods,basedonlinearmodelsofcauseandeffect,includestrategicplanning,settingspecificoutcomes,andusingresearchdesignssuchasrandomizedcontrolledtrials(RCT).
01.W
hatW
eMeanandWhyItM
atters
|15
![Page 19: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/19.jpg)
01.W
hatW
eMeanandWhyItM
atters
HierarchicalStructureswithinComplexSystems
|16
v FundamentalStructure:hierarchical,mechanistic
v Dynamic:controlled,stable,predictable,“regular,”causeandeffect
v Patterns:linear,repetitive,interchangeableparts
v Examples:bureaucracies,assemblylines
![Page 20: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/20.jpg)
01.W
hatW
eMeanandWhyItM
atters
Thecomplexityscienceshavetheirbeginningscenturiesago,buttheideasstartedtoreceiveseriousattentioninthescientificcommunityinaboutthe1980s(Capra,2014).Theunderstandingandapplicationsoftheconceptsofnetworkedstructureswithemergent(or“self-organizing”)dynamicsarestillintheirearlystagesofdevelopment.Theconceptoriginatedinthephysical,biological,andecologicalsciences;ithasmathematicalunderpinnings.
Inanetworkedstructure,weseetheemergenceofnew,unexpectedprocessesandpatternsofmovement.Thesehigher-levelemergentdynamicsunexpectedlyresultfromtheactionsofamultiplicityofsmalloccurrenceswithinthesystem.Thesmalloccurrenceswerenotplannedtocreatetheneworder.Theemergenceofthenewisnotcontrolledbyasingleentity,butresultsfromsemi-independentinteractionsofmanyagents.Theyareadaptingtoeachotherandtheenvironmentasawhole.Theseinteractionscanformpatternsovertimeandlocations.Thosepatternsmaynotbevisibletotheagentsthemselvesandthetimingofthepatternsisoftenunpredictable.
Networkedstructuresmaybedisturbedbyexternalinfluencesbuttheytendtobewithoutexternalcontrol.NetworkedstructuressuchasthehumannetworksontheInternetdisplayemergentdynamics:theyareflexibleandadaptive;noleaderisincontrol.
Networkstructuresfunctionfarfromequilibrium.Theycancreateemergentdynamics withnew,unexpected,self-organizingpatterns,properties,and/orprocesses.
??NetworkedStructures
|17
![Page 21: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/21.jpg)
01.W
hatW
eMeanandWhyItM
atters
Networkedstructures also:
• aresensitivetoinitialconditions.Incomplexsystems,verysmalldifferencesininitialconditionscanhaveadisproportionatelylargeimpactonfutureevents.
• areinfluencedbyiterativefeedback.Theagentsare“learning”fromoneanotherandthecontextastheygetsignalsfromotheragentsandmodifytheirbehavior.
• co-evolve.Co-evolutionreferstotheinterdependentevolutionoftwoormorestructureswithinalargerecologicalsystem.Cooperation,competition,andinterdependenceinrelationtothesamelimitedresourcescreatefeedbackamongtheentitiesthroughthestructuresinacomplexsystem.
• haveactiveboundaries.Asystem’sboundariesarecreatedbythestructureitself.Dynamicstructuressuchasanetwork’shaveactiveboundariesastheyinteractwiththeirenvironments.
• mayoperatefarfromequilibrium.Theycanhaveawiderangeofemergentdynamicswithpatternsofmovementthatmaybeverycomplex.
• canshiftintonewstructures.Theshiftintoaneworganizationalstructurecanarisefromfluctuationsamplifiedbycertaintypesoffeedbackloops.
Networkedstructurescancreateemergentdynamicswithnew,unexpected,self-organizingpatterns,properties,and/orprocesses.
??CharacteristicsofNetworkedStructures
|18
![Page 22: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/22.jpg)
01.W
hatW
eMeanandWhyItM
atters
v Fundamentalstructure:networksofnodesandconnections
v Dynamic:flexible,creative,emergent
v Pattern:co-evolving,semi-independentagentsinfluencingnearbyagents
v Examples:networks,socialmovements,movementsofmovements,swarmingofbirds,schoolsoffish
NetworkedStructureswithinComplexSystems
|19
![Page 23: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/23.jpg)
01.W
hatW
eMeanandWhyItM
atters
Seldomwouldanidentifiedsystemhavethepurefeaturesofahierarchicalornetworkedstructureasdescribedabove.Rathersocialsystemsareamixofstructuresalthoughtheymaytendinonedirectionortheotherintheiroverallcharacter.Hierarchicalandnetworkedstructuresarelensesthroughwhichweviewcomplexsystems.Eachlensrevealsdifferentaspectsofthecomplexityandgivesusinsightintoitsfunctioning.
Complexlivingsystemsgenerallyhavesomeaspectsthatcanbemodeledwellbythinkingofthemashierarchicaland/ornetworkedstructures.
Organizationalpartnershipsandcommunitiesaregoodillustrationsofcomplexsystemswherehierarchicalandnetworkedstructuresbothcanbeidentifiedfairlyreadily.
ComplexSystems:EntangledHierarchicalandNetworkedStructures
|20
![Page 24: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/24.jpg)
01.W
hatW
eMeanandWhyItM
atters
SystemsChangeWe oftenhearpeopletalkaboutwantingtobringaboutchangeinsystemsthroughtheirinterventions.Whenengaginginevaluationrelatedtochangingsystems,weneedtobeveryawarethatthesystemsnaturallyarealreadyinmotionandchanging.Weneedtobeawarethattheinterventionisnotenteringastaticsystem.Interventionsarelikelytochangethepattern ofchangeratherthansimplycreatechange.
Humansystemsarecontinuallychanginginandofthemselvesandthroughtheirconnectionswithothersystems.Theyarelivingsystemsthatarecontinuallychangingwhetherornotthereisanintervention.Thisisespeciallytrueofsystemswithnetworkedstructures.
Interventionssuchasprograms,policies,people,andinitiativesarechangingthepatternsofchange thatarealreadypresentinthesystems.
Itisalsoimportanttorecognizethataninterventionmaybeaffectingmoresystemsthanthosefocusedonbytheintervenor.Thecomplexityoftheinterconnectedsystems,theunpredictabilityofpatternsofchangeinthesesystems,andthesustainabilityofchangesallneedtobetakenintoaccountwhenevaluatinganinterventionincomplexlivingsystems.
Complexlivingsystemsarecontinuallychangingbasedontheirinternaldynamicsaswellasoutsideinfluences.
The influenceofaninterventiondependsontheexistingstructuresanddynamicsofthesystemsbeingaffected.
Programs,policies,peopleandotherinterventionsinfluencethepatternsofchangethatarealreadypresentinthesystems.
??
|21
![Page 25: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/25.jpg)
01.W
hatW
eMeanandWhyItM
atters
SystemConditionsSystemstheoristshaveidentifiedcommonfeaturesorconditionsofsystems.DrawingontheworkofWilliamsandHummelbrunner(2009)andEoyang(1997),weuseInterconnections,Boundaries,andPerspectivesasabasicwaytoframetheconditionsthatdeterminethenatureofhumansystemsandtheirspecialrelevancetoevaluation.Asweconductacomplex-systemsevaluation,weneedtopayspecialattentionto:
• Understandinginterconnections(interrelationships)
• Maintaininganawarenessofboundaries
• Recognizingmultipleperspectives
Interconnections,boundaries,andperspectivesarenotisolatedfromoneanother.Asyouchangeone,youaffectanother.Thinkofthethreeconditionsastheanglesofatriangle.Asyouchangeoneangleyouaffecttheothertwo.
Theseconditionsarerelevantatanyphaseofevaluation.Theycanalsobeusedtounderstandcomplexsystemswiththeirentangledcombinationsofhierarchicalandnetworkedstructures.
Understandingtheinterplayamongtheseconditionsispartofthechallengeofthinkinginsystems.Weneedtoapproachitwiththe“forestandthetrees”mindset.Wehavetobeabletoshiftinandoutoffocusbetweenseeingtheforestandseeingtheindividualtrees.
Thesesystemconditionscanbeusedtohelpyouidentifythestructuresandtheirdynamicsinthesystemsthatyouaregoingtobeevaluatingandchooseevaluationdesignsthatarecongruentwiththesestructures.
Humansystemshavethreegeneralcommonsystemconditions:
Interconnections
Boundaries
Perspectives
I
B P
??
|22
![Page 26: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/26.jpg)
01.W
hatW
eMeanandWhyItM
atters
ImplicitBiasAnimplicitbiasisdifferentfromasuppressedthought thatsomeonemightconceal.Implicitbiasesare“biasesthatareactivatedinvoluntarilyandbeyondourawarenessorintentionalcontrol”(Staats,Capatosto,&Contractor,2015,p.5).Implicitbiasesoftenshowupinourlanguage.
“Unwantedimplicitbiasescanbemitigated….Bytakingthetimetounderstandyourpersonalbiases,youcanbegintomitigatetheireffects....Implicitbiaseshavehugeimplications;thusitisimportanttoidentifyyourownbiasesandthenactivelyengageindebiasingtechniquestoaddressthem.” (Staats,Capatosto,&Contractor,2015,p.5).
Thereisgrowingattentiontostructuralracismastheresultofimplicitbiases.Structuralracismreferstothenormalizationandlegitimizationofinstitutionalpoliciesandpracticesthatroutinelyadvantagewhiteswhileproducingcumulativeandchronicadverseoutcomesforpeopleofcolor.Structuralracismbecomesasystemofitsownduetotheseinterconnectedorganizedsystems.
Theterm“implicitbias”canalsobeappliedtowhatishappeninginevaluationeventhoughtheimpactisverydifferentthanthatresultingfromstructuralracism.
Whenitcomestoevaluation,acriticallyimportantimplicitbiasistowardamodelofsystemswithhierarchicalstructuresandcontrolleddynamics.Thispervasivebiashasmanyimplicationsforevaluation.
“Implicitbiasisconcernedwithunconsciouscognitionthatinfluencesunder-standing,actions,anddecisions”. Thisisdifferentfromindividualsconsciouslysuppressingornotsharingtheirbeliefsbecauseothersmayviewtheirbeliefsassociallyundesirable.
??
|23
![Page 27: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/27.jpg)
01.W
hatW
eMeanandWhyItM
atters
Evaluation’sImplicitBias
Evaluation’simplicitbiastowardsahierarchicalormechanisticorientationisalignedwiththewayofthinkingthathasdominatedoursocietysincetheIndustrialRevolutionwhensystemswerethoughtofasstable.Considerthefactorywhereanassemblylineofworkersfitspartstogethertocreateafinalproduct.Themechanisticmodelisalsoreinforcedbybusiness,whichseeksahigh-qualityoutcomethroughefficiencyandeffectiveness.Thehigh-qualityproduct/outcomesbecomestheaimofthework.
There’sawholeschoolofdesignthatcomesoutofthisbusinessmodel.Inthiskindofdesign,peoplegooutintothecommunitytofindoutthepeople’sneeds.Theyaregoodatinvestigatingpeople’slivesinfriendly,non-obtrusiveways.Theygettoknowaculturewiththepurposeofcomingupwithideasaboutproblemsthatcouldbesolvedwithaproductthattheycandevelop.Theydonotidentifyissuesthatcanbesolvedbypeoplesimplyworkingorlivingtogetherdifferentlywithoutmoreproducts.Instead,it’saboutfindingsomethingthatcouldbethebasisofaproductthebusinesscanselltothepeople.Thatmodelofdesignismovingintoevaluationwithoutrecognizingthatthedesignhasanimplicitbiastowardtheproductionofsomething.
Eventhoughevaluationhasapproachessuchasdevelopmentalevaluationandparticipatoryevaluation,evaluators’implicitbiastowardlinearityleadstoviewingnetworkedstructureswithincomplexsystemsasanadd-ontothehierarchicalstructureratherthanasasystemstructureinitsownright.
Evaluationtendstohaveanimplicitbias,unrecognizedbyevaluatorsandevaluationusers,towardassumingsystemschangeinalinearfashion.
Takingacomplexsystemsorientationwillbeoflittleconsequenceunlessweconfrontthepowerandsignificanceoftheimplicitbiaswehavetowardalinearsystemsmodel.
??
|24
![Page 28: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/28.jpg)
01.W
hatW
eMeanandWhyItM
atters
WhyitMattersInsummary,becauseofthetendencyinthefieldofevaluationtofocus
onhierarchicalstructureswithcontrolleddynamicsand/ortobe
unawareofcomplexitysciences,thissectionhasprovidedasetof
vocabularyandconceptsthatareimportantforevaluatorsto
understand.Withoutsuchunderstanding,evaluatorstendtouse
methodologiesthataregroundedinmechanistic(linearmathematical)
modelsthatarecongruentwithhierarchicalstructuresratherthan
developingmodelswherehierarchicalstructuresareentwinedwith
dynamicnetworkstructures.
??
|25
![Page 29: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/29.jpg)
AComplex-SystemsOrientationtoEvaluation
v AComplex-SystemsOrientationtoEvaluation
v EvaluationGroundedinHierarchically-StructuredSystems
v EvaluationGroundedinNetwork-StructuredSystems
v ContrastofEvaluativeFocusinHierarchicalandNetworkedStructures 02
|26
![Page 30: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/30.jpg)
AComplex-SystemsOrientationtoEvaluationAsystemisacollectionofentitiesthatare‘seen’bysomeoneasinteractingtogethertodosomething.Complexsystemsaremassivelyentangledsystems thatincludebothhierarchicalandnetworkedstructures.
Thissectionofthepapershiftsthefocustotakingacomplex-
systemsorientationtoevaluation.Thatis,asevaluators,weaccept
thatweareworkingwithawebofentangledstructureswithmany
gradationsofdynamicsthatrangefromcontrolledandstableto
emergentandflexible.Withinthisdynamicweb,weworkwith
evaluationuserstoidentifythesystemswithinthiswebthatwe
aregoingtoinvestigatethroughtheevaluation.
Recallourearlierdefinitionofasystem:“Asystemisacollectionof
entitiesthatare‘seen’bysomeoneasinteractingtogethertodo
something.”Thus,ourinitialstepinconductingacomplex-systems
orientedevaluationistointeractivelydevelopapicture(literallyor
figuratively)ofthemultiplicityofhierarchicalandnetworked
structuresasseenbyanarrayofstakeholdersrelevanttothe
work.Fromthere,wehighlightandputourfocusonthose
deemedtobethemostimportantstartingpointforthe
evaluation.Wewillnotbediscussingthedetailsofhowtoengage
insuchdeterminationsinthispaper.
|27
02.ACom
plex-SystemsO
rientationtoEvaluation
![Page 31: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/31.jpg)
02.ACom
plex-SystemsO
rientationtoEvaluation
Complex-SystemsOrientedEvaluationDesign
Theevaluatorlooksforthepresenceofthetwobasicsystemstructures—hierarchies(controlled
dynamics)andnetworks(emergentdynamics)—inthesystemsemphasizedintheevaluation.The
evaluatorconsiderstheimplicationsofthestructures andtheirdynamicsfortheevaluation.
(Althoughtherearemanyotheraspectsofcomplexsystemsthatwecouldaddress,thisisourfocus
inthispaper.)
Wealigntheevaluationdesignwiththefundamentalstructures.Asnotedintheabovediscussion,
anygivenevaluationislikelytohaveamixofmethodsandorientation.However,atthispointwe
wanttoclarifythesystemstructuresinwhichtheevaluationisfundamentally“grounded.”
• Istheevaluationgroundedprimarilyinaffectingthehierarchicalstructureswithinwhichthe
interventionislocated?
AND/OR
• Istheevaluationgroundedininfluencingthedynamicnetworksofnodesandconnections?
|28
![Page 32: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/32.jpg)
EvaluationGroundedinHierarchicalStructures
Whenthefundamentalstructureisahierarchicalone,theevaluationemphasisismostlikelygoingtobeonfeaturesofhierarchicalstructuressuchasprograms,policies,andpracticeswithintherelevantsystems.Althoughadaptationofaninterventionmaybeamajorareaofconcernandthesystematicinquiryprocessitselfmayconsumethemajoramountoftime,theevaluative aspectislikelytoemphasize:
v Measuringoutcomesv Linkingoutcomesandactivitiesv Applyingpredeterminedevaluationcriteriav Buildingevaluationcapacityaroundtechnicalaspectsofdeterminingoutcomesandevaluation
criteria
|29
02.ACom
plex-SystemsO
rientationtoEvaluation
![Page 33: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/33.jpg)
EvaluationGroundedinNetworkedStructures
Whenthefundamentalstructureisanetwork,theevaluationfocusesonthefundamentalfeaturesofnetworks—nodesandconnections.Thesenodesandconnectionsmaywellincludehierarchicalfeaturesbuttheemphasisisonthenetworkedaspects.Thustheevaluative aspectislikelytoemphasize:
v Mappingpatternsofmovementandchangerelatedtovaluesv Facilitatingconversationsaboutmerit,worth,andsignificancev Buildingthecapacityofnetworkstosustaincontextuallyappropriateattentiontomerit,worth,and
significanceamongmultiplestakeholders
|30
02.ACom
plex-SystemsO
rientationtoEvaluation
![Page 34: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/34.jpg)
Neighborhood
Faith-BasedOrganization
SocialMovement
GovernmentalAgency
NationalCommunityLeadershipNetwork
ExampleofEvaluationGroundedinNetworked-StructuresAssumethattheinterventionofinterestisanationalnetworktosupportacultureofhealth.Iftheevaluationistobegroundedinnetworkedstructures,theevaluationmayfocusonfollowingvariousmembersofthenationalnetworkastheyworkthroughnetworksintheircommunitylocation.Thediagrambelowillustratesthenetworksofonememberofanationalnetwork.Thenetworkontheleftisaportionofthelocalcommunity;onememberofthenetworkisbeingfollowedintohercommunity.Therestofthediagramshowsthatpersoninherlocalnetworks.
|31
02.ACom
plex-SystemsO
rientationtoEvaluation
![Page 35: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/35.jpg)
Whentheevaluatorofthenationalnetworkconductsanevaluativefacilitationandinquiryinthisnetworkmember’scommunity,shemightfocusherevaluationonequity,leadership,andpowerdifferentialsinthevariousstructuresinthecommunityasillustratedbelow.
Neighborhood
Faith-BasedOrganization
SocialMovement
GovernmentalAgency
NationalCommunityLeadershipNetwork
Inquiryfocusesonunderstandingdifferencesinthe
powerofhierarchicaland
networkedstructures.
ExampleofEvaluationGroundedinNetworked-Structures
|32
02.ACom
plex-SystemsO
rientationtoEvaluation
![Page 36: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/36.jpg)
v Mappatternsofmovementandchangerelatedtovalues
v Facilitateconversationsaboutmerit,worth,andsignificance
v Buildcapacityofnetworkstosustaincontextuallyappropriateattentiontomerit,worth,andsignificance
HierarchicalStructures NetworkedStructures
v Measureoutcomesv Linkoutcomesandactivitiesv Applyevaluationcriteriav Buildevaluationcapacityaboutaspectsofsystematicinquiry
ContrastofEvaluativeFocusinHierarchicalandNetworkedStructures
|33
02.ACom
plex-SystemsO
rientationtoEvaluation
![Page 37: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/37.jpg)
AComplex-SystemsOrientationtoEvaluationthatSupportsa
CultureofHealthv Culturev ACultureofHealthv ACultureofHealthActionFrameworkv ExampleofanEvaluationSituationandApproach
§ ExampleEvaluationBackground§ ExampleEvaluationApproach§ ExampleEvaluationQuestions:Interconnections,Boundaries,
Perspectives 03|34
![Page 38: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/38.jpg)
CultureCultureisdefinedinanumberofways.Herearethree:
• Cultureisasharedsetofvalues,customs,andbeliefscommontoaparticulargrouporsocietythatproducespatternsofbehaviorsandexperiences.
• Cultureasmultiplediscourses,occasionallycomingtogetherinlargesystemicconfiguration,butmoreoftencoexistingwithindynamicfieldsofinteractionandconflict.(Dirks,Eley,&Ortner,1994,p.4)
• Incontrastwiththeclassicview,whichpositscultureasaself-containedwholemadeupofcoherentpatterns,culturecanarguablybeconceivedasamoreporousarrayofintersectionswheredistinctprocessescrisscrossfromwithinandbeyonditsborders.(Rosaldo,1993,p.20)
Allsaid,cultureisacomplexsystemwith massivelyentangledentitiesandhierarchicalandnetworkedstructures.Itsboundariesarepermeableanditisdispersedthroughsociety.Someculturalaspectsareimposed byoutsidegroupsorindividuals whileothersarisefromwithinthegroup.
Cultureisoftenthoughtofintermsofracialorethnicgroups.Cultureemergesinmanygroups.Abusiness,neighborhood,community,sportsteam,family,ornetworkdevelopsitscultureovertime.Thesharedvalues,customs,andbeliefsmayormaynotbeexplicitandamemberofthegroupmaybeconsciousorunconsciousoftheculture.
Culture isacomplexsystemwithmassivelyentangledentitiesandhierarchicalandnetworkedstructures.
|35
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
![Page 39: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/39.jpg)
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
ACultureofHealthTheaimistofocusonbuildingacultureofhealthratherthan
focusingonlyonhealthcareservices.
Toaccomplishthis,changesmustbemade
inbeliefsandassumptions,andthe
structuresofsocialsystems.
“Recognizethathealthandwell-beingcanbegreatlyinfluencedbycomplexsocialfactors:wherewelive,howwework,thesoundnessandsafetyofoursurroundings,andthestrengthandresilienceofourfamiliesandcommunities”*
“ComprehensiveCultureofHealth…enablesallinourdiversesocietytoleadhealthierlives,nowandforgenerationstocome”*
“Creatingasocietythatgivesallindividualsanequalopportunitytolivethehealthiestlivespossible,whatevertheirethnic,geographic,racial,socioeconomic,orphysicalcircumstanceshappentobe”*
RWJFisfocusedonbuildingacultureofhealththatemphasizesequityandanunderstandingthathealthisdeterminedbymanysocialconditions.BylabelingtheirworkasCulture ofHealth,RWJFrecognizestheimportanceofhealthbeingbasedonsharedvalues,customs,andbeliefsandhowtheseneedtochangewithinvariousgroupstohavepopulationhealth,well-being,andequity.RWJF’sapproachcontraststoonethatfocusesonlyonhealth-careservices.Thehealth-care-serviceapproachassumesthatindividualsarelargelyresponsible fortheirownhealth;itignorestheinequitiesandtheconfusingprocessesandmultipleorientationswithinhealthcareandothersocialdeterminantsofhealth.RWJF’sapproachfocusesonpeoplechangingtheirbeliefsandassumptionsandthestructuresoftheirsocialsystems,i.e.,changingtheculturetoaCultureofHealth.
*Copyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth.
|36
![Page 40: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/40.jpg)
ACultureofHealthActionFrameworkBasedonthesegeneralbeliefsandassumptions,theFoundationhasidentifiedfourareasforactiontobringaboutthecultureshift.Thefoundationhasalsostatedanencompassingoutcomethatwouldresultfromthisshift.
Notethattheframeworkisbroad.Itdoesnotgivespecificactionsforpeopletotakebutprovidestheareasonwhichtofocusandincorporatesthebeliefsthatarekeytothenewculture.Itisgivingpeopleatanyplaceinsocietyareasinwhichtofocus.
Buildinganewculturemaybelessaboutactionswedoandmoreaboutadialoguethatflowsthroughandamongus.
Imagecopyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth. |37
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
![Page 41: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/41.jpg)
ExampleofanEvaluationSituationandApproach Acoalitionthatrecentlyreceivedagranthasrequestedsuggestionsabout
conductingitsevaluationtodetermineifthemanysystemsinvolvedintheirworkareactuallyfunctioninginawaythatchangesthecultureregardinghealth.TheyareusingtheRWJFCultureofHealthActionFrameworkastheirguide.
Theexampleprovidessuggestionsrelatedtothegeneralevaluationapproachandevaluationquestions.
Here’sthegranteesituation:
Overa20-yearperiod,acoalitionhasguidedthelocatingofawiderangeofhealth-careservicesinonehealth-carecomplex.Thecoalition,locatedinalow-incomeregionofasouthwesternstate,includesvariousentitiessuchasnonprofitorganizations,health-careserviceproviders,andresidentsfromtheurbanneighborhoodsandruralareas.Severalyearsago,withcommunityinput,thecoalitiondevelopedastatementofsharedvalues.
Acommunitysurveyrevealedaneedforaccesstomorehealthyfood.So,thecoalitionisnowplanningtoaddagrocerystoreinthehealth-carecomplex.
Thecoalitionmembershaveanunderstandingofcomplexsystemsfromarecentroundofseminars.Theyareparticularlyinterestedinthenodesandconnectionsofthenetworksandwebsoftheircommunityandwhetherthey,asmembersofacoalition,areanimportantnodeinbuildinganeconomically-equitablecommunity.Theyarealsoawarethatcommunityorganizerswhoarenewmembersofthecoalitionthinkthatthecommunitysurveyprovidedaninadequateunderstandingofthecommunityasawhole.Photos(exceptfarmworkers)copyright2015RobertWood
JohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth;photosbyTyroneTurner.Farmworkers’photosretrievedfrompublicdomain. |38
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
![Page 42: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/42.jpg)
ExampleEvaluationSituation
Coalitionmembersarepleasedwiththeprogresstheyhavemadeonoverallindicatorsofpopulationwell-beingandhealth,buttheyarenotsureifdeepchangesintheculture(changesinvalues,beliefs,andassumptions)areoccurringthatwillsupportasustainablecultureofhealthasdescribedbytheRWJFCultureofHealthactionframework.
Thecoalitionhasworkedcollaborativelyinthehealthcomplextobuildawidespreadunderstandingofthecentralityofhealthinthegovernmentalagencies,businesses,schools,anduniversities.Health-careserviceprovidersareintegratingtheirtechnologicalsystemstosharedataandmakeiteasierforresidentstoworkwithmultipleproviders.Theynowrecognizethattheseeffortsareallfocusedonhierarchicalstructures.
Theyareuneasyaboutwhethersufficientattentionisbeinggiventobuildinganeconomically-equitablecommunity,astructurethatismoregroundedinnetworks.
Intheirnextphaseofevaluationtheywanttofocusspecificallyonthefunctioningoftheircoalitionasanodewithinthemanyinterconnectednetworksandwebsoftheircommunitythatareespeciallyrelevanttobuildinganeconomically- equitablecommunity.
Photos(exceptfarmworkers)copyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth;photosbyTyroneTurner.Farmworkers’photosretrievedfrompublicdomain.
|39
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
![Page 43: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/43.jpg)
Workingwithasystems-orientedevaluatorfromanonprofitorganizationthatisamemberofthecoalition,thecoalitionmembersbeganbyidentifyingtheseguidelinesforconductingasystems-orientedandnetwork-focusedevaluation.
• Usebroadlydefinedboundaries fordeterminingwhattypesofnetworksandwebsexistinthecommunitythatmayberelevant.
• Attendtothe interconnections (especiallypowerrelationships) amongthefullrangeofstakeholders(individualsandcollectiveentities)inthesituation.
• Lookforpatternsofinterconnectednodes;lookformanytypesofresources;andlookathowthesevariousresourcesflowthroughthenetworkandareexchangedinthenodesofthenetworks.
• Encouragetransparencyofperspectives (values,beliefs,andassumptions)inallphasesoftheevaluation.
• Emphasizefacilitationthatdeterminesmerit,worth,andsignificancethroughouttheprocessasitrelatestohealthforall.
• Focusonaskingthe“right”questionsforpurposesofsharedreflectionandinquiryratherthannecessarilyansweringthequestionsinafinalsense.
• Emphasizebuildingthecapacityofcoalitionmemberstoengageinevaluative,reflectiveinquirythathelpsthemexpendtheleastefforttobuildtheself-organizingcapacityofthenetworkstosupportaneconomically-equitablecommunity.
• Drawonexpertisefrommultipledisciplinesrelevanttothesituation.
ExampleEvaluationApproach
|40
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
![Page 44: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/44.jpg)
Herearesomeofthequestionstheyareconsidering.Theyasksomequestionstounderstandthecurrentandpastinteractionsabouttheeconomicallyequitablenatureoftheircommunityandhowtheestablishmentofagrocerystoreinthehealthcarecomplexmightrelatetothisdesiredcondition.Theyaskotherquestionstounderstandthebeliefs,assumptions,perspectives,andperceptionsaboutthemeaningandnatureofaneconomicallyequitablecommunity.
Interconnections
• Whatpatternsofpowerrelationshipsarepresentandhowaretheyshifting?
• Whatistheebbandflowofchangesbetweentheformalhealthorganizationsandtheresidentsofthecommunity?
• Whatshiftsareoccurringintheflowofinformation?
Boundaries
• Whatboundarieswithinandacrossorganizationsandwiththecommunityanditsneighborhoodgroupsareshifting?
• Whatdifferencesareevidentinthepermeabilityoftheboundarieswithinandacrossorganizationsandwiththecommunityanditsneighborhoods?
• Whatshiftsareoccurringinwhoisinvolvedandinwhatwaysinvarioustypesofdecision-making?
• Howarepatternsofactionandthoughtshiftingregardinghealthacrossgenerations?Whoisincluded?Whoisnot?
Perspectives
• Whatshiftsinvalues,beliefs,andassumptionsareoccurringrelatedtoequityanddemocracy?
• Whatshiftsinvalues,beliefs,andassumptionsareoccurringregardingtheimportanceofhealth?
• Whatshiftsinvalues,beliefs,andassumptionsareoccurringregardingwhatconstituteshealth?
ExampleEvaluationQuestions:Interconnections,Boundaries,andPerspectives
|41
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
![Page 45: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/45.jpg)
CalltoAction
v ImplicationsforEvaluationandEvaluators
04|42
![Page 46: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/46.jpg)
04.CalltoActio
n
ImplicationsforEvaluationandEvaluatorsThispapercallsonevaluatorsandevaluationuserstoattendtotwofundamentalstructuresofcomplex
socialsystems—hierarchiesandnetworks.Asastarter,considerthefollowing:
• Engageinevaluationwithattentiontobothhierarchicalstructureswithcontrolleddynamicsand
networkedstructureswithemergentdynamics.
• Recognizethatevaluationcapacitybuildingislikelytolookdifferentwhenworkingwithhierarchical
structuresversusnetworkedstructures.
• Payspecialattentiontoevaluativefacilitationaswellasinquiryprocesses.
• Recognizethatoutcomesinahierarchicalstructurearelikelytobedefinableproducts,knowledge,or
services,whereasoutcomesinanetworkedstructurearelikelytobechangesinpatternsofactionand
thinkingacrosstimeandlocations.
• Recognizethedifferenceinrolesofthetwotypesofstructuresinsustaining,restoring,andregenerating
society.
• Engageinconversationswithcolleaguesabouttheapplicationofthesesystemsconceptsandsharewith
uswhatyouarelearning.
|43
![Page 47: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/47.jpg)
Resources
v Referencesv Acknowledgements
|44
![Page 48: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/48.jpg)
ReferencesCapra,F.&Luisi,P.(2014).Thesystemsviewoflife:Aunifyingvision.Cambridge,MA:CambridgeUniversityPress.Dirks,N.B.,Eley,G.,&Ortner,S.B.(1994).Introduction.InN.B.Dirks,G.Eley,&S.B.Ortner(Eds.),Culture/power/history:Areaderin
contemporarysocialtheory(pp.3-45).Princeton,NJ:PrincetonUniversityPress.Eoyang,G.(1997).Conditionsforself-organizinginhumansystems.Unpublisheddoctoraldissertation,UnionInstituteandUniversity,
Cincinnati,OH.Fairclough,N.andWodak,R.,(1998)."CriticalDiscourseAnalysis"inDijk,T.A.(Ed.)(1998)DiscourseasSocialInteraction:Discoursestudies:A
multidisciplinaryIntroduction(Vol.2).LondonSAGE.Girard,JohnP.&Girard,JoAnnL.(2015)."Definingknowledgemanagement:Towardanappliedcompendium"(PDF). OnlineJournalofApplied
KnowledgeManagement.3 (1):1–20.Jackson,M.C.(2003).Systemsthinking:Creativeholismformanagers.WestSussex,England:JohnWileyandSonsLtd.KelloggFoundation,W.K.(2007).Designinginitiativeevaluation:Asystems-orientedframeworkforevaluatingsocialchangeefforts. Battle
Creek,MI:W.K.KelloggFoundation.Mathison,S.(2005).Encyclopediaofevaluation.ThousandOaks,CA:SAGEPublications.Meadows,D.H.(2008).Thinkinginsystems.WhiteRiverJunction,VT:ChelseaGreenPublishing.OpenUniversity.(2011).Systemsthinkingandpractice.Retrievedfromhttp://www.open.edu/openlearn/science-maths-
technology/computing-andict/systems-computer/systems-thinking-and-practice/content-section-0Parsons,B.(2012).Usingcomplexityscienceconceptswhendesigningsysteminterventionsandevaluations(3rded.).Retrievedfrom
http://insites.org/resources/using-complexity-science-concepts-when-designing- system-interventions-and-evaluations/Ramage,M.,&Shipp,K.(2009).Systemsthinkers.London,England:Springer.Rosaldo,R.(1993).Culture&truth:Theremakingofsocialanalysis.Boston,MA:BeaconPress.Scriven,M.(2013).Thefoundationandfutureofevaluation.InThefutureofevaluationinsociety:AtributetoMichaelScriven.Donaldson,S.
(Ed.).Charlotte,NC:InformationAgePublishing,Inc.Staats,C.,Capatosto,K.,&Contractor,D.(2015).Stateofthescience:ImplicitBiasReview2015. Columbus,Ohio:KilwanInstitutefortheStudy
ofRaceandEthnicity.Williams,B.&Hummelbrunner,R.(2009).Systemconceptsinaction:Apractitioners’toolkit. Stanford,CA:StanfordUniversityPress.Zimmerman,B.,Lindberg,C.,&Plsek,P.(2001).Edgeware:Insightsfromcomplexityscienceforhealthcareleaders. Irving,TX:VHA.
Websiteshttp://www.insites.org/
http://www.fsg.org/http://www.evaluationinnovation.org/
https://www.aspeninstitute.org/
Resources
|45
![Page 49: A Complex-Systems Orientation to Evaluation that Supports ...insites.org/wp-content/uploads/58c3050dc99d1-InS.17.ws.resource... · Evaluative Thinking Example Evaluative thinking](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f852cb91e6e11e5d349fe1/html5/thumbnails/49.jpg)
AcknowledgementsBeverlyParsons,PatriciaJessup,andMarahMoorewrotethispaperwithsupport
fromtheRobertWoodJohnsonFoundation.Theviewsexpressedheredonot
necessarilyreflecttheviewsoftheFoundation.
WewishtoexpressourthankstoLauraLevitonoftheRobertWoodJohnson
Foundationforherinsights,review,andfeedbackthroughoutthecompletionof
thegrant.
OurappreciationisalsoexpressedtoourcolleaguesintheAmericanEvaluation
Association’sSystemsinEvaluationTopicalInterestGroupforthemany
conversationsthathaveinformedthethinkingreflectedinthispaper.
WearealsodeeplygratefultoRosemaryReinhartforhereditingandKathy
Wyckoffforherformattinganddesignofthepaper.
Resources
|46October2016