NSF-EHR Framework for Improving Science, Technology, Engineering and Mathematics Education R & D...
-
Upload
brittney-owen -
Category
Documents
-
view
218 -
download
0
Transcript of NSF-EHR Framework for Improving Science, Technology, Engineering and Mathematics Education R & D...
NSF-EHR Framework for NSF-EHR Framework for Improving Improving Science, Technology, Science, Technology, Engineering and Mathematics Engineering and Mathematics
Education R & D Education R & D
Ferdie RiveraProgram Director (Part time; Intermittent Expert)
Division of Research on Learning in Formal and Informal Settings (DRL)Directorate for Education and Human Resources (EHR Directorate)
National Science Foundation (NSF)
National Science FoundationNational Science FoundationDirectoratesDirectorates
Social, Behavioral, and Economic Sciences
Biological Sciences
Engineering
Education and Human Resources
Computer and Information Sciences and Engineering
Mathematics and Physical Sciences
Geosciences
Directorate for Education and Directorate for Education and Human ResourcesHuman Resources
Division of Undergraduate Education Division of Undergraduate Education (IUSE; ATE; NOYCE; STEM-C)(IUSE; ATE; NOYCE; STEM-C)
Division of Graduate Education (GRFP; Division of Graduate Education (GRFP; IGERT; GK 12)IGERT; GK 12)
Division of Human Resource Development Division of Human Resource Development (ADVANCE; AGEP; HBCU; TCUP)(ADVANCE; AGEP; HBCU; TCUP)
Division of Research on Learning in Division of Research on Learning in Informal and Formal Settings (CORE; Informal and Formal Settings (CORE; ITEST; DR K 12; AISL; STEM-C) ITEST; DR K 12; AISL; STEM-C)
Characteristics of Strong STEM R & D ProjectsCharacteristics of Strong STEM R & D Projects
They essentially have a They essentially have a coherentcoherent and and deep scientific deep scientific researchresearch base that informs/guides efforts to meet base that informs/guides efforts to meet national STEM education and workforce development national STEM education and workforce development needs/prioritiesneeds/priorities
They address CORE They address CORE research issues research issues that address that address broad, essential, enduring, and challenges of great broad, essential, enduring, and challenges of great importance and are importance and are designed to inform the designed to inform the transformation of STEM learning and educationtransformation of STEM learning and education
R & D projects are potentially transformative and R & D projects are potentially transformative and innovativeinnovative
NSF EHR CORENSF EHR CORE
STEM LEARNING
STEM LEARNING ENVIRONMENTS
STEM WORKFORCE
DEVELOPMENT
BROADENING PARTICIPATION
IN STEM
Learning of specific STEM disciplinary Learning of specific STEM disciplinary knowledge and practices across different knowledge and practices across different groups of learners, contexts, and groups of learners, contexts, and environmentsenvironments
Learning includes cognitive, social and Learning includes cognitive, social and behavioral competenciesbehavioral competencies
Range of theoretical and methodological Range of theoretical and methodological approaches, including the use of approaches, including the use of ““big databig data””
Focus on learning at the frontiers of STEM Focus on learning at the frontiers of STEM disciplines and interdisciplinaritydisciplines and interdisciplinarity
Core Issues in STEM LearningCore Issues in STEM Learning
Range of formal and informal learning Range of formal and informal learning environments across the K-16 landscapeenvironments across the K-16 landscape
Develop an understanding of how characteristics Develop an understanding of how characteristics of learning environments interact with or support of learning environments interact with or support multiple aspects of STEM learningmultiple aspects of STEM learning
Special focus on emerging learning environments Special focus on emerging learning environments and evidence-based approaches to undergraduate and evidence-based approaches to undergraduate STEM instructionSTEM instruction
Multiple theoretical and methodological Multiple theoretical and methodological perspectivesperspectives
Core Issues in STEM Learning Core Issues in STEM Learning EnvironmentsEnvironments
Preparation of a diverse, globally-prepared and Preparation of a diverse, globally-prepared and highly-skilled STEM workforce - including highly-skilled STEM workforce - including teachersteachers
Focus on entire STEM workforce continuum, Focus on entire STEM workforce continuum, from Pre-K to postdoctoral training to careerfrom Pre-K to postdoctoral training to career
Emphasis on academic and non-academic Emphasis on academic and non-academic STEM career pathways and transitions across STEM career pathways and transitions across changing contexts and climateschanging contexts and climates
Connecting workplace expectations to design Connecting workplace expectations to design of interventionsof interventions
Core Issues in STEM Workforce Core Issues in STEM Workforce DevelopmentDevelopment
Practices that broaden participation, retention, and Practices that broaden participation, retention, and success of individuals underrepresented in STEM success of individuals underrepresented in STEM
Preparing students for successful transition to Preparing students for successful transition to further education or training, or the STEM workplace further education or training, or the STEM workplace
Study of accessibility and the impacts of technology Study of accessibility and the impacts of technology on diverse populationson diverse populations
Measures, processes and metrics to assess impacts Measures, processes and metrics to assess impacts and outcomes of broadening participation and and outcomes of broadening participation and institutional capacity building (e.g. on STEM institutional capacity building (e.g. on STEM innovation/productivity)innovation/productivity)
Core Issues in Broadening Participation Core Issues in Broadening Participation in STEMin STEM
NSF projects should be of the highest quality NSF projects should be of the highest quality and have the potential to advance, if not and have the potential to advance, if not transform, the frontiers of knowledgetransform, the frontiers of knowledge
NSF projects should contribute more broadly to NSF projects should contribute more broadly to achieving societal goalsachieving societal goals
Meaningful assessment and evaluation of NSF Meaningful assessment and evaluation of NSF funded projects should be based on appropriate funded projects should be based on appropriate metrics, keeping in mind the likely correlation metrics, keeping in mind the likely correlation between the effect of broader impacts and the between the effect of broader impacts and the resources provided to implement projectsresources provided to implement projects
NSF Merit Review PrinciplesNSF Merit Review Principles
NSF Review CriteriaNSF Review Criteria All proposals are evaluated against the All proposals are evaluated against the
two NSB-approved review criteriatwo NSB-approved review criteria: : Intellectual Merit:Intellectual Merit: criterion encompasses criterion encompasses
the potential to advance knowledgethe potential to advance knowledgeBroader ImpactsBroader Impacts: criterion encompasses : criterion encompasses
the potential to benefit society and the potential to benefit society and contribute to the achievement of specific, contribute to the achievement of specific, desired societal outcomesdesired societal outcomes
NSF Suggested Questions for Intellectual Merit NSF Suggested Questions for Intellectual Merit & Broader Impact& Broader Impact
1.1. What is the potential for the proposed activity to: What is the potential for the proposed activity to: Advance knowledge Advance knowledge and understanding within its own field or across and understanding within its own field or across
different fields (Intellectual Merit); and different fields (Intellectual Merit); and Benefit societyBenefit society or advance desired societal outcomes (Broader or advance desired societal outcomes (Broader
Impacts)?Impacts)?
2.2. To what extent do the proposed activities suggest and explore To what extent do the proposed activities suggest and explore creative, original, or creative, original, or potentially transformativepotentially transformative concepts? concepts?
3.3. Is the Is the planplan for carrying out the proposed activities well-reasoned, for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to incorporate a mechanism to assessassess success? success?
4.4. How How well qualified well qualified is the individual, team, or organization to is the individual, team, or organization to conduct the proposed activities?conduct the proposed activities?
5.5. Are there Are there adequate resourcesadequate resources available to the PI (either at the available to the PI (either at the home organization or through collaborations) to carry out the home organization or through collaborations) to carry out the proposed activities?proposed activities?
Additional Proposal ConsiderationsAdditional Proposal Considerations Project evaluationProject evaluation
Evaluation plan appropriate to project goals and Evaluation plan appropriate to project goals and descriptive of project assessment approachdescriptive of project assessment approach
Substantive external expert review mechanismSubstantive external expert review mechanism Be very clear that the evaluation and research plans are Be very clear that the evaluation and research plans are
distinct. If they are the same, present a strong justification. distinct. If they are the same, present a strong justification.
Project personnel and managementProject personnel and management Research and management roles of senior personnelResearch and management roles of senior personnel Roles and responsibilities of students and other traineesRoles and responsibilities of students and other trainees Mentoring plan for postdoctoral researchersMentoring plan for postdoctoral researchers
Dissemination strategyDissemination strategy
Additional Proposal ConsiderationsAdditional Proposal Considerations
Letters of agreement to participateLetters of agreement to participate All appropriate organizationsAll appropriate organizations Members of advisory committeesMembers of advisory committees
Research design and methodologyResearch design and methodology Appropriate and rigorous research methodsAppropriate and rigorous research methods Well-documented usable, and replicable models, Well-documented usable, and replicable models,
frameworks, data, literature, and measuresframeworks, data, literature, and measures Well-justified methods, consonant with theory, and Well-justified methods, consonant with theory, and
suited to the stated questions or hypothesessuited to the stated questions or hypotheses Justify measures and instruments that are needed to Justify measures and instruments that are needed to
establish evidence establish evidence
Common GuidelinesCommon Guidelinesfor Education Research and Developmentfor Education Research and Development
U.S. National Science Foundation and U.S. National Science Foundation and U.S. Department of EducationU.S. Department of Education
A cross-agency framework that describes:A cross-agency framework that describes:Broad types of research and development Broad types of research and development The expected The expected purposes, justifications, purposes, justifications, and and contributions contributions of various types of research to of various types of research to knowledge generation about interventions and knowledge generation about interventions and strategies for improving learningstrategies for improving learning
1616
What do we mean by What do we mean by ““Common Guidelines?Common Guidelines?””
The American education system needs stronger The American education system needs stronger evidence provided at a faster paceevidence provided at a faster pace
More constrained federal resources demand that More constrained federal resources demand that NSF and ED purposefully build on each other’s NSF and ED purposefully build on each other’s research and development portfoliosresearch and development portfolios
A cross-agency vocabulary and set of research A cross-agency vocabulary and set of research expectations is critical for effective expectations is critical for effective communicationcommunication
Why do we need “Common Guidelines?”Why do we need “Common Guidelines?”
1717
Guidelines can help stakeholders develop a Guidelines can help stakeholders develop a better understanding of what different stages better understanding of what different stages of education research should address and of education research should address and might be expected to producemight be expected to produce Helps stakeholders understand what to expect from Helps stakeholders understand what to expect from
different types of research findingsdifferent types of research findings Supports more informed decisions based on the level of Supports more informed decisions based on the level of
evidenceevidence Provides a shared sense of what is needed as stakeholders Provides a shared sense of what is needed as stakeholders
engage with research to improve educational practicesengage with research to improve educational practices
1818
Implications of the NSF/IES Common Implications of the NSF/IES Common GuidelinesGuidelines
1919
Is not strictly linear, but multi-directional. Is not strictly linear, but multi-directional. Investigations can Investigations can sometimes move directly from development of core knowledge to Scale-sometimes move directly from development of core knowledge to Scale-up. Investigations can sometimes move directly from development of core up. Investigations can sometimes move directly from development of core knowledge to Scale-up. knowledge to Scale-up.
Requires efforts of researchers and practitioners representing Requires efforts of researchers and practitioners representing a range of disciplines and methodological expertisea range of disciplines and methodological expertise
May require more studies for basic exploration and design May require more studies for basic exploration and design than for testing the effectiveness of a fully-developed than for testing the effectiveness of a fully-developed intervention or strategyintervention or strategy
Requires assessment of implementation—not just estimation Requires assessment of implementation—not just estimation of impactsof impacts
Includes attention to learning in multiple settings ( formal and Includes attention to learning in multiple settings ( formal and informal)informal)
Knowledge Development in EducationKnowledge Development in Education
2020
2121
CORE KNOWLEDGE
•Foundational Research•Exploratory/Early Stage Research
2222
Foundational Research Fundamental knowledge that may contribute to improved learning & other education outcomes
Studies of this type:◦Test, develop or refine theories of teaching or learning◦May develop innovations in methodologies and/or technologies that influence & inform research & development in different contexts
2323
Early-Stage or Exploratory Research
Examines relationships among important constructs in education and learningGoal is to establish logical connections that may form the basis for future interventions or strategies intended to improve education outcomesConnections are usually correlational rather than causal
2424
Design and Development Research
2525
Design and Development Research
Draws on existing theory & evidence to design and iteratively develop interventions or strategies
◦ Includes testing individual components to provide feedback in the development process
Could lead to additional work to better understand the foundational theory behind the results Could indicate that the intervention or strategy is sufficiently promising to warrant more advanced testing
2626
STUDIES OF IMPACT
•Efficacy Research•Effectiveness Research•Scale-Up Research
2727
Studies of Impact generate reliable estimates of the ability of a fully-developed intervention or strategy to achieve its intended outcomes
Efficacy Research tests/examines whether an intervention or strategy can improve outcomes under what are sometimes called “ideal” conditions
Effectiveness Research tests/estimates the impacts of an intervention or strategy when implemented under conditions of routine practice
Scale-Up Research examines/estimates the impacts of an intervention or strategy under conditions of routine practice and across a broad spectrum of populations and settings
Another Way of Thinking About Another Way of Thinking About Different Impact Studies (Dearing)Different Impact Studies (Dearing)
Internal validity research is about Internal validity research is about efficacyefficacyExternal validity research is about External validity research is about
effectivenesseffectivenessDissemination research is about Dissemination research is about
efficiencyefficiency Implementation research is about Implementation research is about
compatibilitycompatibility
Important Features of Each Type of ResearchImportant Features of Each Type of Research
2929
PurposeHow does this type of research contribute to the evidence base?
Justification
How should policy and practical significance be demonstrated?
What types of theoretical and/or empirical arguments should be made for conducting this study?
Important Features of Each Type of ResearchImportant Features of Each Type of Research
3030
Outcomes
Generally speaking, what types of outcomes (theory and empirical evidence) should the project produce?
What are the key features of a research design for this type of study?
3131
Comparison, in brief: JUSTIFICATIONComparison, in brief: JUSTIFICATION
A clear description of the practical education problem and a compelling case that the proposed research will inform the development, improvement, or evaluation of education programs, policies, or practices
A strong theoretical and empirical rationale for the project, ideally with citations to evidence
Exploratory/Early Stage
Research
3232
Comparison, in brief: JUSTIFICATIONComparison, in brief: JUSTIFICATION
A clear description of the practical problem and the initial concept for the planned investigation, including a well-explicated logic modelIn the logic model, identification of key components of the approach, a description of the relationships among components, and theoretical and/or empirical supportExplanation of how the approach is different from current practice and why it has the potential to improve learning
.
Design and Development
Research
3333
Comparison, in brief: JUSTIFICATIONComparison, in brief: JUSTIFICATION
Clear description of the intervention/ strategy and the practical problem it addresses; how intervention differs from others; and connection to learning
Empirical evidence of promise from a Design and Development pilot study, or support for each link in the logic model from Exploratory/Early Stage research, or evidence of wide use
Justification for examining impact under ideal circumstances, rather than under routine practice conditions
Efficacy Research
3434
Comparison, in brief: OUTCOMESComparison, in brief: OUTCOMES
Empirical evidence regarding associations between malleable factors and education or learning outcomes
A conceptual framework supporting a theoretical explanation for the malleable factors’ link with the education or learning outcomes
A determination, based on the empirical evidence and conceptual framework, of whether Design and Development research or an Efficacy study is warranted, or whether further Foundational or Exploratory/Early-Stage research is needed
Exploratory/Early Stage
3535
Comparison, in brief: OUTCOMESComparison, in brief: OUTCOMES
A fully-developed version of the intervention or strategy
A well-specified logic model
Descriptions of the major design iterations, resulting evidence, and adjustments to logic model
Measures and data demonstrating project’s implementation success
Pilot data on the intervention’s promise for generating the intended outcomes
.
Design and Development
Research
3636
Comparison, in brief: Comparison, in brief: OUTCOMESOUTCOMES
Detailed descriptions of the study goals, design and implementation, data collection and quality, and analysis and findings
Implementation documented in sufficient detail to judge applicability of the study findings; when possible, relate these factors descriptively to the impact findings
Discussion of the implications of the findings for the logic model and, where warranted, make suggestions for adjusting the logic model to reflect the study findings
Efficacy Research
Project NarrativeProject NarrativeProject Outcomes Project Outcomes
• • Detailed descriptions of the study goals, design and implementation, Detailed descriptions of the study goals, design and implementation, data collection and quality, and analysis and findings data collection and quality, and analysis and findings
Research Plan Research Plan Study design used to estimate causal impact of the intervention on Study design used to estimate causal impact of the intervention on
the outcomes of interest the outcomes of interest Key outcomes of interest for the impact study and the minimum Key outcomes of interest for the impact study and the minimum
impact that would have policy or practical relevance impact that would have policy or practical relevance Study setting(s) and target population(s) Study setting(s) and target population(s) The sample The sample The data collection plan The data collection plan The analysis and reporting plan The analysis and reporting plan
DeadlinesDeadlines
Please go to nsf.gov for deadlines. Please go to nsf.gov for deadlines.
??
3939
QuestionsQuestions
Thank You!!!Thank You!!!