Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North...
-
Upload
bathsheba-griffin -
Category
Documents
-
view
218 -
download
0
Transcript of Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North...
![Page 1: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/1.jpg)
Assessment & Review of Graduate Programs- Doctoral
Duane K. Larick & Michael P. Carter
North Carolina State University
Council Of Graduate SchoolsPre-Meeting Workshop
December, 2006
![Page 2: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/2.jpg)
Assessment and Review
Outline of Presentation• Why review/assess graduate
programs
• A review process incorporating periodic external reviews and continuous program assessment
![Page 3: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/3.jpg)
Marilyn J. Baker
Revised and Updated by:
Margaret King, Duane Larick, and Michael Carter
NC State University
![Page 4: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/4.jpg)
Background Information About Our Audience
How many of you are responsible for graduate program review at your institutions?
How many of you have this as a new responsibility?
How many of you have recently (or are considering) changing your procedure?
![Page 5: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/5.jpg)
The primary purpose should be to improve in the quality of graduate education on our campuses• By creating a structured, scheduled
opportunity for a program to be examined, program review provides a strategy for improvement that is well-reasoned, far-seeking, and as apolitical as possible
Why Review/AssessGraduate Programs?
![Page 6: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/6.jpg)
Why Review/AssessGraduate Programs?
External Considerations• To help satisfy calls for accountability
Especially at the State level
• Requirement for regional accreditation, licensure, etc.
![Page 7: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/7.jpg)
SACS Principles of Accreditation
Core requirement #5: “The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement and (b) demonstrates that the institution is effectively accomplishing its mission.”
![Page 8: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/8.jpg)
Why Review/AssessGraduate Programs?
Internal Considerations• Meet long-term (strategic) College &
Institutional goalsCreation of new degree programsElimination of existing programsFunding allocation/reallocation
• Advanced understand of graduate education and factors influencing graduate educationAids in identification of “common” programmatic
needs
![Page 9: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/9.jpg)
Why Review/AssessGraduate Programs
Internal Considerations• Creates an opportunity to focus on key issues
impacting graduate educationCauses of retention/attrition among students and
faculty
• Meet short-term (tactical) objectives or targets at the program levelDocuments achievements of faculty & studentsIndicates the degree to which Program outcomes
have been achievedSuggests areas for improvementHelps chart new programmatic directions
![Page 10: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/10.jpg)
So The Questions We Need To Ask Ourselves Are
What are we currently doing? Why are we currently doing it? Is what we are currently doing
accomplishing the external goals described above?
Is what we are currently doing accomplishing the internal goals described above?
Is there a better way?
![Page 11: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/11.jpg)
Graduate ProgramReview – A Two Phase Process
Periodic formal review of graduate programs (external review)
Outcomes-based assessment (internal review that is a continuous and ongoing process)
![Page 12: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/12.jpg)
Key Features of Formal Reviews
Evaluative, not just descriptive Forward-looking: focus on improvement of
program, not just current status Based on program’s academic strengths
and weaknesses, not just ability to attract funding
Objective Independent, stands on its own Action-oriented: clear, concrete
recommendations to be implemented
![Page 13: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/13.jpg)
Questions Answered by Formal Review
Is the program advancing the state of the discipline or profession?
Is its teaching and training of students effective?
Does it meet institutional goals? Does it respond to the profession’s
needs? How is it assessed by experts in the
field?
![Page 14: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/14.jpg)
Issues to be Resolved Before Beginning
Locus of control Graduate-only or comprehensive program
review Counting—and paying—the costs Master’s and doctoral programs Coordination with accreditation reviews Scheduling the reviews Multidisciplinary and interdisciplinary
programs
![Page 15: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/15.jpg)
Clear, Consistent Guidelines• The purpose of graduate program review
• The process to be followed
• Guidelines for materials to be included in each phase
• A generic agenda for the review
• The use to which results will be put
Key Elements of a Successful Program Review
![Page 16: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/16.jpg)
Administrative Support• Departmental resources: time, funding,
secretarial help, etc.
• Central administrative support for larger review process
• Adequate and accurate institutional data, consistent across programs
Key Elements of a Successful Program Review
![Page 17: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/17.jpg)
Program Self-Study• Engage the program faculty in a thoughtful
evaluation of:The program’s purpose(s)The program’s effectiveness in achieving these
purposesThe program’s overall qualityThe faculty’s vision for the program
Key Elements of a Successful Program Review
![Page 18: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/18.jpg)
Surveys/Questionnaires• Surveys from current students, faculty,
alumni, and employers
• Factors to be considered:Time and expense to develop, distribute and
collect responsesLikely response rateAdditional burden on respondentsUniqueness of information to be gained
Key Elements of a Successful Program Review
![Page 19: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/19.jpg)
Student Participation• Complete confidential questionnaires
• Provide input into self-study
• Be interviewed collectively and individually by review team
• Serve on review teams and standing committees
Key Elements of a Successful Program Review
![Page 20: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/20.jpg)
Review Committee• On-Campus Representation
A representative of the Graduate SchoolInternal reviewer from a field that gives him/her
some understanding of the program(s) being reviewed
• External Reviewer(s)Number of reviewers depends on scope and kind
reviewSelection process can vary – programs can have
input but should not make the final decision
Key Elements of a Successful Program Review
![Page 21: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/21.jpg)
Final Report by Review Team• Brief overview of program
• Strengths of program
• Areas for improvement
• Recommendations for improvement
Key Elements of a Successful Program Review
![Page 22: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/22.jpg)
Program Faculty’s Response to Report• Clear up errors or misunderstandings
• Respond to the recommendations (have implemented, will implement, will consider implementing, cannot implement and why)
Key Elements of a Successful Program Review
![Page 23: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/23.jpg)
Implementation• One or more meetings of key administrators
(department, college, graduate school, and university) to discuss recommendations
• An action plan or memorandum of understanding drawn up and agreed on by all participants
• Discussion of the recommendations with program faculty for implementation
• Integration of the action plan into the institution’s long-range planning and budget process
Key Elements of a Successful Program Review
![Page 24: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/24.jpg)
Follow Up• An initial report on progress toward
implementation of action plan (1 or 2 years out)
• Follow-up reports until action plan is implemented or priorities change
• Discussion of recommendations and implementation in self-study for next review
Key Elements of a Successful Program Review
![Page 25: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/25.jpg)
Questions Relative to External Program Review?
![Page 26: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/26.jpg)
What is Outcomes-Based Assessment?
It is a process that engages program faculty in asking 3 questions about their programs• What are our expectations for the program?• To what extent is our program meeting our expectations?• How can we improve our program to better meet our
expectations?
It is a process that provides program faculty the means to answer these questions• By creating objectives and outcomes for their program• By gathering and analyzing data to determine how well the
program is meeting the objectives and outcomes• By applying the results of their assessment toward improving
their program
![Page 27: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/27.jpg)
What is Outcomes-Based Assessment? continued
It entails a shift in emphasis from inputs to outcomes
It is continuous rather than periodic It involves regular reports of program
assessment to the institution Its results are used by the program and
institution for gauging improvement and for planning
![Page 28: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/28.jpg)
What is Outcomes-Based Assessment? continued
• Faculty generate program objectives and outcomes
• Faculty decide how outcomes will be assessed
• Faculty assess outcomes• Faculty use assessment findings to
identify ways of improving their programs
![Page 29: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/29.jpg)
Benefits of Outcomes Assessment
It provides the groundwork for increased responsiveness and agility in meeting program needs
It gives faculty a greater sense of ownership of their programs
It provides stakeholders a clearer picture of the expectations of programs
It helps institutions meet accreditation requirements
![Page 30: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/30.jpg)
SACS Criterion for Accreditation
Section 3 – Comprehensive Standards - #16
“The institution identifies outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.”
![Page 31: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/31.jpg)
Drive Toward Greater Accountability on Our Campus
Professional accreditation agencies (e.g., engineering, social work, business)
Undergraduate assessment Assessment of general education
![Page 32: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/32.jpg)
Outcomes Assessment: A Process
Phase I: Identifying Objectives and Outcomes
Phase II: Creating Assessment Plans Phase III: Implementing Assessment
Plans Phase IV: Reporting Assessment
Results
![Page 33: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/33.jpg)
A Procedure for Implementing Outcomes Assessment
Identify pilot programs to create assessment materials for each phase
Use pilot materials as a basis for DGP workshops for each phase
Offer individual support to DGPs as they created materials and assessed programs
Create online tools to aid DGPs
![Page 34: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/34.jpg)
Phase I: Identifying Objectives and Outcomes
![Page 35: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/35.jpg)
What Are Objectives?
Program objectives are the general goals that define what it means to be an effective program.
![Page 36: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/36.jpg)
Three Common Objectives
Developing students as successful professionals in the field
Developing students as effective researchers in the field
Maintaining/enhancing the overall quality of the program
![Page 37: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/37.jpg)
What Are Outcomes?
Program outcomes are specific faculty expectations for each objective that define what the program needs to achieve in order to meet the objectives.
![Page 38: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/38.jpg)
Example for Outcome 1: Professional Development
1. To enable students to develop as successful professionals for highly competitive positions in industry, government, and academic departments, the program aims to provide a variety of experiences that help students to:
a. achieve the highest level of expertise in XXXX, mastery of the knowledge in their fields and the ability to apply associated technologies to novel and emerging problems
b. present research to local, regional, national, and international audiences through publications in professional journals and conference papers given in a range of venues, from graduate seminars to professional meetings
c. participate in professional organizations, becoming members and attending meetings
d. broaden their professional foundations through activities such as teaching, internships, fellowships, and grant applications
![Page 39: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/39.jpg)
Example for Outcome 2: Effective Researchers
2. To prepare students to conduct research effectively in XXXX in a collaborative environment, the program aims to offer a variety of educational experiences that are designed to develop in students the ability to:
a. read and review the literature in an area of study in such a way that reveals a comprehensive understanding of the literature
b. identify research questions/problems that are pertinent to a field of study and provide a focus for making a significant contribution to the field
c. gather, organize, analyze, and report data using a conceptual framework appropriate to the research question and the field of study
d. interpret research results in a way that adds to the understanding of the field of study and relates the findings to teaching and learning in science
Etc.
![Page 40: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/40.jpg)
Example for Outcome 3:Quality of Program
3. To maintain and improve the program’s leadership position nationally and internationally, the program aims to:a. continue to be nationally competitive by attracting high-
quality students
b. provide effective mentoring that encourages students to graduate in a timely manner
c. place graduates in positions in industry and academics
d. maintain a nationally recognized faculty that is large enough and appropriately distributed across XXXX disciplines to offer students a wide range of fields of expertise
![Page 41: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/41.jpg)
Phase II: Creating Assessment Plans
![Page 42: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/42.jpg)
Four Questions for Creating an Assessment Plan
1. What types of data should we gather for assessing outcomes?
2. What are the sources of the data?3. How often are the data to be
collected?4. When do we analyze and report the
data?
![Page 43: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/43.jpg)
Types of Data Used
1. Take advantage of what you are already doing
Preliminary exams Proposals Theses and dissertations Defenses Student progress reports Student course evaluations Faculty activity reports Student exit interviews
![Page 44: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/44.jpg)
Types of Data Used
2. Use resources of Graduate School and institutional analysis unit
Enrollment statistics Time-to-degree statistics Student exit data Ten-year profile reports Alumni surveys
![Page 45: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/45.jpg)
Types of Data Used
3. Use your imagination to find other types of data
• Dollar amount of support for faculty
• Student activity reports
• Faculty surveys
![Page 46: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/46.jpg)
Data: Two Standards to Use in Identifying Data
1. Meaningful: Data should provide information that is suitable for assessing the outcome
2. Manageable: Data should be reasonable to attain (time, effort, ability, availability, resources)
![Page 47: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/47.jpg)
Four Questions for Creating an Assessment Plan
1. What data should we gather for assessing outcomes?
2. What are the sources of the data?3. How often are the data to be
collected?4. When do we analyze and report the
data?
![Page 48: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/48.jpg)
Sources of Data
Students Faculty Graduate School Graduate Program Directors Department Heads Registration and Records Advisory Boards University Planning and Analysis
![Page 49: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/49.jpg)
Four Questions for Creating an Assessment Plan
1. What data should we gather for assessing outcomes?
2. What are the sources of the data?3. How often are the data to be
collected?4. When do we analyze and report the
data?
![Page 50: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/50.jpg)
Frequency of Data Collection
Every semester Annually Biennially When available from individual graduate
students• At the preliminary exam
• At the defense
• At graduation
![Page 51: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/51.jpg)
Four Questions for Creating an Assessment Plan
1. What data should we gather for assessing outcomes?
2. What are the sources of the data?3. How often are the data to be
collected?4. When do we analyze the data?
![Page 52: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/52.jpg)
Creating a Timeline for Analyzing Assessment Data
According to objective: year 1-objective 1; year 2-objective 2; year 3-objective 3; year 4-objective 1; etc. (3-year cycle)
More pressing outcomes earlier and less pressing ones later
Outcomes easier to assess earlier and outcomes requiring more complex data gathering and analysis later
Approximately the same workload each year of the assessment cycle
![Page 53: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/53.jpg)
Four Questions for Creating an Assessment Plan
1. What data should we gather for assessing outcomes?
2. What are the sources of the data?3. How often are the data to be
collected?4. When do we analyze and report the
data?
![Page 54: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/54.jpg)
Assessment Plan
![Page 55: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/55.jpg)
Phase III: Implementing Assessment Plans
Collecting, Analyzing, and Evaluating Data and
Improving the Program
![Page 56: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/56.jpg)
Collecting Data
Goal:To have data readily accessible
when it is time to analyze the data.
![Page 57: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/57.jpg)
Typical Modes of Data Collection
Rubrics for prelims and defenses Student Activity Reports/CVs Statistics provided by Graduate School Faculty Activity Reports Student exit surveys or interviews
![Page 58: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/58.jpg)
Suggestions for Collecting Data
Identify the kinds of data you need to collect, who is responsible for collecting them, and when they are to be collected.
Determine where the data are to be stored and check periodically to be sure data are up to date.
Make data collection and storage as much a departmental routine as possible.
![Page 59: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/59.jpg)
Analyzing Data
Goal:To put data into a form that will
allow faculty to use them to evaluate the program.
![Page 60: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/60.jpg)
Spreadsheet for Rubrics for Prelims and Defenses
![Page 61: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/61.jpg)
Graphs from Graduate School Statistics
![Page 62: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/62.jpg)
Evaluating Data
Goal:To use the data to judge the extent to which the program is meeting
faculty expectations.
![Page 63: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/63.jpg)
Suggestions for Evaluating Data
In most cases, the primary criterion for evaluation is faculty expectations. Allow faculty to discuss their expectations as a way of defining criteria for evaluation.
Guide faculty discussion by asking them to identify strengths of the program and areas of concern.
Evaluation is typically a judgment call; encourage faculty to trust their judgments.
![Page 64: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/64.jpg)
Making Decisions for Improving the Program
Goal:To apply what has been learned in evaluating the data to identifying
actions that address areas of concern.
![Page 65: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/65.jpg)
Suggestions for Making Decisions for Improving Programs
Lead faculty in brainstorming; try to elicit multiple suggestions for actions.
All suggestions should be evaluated for feasibility and validity (do they offer a good chance of affecting the area of concern?).
It’s OK to conclude that change is not yet warranted, more data need to be collected.
Also encourage faculty to address the need for changes in assessment procedures.
![Page 66: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/66.jpg)
Phase IV: Reporting Assessment Results
![Page 67: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/67.jpg)
Reporting Assessment Results
Goal:To submit a report every two years
in which you summarize your assessment process and findings.
![Page 68: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/68.jpg)
Creating a Timeline for Reporting Assessment Data
Standard practice appear to call for an annual or biennial assessment report
Longer cycles lose the impact on the continuous and ongoing nature
When possible correlate with pre-existing external review program
![Page 69: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/69.jpg)
Two Purposes of Assessment Reports
1. Primary: To maintain a record of assessment and improvements for you and subsequent DGPs to be used for self-studies, accreditation agencies, boards of advisors, etc.
2. Secondary: To provide evidence of a process of accountability for the university.
![Page 70: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/70.jpg)
Questions to Guide Reports
1. What outcomes were you scheduled to assess during the present biennial reporting period? What outcomes did you assess?
2. What data did you collect? Summarize your findings for these data.
3. What did you and your faculty learn about your program and/or your students from the analysis of the data? What areas of concern have emerged?
![Page 71: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/71.jpg)
Questions to Guide Reports
4. As a result of your assessment, what changes, if any, have you and your faculty implemented or considered implementing to address areas of concern?
5. What outcomes are you planning to assess for the upcoming biennial reporting period?
![Page 72: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/72.jpg)
![Page 73: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/73.jpg)
What We Have Learned
The process of change takes time Communication is the key to success It is important to pilot assessment
processes before taking it to all graduate programs.
![Page 74: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/74.jpg)
What We Have Learned continued
This kind of review process must be ground (faculty) up not top (administration) down
This kind of review process requires significant human resources• Training, data collection, analysis, and
interpretation, etc.• A key to our success is how much of this can be
institutionalized
![Page 75: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/75.jpg)
Managerial Tools Created for Program Review - Website
![Page 76: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/76.jpg)
Assessment and Review - Connecting the Two
Both must be owned by the faculty• The self-study required for formal program
review must have input from the entire facultyThe resulting “action plan” must also be agreed
on by the faculty in the Program
• The objectives, outcomes and assessment plan for outcome based assessment must have buy in and participation by all faculty
![Page 77: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/77.jpg)
Continuous and ongoing review should inform and enhance formal program review• Formal review self-study should include a
summary of the assessment findings and changes implemented.
• Ideally, these incremental improvements will have resulted in a stronger program and fewer “surprises” at the time of the formal review.
Assessment and Review - Connecting the Two
![Page 78: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/78.jpg)
The formal review process may suggest additional or revised program outcomes and assessment measures• Formal review self-study should include an
outline of the program outcomes and assessment plan for reviewer comment
Assessment and Review - Connecting the Two
![Page 79: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/79.jpg)
`
Questions & Discussion
![Page 80: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/80.jpg)
Managerial Tools Created for Program Review - Website
![Page 81: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/81.jpg)
Managerial Tools Created for Program Review - Website
![Page 82: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/82.jpg)
Managerial Tools Created for Program Review - Website
![Page 83: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/83.jpg)
Managerial Tools Created for Program Review – Review Document Management
![Page 84: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/84.jpg)
Managerial Tools Created for Program Review – Review Document Management
![Page 85: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/85.jpg)
Managerial Tools Created for Program Review – Review Document Management
![Page 86: Assessment & Review of Graduate Programs- Doctoral Duane K. Larick & Michael P. Carter North Carolina State University Council Of Graduate Schools Pre-Meeting.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649db15503460f94aa028d/html5/thumbnails/86.jpg)