Post on 31-Dec-2015
description
2008 FAEIS Triennial Evaluation
The purpose of the FAEIS Triennial Evaluation is to conduct a comprehensive assessment of FAEIS once every three years in order to improve the usability of the information provided by FAEIS, monitor progress during the three-year grant cycle and plan for the future.
Objectiveso Describe selected demographic and other variables that have potential association with FAEIS use.
o Determine the level of usefulness of FAEIS data resources and applications.
o Determine the level of satisfaction with selected components of FAEIS.
o Determine levels of adoption of FAEIS and factors that explain the levels of adoption.
o Determine the relationship of selected demographic and usage variables (involvement, use, experience, primary job title) with perceived usefulness and satisfaction with FAEIS.
Methodology
o FAEIS Team developed first draft of questionso Peer Panel and USDA partners reviewed and made suggestions to draft questionso Peer Panel members were used to pilot test the on-line survey instrument. Changes made accordingly.o Web survey administered by the VT Center for Survey Researcho 1,145 email addresses from FAEIS users used as survey sample group. Reduced to 1,120 by request of sample members.o Follow-up reminders sent to sample memberso 396 respondents completed surveys for a response rate of 35 %
Question 1: My professional affiliation is best described as ?
2007 Percent of
Total (N=308)
2008 Percent of Total (N=285)
2009 Percent of
Total (N=397)
Staff 49 46.7 30.7
Faculty 18 7.7 11.8
Department Head 6 19.3 19.6
College Associate/Assistant Dean 16 15.4 15.1
College Dean or VP 5 7.7 9.3
University-level Administration 5 2.5 4.5
Other College-level Administration 5.0
Government -- 0.7 0.0
Non-Govt., Organization 0.0
Institutional Research 3.5
No answer 1 --
Analysis – fewer staff respondents in ‘09; greater number of faculty and administrator respondents in 2009 compared to 2007 & 2008.
# of Respondents by Institution Type
1862 – 1831890 – 651994 – 6PNLG – 109Private – 21Did not report -12
n=396
Question 2 2009 Respondents by Institution Type
n=384
Question 3 Indicate which of the following organizations, if any, is your
primary affiliation.
Organization Percentage of Responses
APS 10.7 %
AASCARR 9.6 %
AEA/ARD 1.3 %
BOHS 8.6 %
SAF 3.4 %
AAVMC 0.8 %
CAFCS 6.5 %
NAUFRP 4.7 %
APLU 16.1 %
USDA 1.8 %
Other 10.2 %
Does not apply 26.3 %
Question 45Years in your current position?
N=383
Questions 43 & 44 Demographics
Female 55.70 %
Male 44.30 %
Asian 1.60 %
African American/ Black 8.80 %
American Indian / Alaskan Native 0.50 %
Hispanic 0.80 %
Native Hawaiian/ other Pacific Islander
0.50 %
White 85.20 %
Other 2.60 %
Three-Year Trends: Question 4 : My total experience using FAEIS is ?
Analysis – Users with > 3 yrs experience is trending upward while the other two groups are trending down.
Question 5My Involvement with FAEIS Includes:
(select as many as apply; all possible combinations displayed)
N=396
2009 Results
Three-Year Trends:Question 5: Involvement with FAEIS of Survey Respondents
( “ none” and “ no answer” responses eliminated in ’07 & ‘08 )
Analysis – Data Entry and Product Developer survey respondents trending downward; Users of FAEIS products is trending upward.
Three-Year Trends: Question 6: How often have you used FAEIS in the past 12 months ?
Analysis –The more frequent use categories (3-4 times & >4 times) are trending downward. The less frequent use categories (not at all & 1-2 times) are trending upward.
Three-Year Trends:Question 7: “My level of adoption (use) of FAEIS is best described as:”
(no answer – eliminated from analysis in ‘07 & ‘08)
Analysis – Only “unaware” and “use it but not sure of long-term use” are trending upwards. “Do not use” trending downward.
“ My level of adoption (use) of FAEIS is best described as:” (no answer – eliminated from analysis in ‘07 & ‘08)
2007Mean
2008Mean
2009Mean
Data Entry Users Only 2.49(N=159)
2.30(N=152)
2.32(N=157)
All Other Users 3.16(N=117)
3.24(N=125)
2.93(N=240)
Total 2.78(N=276)
2.73(N=277)
2.69(N=396)
1=unaware
2=aware of it but do not use it
3=Use it but not sure of long term use
4=Use it and it is integral to my job
5=Integral to my job and expect long term use
6=So important as to never be without FAEIS
Analysis – Average level of adoption of FAEIS decreased in ’09 for non-data entry respondents
Question 21Who are the users of FAEIS data at your institution/organization/agency ?
Figure 13. Users of FAEIS at Survey Respondent
Institutions/Organizations/Agencies
1.1%
16.6%
2.4%
39.2%
40.6%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Policy makers
Other
Faculty/researchers
Do not know
University administrators
n=391
Question 9Is FAEIS collecting data from the institutions which you consider
your peers?
n=395
n=271
Question 10If you answered no to:
“Is FAEIS collecting data from the institutions which you consider your peers?,” Which Institutions are missing ?
o Private schools with agriculture programs o Family and Consumer Sciences programs at NLG
Question 11Do a sufficient number of institutions report data to allow for you to
have a meaningful and helpful peer analysis?
n=389
n=210
Three-Year Trends: Question 13: Describe your level of satisfaction with FAEIS components.
(i.e., newsletter, help desk, data entry, report builder, instructions) (1=not satisfied; 5 = very satisfied)
Analysis – Higher and lower satisfaction ratings are trending downward. The neutral rating (3) is trending upward.
Question 14How useful to you are the following FAEIS products and resources?
Figure 9. Respondent Ratings of Utility of FAEIS Products and Resources
5.8
6.6
8.6
5.3
6.1
8.1
7.6
3.5
5.3
5.3
6.8
5.1
7.3
5.1
24.7
30.1
11.9
23.7
18.2
9.1
11.9
18.9
13.4
8.1
19.7
11.6
5.3
7.3
0.0 10.0 20.0 30.0 40.0
FAEIS website
FAEIS electronic newsletter
FAEIS data CD
Classification of Instructional Programs (CIP) code lists
Resource links on the FAEIS website
Teaching Workshop and Awards website
USDA/Science and Education Resources Development(SERD) Funded Projects information
Not Useful At All Not Very Useful Somewhat Useful Very Useful
Highest Ratings:
o CIP listso Newslettero Web site
Lowest Ratings:
o Teaching Awards / Workshops websiteo FAEIS Data CD
n=396
Question 14 hAre there any other FAEIS products or resources you would like to
add to the list ?
Some Responses:
o Salary data for placement data o Trend data such as degree programs that are changing over a 10 year cycle
o Summary of IPEDS major trends that influence food and agricultural sciences o Similar data from 1994 land grant universities obtained from sister institutions.
Question 8 Does the CIP system adequately allow you to classify your
instructional programs and/or address your reporting needs?
n=392
Question 12What aspect of FAEIS is most important to you ?
Most Common Responses: o Student Enrollment data
o Comparisons between institutions
o Trends in enrollment, degrees awarded
o Faculty salaries
Question 15For which of the following purposes have you used FAEIS data
and/or reports? ( Please check all that apply)
N=393
2009 Results
Analysis – Benchmarking and trend analysis are the two most common uses of FAEIS data.
Question 16 Please rate the usefulness of the data resources in FAEIS (for student
enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.) (1 = not useful, 5 = very useful)
2009 Results
N=397
Three-Year Trends: Question 16 : Please rate the usefulness of the data resources in
FAEIS (for student enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.).
Analysis – The higher usefulness scores (4, 5) are trending upward. The lower usefulness scores (1, 2) are stable & trending downward.
Question 17Please indicate how much confidence you have in the following
FAEIS data resources.
Table 15. Respondent Confidence in Types of FAEIS Data
Not Confident
At AllNot Very Confident Neutral
Somewhat Confident
Very Confident
Do Not Know
Total college enrollment 2.3 3.1 19.4 28.2 26.7 20.2
Student enrollment by CIP 2.6 6.5 18.8 29.8 19.6 22.7
Student demographics 1.8 3.4 20.7 32.1 21.5 20.5
Degrees awarded by CIP 2.3 3.4 20.0 29.9 22.3 22.1
Student placement by CIP 2.6 8.1 24.4 26.0 15.1 23.9
Faculty headcounts 1.3 3.1 19.8 28.9 25.3 21.6
Faculty demographics 1.3 2.9 21.1 27.9 24.7 22.1
Faculty salaries 1.8 4.1 20.7 28.2 23.1 22.0
International programs 1.3 3.4 26.2 24.1 15.3 29.8
1st
3rd
2nd
= Least confident
= Most confident n=386
Question 18Is FAEIS the primary place you go to seek higher education data and other information relating to the food and agricultural sciences, natural resources
and conservation, and family and consumer sciences/human sciences ?
n=391
Question 19Do you use any of the following databases ?
Figure 12. Databases Utilized by FAEIS Survey Respondents(Percentage of Respondents Selecting Database on Survey)
11.9
14.6
17.7
17.9
22.7
44.9
0 5 10 15 20 25 30 35 40 45 50
Other
Oklahoma State Faculty Salary Data
Department of Education NationalCenter for Education Statistics (NCES)
College and University ProfessionalAssociation for Human Resources
National Science Foundation (NSF) -Survey of Earned Doctorate Data
Food and Agricultural EducationInformation System (FAEIS)
Other:o CRIS, o NAUFRP, o USDA databases n=280
Question 20To what extent does FAEIS add value to other information sources that
are available ?
n=384
Question 24 Have you contacted the FAEIS Help Desk for assistance ?
Figure 14. Contact with FAEIS Help Desk Among
Survey Respondents
Yes, emailed the help desk
14.4%
Yes, telephoned the help desk
11.3%
No, have not contacted the help
desk61.6%
Yes, telephoned and emailed the
help desk12.6%
n=388
Question 25How would you rate the support you received from the Help Desk ?
Figure 15. Respondent Ratings of FAEIS Help Desk
Excellent62.1%
Good29.4%
Poor0.7%
Fair5.2%
Do not know2.6%
n=153
Question 26If you have contacted the FAEIS Help desk , how could its operation be
improved ? (n=146)
22 % - No improvement needed. 1 % - Increase hours of operation 0.3 % - Improve email response time12.4 % - Do not know 1.8 % - Other suggestions:
o Help Desk staff is limited by the lack of thorough, well-edited, accurate and consistent documentation, web screen content and communications.o Information provided by the Help Desk is not consistent from time-to-time.o Develop databases more thoroughlyo Help Desk employees need better trainingo Clear understanding of the parameters of the data being enteredo More understanding of the difficulties institutions are having entering the data
Question 27How would you rate the ease of navigation of the FAEIS web site?
Figure 16. Respondent Ratings of Ease of Navigating the FAEIS Web Site
Very difficult1.3%
Difficult7.4%
Neither difficult nor easy34.9%
Easy26.2%
Very easy5.4%
Do not know24.9%
n=390
Question 28Have you used customized FAEIS data reports produced by the FAEIS
Report Builder application ?
Figure 17. Levels and Types of Usage of the FAEIS Report Builder Application for Customized Data Reports
No, I have not used any
customized FAEIS data
reports70.7%
Yes, I developed the customized FAEIS reports
myself9.7%
Yes, my staff have developed
customized reports for me
2.3%
Yes, the FAEIS Help Desk
developed the customized
reports for us4.1%
Do not know6.4%
Does not apply6.9%
n=393
Question 29 Have you personally used the FAEIS Report Builder application ?
n=390
Question 30 How well does the FAEIS Report Builder meet your needs ?
Figure 18. Survey Respondent Ratings of How Well the FAEIS Report Builder Meets Their Needs
Excellent11.3%
Do not know5.6%
Poor5.6%
Fair25.4%
Good52.1%n=71
Question 31How easy is it to use the FAEIS Report Builder ?
Figure 19. Respondent Ratings of the Ease of Use of the FAEIS Report Builder (Percentages Reporting)
51.4
24.3
17.1
2.9
2.9
1.4
0 10 20 30 40 50 60
Very difficult
Very easy
Do not know
Difficult
Easy
Neither difficult nor easy
n=70
Question 32Are the instructions for the FAEIS Report Builder clear ?
Figure 20. Percentages of Survey Responses For Survey Item:
"Are the Instructions for the FAEIS Report Builder Clear?"
Yes59.2%
No23.9%
Do not know16.9%
n=71
Question 33Would training on how to use the FAEIS Report Builder be useful to you?
Figure 21. Percentage of Responses to "Would Training on How to Use the FAEIS Report Builder be Useful to You?"
No26.8%
Yes54.9%
Do not know18.3%
n=71
Question 34Do you enter FAEIS data, or are you responsible for data
entered into FAEIS ?
n=389
Question 35How would you rate your ability to obtain the following data in the format
required by FAEIS ?
Enrollments by degree level, discipline,
gender, and ethnicityDegrees awarded by degree level, discipline, gender, and ethnicity
N=262 N=260
Question 35How would you rate your ability to obtain the following data in the format
required by FAEIS?
Faculty by discipline, rank, gender, and
ethnicityPlacement by degree level into various CIP
disciplines and work categories
N=257 N=260
Question 36 Are data on placement of students collected at your institution ?
Figure 22. Survey Findings Related to Collection of Student Placement Data at Respondent Institutions (Respondents Could Select All Response Options
That Applied)
1.1%
1.5%
1.9%
13.7%
16.0%
22.5%
26.3%
30.2%
Yes, data collected but not available
Does not apply
Other
Do not know
Yes, other office in college collects these data
No, placement data are not collected
Yes, our office collects these data
Yes, another office at institution collects these data
N=134
Question 37At your institution, from where do you acquire the data for the various
surveys ?
Figure 23. Sources Within Institution for Acquiring Data for Various Surveys (Respondents Could Select All Survey Response Options That Applied)
18%
15%
32%
1%
18%
16%
28%
1%
12%
21%21%
15%
1%
5%
22%
14%
9%
0%
12%13%
10%
17%
12%11%
0%
0%
5%
10%
15%
20%
25%
30%
35%
Department College Institutional Research Human Resources Other Office onCampus
Student Enrollment Degrees Awarded Faculty Student Placement International Programs
N=257-261
Question 38What time of year would be best to enter the following types
of FAEIS data ?
Figure 25. Respondent Perceptions of Best Time of Year to Enter FAEIS Data by Type of Data
19.2%
11.6%
6.1%
13.4%
8.6%
6.1%
10.1%
25.5%
15.4%
8.6%
5.8%
15.4%
8.3%
5.8%
6.6%
21.0%
6.8%
5.1%
3.8%
12.6%
0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0%
Fall
Winter
Spring
Summer
Student enrollment Degrees Awarded Faculty Student Placement International programs
N=253-260
Question 9What data entry method do you use for the following surveys ?
N=244-256
Figure 24. Respondent Institution Data Entry Methods for Different Data Types
12%
6%
9%
6% 5%6%
10%
6% 7%
4%
13%
26%
15%
21%
13%
15%
20%
24%
36%
15%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Student enrollment Degrees Awarded Faculty Student Placement International programs
Online data entry Spreadsheet data entry Email Telephone call to FAEIS help desk
Question 41Are the instructions clear for FAEIS data entry methods that you use ?
N=240
If answered no, which instructions need to be improved?
o Would prefer questionnaire by mailo Navigation on screen a paino Difficult to discern how to move from in-progress to completedo Finishing an entry is difficulto Difficult to navigate and decide when to saveo Not sure what majors you want to be collectedo Submission instructions are vagueo Time frames for info requested needs to be stated clearly
Question 22How can FAEIS be more useful to your institution or be improved ?
Most Common Responses:
o Data input is cumbersome, needs to be easier, bothersome. o Need more information about what FAEIS is and what it can do for us.
o I need to use it more. o Concerned about the accuracy and credibility of the data.
o Complaints about CIP codes
Question 23 Please share any comments about when FAEIS was helpful to you or
about when you had a problem with FAEIS.
Most common “Helpful” Responses:
o Staff was helpful. (21)o Useful for faculty salary comparisons.(4)o Helpful in grant boilerplates (1)o Student diversity by CIP info (1)o Dropping / adding programs (1)o Growth of programs, trends in majors (2)o Long-term planning (1)
Most Common “Problem” Responses:
o Data not dependable or accurate.(6)o Data entry not user friendly (5)o Report builder not user friendly. (2)o Poor job of marketing / facilitating use of FAEIS (1)
SUMMARY
o The more years using FAEIS the greater the satisfaction with FAEISo Greatest satisfaction of FAEIS data and its use: enrollment, institutional benchmarking, trend analysis, salary comparisons o Greatest satisfaction of FAEIS products: CIP code list, e-newsletter, websiteo FAEIS report builder—good, but room for improvemento Most confident in FAEIS data: college enrollment, faculty headcount, student/faculty demographics, degrees awarded: least confident: international/student placemento Most users believe FAEIS adds-value & is used twice as often as any other similar databaseo Most users have not contacted help desk; but of those that have---90% rated it excellent or good.o Users get FAEIS data from many places—Depts/Colleges/IR/HR