FAEIS Project User Opinion Survey 2005 Thursday, June 23, 2005 Washington, D.C. H. Dean...

FAEIS ProjectFAEIS ProjectUser Opinion Survey User Opinion Survey

20052005

Thursday, June 23, 2005Washington, D.C.

H. Dean Sutphin Yasamin MillerProfessor Director, SRI

Agriculture & Extension Education Cornell UniversityVirginia Tech University [email protected]

[email protected] www.sri.cornell.edu

Food and Agricultural EducationFood and Agricultural EducationInformation System (FAEIS) ProjectInformation System (FAEIS) Project

User Opinion Survey 2005User Opinion Survey 2005


20052005

• To understand from the FAEIS users, their experiences with the system and its usefulness

• To gather input from FAEIS users in higher education on improvements and changes that are needed in the future

Purpose


20052005

1. Solicit input about the user friendliness of FAEIS

2. How well is FAEIS meeting its goal of supporting higher education in food, agriculture, and natural resources sciences

3. How well is FAEIS meeting its goals to support the national network of universities

4. Determine the comprehensive view of current and future use of FAEIS as reported by feedback from expert in the field

Objectives


20052005

5. Determine the effectiveness of FAEIS and future direction in terms of context, input, process, and product

5.1 CONTEXT

Includes the elements of the university system in which FAEIS is operating including alignment with needs of the national network of universities, reliability of FAEIS, and interactions that university personnel have with FAEIS

Objectives


20052005

5.2 INPUT

Asks the user to consider the human, fiscal, and other resources required for the FAEIS to operate within the university system such as commitment of staff time and variables related to data entry.

Objectives


20052005

5.3 PROCESS• The user is asked to evaluate process variables

(i.e. timing of data entry and help desk activity) required for development, maintenance, and continuous improvement.

Objectives


20052005

5.4 PRODUCT• Characteristics of project outcomes and uses of

the system such as quality of report generators, reliability of the system components, and need for further refinements are evaluated.

Objectives


20052005

Methodology• Survey Research using Case Study

Methodology

• Administer Survey Instrument to Population of FAEIS User Group

• Focus Group using Expert Panel


20052005

Structure of evaluation distributed to FAEIS panel of experts Fall 04 for review and reaction Review and refinement of structure based on input Decision to delay evaluation until Spring 05. The survey instrument was developed by SRI in collaboration with the FAEIS central project staff and USDA Higher Education administration responsible for the project.

Methodology


20052005

Population and Sample

1862 Land-Grant7791890 Land-Grant1351994 Land-Grant26Non-Land-Grant 419Other (US, USDA, Priv, Org) 57


20052005

1. Outlining overall survey objectives

2. Development of survey dimensions

3. Development of items within dimensions

4. Internal testing-administration of survey to individuals on the FAEIS administrative team

5. Revision of instrument based upon internal testing

Instrument Development


20052005

6. Pilot test - via web• N=35 individuals

10 pre-selected due to qualifications

25 randomly selected• Personalized invitation e-mails sent with unique

URL links on 5/26/05 • Reminder e-mail sent to non-respondents on

5/31/05• Total: 17/35 completions• Response rate: 48.6%



20052005

7. Revision of questionnaire based upon pilot results• Panel of experts used to determine instrument

validity• Variance in responses• Not all questions applicable to all respondent types• Additional response categories added, few minor

changes made• Conclusion: revised instrument was valid and

reliable and able to capture the desired information



20052005

8. Launching of tested instrument• Personalized invitation e-mail with unique URL

link sent on June 6, 2005

Data Collection


20052005

Number of Responses by Institution TypeJune 17

Institution Type Total Completions Refusals Response Rate

1862 Land-Grant 779 216 28 27.7%

1890 Land-Grant 135 25 3 18.5%

1994 Land-Grant 26 4 0 15.4%

Non-Land-Grant 419 99 10 23.6%

Other (US, USDA, Priv, Org) 57 8 3 14.0%

TOTAL 1,416 352 44 24.8%

Response Rate


20052005

• Send up to five reminder e-mails to non-respondents

• Identify institution types with low response rates and follow-up with reminder phone calls

Follow Up of Non Respondents


20052005

Control for Non Repondent Error

• Comparison of non respondent to respondents data

• Determine reason for lack of participation in survey


20052005

Indepth Analysis and Probing

• Use FAEIS Expert Panel and Focus Group

• Data Transcribed and Reported Internally

• Anonymity of Response


20052005

• Analyses to portray variables and comparison on dimensions of particular interest

• Conclusions

• Recommendations

• Executive Summary

Final Report


20052005

Who are the respondents:

352 respondents out of 1416 by June 17

• 82.4% use the FAEIS system and enter data• 4.4% do not enter data, but use the system• 13.2% do not use the system at all

(of these significantly more NLG (20.2%) and Other (62.5%) are non-users)

Preliminary Findings


20052005

Q: Can you obtain student placement data? (significantly more NLG (30%) cannot get student placement data)

Q: Are we collecting data from the institutions which you consider your peers?

Q: Do a sufficient number of institutions report data to allow for meaningful and helpful analysis?

Student Placement Data

52.6

36.0

17.65.1

7.4

29.8

42.5

56.6

52.4

0

10

20

30

40

50

60

70

80

90

100

Can obtain student placementdata

Collecting data from peerinstitutions

Sufficient number of institutionsreport

Per

cen

tag

e o

f re

spo

nd

ents

Do not know

No

Yes

Preliminary Findings: Context


20052005

Q: How would you rate your ability to obtain the following data in the format required by FAEIS:

Ratings of Ability to Obtain Data in the Format Required by FAEIS

9.13.5

31.029.6

11.4

26.628.7

21.9

7.9 8.7

23.2

11.4

21.0 21.728.5

11.812.8

32.7

23.5

6.6

2.21.71.3

23.0

0

10

20

30

40

50

60

70

80

90

100

Faculty discipline by rankby gender by ethnicity

Degrees awarded by levelof study by gender by

ethnicity

Enrollments by level ofstudy by gender by

ethnicity

Placement by level ofstudy into various sectors

Per

cen

tag

e o

f re

spo

nd

ents

Very easy Easy Neither difficult nor easy Difficult Very difficult Do not know

Preliminary Findings: Input


20052005

Q: Are the instructions in the FAEIS system clear?

Clarity of Instructionsin the FAEIS System

Yes, instructions

are clear83.0%

No, instructions

are not clear

17.0%

Preliminary Findings: Process


20052005

Q: Is the support offered by the Help Desk adequate?

Whether Support Offered byHelp Desk is Adequate

(among those who have contacted Help Desk)

Yes95.6%

No4.4%

Preliminary Findings: Process (continued)


20052005

Q: How useful is the information collected in FAEIS:Ratings of Usefulness of Different Types of Information

Collected in FAEIS

19.7 18.614.5 10.8

26.3

30.5 30.0 27.9

26.225.7

4.4 8.0 9.3 7.3

8.98.7

3.6 3.6 3.63.2

6.9

4.1

37.1 38.2 38.2 38.5 41.3 43.550.6

29.5 27.520.2

28.3

2.82.4

0

10

20

30

40

50

60

70

80

90

100

Degreesawarded

Enrollment Ethnicity Gender Faculty Placement ofgraduates

CIP

Per

cen

tag

e o

f re

spo

nd

ents

Very useful Somewhat useful Not very useful Not useful at all Do not know

Preliminary Findings: Product


20052005

Q: To what extend does FAEIS add value to other information sources that are available?

Value-Added of FAEIS Over Other Data Sources

11.6

8.010.4

32.5

37.3

0

10

20

30

40

50

High degree Moderate degree Low degree None Do not use orsearch otherdatabases

Per

cen

tag

e o

f re

spo

nd

ents

Preliminary Findings: Product (continued)


20052005

• 84.8% do not use Custom Report Builder



20052005

Q: Who are users of FAEIS data at your institution?

Users of FAEIS at Institution(among those using data; multiple users allowed)

36.0

1.5

24.1

62.1

4.4

0 10 20 30 40 50 60 70 80 90 100

Do not know

Other

Government officials

Faculty/Researchers

Universityadministrators

Percentage of respondents



20052005

Q: In general, to what extent is it useful for your institution to participate in the FAEIS database?

Usefulness of Institution's Participation in FAEIS Database

16.3

8.3

4.2

28.8

42.4

0

10

20

30

40

50

Very useful Somewhat useful Not very useful Not useful at all Do not know

Per

cen

tag

e o

f re

spo

nd

ents



20052005

Preliminary Thoughts on Conclusions and Recommendations

• Need further analyses to explain the high rate of “don’t know”. Is this response isolated to a particular population? How can we correct this lack of information?

• Preliminary data show a positive view of FAEIS on most variables• Need to determine why it is more difficult for non land grants to

access data• Custom report builder is in early stages of release and current data

may not reflect full potential


20052005

Preliminary Thoughts on Conclusions and Recommendations

• FAEIS seems to be satisfactory in terms of comprehensive elements involving context, input, process, and product

• Respondents were not forthcoming with significant new and innovative future development recommendations

• FAEIS seems to be meeting its goals in terms of user friendly and meeting needs of the university community.


20052005

Final Report Format

• Are there recommendations for the format, style or types of analyses?

• Are there recommendations on developing the web report or the written report?

• Any suggestions on dissemination of findings?


20052005

Your thoughts – opinions?

Questions


20052005

Thank you.

FAEIS Project User Opinion Survey 2005 Thursday, June 23, 2005 Washington, D.C. H. Dean...

Documents

Transcript of FAEIS Project User Opinion Survey 2005 Thursday, June 23, 2005 Washington, D.C. H. Dean...