2006 2007 Assessment Report - ecs.csus.edu index/assessment report.pdf · 6/25/2007 · H....

2006‐2007 Assessment Report

Department of Computer Science

California State University,

Sacramento

Du Zhang, Department Chair

Assessment Committee: Robert Buckley

Mary Jane Lee, Coordinator William Mitchell

Anne-Louise Radimsky Richard Smith

Submitted June 25, 2007

2

Option 1: Narrative Submission 1. What goals or learning objectives/outcomes were assessed in AY 2006-2007?

Objectives are desirable characteristics/abilities of our B.S. degree graduates five or more years after graduation. The objectives of the Computer Science program are listed in Attachment A. Student learning outcomes or program outcomes are skills/capabilities of students at the time of graduation. Program outcomes are listed in Attachment B. The objectives/outcomes matrix is provided in Attachment C.

Assessment of Objectives

On March 27, 2007, the Computer Science (CSC) Industry Advisory Committee (IAC) members were emailed a survey to complete. Members were asked to list three characteristics of a successful employee five or more years after completion of a B. S. CSC degree in each of three categories of essential, important, and desirable characteristics. The purpose of this survey was to determine how well the objectives of IAC members matched the objectives specified by Computer Science faculty. At the IAC meeting on April 20, 2007, members discussed their recommended characteristics/abilities. A total of nine IAC members responded to the survey in writing or at the meeting. Characteristics mentioned by at least three members are listed below with frequencies in parentheses:

Characteristics/Abilities CSC Objective or Outcome Essential

Communication skills and interpersonal skills (7) Objective #5, #6 Analytical skills (6) Outcome J Team player (4) Objective #2

Important Relevant internship or part-time work experience (5) None

Desirable Knowledge of subject area (6) Objective #1

Ethical and moral values (3) Objective #7

IAC members matched five of the seven objectives of the Computer Science faculty. Two objectives, #3 To pursue professional development opportunities or graduate studies and #4 To assume a leadership role in a chosen career and profession, were not mentioned by IAC members. One IAC response, relevant internship/work experience, though important, is not considered to be a program objective or outcome. We encourage students to participate in internships and co-op programs as part of their college education. Currently two units of work experience counts toward the degree as experiential units. In general, we believe that the list of objectives of IAC members matched reasonably well with those defined by the Department.

3

Assessment of Student Learning Outcomes All student learning outcomes (see Attachment B and listed below) were assessed this year: A. An ability to apply knowledge of computing and mathematics appropriate to the discipline. B. An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution. C. An ability to design, implement, and evaluate a computer-based system, process,

component, or program to meet desired needs. D. An ability to function effectively on teams to accomplish a common goal. E. An understanding of professional, ethical, legal, security, and social issues and

responsibilities. F. An ability to communicate effectively with a range of audiences. G. An ability to analyze the local and global impact of computing on individuals,

organizations, and society. H. Recognition of the need for, and an ability to engage in, continuing professional

development. I. An ability to use current techniques, skills, and tools necessary for computing practice. J. An ability to apply mathematical foundations, algorithmic principles, and computer

science theory in the modeling and design of computer-based systems in a way that demonstrates comprehension of the tradeoffs involved in design choices.

K. An ability to apply design and development principles in the construction of software systems of varying complexity.

2. How were these learning outcomes assessed? a. Describe the measures you used and the information gathered. (Description, date administered, results)

CSC 195 Field Work in Computer Science and CSC 195A Professional Practice A survey was completed by supervisors at the end of Fall 2006 and Spring 2007 semesters. Outcomes C, D, E, and F were assessed. A summary of survey results appear in Attachment D.

Responses by supervisors indicate average, above average, and outstanding student performance in these outcomes. There was only one below average response and that appeared in outcome F - effective oral communication. In general, other than in the category of oral communication, employers are satisfied with the performance of our junior and senior-level majors who participated in CSC-related internships and work experiences. CSC 131 Computer Software Engineering Student oral presentations were assessed by Professors Lee, Radimsky, C. Zhang, and D. Zhang on May 23, 2007. Outcomes F and J were assessed. Summary results are provided in Attachment E.

4

Mean scores of 2.75 and above (out of a 4 point scale) were considered to be satisfactory. Students demonstrated competency in presentation organization, use of audio-visual materials, responding to questions, appropriate vocabulary, correct grammar, application of appropriate methodologies and techniques, and understanding of the problem. Areas in need of improvement include (1) personal appearance and (2) delivery of presentation (speaks clearly and distinctly, speaks to the audience, uses techniques to engage the audience, and communicates an interest in and enthusiasm for the material presented). CSC 190 Senior Project: Part I Oral presentations were assessed by Professors Lee, Mitchell, Radimsky, Rao, and Smith on May 7 and May 14, 2007. Outcomes F and J were assessed. Summary results are provided in Attachment F. Mean scores of 3.0 and above (out of a 4 point scale) were considered to be satisfactory. Students demonstrated competency in the assessed categories of presentation organization, language, and understanding of the problem. In the category of presentation delivery, there is a need for improvement in (1) uses techniques that engage the audience and (2) communi-cates an interest in and an enthusiasm for the material presented. In addition, improvement is needed in applying appropriate methodologies/techniques. CSC 191 Senior Project: Part II Final senior project oral presentations were assessed by Professors Lee, Mitchell, Radimsky, Rao, and Smith on May 9 and 16, 2007. Outcomes F and J were assessed. Summary results are provided in Attachment G. Mean scores of 3.0 and above (out of a 4 point scale) were considered to be satisfactory. Students demonstrated competency in the assessed categories of presentation organization, language, and understanding of the problem. In the category of presentation delivery, improvement is needed in (1) speaks clearly and distinctly and with sufficient volume, (2) speaks to the audience, and (3) uses techniques that engage the audience. Similar to CSC 190 results, appropriate application of methodologies/techniques also needed improvement. Accenture (Information Technology and Business Consultants) – Industry visit to CSUS on May 22, 2007. Accenture employers who participated include one 2002 UCD Liberal Arts graduate and four recent (2004, 2005) CSUS CSC graduates. Accenture representatives discussed and completed a survey assessing all outcomes. Survey results are provided in Attachment H. Preliminary analysis of responses indicate that our graduates perform quite well in outcomes A, B, C, D, H, I, J, and K and satisfactory in outcomes E (An understanding of professional, ethical, legal, security, and social issues and responsibilities), F (An ability to communicate effectively with a range of audiences), and G (An ability to analyze the local and global impact of computing on individuals, organizations, and society). Respondents stated that greater experience with real-world problems would assist graduates with outcome G.

5

b. As a result of these assessments, what did you learn about the program’s success in helping its students achieve these learning outcomes?

From the perspective of employers and faculty, our students have been relatively successful in achieving expected learning outcomes.

c. In what areas are students doing well and achieving expectations?

Employers believe our recent graduates are competent in almost all outcomes.

d. What areas are seen as needing improvement within your program?

Improvement is clearly needed in outcome F: effective communication, particularly in oral communication.

3. As a result of faculty reflection on these results, are there any program changes anticipated?

a. If so, what are those changes?

The Department plans to provide an instructional module on effective oral presentation in CSC 131 and in CSC 190/191 during the Fall 2007 semester. It will focus on effective presentation delivery techniques. b. How will you know if those changes achieve the desired results?

The Department will administer the oral presentation rubric again after implementation of an instructional module on oral communication in CSC 131, 190, and 191 classes in the next academic year. While the Department hopes to see improvement in oral presentation for all three classes in Fall 2007, we expect to see improvement in the CSC 191 assessment compared to the CSC 190 results from Spring 2007.

4. Did your department engage in any other assessment activities, such as, the development of rubrics, course alignment?

A rubric was developed for oral communication and implemented in CSC 131, 190, and 191 last semester. See attachments E, F, and G. The faculty may refine this instrument before administering it in the same courses this Fall.

5. What assessment activities are planned for the upcoming academic year?

(a) With assistance from Communications Studies, the Department will develop an instructional module for at least one lecture covering techniques for effective oral communication with emphasis on delivery.

6

(b) The Department plans to consult with Prof. Dan Melzer, Writing and Reading Coordinator, English Department, to develop and implement an assessment tool for evaluating written communication.

(c) While current assessment efforts have focused on employer and faculty surveys of

student learning outcomes, this coming academic year, the Department plans to develop direct methods to assess CSC core concepts. A matrix of student learning outcomes and CSC core courses was developed in Spring 2007. The Assessment Committee and the Undergraduate Curriculum Committee will work with the faculty to decompose student learning outcomes into specific CSC core concepts to be assessed.

7

Attachment A

Department of Computer Science Program Objectives

1. To be a significant contributor in one or more of the aspects associated with the

development, maintenance, and support of real world computing systems. 2. To be an effective and contributing member of a project team. 3. To be continually engaged in the pursuit of professional development opportunities,

and/or pursue postgraduate studies. 4. To assume a leadership role in their chosen career and profession. 5. To demonstrate competence in producing high quality written documents, and in

reviewing and revising those prepared by others. 6. To demonstrate competence in developing and delivering technical and non-technical

presentations of high quality to a variety of audiences. 7. To abide by the ethical standards of the profession and understand the social implications

of their professional activities.

8

Attachment B

Department of Computer Science Student Learning Outcomes

ABET Program Outcomes

A. An ability to apply knowledge of computing and mathematics appropriate to the discipline. B. An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution. C. An ability to design, implement, and evaluate a computer-based system, process,

component, or program to meet desired needs. D. An ability to function effectively on teams to accomplish a common goal. E. An understanding of professional, ethical, legal, security, and social issues and

responsibilities. F. An ability to communicate effectively with a range of audiences. G. An ability to analyze the local and global impact of computing on individuals,

organizations, and society. H. Recognition of the need for, and an ability to engage in, continuing professional

development. I. An ability to use current techniques, skills, and tools necessary for computing practice.

CAC Program Outcomes for Computer Science

J. An ability to apply mathematical foundations, algorithmic principles, and computer science theory in the modeling and design of computer-based systems in a way that demonstrates comprehension of the tradeoffs involved in design choices.

K An ability to apply design and development principles in the construction of software systems of varying complexity.

9 Attachment C

Objectives/Outcomes Matrix

6/24/2007 CSC ASSESSMENT AND EVALUATION Outcomes for Program Outcomes Computer Science

Program Objectives A B C D E F G H I J K

1 x x x x x x x x 2 x x x x 3 x x x 4 x x x x x x x x 5 x x x 6 x x x x x x 7 x x

CSC Department Program Objectives (Broad statements that describe career & professional accomplishments) 1. To be a significant contributor in one or more of the aspects associated with the development, maintenance, and support of real world computing systems. 2. To be an effective and contributing member of a project team. 3. To be continually engaged in the pursuit of professional development opportunities, and/or pursue postgraduate studies. 4. To assume a leadership role in their chosen career and profession. 5. To demonstrate competence in producing high quality written documents, and in reviewing and revising those prepared by others 6. To demonstrate competence in developing and delivering technical and non-technical presentations of high quality to a variety of audiences. 7. To abide by the ethical standards of the profession and understand the social implications of their professional activities. ABET Program Outcomes for Accrediting Computing Programs (What students are expected to know at time of graduation) A. An ability to apply knowledge of computing and mathematics appropriate to the discipline B. An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution C. An ability to design, implement, and evaluate a computer-based system, process, component, or program to meet desired needs D. An ability to function effectively on teams to accomplish a common goal E. An understanding of professional, ethical, legal, security, and social issues and responsibilities F. An ability to communicate effectively with a range of audiences

10 G. An ability to analyze the local and global impact of computing on individuals, organizations, and society H. Recognition of the need for and an ability to engage in continuing professional development I. An ability to use current techniques, skills, and tools necessary for computing practice. CAC Program Outcomes for Computing Sciences (page 17) J. An ability to apply mathematical foundations, algorithmic principles, and computer science theory in the modeling and design of

computer-based systems in a way that demonstrates comprehension of the tradeoffs involved in design choices K. An ability to apply design and development principles in the construction of software systems of varying complexity.

11

Attachment D

COMPILATION OF SUPERVISOR EVALUATIONS OF STUDENT INTERNSHIPS CSC 195/195A

Semester: Fall 2006 N = 27

Ability to develop a computerized solution to a real life problem using appropriate tools: Outstanding Above Average Average Below Average Weak Did Not Observe

14 9 4 Ability to function as a team member: Outstanding Above Average Average Below Average Weak Did Not Observe

14 12 1 Effective oral communication: Outstanding Above Average Average Below Average Weak Did Not Observe

10 10 6 1

Effective written communication: Outstanding Above Average Average Below Average Weak Did Not Observe

10 9 7 1 Appropriate use of presentation tools: Outstanding Above Average Average Below Average Weak Did Not Observe

8 6 4 9 Awareness of ethical and societal concerns: Outstanding Above Average Average Below Average Weak Did Not Observe

14 8 4 1 Additional Comments:

• “[He] has excellent leadership and team building skills… [H]e offered valuable insights, ideas, and problem-solving skills. He has been an asset to our team, contributing to the success of our project.”

• “[He] is an outstanding employee and person and I hope we have the opportunity to work together in the future.”

• “[He] came in with critical skills and has been able to deliver solutions in a timely manner, especially impressive for a new hire. He gets along well with other team members, is willing to ask questions and share solutions. He is conscientious and works hard to get his assignments done.”

• “[He] is a wonderful person to work with.”

12

• “[He] has been a great contributor to desktop and enterprise-wide solutions. He has good ideas and problem-solving skills; he is well-recommended.”

• “Works well in a team environment with frequently changing requirements. Friendly attitude and good work ethic. Needs to be more self-motivated to find solutions to problems.”

• “[He] has been an excellent addition to our team. His contributions are on par with an experienced SW engineer.”

• “We’ve really enjoyed [him]!!!!” • “I am very proud to work with [him]. He has proven to be an asset in the workplace.” • “Although [he] worked independently for the most part, he made full use of team

resources for knowledge and advice. He was able to complete all tasks assigned to him in a very timely fashion, and his work was professional grade. [He] is a pleasure to have on the team.”

• “[He] was given several tasks over his internship period including updating tools with additional features and software (driven) replication. These tasks were executed and completed by [him] successfully without much supervision. He was always seeking areas that he could contribute to add value to the group. I would say, he is a self-starter, good planner who can handle multitasks by having good prioritization skill. Overall, he had [a] very successful internship with Intel Corporation, and we really appreciated his hardworking effort and time.”

• “[He] has outstanding ability to juggle multiple tasks under distraction and stay focused. He is aggressive and enthusiastic about embracing new technologies and solutions. Great problem-solver and great at balancing opposing challenges, e.g. cost vs. benefit.

• “[He] has demonstrated strong technical skills toward instrumenting [sic] the software and collecting code coverage data.”

13

COMPILATION OF SUPERVISOR EVALUATIONS OF STUDENT INTERNSHIPS CSC 195/195A

Semester: Spring 2007 N = 14

Ability to develop a computerized solution to a real life problem using appropriate tools: Outstanding Above Average Average Below Average Weak Did Not Observe

8 2 2 2 Ability to function as a team member: Outstanding Above Average Average Below Average Weak Did Not Observe

6 5 3 Effective oral communication: Outstanding Above Average Average Below Average Weak Did Not Observe

3 3 8

Effective written communication: Outstanding Above Average Average Below Average Weak Did Not Observe

6 4 4 Appropriate use of presentation tools: Outstanding Above Average Average Below Average Weak Did Not Observe

5 2 4 3 Awareness of ethical and societal concerns: Outstanding Above Average Average Below Average Weak Did Not Observe

6 3 4 1 Additional Comments:

• “[He] was so good we offered him a job! I was extremely pleased with [his] performance. A+.”

• “I’m not being nice – he earns those marks.” • “[She] is a good team player, we value her contributions!” • “A very impressive future Intel engineer! [He] was very easy to direct, and executed

his tasks at a higher level than expected.” • “He has been extremely effective. His ‘average’ rating in Ability to function as a

team member is solely due to a reluctance to take initiative to meet and engage with other team members. I have already seen positive signs that [he] is improving in this.”

• “During his internship… [he] consistently meets expectations …[He] works hard and finished tasks assigned to him.”

14

Attachment E MEAN SCORES OF FACULTY RATINGS OF ORAL PRESENTATIONS

Instructor: Salem CSC Course: CSC 131 ` May 23, 2007 Total N* = 21 Assessors: Lee, Radimsky, C. Zhang, D. Zhang *Total number of evaluations submitted. Responses of “n/a” are not included in calculations.

Exce

llent

Com

pete

nt

Nee

ds w

ork

Una

ccep

tabl

e

Not

App

licab

le

ORAL PRESENTATION Mean Score

Organization

4 3 2 1 n/a Includes introduction, body, and conclusion. 3.0

4 3 2 1 n/a Organizes content logically and sequentially. 2.9

4 3 2 1 n/a Presents ideas and arguments logically and clearly. 2.9

4 3 2 1 n/a Cites sources appropriately. 2.0 (3 people said “n/a” for all teams)

Delivery

4 3 2 1 n/a Speaks clearly, distinctly, and with sufficient volume. 2.5

4 3 2 1 n/a Speaks to audience (e.g., does not just read the material and/or talk to screen). 2.7

4 3 2 1 n/a Uses audio-visual materials appropriately. 2.8

4 3 2 1 n/a Uses techniques that engage the audience. 2.5

4 3 2 1 n/a Communicates an interest in and an enthusiasm for the material presented. 2.7

4 3 2 1 n/a Time permitting, responds effectively to questions and comments. 3.1

Language

4 3 2 1 n/a Uses technical vocabulary appropriate for the audience. 2.9

4 3 2 1 n/a Uses correct grammar. 2.9

Appearance

4 3 2 1 n/a Personal appearance (attire, posture, and body language) is appropriate. 2.7

Exce

llent

Com

pete

nt

Nee

ds w

ork

Una

ccep

tabl

e

Not

App

licab

le

PROJECT TECHNICAL CONTENT Mean Score

Methodologies & Techniques

4 3 2 1 n/a Applies methodologies and techniques appropriate for the project. 3.0

4 3 2 1 n/a Demonstrates a good understanding of the problem area. 3.1

15

Attachment F MEAN SCORES OF FACULTY RATINGS OF ORAL PRESENTATIONS

Instructor: Buckley CSC Course: CSC 190 ` May 7 & 14, 2007 Total N* = 21 Assessors: Lee, Mitchell, Radimsky, Rao, Smith *Total number of evaluations submitted. Responses of “n/a” are not included in calculations.

Exce

llent

Com

pete

nt

Nee

ds w

ork

Una

ccep

tabl

e

Not

App

licab

le


Organization




4 3 2 1 n/a Cites sources appropriately. 3.4

Delivery







Language



Appearance


Exce

llent

Com

pete

nt

Nee

ds w

ork

Una

ccep

tabl

e

Not

App

licab

le





16

Attachment G MEAN SCORES OF FACULTY RATINGS OF ORAL PRESENTATIONS

Instructor: Buckley CSC Course: CSC 191 ` May 9 &16, 2007 Total N* = 20 Assessors: Lee, Mitchell, Radimsky, Rao, Smith *Total number of evaluations submitted. Responses of “n/a” are not included in calculations.

Exce

llent

Com

pete

nt

Nee

ds w

ork

Una

ccep

tabl

e

Not

App

licab

le


Organization




4 3 2 1 n/a Cites sources appropriately. 3.0 (4 people said “n/a” for all teams)

Delivery







Language



3.0


Exce

llent

Com

pete

nt

Nee

ds w

ork

Una

ccep

tabl

e

Not

App

licab

le





17

Attachment H

2006 2007 Assessment Report - ecs.csus.edu index/assessment report.pdf · 6/25/2007 · H....

Documents

Transcript of 2006 2007 Assessment Report - ecs.csus.edu index/assessment report.pdf · 6/25/2007 · H....