Running head: OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 1

Objective Two – Data Collection and Analysis – Educational

Instructional Systems Design: Training Evaluation Strategies

Kayla Conklin

McDaniel College

OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 2

Objective Two – Data Collection and Analysis – Educational

Instructional Systems Design: Training Evaluation Strategies

The Instructional Systems Design (ISD) course required me to design a complete training

program that was relevant to my workplace. My design plan of the final project was chosen as an

educational artifact to demonstrate my competency in objective two of McDaniel College’s

Human Resources Development (HRD) program. This reflection will focus on my choice of

evaluation strategies which can be found on pages 3-4 of the design proposal artifact. While

creating the evaluation strategies for the ISD course, I was not cognoscente of the reasoning

behind my data collection methods. I simply wanted to incorporate various techniques in order to

gather a wealth of data from participants. Looking back at my design, I realized the knowledge I

gained from the Research Methods course was utilized in my evaluation methodologies. The

following provides proof of my ability as a human resources (HR) professional to identify,

evaluate, and summarize relevant data to assist organizations and individual employees.

Research Methods introduced quantitative and qualitative methods of data collection, in

addition to the benefits and drawbacks to each. Quantitative data is typically associated with pure

science and experimental research as it involves definable data like numbers and specific

measurements (Creswell, 2013). In opposition, qualitative data cannot be measured because it is

descriptive information such as, feelings, emotions, and behaviors (Creswell, 2013). Since

quantitative methods result in actual numerical information, it is easier to compile into a chart or

graph to show significance. This provides an advantage over qualitative data which is difficult to

summarize in a visual representation. Another advantage to quantitative research methods is that

a large amount of information about a population can be gathered rather quickly (Creswell,

OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 3

2013). Qualitative research involves observation methods, an abundance of note taking, thorough

review of participant’s written responses, etc., which are all time consuming processes.

While it may be difficult to measure qualitative data, it often provides information that

quantitative methods cannot. A major advantage to qualitative data is that it allows participants

to provide in-depth responses which include detail and opinion (Goertz & Mahoney, 2012). For

example, a quantitative survey question would provide participants with multiple choice options,

whereas qualitative questions may require them to give short answer responses. In quantitative

questions, participants are required to choose a predetermined option formulated by the

researcher; however, their true opinion may not be reflected if it is not listed as one of the

options (Creswell, 2013). Qualitative research attempts to eliminate these biases in the data

collection design and the researcher’s analysis of the information (Goertz & Mahoney, 2012). In

regards to my evaluation strategies it is apparent that I attempted to incorporate methods to

gather both qualitative and quantitative data.

Controversy over which research method should be utilized has sparked many debates

among professionals in the social science fields. Those in favor of quantitative methods believe it

is the only way to make social sciences truly scientific as they are often regarded as the “softer”

sciences (Goertz & Mahoney, 2012). They want their research to be supported by hard,

statistically significant data. Professionals in support of qualitative methods see the importance

of numerical data, yet they believe it can affect the true nature of responses because it neglects

non-measurable factors (Goertz & Mahoney, 2012). They argue that these factors, like feelings

and emotions, can often be the most important in social sciences (Goertz & Mahoney, 2012).

There are others such as myself who believe that both have their place in data collection and a

combination of these methods is best when designing research plans.

OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 4

In support of “mixed methods research,” John Creswell stated, “if we assume that each

type of data collection has both limitations and strengths, we can consider how the strengths can

be combined to develop a stronger understanding of the research problem or questions (and, as

well, overcome the limitations of each). This ‘mixing’ or blending of data provides a stronger

understanding than either by itself” (2013, p. 215). My evaluation strategies for the ISD course

were developed around this idea of mixed methods research which provided increased reliability

and validity in any findings from gathered data.

The first method of evaluation involved a training satisfaction survey which asked

participants to rate the instructor, course material, and learning environment using a mixed

question format. The HRD program has consistently reinforced the importance of HR

professionals earning their seat at the corporate table by providing bottom-line significance to

workplace issues. I knew that the simplicity of this survey would not be enough to show the

impact of the training to the executives at my organization. I incorporated a second survey

immediately following the training session to gather information on how participants felt about

their current workplace environment as this related to the conflict management training topic.

This survey was to be used as a baseline for additional surveys after the training. The same

survey was distributed to participants each quarter following the training, and the results were

compared to those from the original survey.

The training was to be provided to all 143 employees within the organization. Not only

was this a large group to gather information on but the company had offices all over the United

States so employees were not easily accessible in person. For those reasons, I chose to

incorporate surveys into my evaluation methods. Surveys are less time-invasive to employees at

work, and they can easily be distributed and collected electronically (“Using surveys for,” 2006).

OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 5

The satisfaction survey was to contain quantitative questions such as multiple choice, yes or no

answers, and rating scales; while the baseline survey contained qualitative questions, in addition

to quantitative, like short answer and explanation. By selecting the baseline survey technique, my

design provided a way to monitor and assess the effectiveness of the training over time

(Anyaegbunam, Mefalopulos, & Moetsabi, 2004). Ideally, these surveys would be increasingly

positive in their report of the workplace environment and conflict resolution following the

training. I knew that showing this progression to executives would allow me attribute the

improved work environment to the implementation of my training leading to executive buy-in

for continued training needs. It is important to determine if training efforts were effective to

communicate to an organization’s decision makers the value of the time, funding, and resources

used in the process.

Another method of evaluation that I chose for my training program was a post-training

test for all participants. The idea behind this data collection method was to analyze retention of

knowledge among those who attended the training. “Tests provide a way to assess subjects’

knowledge and capacity to apply this knowledge to new situations” (“An overview of,” 2002, p.

56). The purpose of training is to equip employees with the proper skills and knowledge to do

their job efficiently and effectively (“Purpose of training,” 2008). I chose to administer these

tests two weeks after the conclusion of the training in order to gauge some long term retention of

the information presented to them. Looking back at my design plan, I realized that I did not

specify the form this test would be in. Ideally, the test would have been in the form of an e-

learning model that incorporated both qualitative and quantitative measures. Due to the

geographical barriers my company faced, doing an online based test would have realistically

been the only option. In addition, I could have distributed these tests on more than one occasion.

OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 6

This would have allowed the HR department to identify if participants were consistently utilizing

the information months after the training. The test results would have provided additional

numerical data to strengthen my argument to executives that the training was a success.

In order to gather more qualitative data, I chose to do bi-monthly random interviews from

various departments, as well as bi-monthly naturalistic observations of employees in the

workplace. Interviews typically yield the richest data and detail among other collection methods,

however interviewees may distort information in an attempt to please the interviewer (“An

overview of,” 2002). “Observational techniques are methods by which an individual or

individuals gather firsthand data on programs, processes, or behaviors being studied” (“An

overview of,” 2002, p. 53). Naturalistic observations allow an evaluator to “learn about issues the

participants or staff may be unaware of or that they are unwilling or unable to discuss candidly in

an interview” (“An overview of,” 2002, p. 54). While participants will likely behave naturally as

they are unaware they are being observed, the perception of the evaluator can distort gathered

data (“An overview of,” 2002).

In retrospect, I could have incorporated a focus group collection method to my evaluation

strategies. Focus groups combine the fundamentals of interviews and observation into one

collection method (“An overview of,” 2002). A focus group would have allowed me to ask

interview questions to a group in order to observe their group dynamics, behaviors, attitudes, etc.

(“An overview of,” 2002). In a sense, I would have killed two birds with one stone and

eliminated some of the time required to do both collection methods separately. As with each

collection method, there are both advantages and disadvantages to focus groups. While it would

be less time consuming, I would be eliminating a naturalistic collection method as the

participants would have knowledge of the observation. This could result in reactivity which is

OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 7

when participants alter their performance or behavior because they know they are being watched

(Lavrakas, 2008).

My knowledge of data collection and analysis techniques has assisted me in my

professional endeavors by allowing me to create and administer training to develop employees.

During my time at Sales Focus, Inc. the HR department began implementing the training

program I designed in the ISD course. Unfortunately, I left the organization before the post-

training evaluations could be fully developed. The participants were provided with the simple

satisfaction survey at the end of the training session. Courses in the HRD program like research

methods have afforded me the knowledge to understand that the current data collection method

Sales Focus, Inc. is administering is not sufficient in determining training effectiveness. As a

result of the HRD program, I now have transferable skills and knowledge to assist me at any

organization. I am certain that I have mastered objective two in my ability to identify, evaluate,

and summarize relevant data to assist organizations and individual employees which is supported

in the reflection above.

OBJECTIVE TWO – DATA COLLECTION AND ANALYSIS 8

References

Anyaegbunam, C., Mefalopulos, P., & Moetsabi, T. (2004). Participatory rural communication

appraisal. (2nd ed.). Rome, Italy: Merchant House. DOI: ftp://ftp.fao.org/docrep/fao/008/

y5793e/y5793e00.pdf

Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods

approaches. Thousand Oaks, California: Sage Publications

Goertz, G., & Mahoney, J. (2012). A tale of two cultures: Qualitative and quantitative research

in the social sciences. Princeton, New Jersey: Princeton University Press

Lavrakas, P. J. (2008). Encyclopedia of survey research methods. (2nd ed.). Thousand Oaks,

California: Sage Publications

National Science Foundation, (2002). An overview of quantitative and qualitative data collection

methods. Retrieved from website: http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_4

.pdf

Purpose of training and development. (2008). Retrieved from http://www.scribd.com/doc/64085

89/Purpose-of-Training-and-Development

Using surveys for data collection in continuous improvement. (2006). Innovation Insights.

Retrieved from http://www.psu.edu/president/pia/innovation/insights014.pdf