CNM HANDBOOK FOR

EDUCATIONAL RESEARCH

Central New Mexico Community CollegeUrsula Waln, Director of Student Learning Assessment, and Fred Ream, Mathematics Instructor

2014

Table of Contents

Purpose of this Handbook......................................................................................................................................1IRB and Educational Research...............................................................................................................................1

How the IRB Came to Be...................................................................................................................................1The Role of the IRB............................................................................................................................................2Research Involving Human Subjects..................................................................................................................2IRB Exemptions..................................................................................................................................................3IRB Criteria........................................................................................................................................................4Informed Consent...............................................................................................................................................5

Research Design.....................................................................................................................................................6Quantitative and Qualitative Options.................................................................................................................6Sampling Methods..............................................................................................................................................9

Statistics................................................................................................................................................................10Data Types........................................................................................................................................................10Analytic Approaches........................................................................................................................................10

Figure 1: Statistical Test Decision-Making Flowchart.................................................................................11Refresher of Basic Concepts.............................................................................................................................12

Figure 2: Normal Distribution.......................................................................................................................14Figure 3: Illustration of Outliers...................................................................................................................15

Analytic Software Programs.............................................................................................................................16Preliminary Data Analysis Tips........................................................................................................................16

Glossary................................................................................................................................................................17References............................................................................................................................................................19Appendix A: Recommendations from CNM’s IRB.............................................................................................20Appendix B: Preview of CNM IRB Application Form........................................................................................21

PURPOSE OF THIS HANDBOOK

This handbook was developed to foster depth of intentionality and critical analysis in CNM student outcomes assessment. It builds on the information presented in the CNM Handbook for Outcomes Assessment and is based on the premises that 1) student outcomes assessment is a form of educational research, and 2) considering one’s assessment efforts as research may suggest more creative, purposeful, and meaningful approaches. The following pages offer information and tips for those who wish to develop a greater degree of sophistication in their assessment approach and/or have their assessment efforts render more useful information.

IRB AND EDUCATIONAL RESEARCH

If you are interested in conducting research that is not purely operational in nature, is experimental, is for a dissertation or master’s thesis, or is going to be used outside of CNM in any way, your project requires prior approval by the CNM Institutional Review Board (IRB). If you think there is any possibility at all that you will ever want to publish your findings, use the data for other publishable research, or do anything else with your findings outside of CNM, please get IRB approval before beginning any data collection. The question of whether your particular study must have IRB approval hinges on whether or not it meets the definition of ‘research involving human subjects.’ However, for any study that is not operational in nature and going to be used only within CNM, the IRB, not the researcher, must make this determination, based on an application. If you have any doubt at all, submit an application. The process itself can help you organize your research plans, so the time won’t be wasted, and you’ll know you covered your bases.

A requirement for IRB approval need not deter faculty from pursuing educational research projects. Indeed, structured research into student learning not only has great potential for informing internal improvement efforts, but also can produce insights with the potential to impact the broader profession and/or instructional community. The application form for submitting a project for IRB approval is shown in Appendix B and can be accessed at http://www.cnm.edu/depts/planning/instres/irb.

If you are absolutely certain your project does not require IRB approval and you already know all you need to know about the IRB, you may want to skip ahead to the research design section, page 11.

How the IRB Came to Be

The existence of IRBs for certification of research conducted at colleges and university is required under The National Research Act of 1974. Although certainly not the first example of a societal effort to regulate experimentation involving humans, this federal statute came about in response to medical research that violated ethical principles deemed fundamental to American values and basic human rights. Salient examples include the medical experiments conducted in Nazi concentration camps between 1939 and 1945, (which led to the establishment of the first international code of research ethics, the Nuremburg Code) and the U.S. Public Health Service’s Tuskegee Syphilis Study of 1932-1972, in which 400 African American men known to have syphilis had information and treatment withheld so that researchers could observe the progression of the disease (Schneider).

In 1974, Congress established the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research and tasked it with identifying ethical principles and developing guidelines for the conduct of research involving human subjects. In 1979, the Commission published the Belmont Report, which summarized three basic principles:

1. Respect for persons

o Individuals should be treated as autonomous agents1

o Persons with diminished autonomy are entitled to additional protections

2. Beneficence

o Do no harm

o Maximize possible benefits and minimize possible harms

3. Justice

o Requires that individuals and groups be treated fairly and equitably in terms of bearing the burdens and receiving the benefits of research

-- NIH, p. 21

The Belmont Report principles and guidelines were codified in the Title 45 Code of Federal Regulations, Part 46, and oversight was given to the U.S. Department of Health, Education and Welfare (which later split into the Department of Health and Human Services and the Department of Education).

In 1991, The U.S. Department of Education became one of fifteen federal departments and agencies to codify in separate regulations the “Common Rule,” including in its 34 CFR Part 97 language identical to 45 CFR 46, Subpart A (NIH). Today, seventeen federal agencies fall under the Common Rule, and oversight for all is provided by the Office for Human Research Protections, a division of the U.S. Department of Health and Human Services.

The Role of the IRB

At the institutional level, “The IRB is an administrative body established to protect the rights and welfare of human research subjects recruited to participate in research activities conducted under the auspices of the institution with which it is affiliated” (IRB Guidebook, p. 1). IRBs determine “the acceptability of proposed research in terms of institutional commitments and regulations, applicable law, and standards of professional conduct and practice” (45 CFR 46.107).

The major roles of IRBs in the oversight of research are:

1. Evaluating the respect for persons, beneficence, and justice of all research activities proposed under the auspices of the institution and providing approval or disapproval accordingly.

2. Ensuring that the process proposed to collect informed consent meets regulatory requirements.

3. Overseeing the progress and protocols of ongoing research studies.

-- NIH, p. 84

Research Involving Human Subjects

The CFR provides the following definitions:

…(d) Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge. Activities which meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program which is considered research for other purposes. For example, some demonstration and service programs may include research activities.

(e) Research subject to regulation, and similar terms are intended to encompass those research activities for which a federal department or agency has specific responsibility for regulating as a research activity, (for example, Investigational New Drug requirements administered by the Food and Drug

2

Administration). It does not include research activities which are incidentally regulated by a federal department or agency solely as part of the department's or agency's broader responsibility to regulate certain types of activities whether research or non-research in nature (for example, Wage and Hour requirements administered by the Department of Labor).

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual, or

(2) Identifiable private information.

Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the subject or the subject's environment that are performed for research purposes.

Interaction includes communication or interpersonal contact between investigator and subject.

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place, and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example, a medical record).

Private information must be individually identifiable (i.e., the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects…

-- 45 CFR 46.102

IRB Exemptions

The CNM IRB, with endorsement from the VPAA, Deans’ Council, Chairs’ Council, and Faculty Senate, documented its Recommendations for Research in 2012, stating that “Some types of research are exempt from IRB review: student projects for class, data collection (e.g., surveys) by faculty or staff, assessment data collection, student course evaluations, and research by the college that is not designed for publication” (Appendix A).

Whether or not an investigation is subject to IRB approval, the CNM IRB (on its homepage) recommends following the basic rules listed in its document Recommendations for Research. In the same vein, it is advisable for anyone planning educational research at CNM to be familiar with and follow the IRB criteria and informed consent guidelines in the following sections of this handbook, whether or not the research requires IRB approval.

Forming the basis for the CNM statements of exemption are the following excerpts from 45 CFR Part 46, and the corresponding educational regulations in 34 CFR Part 97 (formatting enhanced for readability).

…(b) Unless otherwise required by department or agency heads, research activities in which the only involvement of human subjects will be in one or more of the following categories are exempt from this policy:

(1) Research conducted in established or commonly accepted educational settings, involving normal educational practices, such as

(i) research on regular and special education instructional strategies, or

(ii) research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods.

3

(2) Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior, unless:

(i) information obtained is recorded in such a manner that human subjects can be identified, directly or through identifiers linked to the subjects; and

(ii) any disclosure of the human subjects' responses outside the research could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, or reputation.

(3) Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures, or observation of public behavior that is not exempt under paragraph (b)(2) of this section, if:

(i) the human subjects are elected or appointed public officials or candidates for public office; or

(ii) federal statute(s) require(s) without exception that the confidentiality of the personally identifiable information will be maintained throughout the research and thereafter.

(4) Research involving the collection or study of existing data, documents, records, pathological specimens, or diagnostic specimens, if these sources are publicly available or if the information is recorded by the investigator in such a manner that subjects cannot be identified, directly or through identifiers linked to the subjects…

For the CNM IRB Recommendations for Research, see Appendix A or go to https://www.cnm.edu/depts/planning/instres/irb/documents/RECOMMENDATIONS-FOR-RESEARCH.pdf.

IRB Criteria

The following CFR excerpts hit the highlights of the federal criteria for IRB approval:

1. Risks to human subjects are minimized…

2. Risks to human subjects are reasonable in relation to anticipated benefits, if any, to human subjects and the importance of the knowledge that may reasonably be expected to result…

3. Selection of human subjects is equitable…

4. Informed consent will be sought from each prospective research participant or the prospective research participant’s legally authorized representative in accordance with and to the extent required by §46.116.

5. Informed consent will be appropriately documented in accordance with and to the extent required by §46.117.

6. When appropriate, the research plan makes adequate provision for monitoring the data collected to ensure the safety of subjects

7. When appropriate, there are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data.

-- 45 CFR 46.111

The federal regulations can be accessed in their entirety at http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html or at http://www2.ed.gov/policy/fund/reg/humansub/part97.html.

In addition, an Institutional Review Board Guidebook, published by the HHS, is available at http://www.hhs.gov/ohrp/archive/irb/irb_guidebook.htm.

4

And, the CNM IRB web site and links to documents can be found at http://www.cnm.edu/depts/planning/instres/irb .

Informed Consent

The principle of respect for persons requires that investigators conducting research on human subjects obtain informed consent in the form of “[a] legally-effective, voluntary agreement that is given by a prospective research participant following comprehension and consideration of all relevant information pertinent to the decision to participate in a study” (NIH, p. 117).

Autonomous adults can provide their own informed consent. (The age of majority in New Mexico is eighteen.) However, research involving children or other vulnerable populations requires informed consent from not only the subject (whenever the individual is reasonably capable of providing or denying consent), but also the subject’s legally authorized representative. In some cases (e.g., specific cultural settings), community or family consent may also be required.

Informed consent consists of voluntariness, comprehension, and disclosure:

Voluntariness: Consent must be given free of coercion, and the perceived value of any inducements must be small enough to avoid undue influence on the decision to participate. Care must be taken to ensure that subjects are not manipulated to ignore the risks of participation either through fear of repercussions for refusal or through motivation by excessive or inappropriate rewards. Offering bonus points toward a final course grade would be an example of an inappropriate incentive. However, in some cases, minimal compensation for participants’ time and effort may be appropriate.

Comprehension: To provide informed consent, subjects must first understand the information presented to them sufficiently to make a reasoned, informed decision. Subjects must therefore have the cognitive capacity to understand (or else be represented by legally authorized individuals), and the information must be presented using language and methods of delivery appropriate to the people making the decision.

Disclosure: Information that must be disclosed for participants to be able to provide informed consent include the purpose, risks, and benefits of the research; alternatives to the research protocol; the extent to which confidentiality can be protected; compensation in case of injury; contact information for questions or concerns; and conditions of participation, including the right to refuse or withdraw participation without penalty. (Compensation may be terminated proportionately upon withdrawal, but any offer of compensation for participation prior to the time of withdrawal must be honored. Telling prospective subjects they must complete the study or forfeit all compensation would be an example of a penalty.)

Waivers regarding the disclosure of specific information may be considered by the IRB when the research poses no more than minimal risk to the participants, withholding the information will not adversely affect the rights or welfare of the participants, and the research could not practicably be carried out otherwise. When information is withheld, it should be disclosed following the conclusion of research unless doing so might cause harm.

“No informed consent, whether oral or written, may include any exculpatory language through which the subject or the representative is made to waive or appear to waive any of the subject's legal rights, or releases or appears to release the investigator, the sponsor, the institution or its agents from liability for negligence” (45 CFR 46.116).

Finally, informed consent should be viewed as an ongoing process, not just a one-time event. Participants should be kept informed and given opportunities to ask questions as the research is carried out.

5

RESEARCH DESIGN

For the purposes of CNM student learning assessment, the primary consideration in designing your research should be what you want to know related to the development of your program competencies (student learning outcome statements). Using that as your starting point, you can employ ‘reverse design’ principles to map out the steps you will take toward gaining the insights you seek. Depending on the type of information you want to collect and what you want to do with it, your research design may or may not include the following:

A written statement of the issue to be investigated

Identification of the population to be studied

A description of relevant background information

A literature review

A written rationale

A hypothesis and/or null hypothesis (or just a statement of what you expect to find)

Identification of methodology

o Sampling methods

o Instruments

o Procedures for administration

o Data analysis methods

A timeline

Quantitative and Qualitative Options

Research approaches are often described as quantitative or qualitative, though the two descriptors more accurately represent ends of a continuum than distinct categories. Both extremes seek to answer questions using systematic procedures, collecting evidence, and examining findings. However, where a research project falls on the continuum is determined by its objective(s), the types of research questions posed, the methodologies employed, and the types of data produced. Expectations for researchers differ as well.

Quantitative research tends to have as an objective obtaining generalizable findings. Questions posed tend to focus on associations between variables, methodologies employed tend to involve studying specific variables, and data produced tends to be numerical. Quantitative investigators are typically expected to adhere to scientifically validated research models and techniques and to remain objective.

Qualitative research, on the other hand, tends to have as an objective exploring the dynamics of a specific context. Questions posed tend to focus on understanding factors that influence outcomes, methodologies employed tend to be open-ended and exploratory, and data produced tends to be textual (recorded observations, participant comments, etc.). Qualitative investigators are typically considered free to adapt their research models to fit the changing circumstances of their investigations and to render subjective interpretations.

Randomized controlled trials represent the quantitative end of the spectrum. In a nutshell, such investigations provide an intervention with a randomly selected experimental group and compare the results to those of a control group (a group that does not receive the intervention). Here are some grouping models commonly used in randomized controlled trials:

6

Parallel grouping: Each student is randomly assigned to an experimental or control group; the experimental group receives the intervention and the control group does not – but may receive a dummy/placebo intervention, to minimize the potential for participant expectations to influence the outcomes. (In educational research, a placebo may be, for example, an instructional method that has already been shown to not produce the effect being researched.)

Cluster grouping: Pre-existing groups of students (such as course sections) are randomly selected to receive or not receive the intervention

Crossover grouping: Each student receives or does not receive the intervention in a random sequence

Factorial grouping: Used when more than one intervention is tested simultaneously to determine whether there is a compound effect; each student is randomly assigned a group that receives a particular combination of interventions (AA, AB, AC, & BC, where A = no intervention, B = intervention 1, and C = Intervention 2)

A common crossover approach is to switch the treatment and non-treatment groups at the midpoint of the study. Crossover grouping minimizes the potential for group differences to influence the findings. In educational research, the effectiveness of a cross-over approach depends in part upon whether the intervention being studied is influenced by timing.

For example, let’s say a foreign language instructor randomly selects half of her students to participate in an emersion program during the first half of the semester only and then has the other half of the students participate in the emersion program during the second half of the semester. The timing of the emersion program could turn out to be a significant factor. Students who develop basic vocabulary and syntactic awareness through classroom instruction during the first eight weeks may have a decoding advantage when they enter the emersion program. On the other hand, students who experience the emersion program during the first eight weeks may develop greater awareness of cadence, pronunciation, and speech patterns, and applying these, advance more effectively through the conventional instruction during the second eight weeks. Anticipating these possibilities, the instructor may decide to include in the design a third group, comprised of students who do not participate in the emersion program at all.

A subset of randomized controlled trials, called randomized double-blind placebo-controlled crossover studies, is generally considered the ‘gold standard’ in quantitative scientific research (particularly pharmaceutical research). And, while it need not be the goal of every educational researcher to conduct such a study, an understanding of the rationale supporting the design will help inform design decisions.

Double-blind means neither the researchers in contact with the subjects nor the subjects know who is receiving the ‘treatment’ and who is not. This minimizes the potential for either the researchers or the subjects to influence outcomes based on their expectations. In educational research, conducing a double-blind study would require that the educational intervention be provided by someone other than the investigator.

7

A Source of Some ConfusionThe word quantitative may sometimes seem a little slippery because people use it differently depending on whether they are describing data, measurement methods, or research design.

Quantitative data is information derived from direct numerical measurement. Height, weight, area, pounds per square inch, counts of students, proportions of responses, and GPA are examples. Representing something as a number when it is not directly, numerically measurable (such as pain rated on a scale of 1 to 10 or level of effort on a scale of 0 to 4) does not make it quantitative data. It is a common error to refer to qualitative data that has been numerically represented as quantitative.

Readers will find disagreement in the literature about the classification of measurement methods as quantitative versus qualitative. Applying the same distinction used for classifying data, quantitative measurement methods are those that involve direct, numerical measures. However, common usage lumps together all measurement methods that produce numerical data (especially when conducted in systematic ways) as quantitative measurement methods. As a result, Likert-scale questionnaire items and rubric-based evaluations are commonly referred to as quantitative methods even though the data they comprise is qualitative data.

At the research design level, quantitative essentially means the researchers intend to use inferential statistics to make generalizations

For example, let’s say a researcher wants to find out whether students approached outside of class by a peer and asked to explain core course concepts perform significantly better on subsequent exams than students not questioned in this way. To make this a double-blind study, the students must have no inkling that being questioned by a peer is an intervention, and the researcher must remain ignorant regarding which students are questioned until after the data collection is complete.

At the other end of the spectrum, qualitative research usually seeks to answer a question related to the perspectives of an involved population. Qualitative research can be particularly useful in studying attitudes, values, opinions, perceptions, beliefs, behaviors, emotions, relationships, social situations, experiences, and other aspects of individual, community, or cultural contexts. Although insights gained may have the potential for extension to similar populations or situations, qualitative research usually focuses more on delving into the factors that influence outcomes within the specific study population than on being able to generalize the findings to other populations. Qualitative research explores and interprets how people experience the complex reality of given issues and/or situations.

For example, imagine CNM researchers want to better understand why area employers within a specific sector do not hire as many qualified CNM alumni as might be expected on the basis of job openings. The objective of their investigation will be to identify factors influencing the decisions of the local employers. Their findings might ultimately reveal something about employer perceptions that could be useful, by extension, to other CNM programs or other colleges with similar programs. However, their primary goal is not finding something that can help other programs, but rather, better understanding what their students’ prospective employers want so that the program faculty might better address the preparation of candidates.

Research within the field of education is often driven by context-specific questions posed by faculty and therefore tends toward qualitative design. Indeed, large-scale quantitative educational studies are relatively rare, and generalizations made on their basis tend to be subject to dismissal because of the difficulties inherent in applying scientific models to studies of humans that take place in natural settings, especially when the conclusions contradict strongly-held convictions about teaching and learning. (For an illustrative example, consider Project Follow Through, “the largest, most expensive educational experiment ever conducted” [Adams, 1996].)

Because investigators conducting qualitative research typically care more about the usefulness of the information gathered in supporting interpretation and insight than about the reproducibility of the study, qualitative research tends to be characterized by adjustability and/or fluidity. The design needs to facilitate collection of the desired information, not stand up to scientific scrutiny. Stated another way, validity of the findings is important in qualitative research, but reliability is not.

Observation, interviews, and focus groups are methods commonly used in qualitative research. The data produced may be a mix of direct and indirect measurements and/or textual information with no associated numeric values (such as recorded dialogs, comments, field notes, responses to open-ended questions, etc.). Input using categorization, holistic ratings, Likert scales, rubrics, and/or forced-choice responses may be numerically represented to facilitate statistical analysis.

Note that the use of statistics is not the exclusive domain of quantitative research; however, while quantitative research tends to employ inferential statistics, qualitative research tends to incorporate descriptive statistics. These categories are explored further in the Statistics section of this handbook.

8

Sampling Methods

A population is all of a group of interest (in New Mexico, the whole enchilada). A sample is a subset of the population being studied (in New Mexico, some of the enchilada). Sampling can make it feasible to implement research methods that would otherwise be impossible or too time consuming to consider.

Sampling methods fall into two broad categories: those that offer the potential to support inferences about the entire population (probability methods) and those that do not (non-probability methods). Probability methods are typically used with quantitative research designs and inferential statistics; whereas, non-probability methods are more often (but not always) used with qualitative designs and descriptive statistics.

If you want to be relatively sure that the results you obtain from your sampled students are generalizable to the population of students in your course and/or program, consider using a probability method, such as simple, systematic, stratified, or clustered random sampling. Below are some examples:

Simple random sample: Drawing names from a hat (For the sample to be truly random, all of the possible names must be in the hat to begin with. If you systematically leave out any group of people [e.g., students who are failing the course], then your results will not be generalizable to that group.)

Systematic random sample: Counting off every 5th student who comes through the door

Stratified sample: Randomly selecting 10 first-year students, 10 second-year students, and 10 recent graduates

Cluster sample: Randomly selecting 6 students from each of five sections of the same course

To be confident in your inferences, you will want to select a sample that is big enough to be representative of the population you are studying. Consider using an online sample-size calculator, such as the one provided by Creative Research Systems at http://www.surveysystem.com/sscalc.htm, to find out how many students you need to include in your sample to achieve the level of statistical reliability you want. Then, you can use an online random integer generator, such as the one at http://www.random.org/integers/, to get numbers that you can apply to a numbered list of all students in the research ‘population’ to create your random sample.

On the other hand, if you don’t care about generalizing your sample’s results to a larger population, you might prefer to use non-probability methods such as convenience, purposive, snowball, quota, or theoretical sampling. Just keep in mind that if you choose any of these, the statistics you obtain will not support claims that your findings can be generalized to the entire population. Following are some examples:

Convenience sample: Using your own class as a sample

Purposive sample: Asking all of the African American students in your program to participate in a focus group to provide feedback regarding their perceptions of cultural inclusiveness within the program

Snowball sample: Interviewing a few people and asking them to refer their friends to you for interviews

Quota sampling: Selecting 200 students in such a way that your sample has essentially the same proportionate representation of ethnic groups seen in your program

Theoretical sample: Questioning a few students about something you’re considering studying further

9

STATISTICS

Depending on the type of assessment/research you conduct, you may or may not need to include statistical analyses. This final section is included for those who do plan to use statistics but are not expert statisticians.

If you are planning to conduct a formal study, even an in-house one, and are not a statistician, consider consulting with someone who is a statistician during the planning phase, before you begin collecting any data. This will help ensure that the data collected will be able to provide the information you seek and shed useful light on the research topic. Also, consider including a statistician in your plans for conducting the statistical analysis once you have collected the data.

Data Types

Although as a branch of mathematics statistics involves numbers, many types of information can used for purposes of statistical analysis. Before selecting a method of analysis, it is important to understand what type of data you have. And, considering the method of analysis prior to conducting a study can help to ensure that the data collected will provide the desired functionality. There are four main types of data: nominal, ordinal, interval, and ratio:

Nominal: Numbers or words are used to label, classify, or categorize. For example, each student may be represented by a student ID number. Or, maybe students are grouped according to a characteristic, such as gender, with 1 representing male and 2 representing female. The numbers or words are representations, not measures, and they have no inherent order (1 could just as easily represent female and 2 male), so they cannot be used in mathematical calculations. However, nominal data can be used to indicate similarities and differences.

Ordinal: Numbers or words are used to categorize information on the basis of order. Observations are rank-ordered on the basis of some criterion, but the intervals between the rankings are not necessarily equal. For example, a postgraduate degree is higher than a bachelor’s degree, which is higher than an associate’s degree, which is higher than a high school diploma. Likert-item responses, letter grades, and rubric scores are other examples of ordinal data. Ordinal data is used to indicate more than or less than relationships.

Interval: Numbers represent equal intervals between points on a scale, but the scale lacks an absolute zero. IQ scores, calendar years (e.g. 1983 and 2012), and temperatures are examples.

Ratio: Numbers represent equal intervals between points on a scale, and the scale has an absolute zero, which allows measures of ratio to be taken. We can say that one item is two-thirds the size of another or that one student took twice as long as another to complete a task. Height, weight, time intervals, and counts are additional examples.

Analytic Approaches

Statistical analyses are often classified as parametric or nonparametric:

Parametric analyses, associated primarily with inferential statistics, can be used with interval or ratio data that are normally distributed when the variance within the two groups being compared is the same or similar. Parametric tests include t-tests and analysis of variance (ANOVA). Longitudinal and cross-sectional studies typically employ parametric analyses.

10

Nonparametric analyses, associated more with descriptive statistics, can be used with nominal and ordinal data. They are also useful for interval and ratio data that are not normally distributed and/or do not have variances similar to those of the comparison populations. Chi square tests are non-parametric.

The decision chart below can be used to identify the statistical tests most applicable to your research project.

Figure 1: Statistical Test Decision-Making Flowchart

Adapted from Ruth, D., Practical Statistics for Educators, p. 227

You may have noticed that the flowchart above does not include any guidance for the use of ordinal data. This is because statistical applications using ordinal data must take into account the degree to which the intervals between can be assumed to be equal. In some cases, such as the example of educational attainment given in the previous section, ordinal data is essentially nominal data with some order to it. While one might assign numbers to progressive educational awards (e.g., 1 for High School, 2 for Associate’s, 3 for Bachelor’s, and 4 for Post-grad), it would make no sense to calculate a mean on the data (or to conduct mean comparisons using an independent samples t-test). However, it might make sense to calculate the mode or to calculate median income associated with each educational level.

11

In other cases, when the ranking strongly suggests a degree of continuity across intervals, ordinal data may be treated more like interval data. For example, it is common practice, though not all agree it should be, to calculate mean responses from Likert items. One sees this more in the social sciences and in qualitative research than in the fields of math and science or in quantitative research.

Correlation studies are commonly used in educational research because they are very helpful in determining whether a relationship exists and is worth exploring further. Fortunately, or unfortunately depending on how much you love statistics, a variety of correlation coefficients are available for computation. The Pearson correlation coefficient (a.k.a. the Pearson product-moment correlation, or just Pearson for short) is most widely used, and indeed some analytic software programs offer only the Pearson. Generally speaking, however, it is the type of data that determines which correlation study is most appropriate. Because the differences could be important if you intend to publish your research, the following differentiations are provided:

Pearson is a nonparametric test and can be used when the data is not normally distributed and/or when you want to see if there is a relationship between two interval or ratio variables. As indicated above, however, Pearson is commonly used with normally distributed data as well.

Spearman’s rho (Spearman for short) is a parametric test, intended for use with normally distributed data. You can use it to compute the correlation between two ordinal variables. However, the Spearman is most appropriately used with ordinal data for which the ranking is highly detailed and/or strongly suggests a degree of continuity across intervals. In other words, Spearman is good for analyzing ordinal data that closely resembles interval data.

For ordinal data that is more like nominal data, having five or fewer rankings with some order to them and/or having intervals that are not really approximately equal (such as Likert-item responses), a Gamma correlation may be more appropriate.

Point-biserial correlation studies can be used to analyze dichotomous data. Note, however, that only one of the variables can be dichotomous. Some data is naturally dichotomous, such as gender or Yes/No responses. And, sometimes people artificially dichotomize data, particularly ordinal data (beware however that some people frown on this practice). A common example of artificial dichotomization is taking Likert-items and putting the “Agree” and “Strongly Agree” responses into one category while putting the “Disagree” and “Strongly Disagree” responses in another. Another example would be putting all of the students who passed with grades of C or better into one category and all the students who got D’s and F’s into another.

Refresher of Basic Concepts

The following list highlights some key statistical concepts. Additional definitions are provided in the glossary. Far more extensive glossaries of statistical terminology can be accessed online at sites such as statistics.com and stat.berkeley.edu.

Alternative Hypothesis: Typically a statement made prior to a study asserting (hypothesizing) that there is or will be an

observable effect or difference. (Contrast with Null Hypothesis.)

Averages: 3 main measures of central tendency (though there are others) Mean: The arithmetic calculation of the sum of the numbers divided by the count of the

numbers. This is what most people mean (no pun intended) when they say “average.” It is the balance point of the data. (A deviation is the difference between a data value and the mean. With the mean as the balance point, the sum of the deviations above the mean will cancel the sum of the deviations below the mean.)

12

Median: The middle value between the highest and the lowest of the numbers when the set is arranged in order. For example, if the set is {2, 3, 3, 5, 6, 8, 9}, the median is 5. If you have an even count of numbers, the median is the sum of the two in the middle divided by 2. For example, if the set is {1, 1, 3, 5, 5, 6}, the median is calculated (3+5) / 2 = 4.

Mode: The most frequently occurring of the numbers. In the set {2, 3, 3, 5, 6, 8, 9}, the mode is 3 because that is the only number that occurs twice. A set can have more than one mode, as in {1, 1, 3, 5, 5, 6}, and if no number occurs more than once, there is no mode at all. The mode is considered a measure of central tendency because in a more-or-less normally distributed population, the mode would tend to be at or near the middle, as would the mean and median.

Correlation: A procedure used to quantify the relationship between two quantitative variables. In a positive correlation, an increase in one variable corresponds to an increase in the other. In a negative correlation, an increase in one corresponds to a decrease in the other. Correlations are represented using correlation coefficients, ranging between -1 and 1, with zero indicating no correlation at all and the strength of the correlation increasing toward each end of the range. A correlation of 1 would suggest that an increase in one variable corresponds to an equivalent increase in the other, a phenomenon that rarely seen in educational research.

In interpreting correlations, it is important to keep in mind that they do not necessarily demonstrate causal relationships. Both variables could potentially be influenced by other variables. For example, ACT scores correlate positively with college completion rates, but having higher ACT scores does not make students more likely to finish college. Both are influenced by prior educational experiences, socio-economic status, parental education, cultural/family expectations, etc.

Data Plots: Graphic representations of data sets, often taking the form of scatterplots, histograms (similar to bar graphs), or box plots (a.k.a. box and whisker charts).

Effect Size: A calculation that quantifies the magnitude of a difference between two groups. The effect size is a measure of how far apart the true value of the parameter being tested and the one assumed in the null hypothesis are. Failing to detect the difference is a type II error (interpreting the data as not showing a difference when there really is one).

Sometimes effect size is presented instead of (or in addition to) calculations of statistical significance, in which very large sample sizes tend to cause relatively minor differences to be flagged as significant. Effect size is an indicator of whether the statistically significant information is of practical significance.

Different formulas can be used to calculate effect size, but one way is to divide the difference between the two group means by the standard deviation (of either group or of the two groups averaged). Basically, this tells you how many standard deviations’ difference there is between the two means. For example, an effect size of .70 tells you that the difference between the means represents approximately 7/10 of a standard deviation. Some would consider this a large effect size. However, different scales are used for interpretation, depending on the field of study. Some interpret .25 as small, .50 as medium, and .80 as large. Others consider .10 small, .30 medium, and .50 large.

13

Hypothesis Testing: A formal process of deciding whether to reject or not to reject a null hypothesis based on the

results of a study.

Level of Significance & Confidence Level: The level of significance is referred to as the alpha level. It is used in hypothesis testing and

should be chosen before the study is conducted. The level of significance quantifies the investigator’s willingness to make a type I error (to interpret the data as showing a difference when one truly does not exist) by identifying what p-value will be considered the ‘bar’ (or cut-off) for statistical significance.

After the data is collected, a p-value is calculated from the sample, and if it is smaller than the level of significance, then the null hypothesis is rejected (and the relationship deemed to be statistically significant).

Confidence Level is the likelihood that the corresponding confidence interval is one of the intervals that contains the parameter being estimated. The confidence level is one minus the level of significance. A confidence interval can be interpreted as a two-sided or two-tailed hypothesis test at the alpha level of significance.

Normal Curve: A frequency plot showing the distribution of a characteristic over a population. Also known as a

bell curve, the graphic representation is bell-shaped, with the majority of people clustering near the mean. A perfectly normal distribution is one in which mean, median, and mode all have the same value and the tails (lowest and highest ends) remain slightly above the horizontal axis, as shown in Figure 2.

Figure 2: Normal Distribution

Image retrieved from http://www.lesn.appstate.edu/olson/normal_curve.htm

Null Hypothesis: Specifies a value for a parameter. Usually, the choice is based on there being no relationship or

effect, no difference between groups, or no difference from a previous value. Like the assumption of innocence until proven guilty, the null hypothesis places the burden upon the investigator to show with significant evidence, beyond reasonable doubt, that a difference exists.

14

Outliers: Data points that are significantly above or below the trend or deviate from the pattern. The following scatterplot illustrates by pointing out two outliers:

Figure 3: Illustration of Outliers

Power: An estimate of the sensitivity of the test in detecting a difference. The power of a test is the chance that the test correctly rejects a null hypothesis, i.e., the chance of not making a type II error.

p-Value: The “p” in p-value stands for probability. A p-value represents the probability that an experimental result is due to a random occurrence under a true null hypothesis. (It is essentially the opposite of the probability of the phenomenon occurring in any given case, which is known as the confidence interval.) P-values are presented with tests of statistical significance to communicate the level of reliability of the statistical conclusions. The smaller the p-value, the greater the reliability of the statistical significance.

P-values are usually derived from the “Sig” (significance) data in analytical output tables and are written as “p ≤ 0.01,” which equates to a 99% confidence interval (i.e., a 99% or greater probability of the relationship holding true), or as “p ≤ 0.05,” which corresponds to a 95% confidence interval. Data with a p-value > 0.05 is usually interpreted as not rising to the level of statistical significance.

Standard Deviation (SD): A measure of how spread out (dispersed) the numbers are from the mean, symbolized as σ (lower

case sigma) for the population standard deviation and “s” for the sample standard deviation. The more closely grouped the numbers are around the mean, the smaller the SD will be. SD is expressed in the same units used to express the mean, so it makes it relatively easy for people to understand just how big the spread is above and below the mean. SD is the square root of the mean of the squared differences from the mean value, i.e., the square root of the variance.

Other measures of the spread: Standard Error is used when the standard deviation for an experiment is estimated from its data. Although the sample SD will appear in it, the correct form will depend on the experiment. Margin of Error is the product of a probability factor, to insert the confidence, and the standard error as a scale factor based on the data. It is the number behind the plus or minus in a confidence interval and represents half the width of the interval. For example, with 95% confidence, the true mean μ is thought to lie on the interval 5.25 ± .37.

Type I and Type II

15

Errors: A type I error is a false positive where a true null hypothesis is rejected. A type II error is a false negative where a false null hypothesis is not rejected.

Variance: Variance is way of measuring how spread out (or dispersed) the numbers are from the mean. The obvious measure of the spread would be the mean of the deviations (the differences between the data values and the mean); however, the mean is the balance point of the data, so the sum of the deviations above the mean cancel those below, and the mean deviation therefore is always zero. To avoid this problem, the deviations are squared. Variance, then, is the mean of the squared deviations. This result is the variance for the population. There is an inherent error caused by sampling. This is corrected for by dividing by one less than the number of data values to obtain the variance for the experimental sample.

Analytic Software Programs

Open-source and public-domain platforms and online calculators are readily available to support most analytic needs. Here are just a few examples:

Dataplot: National Institute of Standards and Technology

PSPP: The Free Technology Academy

Statistical Calculators: MathPortal.org

Sample Size Calculator: National Statistical Service

Sample Size Calculator: Creative Research Systems

Preliminary Data Analysis Tips

Clean up your data: Before beginning an in-depth analysis, look through the data for typos and errors in formatting. If you are using Excel, be sure to format the cells consistently throughout each field. Look for number fields formatted as text, and check decimal settings, time and date formatting, etc. If you are using PSPP, SPSS, or some other statistics software, be sure the “Variables” view has the correct data type selected. Scan the data for things like ID numbers with too many or too few digits or income figures with inconsistent formatting (ranges, dollar signs, text, etc.). Consider sorting the records in ascending or descending order by each data field to make errors more apparent.

Develop some first impressions: Look for general trends and/or groupings and anything else that stands out. Often it is helpful to look at the data of interest in a scatter plot before running tests on it to see if any linear or grouping trends are apparent, to check for outliers, etc. Having familiarized yourself with the data in this way will stimulate ideas about what to explore and will help you interpret the output when you run your analyses.

Check to see if the requirements for the model you chose are met: For example, for a nearly normal model, look for a single peak and symmetry in the data using a histogram or a normal probability plot. For a linear regression, check for linearity using a scatterplot. After fitting the regression line to the data, look for structure in the residuals that indicate problems in the original linear model.

GLOSSARY

ANOVA Analysis of Variance, a parametric test used to compare the means of two or 16

more independent samples and check for statistically significant differences.Case Studies Anecdotal reports prepared by trained observers. When numerous case studies

are analyzed together, the findings are often considered empirical evidence.Chi Square (χ2) A non-parametric test used to compare the frequencies of counts, using

nominal or ordinal data, to expected frequencies. The Goodness of Fit Chi Square is used for one independent variable, the Chi Square Test of Independence for two independent variables.

Content Validity The degree to which an instrument actually measures the content it is intended to measure.

Construct A theoretical entity or concept such as aptitude, creativity, engagement, intelligence, interest, or motivation. Constructs cannot be directly measured, so we assess indicators of their existence. For example, IQ tests assess prior learning as an indicator of intelligence.

Construct Validity The ability of a research approach to actually represent the construct it purports to address. Since constructs cannot be directly measured, construct validity depends upon how much people can agree upon a definition of the construct and how truly the indicators represent that definition.

Correlation A process of quantifying a relationship between two quantitative variables in which an increase in one is associated with an increase in the other (a positive correlation) or a decrease in the other (a negative correlation).

Cross-Sectional Analyses Studies comparing, at a single point in time, people representing different stages of development, for the purpose of drawing inferences about how people change/develop over time.

Descriptive Statistics Tests used to quantitatively describe phenomena. (Compare nonparametric analyses. Contrast inferential statistics.)

Empirical Evidence Usually refers to what has been systematically observed. May or may not be obtained through scientific experimentation.

Hawthorne Effect A phenomenon in which the behavior of participants in a study is affected by their knowing they are being studied.

Inferential Statistics Tests used to quantify the degree to which generalizations can be made to a population based on outcomes observed in a sample. (Compare parametric analyses. Contrast descriptive statistics.)

Internal Validity The extent to which variables in a study can be controlled sufficiently to ensure that any observed change is actually attributable to the intervention.

Inter-Rater Reliability The degree of consistency in ratings among different evaluators. For example, if five judges score a performance with nearly identical ratings, the inter-rater reliability is said to be high.

Interval Data Numbers on a scale comprised of equal intervals but having no absolute zero.Longitudinal Analysis A comparison of outcomes over time within a given group, based on changes at

an individual level (versus differences in group averages). E.g., comparing pre-test and post-test scores to assess learning gains.

Mean The average, arithmetic mean, derived by dividing the sum of all scores by the total number (count) of scores. It is the balance point of the data.

Median The midpoint, were 50% of scores are above it and 50% are below it (in position, not value) when all of the scores are ranked in ascending order.

Mode The most frequently occurring score.Nominal Data Numbers (usually) represent individuals or categories with no associated order.

E.g., numbers assigned to ethnic groups or student ID numbers.17

Nonparametric Analyses Studies using nominal and/or ordinal data or any data type that does not follow a normal distribution and/or does not have variances similar to those of the comparison population. Nonparametric tests do not support population inferences. They are used primarily in qualitative research.

Null Hypothesis A statement to be tested through systematic investigation, asserting that no relationship or effect exists between the variables being studied, that there is no difference between groups, or that there is no difference from a previous value. In scientific research, the assumption is that no claims can be made regarding relationships between variables unless significant evidence exists to reject the null hypothesis. Even so, a cause-and-effect relationship cannot be established.

Ordinal Data Numbers (usually) represent observations that have an underlying order or magnitude but the intervals cannot be assumed to be equal.

Outliers Data points that are noticeably outside the pattern of the point distribution.Parametric Analyses Studies using interval or ratio data that are more-or-less normally distributed

when the variance within the two groups being compared is the same or similar. Parametric analyses can be used to support generalizations/inferences. (Compare inferential statistics.)

Qualitative Evidence Evidence that cannot be directly measured, such as people’s perceptions, valuations, opinions, effort levels, behaviors, etc. Many people mistakenly refer to qualitative evidence that has been assigned numerical representations (e.g., Likert-scale ratings) as quantitative data. The difference is that the qualitative evidence is inherently non-numerical.

Quantitative Evidence Evidence that occurs in traditionally established, measurable units, such as counts of students, grade point averages, percentages of questions correctly answered, speed of task completion, etc.

Randomized, Controlled Trials (RCTs)

A model for scientific experimentation. Participants are randomly selected and divided into groups, experimental and control, for outcome comparison.

Ratio Data Numbers on a scale comprised of equal intervals and having an absolute zero. Reliability Consistency in outcomes over time or under differing circumstances. For

example, if a student challenges a placement exam repeatedly without doing any study in between and keeps getting essentially the same result, the test has a high level of reliability.

Standard Deviation (SD) A measure of the spread of scores about the mean. SD is the square root of the variance. There are two, one for a population and one for a sample; these are calculated slightly differently.

t-Test Originally derived for small samples, current practice is to use this distribution when the population standard deviation is not known.

Validity The ability of a research approach to measure what it is intended to measure. An aptitude test in which cultural expectations or socio-economic level significantly influence outcomes may have low validity.

REFERENCES

Adams, G., & Engelmann, S. (1996). Research on direct instruction. Seattle, WA: Educational Achiebement Systems.

18

Grossen, B. (1996). The story behind Project Follow Through. Effective school practices, Vol. 15, 1. Retrieved from http://darkwing.uoregon.edu/~adiep/ft/grossen.htm.

Institutional Review Board (2004). CNM institutional review board [PDF document]. Central New Mexico Community College. Retrieved from http://www.cnm.edu/depts/planning/instres/irb/documents/CNM_Institutional_Review_Board.pdf

National Institute of Health Office of Extramural Research. Protecting human research participants. [IRB training PDF]. Retrieved from http://phrp.nihtraining.com/

U.S. Department of Health and Human Services (1993). Institutional review board guidebook. Retrieved from http://www.hhs.gov/ohrp/archive/irb/irb_chapter1.htm.

Schneider, W. H. The establishment of institutional review boards in the U.S. background history. Indiana University-Purdue University Indianapolis History Department. Retrieved from http://www.iupui.edu/~histwhs/G504.dir/irbhist.html.

APPENDIX A: RECOMMENDATIONS FROM CNM’S IRB

19

https://www.cnm.edu/depts/planning/instres/irb/documents/RECOMMENDATIONS-FOR-RESEARCH.pdf

20

APPENDIX B: PREVIEW OF CNM IRB APPLICATION FORM

21

22

23

24

25

26

http://www.cnm.edu/depts/planning/instres/irb

27