Unit 4 Study Guide

Chapter 12: Agency Records, Content Analysis, and Secondary Data

Learning Objectives:

1. Recognize that public organizations produce statistics and data that are often useful for criminal justice researchers.

2. Provide examples of nonpublic agency records that can serve as data for criminal justice research.

3. Understand why the units of analysis represented by agency data may be confusing for researchers.

4. Explain why researchers must be attentive to reliability and validity problems that might stem from agency records.

5. Summarize why “follow the paper trail” and “expect the expected” are useful maxims to follow when using agency records in research.

6. Summarize content analysis as a research method appropriate for studying communica-tions.

7. Describe examples of coding to transform raw data into a standardized, quantitative form.8. Summarize how secondary analysis refers to the analysis of data collected by another re-

searcher for some other purpose.9. Be able to access archives of criminal justice data that are maintained by the ICPSR and

the NACJD.10. Understand how the advantages and disadvantages of secondary data are similar to those

for agency records.

Chapter Summary:

• Many public organizations produce statistics and data for the public record, and these data are often useful for criminal justice researchers.

• All organizations keep nonpublic records for internal operational purposes, and these records are valuable sources of data for criminal justice research.

• Public organizations can sometimes be enlisted to collect new data-- through observa-tions or interviews-- for use by researchers.

• The units of analysis represented by agency data may not always be obvious, because agencies typically use different, and often unclear, units of count to record information about people and cases.

• Researchers must be especially attentive to possible reliability and validity problems when they use data from agency records.

• “Follow the paper trail” and “Expect the expected” are two general maxims for re-searchers to keep in mind when using agency records in their research.

• Content analysis is a research method appropriate for studying human communications. Because communication takes many forms, content analysis can study many other aspects of behavior.

• Coding is the process of transforming raw data-- either manifest or latent content-- into a standardized, quantitative form.

• Secondary analysis is the analysis of data collected earlier by another researcher for some purpose other than the topic of the current study.

• Archives of criminal justice and other social data are maintained by the ICPSR and the NACJD for use by other researchers.

• The advantages and disadvantages of using secondary data are similar to those for agency records-- data previously collected by another researcher may not match our own needs.

Key Terms:

Content analysis (Page 348)Interuniversity Consortium for Political and Social Research (ICPSR) (Page 353)Latent content (Page 349)Manifest content (Page 348)National Archive of Criminal Justice Data (NACJD) (Page 353)Published statistics (Page 331)Secondary analysis (Page 330)Social production of data (Page 334)

Chapter 13: Evaluation Research and Problem Analysis

Learning Objectives:

1. Summarize evaluation research and problem analysis as examples of applied research in criminal justice.

2. Describe how different types of evaluation activities correspond to different stages in the policy process.

3. Explain the role of an evaluability assessment.4. Understand why a careful formulation of the problem, relevant measurements, and crite-

ria of success or failure are essential in evaluation research.5. Describe the parallels between evaluation research designs and other designs.6. Explain the advantages, requirements, and limits of randomized field experiments.7. Summarize the importance of process evaluations conducted independently or in connec-

tion with an impact assessment.8. Describe the role of problem analysis as a planning technique that draws on the same so-

cial science research methods used in program evaluation.9. Explain how the scientific realist approach focuses on mechanisms in context, rather than

generalizable causal processes.10. Present an example of how criminal justice agencies are increasingly using problem anal-

ysis tools, crime mapping, and other space-based procedures.11. Explain how evaluation research entails special logistical, ethical, and political problems.

Chapter Summary:

• Evaluation research and problem analysis are examples of applied research in criminal justice.

• Different types of evaluation activities correspond to different stages in the policy process-- policy planning, process evaluation, and impact evaluation.

• An evaluability assessment may be undertaken as a scouting operation or a preevaluation to determine whether it is possible to evaluate a particular program.

• A careful formulation of the problem, including relevant measurements and criteria of success or failure, is essential in evaluation research.

• Organizations may not have clear statements or ideas about program goals. In such cases, researchers must work with agency staff to formulate mutually acceptable statements of goals before proceeding.

• Evaluation research may use experimental, quasi-experimental, or nonexperimental de-signs. As in studies with other research purposes, designs that offer the greatest control over experimental conditions are usually preferred.

• Randomized designs cannot be used for evaluations that begin after a new program has been implemented or for full-coverage programs in which it is not possible to withhold an ex-perimental treatment from a control group.

• Process evaluations can be undertaken independently or in connection with an impact as-sessment. Process evaluations are all but essential for interpreting results from an impact as-sessment.

• Problem analysis is more of a planning technique. However, problem analysis draws on the same social science research methods used in program evaluation. Many variations on problem analysis are used in applied criminal justice research.

• The scientific realist approach to applied research focuses on mechanisms in context, rather than generalizable causal processes.

• Criminal justice agencies are increasingly using problem analysis tools for tactical and strategic planning. Crime mapping and other space-based procedures are especially useful ap-plied techniques.

• Problem solving, evaluation, and scientific realism have many common elements. • Evaluation research entails special logistical, ethical, and political problems because it is

embedded in the day-to-day events of public policy and real life.

Key Terms:

Evaluation research (Page 362)Evidence-based policy (Page 363)Impact assessment (Page 366)Problem analysis (Page 362)Problem-oriented policing (Page 384)Problem solving (Page 384)Process evaluation (Page 367)Stakeholders (Page 370)

Chapter 14: Interpreting Data

Learning Objectives:

1. Understand that descriptive statistics are used to summarize data under study.2. Describe a frequency distribution in terms of cases, attributes, and variables.3. Recognize that measures of central tendency summarize data, but they do not convey the

idea of original data.4. Understand that measures of dispersion give a summary indication of the distribution of

cases around an average value.5. Provide examples of rates as descriptive statistics that standardize some measure for com-

parative purposes.6. Describe how bivariate analysis and subgroup comparisons examine relationships be-

tween two variables.7. Compute and interpret percentages in contingency tables.8. Understand that multivariate analysis examines the relationships among several variables.9. Explain the logic underlying the proportionate reduction of error (PRE) model.10. Describe the use of lambda and gamma, and and Pearson’s product-moment correlation

as PRE-based measures of association for nominal, ordinal, and interval/ration variables, respectively.

11. Summarize how regression equations and regression lines are used in data analysis. 12. Understand how inferential statistics are used to estimate the generalizability of findings

arrived at in the analysis of a sample to a larger population.13. Describe the meaning of confidence intervals and confidence levels in inferential statis-

tics. 14. Explain what tests of statistical significance indicate, and how to interpret them.

Chapter Summary:

• Descriptive statistics are used to summarize data under study.• A frequency distribution shows the number of cases that have each of the attributes of a

given variable.• Measures of central tendency reduce data to an easily manageable form, but they do not

convey the detail of the original data. • Measures of dispersion give a summary indication of the distribution of cases around an

average value. • Rates are descriptive statistics that standardize some measure for comparative purposes.• Bivariate analysis and subgroup comparisons examine some type of relationship between

two variables.• The rules of thumb in making subgroup comparisons in bivariate percentage tables are

(1) “percentage down” and “compare across” or (2) “percentage across” and “compare down.”• Multivariate analysis is a method of analyzing the simultaneous relationships among sev-

eral variables and may be used to more fully understand the relationship between two vari-ables.

• Many measures of association are based on a proportionate reduction of error (PRE) model, which measures improvement in predictions about one variable, given information about a second variable.

• Lambda and gamma are PRE-based measures of association for nominal and ordinal vari-ables, respectively.

• Pearson’s product-moment correlation is a measure of association used in the analysis of two interval or ratio variables.

• Regression equations are computed based on a regression line-- the geometric line that represents, with the least amount of discrepancy, the actual location of points in a scattergram.

• The equation for a regression line predicts the values of a dependent variable based on values of one or more independent variables.

• Inferential statistics are used to estimate the generalizability of findings arrived at in the analysis of a sample to the larger population from which the sample has been selected.

• Inferences about some characteristic of a population, such as the percentage that favors gun control laws, must contain an indication of a confidence interval (the range within which the value is expected to be-- for example, between 45 and 55 percent favor gun control) and an indication of the confidence level (the likelihood that the value does fall within that range-- for example, 95 percent confidence).

• Tests of statistical significance estimate the likelihood that an association as large as the observed one could result from normal sampling error if no such association exists between the variables in the larger population.

• Statistical significance must not be confused with substantive significance, which means that an observed association is strong, important, or meaningful.

• Tests of statistical significance, strictly speaking, make assumptions about data and meth-ods that are almost never satisfied completely by real social research. Claiming a “statistically discernible relationship” is more appropriate when assumptions are not satisfied.

Key Terms:

Average (Page 401)Bivariate analysis (Page 408)Central tendency (Page 401)Contingency table (Page 411)Descriptive statistics (Page 399)Dispersion (Page 401)Frequency distributions (Page 400)Inferential statistics (Page 399)Level of significance (Page 423)Mean (Page 401)Median (Page 401)Mode (Page 401)Multivariate analysis (Page 411)Nonsampling error (Page 422)Null hypothesis (Page 425)Proportionate reduction or error (PRE) (Page 417)Range (Page 401)Regression analysis (Page 419)Standard deviation (Page 403)Statistical significance (Page 423)Statistically discernible difference (Page 428)Tests of statistical significance (Page 422)

Univariate analysis (Page 400)