How do numbers sing?

The uses of statistics in policy environments.

Epistemology and semiotics

Numbers are used to tell stories that decision-makers assume are reality

In and of themselves, numbers are not evidence

Numbers are representations of reality The story is the derived reality

Transformations of meaning

STORY (reality #1) shaping, presentation, translation NUMBERS (reality #2) shaping to determine data elements

EVIDENCE (reality #3) determined by the STORY

As representations of reality . . .

Numbers are assembled within metric categories (continuous, categorical, dichotomous) and semantic categories

(e.g. “academic year”)

Once assembled, their representation is

controlled by denominator, that is, the definition of the universe being considered

Quality control at the point of assembly Evidence is gathered and first turned into

numbers at micro levels, and it is at these levels that quality control must function at its highest intensity

Aggregates of data will follow; derived variables will follow.

The further removed from the first construction, the more statistical noise

When numbers become data and then become stories . . .

They serve a propagandistic function in the sense that they are used to set the conditions of policy

The data can be manipulated by categories and denominators to produce good stories or bad stories

But honest stories are a better policy. Why?

We can tell a bad story, a good story, or an honest story. An honest story . . .

Is derived, as much as possible, from unobtrusive evidence, e.g. student records

Always tells you who is in the denominator—and who is not in the denominator

Matches its universe against another external reference point

Does not mix people from different planets Hence, provides more accurate indicators of

where problems may lie

Case #1: “Retention” of students from year 1 to year 2 of university study The statistical definition of “retention” was

based on institutions providing data of the proportion of students who started in the fall term of year 1 who were still enrolled in the fall term of year 2 at the same institution

But a national sample of student records found that (a) 20 percent of the students did not start in the fall term, (b) 62 percent attended during summer terms, and © 57 percent attended more than one institution

Case #1 (continued)

The “propaganda” of low retention rates damages the reputation of institutions

The policy issues involve financial support for institutions, principally by state governments

The honest story—based on unobtrusive evidence (student records), taking into attendance patterns into account, and following the student—results in a very high system retention rate, and relieves pressure on institutions

Case #1 (continued)

The honest story goes on to raise another set of questions about the quality of “retention.” Using student records, it asks

How many credits were earned in the 1st year by students who were “retained”? How many remedial courses did they take? What was their grade point average? And how to these (and other features of student academic history) relate to their course-taking and performance in secondary school

Case #1 (continued again)

The honest data story says that while students can be “retained” in the system at a high rate (88 percent, in fact), that does not mean these students have sufficient momentum to complete degrees

With these data, the honest story can isolate groups of students who can be targeted for special pre-college preparation programs, and/or who can receive better academic advisement in college

Prelude to Case #2: What proportion holds a 1st university degree? Population 25-29 years old 29% All students entering postsecondary 25%

education in 1995 by 2001 Recent secondary school graduates 67%

entering 4-year colleges in 1992, by

2000

The answer is in the denominator

Case #2: What proportion of postsecondary students earns a 1st university degree? People who want to tell bad stories do not

discriminate the student population by age in the term of first attendance

People who want to tell bad stories include students who attend non-degree- granting vocational schools in the denominator

Case #2 (continued)

If student records are used, and The denominator marks all students who

(a) start out in higher education by age 20-21, and, at any time, (b) attend an institution that grants 1st university degrees . . .

Then 2 out of 3 students earns a 1st university degree by age 26/26. That is very good.

Why raise Case #2?

Because the bad stories (low completion rates) dominate the media, they set a tone for the policy environment that offers excuses for reducing public financial support

This is a classic case of the battle between bad and honest stories to determine the policy environment

Case #3: Quality control in data entry: occupational classifications Computer Assisted Telephone Interviews Literal strings of words entered Data entry clerk makes classification When examined later, 2000 out of 12,000

classifications were wrong Problems lie in broad categories such as

Manager: Mid-Level, and in distinctions between Laboratory Technician/Other Technical/Medical Services.

If we do not classify occupations accurately . . . We cannot analyze relationships between

education and the labor market with confidence

Vocational/occupational education policy will not be targeted for highest efficiency

Predictions—and the public funding commitments based on predictions—will not be effective

CASE #4: Problems in coding academic course work: what questions do we ask?

4-year colleges

2-year colleges

Less than 2-year

Telecomm Technology

66% 22% 12%

Information Technology

33 25 42

Veterinary Medicine

55 44 1

CASE #4: These are not numbers—they are data. What do they tell us?

1) We have made a mistake in placing Telecommunications Technologies and Information Technologies in the same aggregate category of “Engineering Technologies”

2) There were gross errors in classifying “Veterinary Medicine” courses

We can fix #2 easily. But #1 is not an easy case.

Confusions in data entry for “Veterinary Medicine” Courses such as ‘Animal Care’ or ‘Clinical

Animal Technology’ are vocational courses for veterinary assistants—Code 170508

Course titles such as ‘Feline Medicine’ or ‘Avian Diseases’ are professional courses for veterinarians---Code 182401

It is not uncommon for a School of Veterinary Medicine to prepare students at both levels, but most vocational offerings are at 2-year colleges

It is worth repeating after these examples . . .

STORY (reality #1) shaping, presentation, translation NUMBERS (reality #2) shaping to determine data elements

EVIDENCE (reality #3) determined by the STORY