1

Data, integrity, and the scientific method: The case of bullet lead as forensic evidence and the JFK

assassination

Cliff Spiegelman

TAMU

Karen Kafadar, Bill Tobin, Dennis James, Simon Sheather, Stu Wexler, and Max Roundhill

2

From FBI Affidavit in Kulbicki case--2006

• 27. The FBI Laboratory established the comparative bullet lead program in response to requests for examination of the bullet fragments recovered from the body of President John F. Kennedy and the ammunition recovered from the firearm believed to have been used in his assassination. Establishment of the examination for routine casework followed from those initial efforts to determine if an association could be made between fragmented bullets and functional ammunition.

3

November 22, 1963

• A truly traumatic event event occurred on November 22, 1963.

• The nation and beyond has been wondering for decades was Lee Oswald the person solely responsible for the assassination.

• We will look closely at the testimony and science underlying the testimony of Dr. Guinn to the HSCA.

4

Testimony of Dr. Vincent Guinn Excerpts--each bullet is unique

• Page 494

• “So we have usually found [ in the general bullet population] much higher antimony levels than in the WCC Mannlicher-Carcano bullets.”

• “The other unusual feature of the WCC Mannlicher-Carcano is that there seems to be no uniformity within a production lot. That is, even when we would take a box of cartridges all from a given production lot, take 1 cartridge out and then another and then another and then another, all out of the same box-boxes of 20, these were-and analyze them, they all in general look different and widely different, particularly in their antimony content. This is not true of most bullet leads that we have ever looked at before, which are very uniform. In general, if you take most boxes”

5

What is the null hypothesis?

If you want to convince people that there was a conspiracy

then H0: There was no conspiracy

If you want to convince people there was no conspiracy then

H0: There was a conspiracy

Thus folks who say on TV you have to believe there was/was

not a conspiracy unless you can prove otherwise usally have

their null hypothesis wrong.

(Aside: This is a major issue in the FBI’s application of

CBLA.)

6

7

So are bullets essentially unique?

• Does not look like that does it?

8

FBI measurements

• The next few slides show antimony measurements from 4 boxes of of

the 4 largest domestic bullet manufacturers

• Does they look qualitatively different from the TAMU measurements of

the WCC bullets?

9

10

11

12

13

Testimony of Dr. Vincent Guinn-- excepts ContinuedTwo and only two bullets

• From Page 503• Mr. WOLF. Is it your opinion then that these all are fragments from WCC Mannlicher-Carcano bullets?• Dr. GUINN. I think that is their most likely origin, yes.• Mr. WOLF. Looking at these results, can you determine how many bullets these evidence specimens came

from?• From Page 504• Dr. GUINN. Yes, sir, I can.• Mr. WOLF. What is the number of bullets, in your opinion?• Dr. GUINN. These numbers correspond to two bullets . Two of the samples have indistinguishable

compositions, indicating that they came from the same bullet, and the other three particles are evidently samples from another bullet.

• Mr. WOLF. So it is your opinion that the evidence specimens represent only evidence of two bullets, is that correct?

• Dr. GUINN. Yes, sir, there is no evidence for three bullets, four bullets, or anything more than two, but there is clear evidence that there are two.

14

Description of our experiment

• We measured 30 WCC bullets from 3 separate boxes (2 lots)

• Lot 6000 and 6003P bullets we measure 3 times each--the FBI stated number of fragments

• Lot 6003 bullets were measured more than 10 times each with some bullets measured in many over 15 times including 3 separate radial locations 3 times each. The purpose of the multiple measurements is to get a sense of compositional bullet homogeneity

• 4 elements measured antimony, silver, arsenic, and copper (all the relevant elements that NAA can reasonably handle).

15

Guinn’s revised measurements

16

17

What is the likelihood of a 3rd bullet within the assassination fragments?

The ratio on the left hand side of the equation is a ratio of the probabilities for two bullets as the source for the five JFK fragments in question versus more than two bullets being the source for the five JFK fragments. The first ratio on the right hand side is the ratio of probabilities for the evidence given two bullets versus having more than two bullets. This ratio allows to decide whether, given the evidence, two bullets or more than two bullets is more likely.

Pr(T|E)Pr(T|E)=Pr(E|T)Pr(E|T)Pr(T)Pr(T)

18

The calculated ratio is:Pr(E|T)Pr(E|T) = .53.80

19

Our finding based upon our experience and NRC report

• JFK fragments need to be remeasured following NRC bullet committee guidelines

• The necessary science needs to be established for counting bullets contributing to the assassination fragments

– This includes:

• characterizing within bullet homogeneity/heterogeneity

• between bullet homogeneity/heterogeneity

• other types of bullet science not considered by the NRC committee -- metallurgical,…

20

Some Aspects of Work on the NRC Bullet Committee(This part is Joint with Karen Kafadar)

21

Committee members

A special thanks to Mike Cohen of the NRC for help editing the statistics sections, Changfu Cxaio for computer support, and Bill Tobin for background and

sharing some of his case information.

22

Background for CABL

• CABL uses seven selected elements—arsenic (As), antimony (Sb), tin (Sn), copper (Cu), bismuth (Bi), silver (Ag), and cadmium (Cd)—in the bullet lead alloy of both the crime-scene and suspect’s bullets.

• Measured by ICP-OES

• Various methods akin to Z or t tests done element by element comparing crime scene and suspects bullets. Sometimes clustering (called chaining by the FBI) is used. If two bullets are alike for all 7 elements a match is declared. The chance of a false match depends upon how close a truly non-matching bullet is

• Error ranges from nearly zero to over 40% if true non match is within 2 SD.

σ2 =σ 2(measurement error)+σ 2(between bullet)+σ 2(within bullet fragments)

23

Some Committee Recommendations

• Recommendation: The FBI should continue to measure the seven elements As, Sb, Sn, Cu, Bi, Ag, and Cd through ICP-OES as stated in the current analytical protocol. Also, the FBI should evaluate the potential gain from the use of high-performance ICP-OES because improvement in analytical precision may provide better discrimination.

• Recommendation: The FBI’s documented analytical protocol should be applied to all samples and should be followed by all examiners for every case.

• Recommendation: The FBI’s statistical protocol should be properly documented and followed by all examiners in every case.

• Finding: Variations among and within lead bullet manufacturers makes any modeling of the general manufacturing process unreliable and potentially misleading in CABL comparisons.

• Recommendation: Interpretation and testimony of examiners should be assessed regularly.

• Recommendation: Laboratory reports and expert witnesses on direct examination should acknowledge that different CIVLs can sometimes coincidentally be analytically indistinguishable.

M

M

24

Requested 71k case data was never made available to the committee

Committee requested “full” data log of ~71,000 bullet lead measurements.

Two files purportedly fulfilling this request were delivered on CD-ROM at

the final committee meeting. The two files together contain 56,260

measurements. These files were incomplete without a record of bullets in

a case

(if any) that were found by the FBI crime lab to match. An partial analysis

of these data later this month.

25

Why case data would be useful

• Compare findings to testimony

• Compare frequency of matches among examiners

• Compare relative standard errors among tech (may not exist)

• Compare matching algorithm among examiners

• Find trends in measurements and bullet compositions over time

• See if practice matches the descriptions given to panel

26

Additional reasons that the data is useful

• Does the data support inter-examiner reliability and consistency?

• Does the data support identification of patterns or inconsistencies in testimonies?

• Did the FBI use different protocols: number of ``slivers'' per bullet?

• How many matches are attributed to the different match criteria that were used?

• What were the different match criteria used?

• Number of measurements per bullet? ("choose 3 of the most consistent matches"?)

27

Scientific Method

• The scientific method is supported by common sense

• Is the nondisclosure of case data to the NRC panel or failing to make all case data generally available to scientific bodies sensible?

• In at least one field of study the NRC has made recommendations about data disclosure. The results can be found in:

– Sharing Publication-Related Data and Materials: Responsibilities of Authorship in the Life Sciences http://books.nap.edu/catalog/10613.html

28

Why the scientific method matters

In Daubert v. Merrell Dow Pharmaceuticals, Inc., the Supreme Court interpreted

an earlier version of Rule 702 to require that scientific evidence meet a reliability

test. The Court wrote that “in order to qualify as ‘scientific knowledge,’ an

inference or assertion must be derived by the scientific method. Proposed

testimony must be supported by appropriate validation—i.e., ‘good grounds,’

based on what is known. In short, the requirement that an expert’s testimony

pertain to ‘scientific knowledge’ establishes a standard of evidentiary reliability.”.

29

The scientific method depends upon common sense(From Scientific Method In Practice, by Gauch,

Cambridge University Press, 2003)

30

Sharing Publication-Related Data and Materials: Responsibilities of Authorship in the Life Sciences

• Community standards for sharing publication-related data and materials should flow from the general principle that the publication of scientific information is intended to move science forward. More specifically, the act of publishing is a quid pro quo in which authors receive credit and acknowledgment in exchange for disclosure of their scientific findings. An author’s obligation is not only to release data and materials to enable others to verify or replicate published findings (as journals already implicitly or explicitly require) but also to provide them in a form on which other scientists can build with further research. All members of the scientific community—whether working in academia, government, or a commercial enterprise—have equal responsibility for upholding community standards as participants in the publication system, and all should be equally able to derive benefits from it.

31

Some of what we have found with limited data sets provided at the last NRC committee meeting

• No case data were provided indicating whether a match was found despite numerous requests from all levels of the NRC committee

• The data that were provided at the end of the last NRC committee meeting contained chemical measurements for 1079 of the roughly 2500 cases that the FBI has handled since 1980. (It has been doing CABL for a much longer period.) It was incomplete with over 85% of cases missing at least all K bullets or all Q bullets. Some cases, as well, appear to be missing all K bullets and some Q bullets.

– Out of these 1079 cases only 138 cases had both K and Q bullets• There are many extreme outliers with standard errors > 1012 , (> 1.2% of the records had

such outliers). No data given to the NRC committee before final deliberations had such outliers. So we never asked how they were handled.

– No indication from the FBI, to the committee on a uniform policy for handling outliers. Hopefully they did not use them.

• Other findings follow…

32

Observations

• No cases identified whether there was a match (or a method(s) for how a match was obtained)

• There are massive inconsistencies among FBI data sets and FBI descriptions of the data sets

• There are differences in the chemical compositions patterns of bullets according to the caliber of the bullets- relevant to how the FBI calculated the probability of false matches.

• Only a few cases, 138, in the data provided have both K and Q bullets

– K bullets are those found at the crime scene

– Q bullets are suspect bullets

• The more K and Q bullets the more matches ( This is a point that is not brought out by the FBI).

33

How the number of matches varies with the number of K & Q bullets (from lead2.xls)

• As expected the number of 2 SD overlap matches increases as the number of K*Q (the Bonferroni count) increases. A similar increase is

seen in the percentage of cases that have matches when plotted against K*Q.

34

U.S. Department of Justice Federal Bureau of Investigation

October 1999 Volume 1 Number 3

Statistics and Trace Evidence:

The Tyranny of NumbersMax M. Houck

Supervisory Physical Scientist Trace

Evidence UnitFederal Bureau of Investigation

Washington, DC

35

Excerpts from “The Tyranny of Numbers”

• How Do We Know All Ravens Are Black?

• The tyranny of numbers is a consequence of an over reliance on deduction and mathematics, and these ultimately limit a discipline by requiring it to fit into a preordained model. Equating quantification with science to justify and validate its "science-ness" indicates that a "faulty notion of science, or no notion at all," is at the heart of the tyranny (Mayr 1982)

• It isn't that they can't see the solution. It is that they can't see the problem." —G. K. Chesterton…

36

Excerpts Con’t

• …How does one determine the frequency of a fiber type? You could look for a specific fiber on unrelated garments (Houck and Siegel 1999) to gain an idea of its commonness or rarity. It could also be possible to determine the frequency of fiber types within a textile population, either by cataloging the fibers (Home and Dudley 1980) or the textile population (Biermann and Grieve 1996a, 1996b, 1998) or by sampling selected environments to determine chance matches (Palmer and Chinherende 1996). Small ad hoc studies could also be performed to answer specific questions of frequency (Deedrick 1998). The difficulty with all of these approaches is that they are not universal in their application, if they can be applied to casework at all. Questions of sampling, randomness, and relationships impinge on this process and complicate matters.… Before trace analysts can comfortably approach using statistics to present their results to a judge or jury, it is critical that these fundamental questions be answered and those answers be accepted by the trace evidence community.

37

How the FBI crime lab takes the committees recommendations

Forensic Significance of Bullet Lead by Koons and Buscaglia, JFS 2005.

38

So is the 1837 dataset is a reasonable representation of the FBI case load?

39

Conclusions

• Forensic science laboratories need to make their data, materials, and methods available to responsible organizations and individuals upon request

• The FBI crime lab needs to embrace all aspects of the scientific method including:

– Recommendations from major scientific bodies such as the NRC

– Mathematical aspects of the scientific method such as sampling, data analysis, inference,…

– Creating an atmosphere of openness

40

Prologue• Dwight, • I hear what you're saying and understand your concern. I would add,

however, that this is a very tight examination now. There will be no such thing as an inconclusive, provided we could do the examination. Either they'll match or they won't using our revised statistical protocol. The exception may be with fragments where we can at least rule out a common source of the lead, but the new technique will not allow us to use a single measurement to claim that we have a match.

• The only reason that we had issues in the past was because we allowed for a subjective variable ... namely "experience "....to somehow figure into whether a call was made or not. Now it is all done behind the scenes with our statistical program. Looking at all the cases that we reported since 1996, only 1.4% of the cases resulted in a different report being issued. To the best of our ability, we have identified why those 7 cases have a different result today and it is mainly due to the use of 'chaining 'I... some of it used quite liberally.

• I think some of your concern may be due to discussions that were held last year when we were looking at the use of the "equivalence test". If we had continued with that method of assessing the data, we would have been making a lot more false exclusions. As you will recall, we switched horses at that point and did the false positive probability study that will allow us to use a very traditional statistical technique ... namely the student t test. I can't imagine anyone really questioning the use of this statistical technique in the scientific community.