The Practice of Firearm/Toolmarks and a Reasonable Path ...€¦ · Firearm toolmarks examinations...
Transcript of The Practice of Firearm/Toolmarks and a Reasonable Path ...€¦ · Firearm toolmarks examinations...
-
Copyright Cliff Spiegelman 1
The Practice of Firearm/Toolmarks and a Reasonable Path Forward
Cliff Spiegelman, Texas A&M UniversityCoauthor William A. Tobin
-
Copyright Cliff Spiegelman 2
Three Parts Of The Talk
What are Firearm Toolmarks
What statements etc. does the scientific method allow?
Explanation of types of error with examples
Three families of experiments to start laying the
statistical foundation for firearm/toolmarks
Discussion
-
Copyright Cliff Spiegelman 3
Firearm/toolmarks
Firearm toolmarks examinations and comparisons are often used in investigations of homicides involving a firearm, spent bullets and/or cartridge cases that are recovered from crime scenes. Most frequently, one or more firearms are recovered during investigation of a shooting incident and typically submitted for forensic comparisons with bullets and/or cartridge cases recovered from the scene. The forensic practice used to associate or eliminate a particular firearm as the murder weapon is based on comparisons of characteristics imparted to bullets and cartridge cases during cycling of a cartridge through the gun, and is known as firearm/toolmarks examination.
-
Copyright Cliff Spiegelman 4
Typical testimony
If it is concluded that the submitted weapon “matches”the crime scene bullets, the firearm/toolmarks examiner typically testifies at trial that the crime scene bullets were fired from the gun to the exclusion of all other possible weapons, although sometimes “to a practical certainty.”
-
What Claims does the Scientific Method Support?
It supports claims that have been reasonably verified by hypotheses tests and corroborated by others. This means repeatable and reproducible and robust reproducible (in the case of broad claims) results
Repeatability: The same experimental set up gets the same resultsReproducibility: Similar setups that use different examiners at different locations get the same experiments resultsRobust Reproducibility: Widely varied experimental setups get the supporting results
Copyright Cliff Spiegelman 5
-
What the scientific method does not allow
Without repeatability claims should be restricted to the (crime) lab and the original experiment. Nothing can be said about any other experiment.
Without reproducibility claims should be restricted to the (crime) labs participating in the experiment and experiments using the same factors and in the same way.
Without robust reproducibility claims cannot be extended beyond the specific factors used in the experiment.
Copyright Cliff Spiegelman 6
-
Error According To The Apple Dictionary
Technical: a measure of the estimated difference
between the observed or calculated value of a
quantity and its true value.
Law: a mistake of fact or of law in a court's opinion,
judgment or order.
Phrases: see the error of one's ways realize or
acknowledge one's wrongdoing.
Copyright Cliff Spiegelman 7
-
Two Categories Of Error Used In This Presentation
Systematic or sometimes referred to as biasBetter know examples:
Publication bias: publishing only positive resultsDifferent instrumentation as in different light sources (much more about this later)
Random errorBetter know examples:
Variation among examiners trained within the same school ('CMS' or classical)Variation in marks from the same firearm
Copyright Cliff Spiegelman 8
-
More Bias Examples
Underreported, summary dismissal, or unreported false IDs
Missouri and other cases reported in “Gun evidence often wide of scientific mark” in the Austin American Statesman
Anthony Ray Hinton exonerated after 30 years on death row
Former Detroit firearm unit (~10% error rate)
Blame the lab or examiner rather than the process
Question? Any such cases in the FBI firearm/toolmark unit?
Present publications in trade journals as publications in science journals
Copyright Cliff Spiegelman 9
-
AAS Article Highlighted By AFTE Website
Copyright Cliff Spiegelman 10
-
More Bias Examples
Myopic view of resultsNot involve the broader scientific community in research investigations but keep them within 'house’
Myopic view of results spreads by publication bias etc..
Restrictions to visible light (more later)Not combining human with computer determinations Using tortured language to deny a practical error rate
‘match to a practical certainty’ or ‘practical impossibility of being wrong’
Copyright Cliff Spiegelman 11
-
More on Bias
An example of publication bias on steroids
Have a trade organization claim it’s publishes a science journal and publish non statistically designed, not double blinded studies and other pseudo-experiments as broad based proof of efficacy.
Copyright Cliff Spiegelman 12
-
More Random Error
Within bullet brand variation in charge and hence variation in striations
Within bullet brand variation in bullet hardness (antimony levels) and hence variation in striations
Random variations in light output
Locations of bullet damage from striking bone or buildings
Within bullet brand variation in cartridge hardness and hence variation in ejector marks
Copyright Cliff Spiegelman 13
-
More Random Error
Within bullet brand variation in primer cup hardness and
hence variation in firing pin impressions
Cleanliness of weapon components
Condition of lubrication system, nested within lubrication
nature and regime.
……………………………etc..
-
Copyright Cliff Spiegelman 15
-
Copyright Cliff Spiegelman 16
Some Factors To Consider During Firearm/Toolmark Experiments
Factors not considered or discussed, and sample size issues:Ammunition typeAmmunition chargeCartridge case hardnessPrimer cup hardnessBreechface hardnessFiring pin hardnessDifferent batches of GlocksNon-Glock firearms“feeds and speeds” of production (or, alternatively, economic conditions of the manufacturing environment)
-
Copyright Cliff Spiegelman 17
Factors To Consider (Continued)Ids from firing pinsIds from ejector marksIds from breechface marksIds from combination of ejector, firing pin, and breechface marksWeapon cleanlinessParticipants’ experience as toolmarks examinersParticipants asked to handle test as casework, but no measures of effectiveness for this instruction reportedBreak-in period for pistols Lubrication regimeCondition of lubrication system, nested within lubrication regime
-
Copyright Cliff Spiegelman 18
Factors (Continued)
Temper of workpieceMicrostructure of workpieceFinishing processes (assertion only by author that “breechface is unaffected during all remaining operations” but no description as to what constituted “remaining operations” for interpretation of possible metallurgical effects)Effect of checking random sample of test cartridges for clear marks by experimenter before proceeding with the experiment was not investigated. This cannot be done in practice, as criminals do not check to insure their cartridges are clearly identifiable before leaving them at crime scenes.
-
Still More Factors
Batch and sample not chosen randomly; Two 9 mm Lugers were not chosen randomlyAcceptable undersized ammunitionBrand of ammunitionResponse measures Fabrication tooling materials, if anyFabrication tooling hardnessType of workpieceAlloy used for workpiece
-
Copyright Cliff Spiegelman 20
Possible Experimental Approaches to begin Laying a Statistical Foundation for Firearm
ToolmarksThree classes of experiments will be presented
1. To build trust and confidence and motivate SOP writing. To build a team that can function together.2. To assess examiners’ abilities to differentiate class, subclass, and individual characteristics3. To assess error rates and assess their dependence upon many factors. To assess the importance of many factors and variables
-
Copyright Cliff Spiegelman 21
Experiments Continued
The experimental approaches should proceed sequentially and under no circumstances should the third set of experiments be performed before the first two.The reason is that without an SOP and without understanding examiners’ ability to distinguish various types of marks, error assessment will not serve a scientific purposes well. They can serve ‘got you’ goals. For example…
-
Copyright Cliff Spiegelman 22
1. Building Confidence And Trust And Starting A Detailed SOP
Pick a moderately priced pistol and film test fires from itSend the same test fired bullets (or cartridge casings) and video to participants
Ask participants if the bullet was fired from the test weapon and to give the complete list of marks used or that could be used to match the bullet to the test gun. A template (2 dimensional blueprint, if need be colors can be used to demonstrate a 3rd dimension) of the bullet should be provided for examiners to show what marks that they used. Lands and groves should be numbered so that results from different examiners can be compared
-
Copyright Cliff Spiegelman 23
Experiment Type 1 Continued
After the results are returned summaries should be sent to all participants for discussion
Discrepancies should be discussedFollow on experiments could remove some lands and groves from the original test bullets and the experiment repeated
Eventually an SOP starts developing
-
Copyright Cliff Spiegelman 24
2. Assess Examiners’ Abilities to Differentiate Class, Subclass, and Individual Characteristics
(We use bullets but a similar approach can be used for cartridge
casings)
Select a few production runs and choose 2 consecutively manufactures runs from each. The runs should be chosen to cover different types of subclass characteristics and unique identifying marks (dirty lubrication vs. clean lubrication)
Before each firing of the pistols make a map of each pistol barrel (using cameras and microscopy)Film test fires from each pistol and label them to the proper pistol and make video available to participating examiners
-
Copyright Cliff Spiegelman 25
Experiment Type 2 Continued
Provide a diagram to the FTE of all marks on the bullets. Ask them to label which ones are class, subclass, and unique characteristics. (Do not ask them to match bullets to guns. )After the results are returned summaries should be sent to all participants for discussion
Discrepancies should be discussed. The goal is to characterize how well different types of marks can be characterized.
Follow on experiments could try different ammunition brands and types (of bullets-jacketed, hollow point etc.)
-
Copyright Cliff Spiegelman 26
3. Error Rate Assessment
Perform screening experiments from among the 50+ factors and variables to find out the most relevant and important ones affecting examiner performance.
Physical factors such as gun type, and ammunition type, microscopes and lighting, etc.Social factors such as training and certification type
These experiments will need complete buy in. Social psychologists should be consulted to help make tests representative of casework
-
Copyright Cliff Spiegelman 27
Example of a screening experiment
-
Subjective Comparison Procedures Still Subject To Statistical Laws And Issues
Multiple comparison issuesModeling errors (misspecified likelihoods particularly in tail regions)
Need confidence statementsPopulations need to be clearly defined
Data with annotation are neededAnalyses need to grounded within the field of statistics
-
A
D
Com
posi
tion
-
A
B
C
D
Com
posi
tion
By chaining, ‘D’ becomes “analytically indistinguishable” from ‘A’
-
A Hypothetical: Example: Madrid Bombing
-
Inference
Deductive inference:The fingerprints correspond…… therefore the suspect is the offender
Not appropriate in most cases in forensic scienceWould involve explicit exclusion of all other sources than the suspected one
-
Inference
Inductive inferenceIf the suspect is the offender, then the fingerprints should correspond…… the fingerprints correspond …… therefore, it is possible that the suspect and the offender are the same person
Appropriate to most cases
-
Inductive inference
Intuitively depends on:
-
Inductive inference
Intuitively depends on:Similarity between trace and control materialRarity of control materialBackground circumstances
-
Current Situation
It is very difficult for an audience to appreciate what the scientist truly means, and how to use the conveyed information to reach a decision
What and How are not considered separately in Forensic Science
No real consensus on the What(not even the beginning of one in the US)No study on the How
PresenterPresentation NotesReinforce that it is not only about knowing what we mean, but also ensuring that the audience unerstand it. NOTE
-
There is no good way to report these conclusions
Weight of evidence convey information regarding:Level of agreement between trace and control objectsLevel of “rarity” of the characteristics of the tracePotentially error rate(s)Potentially relevance, transfer and persistence
It does not:Make assumptions on size of population of potential offendersInvolve considering factors unrelated to the evidence
But weight of evidence conveys the information in an obscure way, and the audience may not be able to readily use it in its decision-making process
-
There is no good way to report these conclusions
Weight of evidence convey information regarding:Level of agreement between trace and control objectsLevel of “rarity” of the characteristics of the tracePotentially error rate(s)Potentially relevance, transfer and persistence
It does not:Make assumptions on size of population of potential offendersInvolve considering factors unrelated to the evidence
But weight of evidence conveys the information in an obscure way, and the audience may not be able to readily use it in its decision-making process
Thompson el al. (2013) Do Jurors Give Appropriate Weight to Forensic
Identification Evidence? Journal of Empirical Legal Studies 10(2) 359-397
-
Psychology of Effective Communication
Three main theories:Frequency theory – Theorizes that human beings are more competent with counts than with probabilities because they have been exposed to them more across evolution.Cognitive experiential approach – Originates from psychodynamics: different personality types, some relying more on numbers, some relying more on intuition. Intuition represents a lower level of development than numeracy.Fuzzy trace theory – Originates from cognitive research: individuals rely on their gist (substance of information –intuition ) and verbatim (exact representation of information - numeracy) to make decisions. Intuition represents a higher level of development than numeracy.
-
Psychology of Effective Communication
Three main theories:Frequency theory – Theorizes that human beings are more competent with counts than with probabilities because they have been exposed to them more across evolution.Cognitive experiential approach – Originates from psychodynamics: different personality types, some relying more on numbers, some relying more on intuition. Intuition represents a lower level of development than numeracy.Fuzzy trace theory – Originates from cognitive research: individuals rely on their gist (substance of information –intuition ) and verbatim (exact representation of information - numeracy) to make decisions. Intuition represents a higher level of development than numeracy.
Human relies on the least precise gist representations
necessary to make a decision
-
Psychology of Effective Communication
According to these theories, comprehension of information (verbal or numerical) and resulting actions/decisions are generally influenced by:
Ability to mentally conceptualize the problemFormat of the informationExpectationSeverity of the possible outcome
Ability to retrieve knowledge/values from memoryPast experienceSpecific contextCueing of relevant knowledge/values to consider
Ability to apply reasoning processesProcessing interferences
-
So right now what would the scientific method allow for testimony
1. In the opinion of the examiner the bullets and/or cartridge casings have marks consistent with being fired from the evidence weapon.
The prosecutor and defense attorney should ask how many weapons might be consistent with this evidence.
The examiner can answer a small percentage but should say that they do not know how small and that a few thousand weapons would be a small percentage.
-
So right now what would the scientific method allow for testimony
2. In the opinion of the examiner the evidence weapon cannot be eliminated as making the marks on the bullets and/or cartridge casings found at the crime scene.
The prosecutor and defense attorney should ask how many weapons could there be that would not be eliminated as providing this evidence?The examiner can answer a small percentage but should say that they do not know how small and that a few thousand weapons would be a small percentage.
-
How can these statements be made stronger?
By using statistically designed experimentsEstimate the probability of misidentifying ‘matches’ due to subclass characteristics.
Use uncertainty intervals in testimonyEstimate the probability of misidentifying a ‘match’ from a weapon that does not share subclass characteristics.
Use uncertainty intervals in testimony
-
Lack of statistical and mathematical input hurts us all
If the wrong person is convicted in part or wholly due to overstated testimony or faulty forensic practice (say dry labbing) then the guilty person is free to commit other crimes, such as murder and rape. Of course long years on death row or worse for the wrongfully convicted are unintended consequences of the current forensic practice. This is a fixable problem.
-
Using IR images instead of visual light
Copyright Cliff Spiegelman 46
-
More IR detail
-
More IR detail
-
The Practice of Firearm/Toolmarks and a Reasonable Path Forward Three Parts Of The TalkFirearm/toolmarksTypical testimonyWhat Claims does the Scientific Method Support?What the scientific method does not allowError According To The Apple DictionaryTwo Categories Of Error Used In This PresentationMore Bias ExamplesAAS Article Highlighted By AFTE WebsiteMore Bias ExamplesMore on BiasMore Random ErrorMore Random ErrorSlide Number 15Some Factors To Consider During Firearm/Toolmark ExperimentsFactors To Consider (Continued)Factors (Continued)Still More FactorsPossible Experimental Approaches to begin Laying a Statistical Foundation for Firearm ToolmarksExperiments Continued1. Building Confidence And Trust And Starting A Detailed SOPExperiment Type 1 Continued2. Assess Examiners’ Abilities to Differentiate Class, Subclass, and Individual Characteristics�(We use bullets but a similar approach can be used for cartridge casings) Experiment Type 2 Continued3. Error Rate AssessmentExample of a screening experimentSubjective Comparison Procedures Still Subject To Statistical Laws And IssuesSlide Number 29Slide Number 30A Hypothetical: Example: Madrid BombingInferenceInferenceInductive inferenceInductive inferenceCurrent SituationThere is no good way to report �these conclusionsThere is no good way to report �these conclusionsPsychology of Effective CommunicationPsychology of Effective CommunicationPsychology of Effective CommunicationSo right now what would the scientific method allow for testimonySo right now what would the scientific method allow for testimonyHow can these statements be made stronger?Lack of statistical and mathematical input hurts us allUsing IR images instead of visual lightMore IR detailMore IR detailSlide Number 49