Steps 3 & 4: Evaluating types of evidence for the Truckee River case study
description
Transcript of Steps 3 & 4: Evaluating types of evidence for the Truckee River case study
![Page 1: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/1.jpg)
Steps 3 & 4: Evaluating types of evidence for the Truckee River case study
![Page 2: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/2.jpg)
2
Define the Case
List Candidate Causes
Evaluate Data from Elsewhere
Identify Probable Cause
Detect or Suspect Biological Impairment
As Necessary: Acquire Data
and Iterate Process
Identify and Apportion Sources
Management Action: Eliminate or Control Sources, Monitor Results
Biological Condition Restored or Protected
Decision-maker and
Stakeholder Involvement
Stressor Identification
Step 3: Evaluate Data from the Case
Step 4: Evaluate Data from Elsewhere
![Page 3: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/3.jpg)
3
Types of evidence using data from the case• Spatial/temporal co-occurrence• Evidence of exposure or biological
mechanism• Causal pathway• Stressor-response relationships from
the field• Manipulation of exposure• Laboratory tests of site media• Temporal sequence• Verified predictions• Symptoms
Types of evidence using data from elsewhere• Stressor-response relationships from other
field studies• Stressor-response relationships from
laboratory studies• Stressor-response relationships from
ecological simulation models• Mechanistically plausible cause• Manipulation of exposure at other sites• Analogous stressors
Use all available types of evidence to make an inferential assessment
italics indicates commonly available types of evidence
![Page 4: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/4.jpg)
4
Basic analysis strategy
• Develop as many types of evidence, for as many candidate causes, as you can
– you won’t have all types of evidence, for all candidate causes– most effective when you can compare results across candidate
causes
• Work through one type of evidence, then set it aside – avoid cognitive overload
• Show your work– make your process transparent & reproducible – make use of appendices
![Page 5: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/5.jpg)
5
Let’s begin by figuring out what types of evidence we have for the Truckee…
![Page 6: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/6.jpg)
6
General vs. specific causation
• General – Does C cause E?– Does smoking cause lung cancer?
– Does increased water temperature reduce bull trout abundance in rivers?
• Specific – Did C cause E?– Did smoking cause lung cancer in
Ronald Fisher?
– Did increased water temperature reduce bull trout abundance in my stream?
![Page 7: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/7.jpg)
7
Specific causation: using data from the case
![Page 8: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/8.jpg)
8
Spatial/temporal co-occurrence
SUPPORTS
WEAKENS
Want paired measurements of proximate stressors & biological impairments, at locations where impairments are & are not observed.
![Page 9: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/9.jpg)
9
Causal pathway
Want paired measurements of other steps in causal pathway & biological impairments, at locations where impairments are & are not observed.
![Page 10: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/10.jpg)
10
Stressor-response relationships from the field
Want paired measurements of proximate stressors (or other steps in causal pathway) & biological impairments, at varying levels of exposure.
![Page 11: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/11.jpg)
11
Other types of evidence using data from case
TYPE OF EVIDENCE SUPPORTING EVIDENCE
Manipulation of exposure Impairment improves after stressor is removed
Laboratory tests of site media
Exposure to site media in lab tests results in effects similar to impairment
Evidence of exposure or biological mechanism
Measurements of biota (e.g., biomarkers, tissue residues) show proposed mechanism of exposure has occurred
Verified predictions Predictions based on stressor’s mode of action are made & confirmed at site
Temporal sequence Exposure to stressor precedes impairment
Symptoms Only one stressor supports observed symptom
![Page 12: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/12.jpg)
12
What types of evidence do we have, using data from the case?
![Page 13: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/13.jpg)
13
General causation: using data from elsewhere?
![Page 14: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/14.jpg)
Stressor-response relationships from other field studies
![Page 15: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/15.jpg)
15
Stressor-response relationships from the lab
![Page 16: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/16.jpg)
16
Other types of evidence using data from elsewhere
TYPE OF EVIDENCE SUPPORTING EVIDENCE
Stressor-response relationships from ecological simulation models
Stressor is at levels associated with impairment in mathematical models simulating ecological processes
Manipulation of exposure Impairment improves after stressor is removed at another site
Mechanistically plausible cause
Relationship between stressor & impairment is consistent with current scientific knowledge
Analogous stressors Stressor is structurally similar to other stressors known to cause impairment
Verified predictions Predictions based on stressor’s mode of action are made & confirmed at other sites
![Page 17: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/17.jpg)
17
What types of evidence do we have, using data from elsewhere?
![Page 18: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/18.jpg)
18
Now that we know what data we have, how do we analyze it?
![Page 19: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/19.jpg)
19
Spatial co-occurrence
Do your impairment and your stressor co-occur in space?
To Do:1. Load relevant data file2. Merge files3. Make boxplots for each
candidate causeSelect ‘reference’ and impaired sites
4. Fill in worksheet
![Page 20: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/20.jpg)
20
Causal Pathway
Does your data support the steps in the causal path between the stressor and the impairment?
To Do:1. Return to the conceptual
diagram2. Identify the steps in the causal
pathway3. Construct table to show
whether data supports the steps between the stressor and the impairment
4. Fill in worksheet
![Page 21: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/21.jpg)
21
Verified Prediction - Traits
Do data support predictions based on stressor’s mode of action?
To Do:1. Load relevant data file2. Merge files3. Make boxplot
Select ‘reference’ and impaired sites
4. Fill in worksheet
![Page 22: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/22.jpg)
22
Verified Prediction - PECBO
Do data support predictions based on stressor’s mode of action?
To Do:1. Load relevant data file2. Merge files3. Run PECBO4. Load PECBO results file into
CADStat5. Merge files6. Make boxplot
1. Sed2. STRMTEMP
7. Fill in worksheet
![Page 23: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/23.jpg)
23
Stressor-response from elsewhere
Does impairment decrease as exposure to the stressor decreases (or increases as exposure increases)?
To Do:1. Listen and ask lots of
questions2. Fill in the worksheet
![Page 24: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/24.jpg)
24
Randomized, controlled experiments
Key elements:• Replication: use of multiple
test units (e.g. tanks, sites)
• Controls: differ only by absence of the treatment
• Randomization: random assignment of test units to “control” or “treated” status
• Statistical analysis: estimate treatment effect (causal)
The scientific standard for establishing cause and effect
0
1
2
3
4
5
6
7
1 2 3 4
Toxin
Surv
ivor
sip
![Page 25: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/25.jpg)
25
Observational studies
Key elements:• Replication: collect data
from multiple test units
• Controls: ?• Randomization: ?• Statistical analysis: identify
associations among variables of interest (non-causal)
Often the only option for large-scale field studies
None
None
![Page 26: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/26.jpg)
26
Trade offs: control vs. realism, scale
Lab Experiment
Field Experiment
Observational Study
control
realism,scale
![Page 27: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/27.jpg)
27
Biomonitoring = Observational
Issues for causal analysis:
• Estimates of stressor effects are confounded by covarying factors
• Analyst can’t randomly assign treatments (stressors) to sites
* Reference sites are not experimental controls
![Page 28: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/28.jpg)
28
Analogous to clinical trials
Does smoking cause lung cancer?
• Estimates of stressor effects are confounded by covarying factors
• Analyst can’t randomly assign treatments (stressors) to subjects
* Non-smokers without lung cancer are not experimental controls
![Page 29: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/29.jpg)
29
Example using western EMAP*
Using propensity scores to infer cause-effect relationships in observational data
– Analysis and slides by Lester Yuan (USEPA), Amina Pollard (USEPA), and Daren Carlisle (USGS)
– Original presentation given at North American Benthological Society conference, May 2008
*EPA Environmental Monitoring and Assessment Program (EMAP)
![Page 30: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/30.jpg)
30
EMAP-West Study Area
Measurements Collected:
•Macroinvertebrates•Substrate composition
(SED)•Stream temperature
(STRMTEMP)•N = 838
Data collected by the EPA Environmental Monitoring and Assessment Program (EMAP)
![Page 31: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/31.jpg)
31
Total N vs. total taxon richness
Data from EMAP Western Pilot
SLOPE = -16.5
![Page 32: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/32.jpg)
32
Total N covaries with many other factors
![Page 33: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/33.jpg)
33
Multiple linear regression
Include covariates in the regression model to control for their effect.
SLOPE = -9.4
Regression model includes: %agriculture, %urban, grazing intensity, %sands/fines, stream temperature, and log conductivity.
SLOPE = -16.5
Correlation of Total Richness and Total N (ug/L)
![Page 34: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/34.jpg)
34
Potential issues with multiple regression
• Must assume that linear relationships are appropriate for all covariates.
• Regression model may extrapolate.
• Inclusion of certain variables may “mask” true effect:– e.g., part of the effect of agriculture may be
attributed to total N
![Page 35: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/35.jpg)
35
Alternate approach: Stratify dataset
r = -0.01 r = 0.15 r = 0.27
r = 0.64
![Page 36: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/36.jpg)
36
Model richness vs. total N within strata
How do we simultaneously stratify on many different covariates?
SLOPE = -10.7 SLOPE = -12.3 SLOPE = -9.7
![Page 37: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/37.jpg)
37
Propensity Score Matching
• Method developed in epidemiology to retroactively control for confounding effects in observational studies
• Sometimes called a quasi-experiment• Intuitively:
1. Model the magnitude of treatment (e.g. nutrient concentration) as a function of the covariates. The predicted magnitude of treatment at each site is its propensity score.
2. Stratify the total set of observations by the propensity scores (i.e., group sites with similar scores). Six strata are typically used.
3. Within each stratum, sites having different treatment levels (e.g. high vs. low nutrients) may be considered to have been “randomly assigned” to those treatment levels, because covariates have effectively been controlled by propensity score matching of “treated” and “control” sites.
![Page 38: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/38.jpg)
38
Propensity Score Model
Total N = f(percent agriculture, percent urban, grazing intensity, percent sand/fines, stream temperature, log conductivity, elevation, log catchment area, canopy cover, sampling day)
![Page 39: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/39.jpg)
39
Define 6 strata based on propensity scores
![Page 40: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/40.jpg)
40
Covariate values within strata
Grazing intensity
Percent agriculture
![Page 41: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/41.jpg)
41
Stratification by propensity score controls covariance of all modeled variables
Original r Min r Max r
Percent agriculture 0.60 0.06 0.24
Percent urban 0.29 -0.12 0.40
Grazing intensity 0.64 -0.21 0.18
Percent sand/fines 0.64 -0.16 0.13
Stream temperature 0.48 -0.13 0.22
log conductivity 0.68 -0.12 0.16
Elevation -0.26 -0.38 0.20
log catchment area 0.48 -0.26 0.21
Canopy cover 0.45 -0.13 0.09
Sampling day -0.17 -0.21 0.10
After stratification
![Page 42: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/42.jpg)
42
Total N vs. total taxon richness
Data from EMAP Western Pilot
SLOPE = -16.5
![Page 43: Steps 3 & 4: Evaluating types of evidence for the Truckee River case study](https://reader036.fdocuments.us/reader036/viewer/2022062502/5681533f550346895dc15d59/html5/thumbnails/43.jpg)
43
Total N vs. total richness: Stratified
SLOPE = -3.3 (n.s.)
SLOPE = -10.0***
SLOPE = -7.1*SLOPE = -7.1*
SLOPE = -10.5*** SLOPE = -8.1***