Competências Básicas de Investigação Científica e de Publicação

Ganesha Associates CC BY 3.0 1

Competências Básicas de Investigação Científica e de Publicação

Lecture 2: Hypotheses, reproducibility and experimental design

9 September 2013


The scientific process involves making models of how things work

• These evolving models are described in the scientific literature

• Sometimes the models are wrong, often they are incomplete

• Scientific progress is driven by the communication and publication of the results of new research, and the reinterpretation of older work

• The tool which makes all of this possible is the hypothesis

9 September 2013


The scientific process involves making models of how things work

9 September 2013

Ganesha Associates CC BY 3.0 49 September 2013


A visual abstract from Cell


Experimental vs. Observational studies

No modification of experimental variablesUseful to discover trends and associationsCannot directly be used to infer causality

Compare responses different treatmentsDesigned to avoid misleading results

e.g. randomisationCan be used to infer cause and effect9 September 2013

9 September 2013 Ganesha Associates CC BY 3.0 9

Main learning points

• Student projects fall into three categories– No hypothesis, i.e. observational– Weak hypothesis– Strong hypothesis

• The work will be published in a – National journal– Low impact factor journal– High impact factor journal

• Starting with strong hypothesis improves your chances of getting published in a good journal


What is a strong hypothesis ?

• A strong hypothesis is based on a series of premises – things that are already known with some certainty

• Each premise must be supported by references back to the (international) primary literature

• So a strong hypothesis will be backed by references to recent papers in high quality journals


Why did the Titanic sink?It was Captain Smith's faultThis was Captain E. J. Smith's retirement trip. All he had to do was get to New York in record time. Captain E. J. Smith said years before the Titanic's voyage, "I cannot imagine any condition which would cause a ship to founder. Modern shipbuilding has gone beyond that." Captain Smith ignored seven iceberg warnings from his crew and other ships. If he had called for the ship to slow down then maybe the Titanic disaster would not have happened.It was the shipbuilder's faultAbout three million rivets were used to hold the sections of the Titanic together. Some rivets have been recovered from the wreck and analysed. The findings show that they were made of sub-standard iron. When the ship hit the iceberg, the force of the impact caused the heads of the rivets to break and the sections of the Titanic to come apart. If good quality iron rivets had been used the sections may have stayed together and the ship may not have sunk.It was Bruce Ismay's faultBruce Ismay was the Managing Director of the White Star Line and he was aboard the Titanic. Competition for Atlantic passengers was fierce and the White Star Line wanted to show that they could make a six-day crossing. To meet this schedule the Titanic could not afford to slow down. It is believed that Ismay put pressure on Captain Smith to maintain the speed of the ship. It was Thomas Andrews' faultThe belief that the ship was unsinkable was, in part, due to the fact that the Titanic had sixteen watertight compartments. However, the compartments did not reach as high as they should have done. The White Star Line did not want them to go all the way up because this would have reduced living space in first class. If Mr Andrews, the ship's architect, had insisted on making them the correct height then maybe the Titanic would not have sunk. It was Captain Lord's FaultThe final iceberg warning sent to Titanic was from the Californian. Captained by Stanley Lord, she had stopped for the night about 19 miles north of Titanic. At around 11.15, Californian's radio operator turned off the radio and went to bed. Sometime after midnight the crew on watch reported seeing rockets being fired into the sky from a big liner. Captain Lord was informed but it was concluded that the ship was having a party. No action was taken by the Californian. If the Californian had turned on the radio she would have heard the distress messages from Titanic and would have been able to reach the ship in time to save all passengers.

9 September 2013


Coin-tossing - an example• I wonder how many heads or tails I will get if I toss

this coin 100 times– No model

• The frequency distribution of heads and tails will be approximated by a binomial distribution with n=100 and p=0.5– Simple model, based on symmetry

• A detailed analysis of the dynamics reveals that the probability of a head is 0.51– Complex model, based on asymmetry, aerodynamics, etc


Coin-tossing – impact on CV1. None, or possibly negative

2. R. A. Fisher and others did perform this experiment in the early days of biological statistics, before the advent of computers, as a proof that the binomial distribution tended towards a normal one at high levels of n.

Interestingly they all found that the probability of a head p was usually slightly higher than 0.5, but this difference was ignored.

3. Persi Diacusis, Susan Holmes and Richard Montgomery (Stanford, 2004) publish a paper on the ‘Dynamical bias in the coin toss’ proving that the lack of total symmetry in a coin means that the probability of a head will always be slightly greater than 0.5.


Coin tossing - relevance• I think that there will be an association (+ or -) between mutations

in gene x and susceptibility to disease y– No causal basis for a relationship given

• I predict that mutations in gene x will increase susceptibility to disease y because patients with disease y often have low levels of gene product x.– Built-in control, patients with normal levels of the gene product should not

have the disease.• I predict that chemically non-neutral mutations in gene x will

increase susceptibility to disease y in patients with low levels of gene product x.– Second level of control – neutral mutations should be asymptomatic


Coin-tossing – moral of the story

• With a strong hypothesis, you:– Avoid following leads which go nowhere – false

positives, fail early– Avoid ignoring unexpected observations that are

of high interest – false negatives– May need to do less work !– Will get published in better journals !


Testing a hypothesis - 1• The first stage of any scientific analysis is to define

the null hypothesis that is to be tested.

• This makes the prediction that our chosen variable has no impact on the outcome we are measuring.

• As an example, the null hypothesis could be that “there is no difference in rainfall levels between urban and adjacent rural areas”.

• The null hypothesis is denoted symbolically as H0


• It is also necessary to state the alternative hypothesis (H1) – that there is an increase in precipitation levels in urban areas relative to adjacent rural areas because of the heating differences of the two surface types (the urban area heats up more and has increased convective uplift).

• Notice, it is a lot easier to specify the H0 version. H1 requires an understanding of convection, thermodynamics and so forth.

Testing a hypothesis - 2


• The H0 hypothesis protects us from trying to explain results that have no significance.

• The H1 hypothesis challenges us to make sure we have thought of all of the other factors that could cause the effects we observe and excluded them by appropriate experimental controls or via additional experiments.



• You can often spot problems with project design by analyzing the Introduction to a research proposal.

• If you edit out repetitive statements and non-essential background information, you should be left with a logical sequence of statements which leads to a clear H1 hypothesis.

• Unlike H0 hypotheses – there is no treatment effect – you have to have a mechanism in mind in order to formulate the H1 hypothesis.

• Without this you have no logical structure to connect your

results back to a known process or mechanism.



• For example, this is the edited version of the introduction to a scientific paper. Other than remove a couple of repetitious statements, I have made no changes.

1. Ectomycorrhizal fungi (EMF) have a fundamental role in nutrient absorption of many plant species. 2. Tree species of ecological and economic relevance in reforestation programs depend on ectomycorrhizal symbiosis, especially in soils contaminated by mining activities.

3. The ability of EMF to reduce the toxicity of heavy metal ions (e.g. copper) in their host plants is accompanied by the decrease of metal concentrations in the aerial part of the plant. 4. Due to increased absorption of these metals by the EMF roots and the accumulation in the extra radical mycelium, greater tolerance to such elements is achieved by the host.

5. Research has been conducted to determine the sensitivity of EMF to a variety of potentially toxic metals to understand the diverse mechanisms through which the fungi may tolerate heavy metals. 6. Enzymatic activity is important for the mobilization and transference of soil nutrients through EM fungi towards the host plant.

7. In this study we investigate the effects of copper and phosphorus concentrations on mycelial growth and enzymatic activities of the EM fungi Pisolithus microcarpus, Chondrogaster angustisporus and Suillus sp. in two growth experiments.



• The logical discontinuity occurs between statements 6 and 7. Since copper is toxic and phosphorous a fertilizer, we are almost certainly going to see effects. But what do they mean ?

• There is actually quite a large literature on copper toxicity in

plants. Several mechanisms have been identified and it would have been possible by choosing the experimental conditions carefully to distinguish which ones were operating under this particular set of experimental conditions.

• Hence the introduction should have continued after point 6 to

develop reasons for a more selective approach. This would form the basis for the missing H1 hypothesis.



Case study: Hummingbird territorial behaviour


Most hummingbird species demonstrate strong territorial behavior

If a bluffing charge attack does not work, the residentmay engage the trespasser in a brief but intense physical battle

So why do hummingbirds defend territories ?

H0: Hummingbirds are randomly distributed in space and time.

Hummingbird territorial behaviour


Hummingbird territorial behaviour

H1

If territory = F(energy), then behavior not species-dependent

If territory = F(mating), then behavior should be species and sex dependent

If…

If…


Territorial behaviour in 1971

• Time, Energy, and Territoriality of the Anna Hummingbird (Calypte anna) Science 173 (1971) 818-821.

• When territory quality decreases defenders may switch to less expensive forms of defense because the energy savings outweigh the loss of resources

• Augmented territorial defense during the breeding season is made possible by increased feeding efficiency due to the availability at this time of very nectar-rich flowers.

• Individuals with large territories are more successful reproductively.


Hummingbird territoriality since• Digestive physiology is a determinant of foraging bout

frequency in hummingbirds. Nature. 1986 Mar 6-12;320(6057):62-3.

• Mitochondrial respiration in hummingbird flight muscles. Proc Natl Acad Sci U S A. 1991 Jun 1;88(11):4870-3.

• Cloning and analysis of the gene encoding hummingbird proinsulin. Gen Comp Endocrinol. 1993 Jul;91(1):25-30.

• Flight and size constraints: hovering performance of large hummingbirds under maximal loading. J Exp Biol. 1997 Nov;200(Pt 21):2757-63.


Hummingbird territoriality since

• Hovering performance of hummingbirds in hyperoxic gas mixtures. J Exp Biol. 2001 Jun;204(Pt 11):2021-7.

• Adipose energy stores, physical work, and the metabolic syndrome: lessons from hummingbirds. Nutr J. 2005 Dec 13;4:36.

• Neural specialization for hovering in hummingbirds: hypertrophy of the pretectal nucleus Lentiformis mesencephali. J Comp Neurol. 2007 Jan 10;500(2):211-21.

• Three-dimensional kinematics of hummingbird flight. J Exp Biol. 2007 Jul;210(Pt 13):2368-82.


Case Study One– Observations

• There is evidence that the central nervous system can influence bone metabolism

• Neurotransmitter (serotonin) receptors are found in bone• Drugs that inhibit neurotransmitter release can affect bone

growth in humans– Hypothesis

• If serotonin plays a role in regulating bone growth, then reducing serotonin release in bone will slow growth

– Experiment• Inject mice with drugs (SSRI’s) that reduce serotonin

release and measure impact on growth of femur over time– Result

• Small but significant reduction in bone growth observed


Case Study One– Observations

• There is evidence that the central nervous system can influence bone metabolism

• Neurotransmitter (serotonin) receptors are found in bone• Drugs that inhibit neurotransmitter release can affect bone

growth in humans– Hypothesis

• If serotonin plays a role in regulating bone growth, then reducing serotonin release in bone will slow growth

– Experiment• Inject mice with drugs (SSRI’s) that reduce serotonin release

and measure impact on growth of femur over time– Result

• Small but significant reduction in bone growth observed– Referees comments

• Is this a direct or an indirect effect ?


Case Study Two– Observations

• A plant extract (E) is used to treat diabetes• Prenatal malnutrition may cause diabetes

– Hypothesis• If E plays a role in regulating blood sugar levels then the

responses to E will be affected by dietary history– Experiment

• Measure acute and chronic effects of E on glucose and insulin levels in normal and prenatal malnourished rats fed either on a standard or high glucose diet

– Result• E does appear to reduce blood sugar levels but with no

significant differences between the experimental groups of animals


Case Study Two– Observations

• A plant extract (E) is used to treat diabetes• Prenatal malnutrition may cause diabetes

– Hypothesis• If E plays a role in regulating blood sugar levels then the responses to E

will be affected by dietary history– Experiment

• Measure acute and chronic effects of E on glucose and insulin levels in normal and prenatal malnourished rats fed either on a standard or high glucose diet

– Result• E does appear to reduce blood sugar levels but with no significant

differences between the experimental groups of animals– Referees comments

• Statistical complexity of experimental design prevents any clear conclusions being drawn about mechanism


Case Study Three– Observation

• Fungal and other microbial symbionts can improve nutrient uptake by plant roots

– Hypothesis• Soil microbial diversity is adversely affected by

conventional farming techniques– Experiment

• Compare microbial diversity associated with root soil in vines cultivated using either conventional or organic techniques

– Result• Diversity and metabolic vigor higher in soils associated

with the roots of organically cultivated plants


Case Study Three– Observation

• Fungal and other microbial symbionts can improve nutrient uptake by plant roots

– Hypothesis• Soil microbial diversity is adversely affected by conventional farming

techniques– Experiment

• Compare microbial diversity associated with root soil in vines cultivated using either conventional or organic techniques

– Result• Diversity and metabolic vigor higher in soils associated with the roots

of organically cultivated plants– Referees comments

• What was the control for this experiment ? Maybe the differences in the two soil areas has nothing to do with agricultural methods.


Case Study Four– Observation

• A patient suffers from an unusually aggressive form of retinoblastoma

– Premise• Patients who develop retinoblastoma also have a

particular type of genetic defect which can be identified cytogenetically

– Hypothesis• This patient has an unusual cytogenetic pattern which

could be useful diagnostically– Result

• The patient does indeed have a unique cytogenetic profile, but there are many differences from the pattern normally seen in retinoblastoma cases


Case Study Four– Observation

• A patient suffers from an unusually aggressive form of retinoblastoma

– Premise• Patients who develop retinoblastoma also have a particular

type of genetic defect which can be identified cytogenetically– Hypothesis

• This patient has an unusual cytogenetic pattern which could be useful diagnostically

– Result• The patient does indeed have a unique cytogenetic profile

– Referees comments.• In this patient there are many differences from the normal

pattern seen in retinoblastoma cases. How can we chose which differences are causal?


Case Study Five– Observation

• Infection by certain human papilloma virus serotypes can lead to cervical cancer. One study has shown that the prevalence of HPV 31 and 33 is much higher in Recife than elsewhere in Brasil

• Mutations to human P53 are often associated with the more severe consequences of HPV infection

– Hypothesis• There is a strong correlation between patients with

advanced HPV 31 and 33 disease and a new p53 mutation which is found mainly in the NE of Brasil

– Result• Patients with HPV 31 and 33 infections have higher levels of

cervical cancer and the new P53 mutation compared to other regions in Brazil.


Case Study Five– Observation

• Infection by certain human papilloma virus serotypes can lead to cervical cancer. One study has shown that the prevalence of HPV 31 and 33 is much higher in Recife than elsewhere in Brasil

• Mutations to human P53 are often associated with the more severe consequences of HPV infection

– Hypothesis• There is a strong correlation between patients with advanced HPV 31

and 33 disease and a new p53 mutation which is found mainly in the NE of Brasil

– Result• Patients with HPV 31 and 33 infections have higher levels of cervical

cancer and the new P53 mutation compared to other regions in Brazil.– Referees comments

• Isn’t this what we would expect from the existing epidemiological data?


Case study summary

• These five hypotheses failed • Either because they were not based on a rigorously

researched mechanism of action • Or because the experimental design was weak• As a consequence, important variables were not controlled

for• And the results could not be explained within the context of

the model• Moral: Hypotheses are often wrong first time around• Moral: Strong hypotheses usually require less work to prove

than weak ones !


Hypothesis lecture learning points

• Good hypotheses build directly onto previous work

• So they need to become technically more sophisticated over time moving from the general to the particular

• A given problem can be associated with a number of very different hypotheses – your experiments should include tests to exclude these alternative explanations


Hypothesis lecture learning points

• Hypotheses can be weak (observational) or strong (mechanism-based)

• For example, a hypothesis which predicts that a tossed coin will end up ‘heads’ 50% of the time is much weaker than one that can predict the exact sequence of ‘heads’ and ‘tails’

• So hypothesis ‘quality’ is important


Break

9 September 2013


Reproducibilty• No research paper can ever be considered to be the final

word• Replication and corroboration of research results is key to

the scientific process. • In studying complex entities, such as animals, ecosystems

and so forth, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of reproducibilty.

• Journals, research laboratories and institutions and funders all have an interest in tackling issues of irreproducibility.

9 September 2013


“Why Most Published Research Findings are False”

• Original paper by John Ioannidis published in PLoS Medicine in 2005 led to much debate.– Small sample sizes, weak hypotheses and experimental design, and bias mean that most

positive findings are incorrect.• Commentary by Steven Goodman, ibid, 2007.

– Agreement that many medical research findings are less definitive than readers suspect– P-values are widely misinterpreted.– Bias of various forms is widespread– Multiple approaches are needed to prevent the literature from being systematically

biased – Need for more data on the prevalence of false claims.

• Sources: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 and http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1855693/

9 September 2013


Case study: Amgen retrospective preclinical study

• Fifty-three papers were identified as ‘landmark’ studies in preclinical cancer research.

• However, scientific findings were confirmed in only 6 (11%) cases.

• Source: www.nature.com/nature/journal/v483/n7391/full/483531a.html

9 September 2013


Amgen retrospective preclinical study• The limitations of preclinical cancer models have been widely reviewed and are

largely acknowledged by the field. • They include:

– the use of small numbers of poorly characterized tumour cell lines that inadequately recapitulate human disease

– an inability to capture the human tumour environment– a poor appreciation of pharmacokinetics and pharmacodynamics– the use of problematic endpoints and testing strategies– in addition, preclinical testing rarely includes predictive biomarkers that, when advanced to

clinical trials, will help to distinguish those patients who are likely to benefit from a drug.

• Learning point: Unfortunately, much published research ignores these

problems– Read “Confiabilidade em crise” and “Nature Special: Challenges in irreproducible research”.

9 September 2013

Competências Básicas de Investigação Científica e de Publicação

Documents

Transcript of Competências Básicas de Investigação Científica e de Publicação