Exploring the Societal Returns of Research Conducted within

Hospitals

New Frontiers in Evaluation ConferenceVienna 2006, April 24th - 25th

Antonio García Romero and Luis Royuela MoralesAgencia Laín Entralgo.

Consejería de Sanidad y Consumo de la Comunidad de Madrid.

Presentation Outline

1.1. IntroductionIntroduction2.2. FrameworkFramework3.3. The studyThe study4.4. ResultsResults5.5. ConclusionsConclusions

1.1. IntroductionIntroduction• How much to spend on medical

research?It could be useful to measure the

societal impacts of medical research

• Difficult but NecessaryDifficult: we have a limited

understanding of how research induce societal effects

Necessary: In US: $70 billion / year saved in

medical care costs$90 billion / year sales of products

unrelated to health care resulted from NIH R&D

Silverstein S. 1995

2. Framework2. Framework

The evaluation of scientific research based mainly bibliometrics and expert judgement (peer review).

These methods are not very useful if the objective is to measure the returns from research in terms of wealth, employment or health.

Development of new indicators and methods are needed

2. Framework2. FrameworkStatement 1: Improved health is a consequence

of several factors and health care quality is only one of them

Health Care (hospitals)

EnvironmentalConditions

Genetics

Life style decisions HEALTH

2. Framework2. FrameworkStatement 2: Health care is a result of the

interaction of several factors such as research, training, health technologies and other organizational characteristics

Health Care Drugs / Health Technologies

Research

Training

2. Framework2. FrameworkStatement 3: The research conducted within

hospitals generates several societal outcomes, in addition to health care

Research within

hospitals

Drugs innovation

OtherHuman Capital

Health Tech.

Innovation

Clinical practice improvement

2. Framework2. FrameworkStatement 4: The relationship between medical

research and its societal outcomes is very complex

Lewison, G From Biomedical Research to Health Improvement, Scientometrics. 54(2): 179-92.

2. Framework2. Framework

A consequence of this complexity is the existence of a wide set of methodological approaches that can be applied to analyze this topic

• Macro approachesUseful to estimate aggregated

effects on employment, wealth or population’s health

• Meso & Micro approachesUseful to estimate direct / indirect

effects, policy analysis, comparative analysis

2. Framework2. Framework

A) The Region of Madrid• 6 mill. Habitants • 26.000 € GDP/per capitaB) The Public Health Care System

of Madrid (SMS)• 24 + 8 hospitals• + 200 Primary Care Centers• Around 10,000 physicians• 2006 Budget: 6,066 mill. €C) Research in SMS• 1,650 researchers (INE)• 13% of their time• 2,000 papers SCI / year

3. The Study3. The Study

• Objective: To analyze the effect that physicians’ research has on their clinical performance

• Two Assumptions:• Research within hospitals should

generate both scientific and societal returns

• research allows physicians to improve their skills and knowledge level.

• Methodology:• Individual data (micro approach)• Motivation, Satisfaction and Group

Structure as explanatory variables• Structural Equation Modeling

3. The Study3. The Study

Baseline model:Comprises two sets of hypotheses

Satisfaction (*)

Gender

SCI

Ph.D

Impact

Motivation

GroupStructure

Negative (-)(0: no Ph.D 1: Ph.D)

Negative (-)(0: Men 1: Women)

Positive (+)Positive (+)

Positive (+) Positive (+)

Positive (+)

Positive (+)

Positive (+)

3. The Study3. The Study

Data 279 physicians-scientists working in hospitals of SMS (Madrid)

Table 1. Population and sample composition.

Questionnaires Number of Respondents

(%)

Number of respondents (%) (physician-scientists)

Hospitals 2.038 410 (20,1) 279 (67,8)

Primary care

1.818 347 (19,1) 74 (21,3)

Total 3.856 757 (19,6) 352 (46,5)

3. The Study3. The StudyVariables:3 observed variables & 4 constructs

Table 2. Definition of constructs and variables used in the study

Construct/ Variable Definition

PhD 1 if respondent has the PhD degree; 0 otherwise

GENDER Gender of the respondent (0: men; 1: women)

SCI Papers SCI Journals (last 3 years)

IMP Construct for perceived impact of R&D on clinical practice

SAT_CAR, SAT_CLI and SAT_SCI

Constructs for job satisfaction

MOT Construct for motivation

GROUP Construct for group structure

We have estimated three models• Satisfaction with professional

career• Satisfaction with job climate• Satisfaction with scientific issues

4. Results4. Results

Table 4. Structural models standardized coefficients

Effect Model 1 SAT_CAR

Model 2SAT_CLI

Model 3SAT_SCI

Satisfaction → Impact research clinical performance

0.59(4.032)

0.64(4.190)

0.67(4.483)

Motivation → Impact research clinical performance

0.33(3.400)

0.30(3.330)

0.31(3.546)

SCI→ Impact research clinical performance 0.10(1.603)

0.14(2.147)

0.10(1.605)

Goodness of Fit Summary

χ 2 (N = 279)(1) 384.99 (p<0.001)

415.78 (p<0.001)

543.87 (p<0.001)

Goodness of fit Index (GFI)(2) 0.88 0.88 0.86

Comparative Fit Index (CFI)(3) 0.92 0.93 0.92

Incremental Fit Index (IFI)(4) 0.92 0.93 0.92

(1)Non-significant χ 2 indicates that the hypothesized model fits the data(2)Similar to R2 in regression analysis(3)CFI > 0,9 indicates a good fit(4)IFI > 0,9 indicates a good fitThese cut off values are recommended by HU et al (1995)

4. Results4. Results

SAT_CLI

Gender

SCI

Ph.D

Impact

Motivation

GroupStructure

-0.24

-0.17

0.21 0.30

0.640.20

0.14

5. Conclusions5. Conclusions

Motivation and Job Satisfaction have a significant and positive effect on the perceived impact of research activity on clinical performance

There is a significant relationship between Research Productivity and gender, academic degree, Motivation and Group structure

These results suggest the strategic value of the human factor within R&D management