Instruction in The Protection of Instruction in The Protection of Human Research SubjectsHuman Research Subjects

A web based modelA web based model

Paul Braunschweiger Ph.D.Paul Braunschweiger Ph.D.

Department of Radiation OncologyDepartment of Radiation Oncology

Director, Office of Research EducationDirector, Office of Research Education

University of MiamiUniversity of Miami

Co-Founder, CITI ProgramCo-Founder, CITI Program

pbraunsc@med.miami.edupbraunsc@med.miami.edu

2-23-06

Talking PointsTalking Points

PHS HSR Training mandatePHS HSR Training mandate Why the web?Why the web?

– Pros and consPros and cons– How to get startedHow to get started

CITI Program as an example of a web based platform CITI Program as an example of a web based platform – CITI Program OverviewCITI Program Overview– How is the CITI course site is usedHow is the CITI course site is used

DL Software Demo and Admin utilities DL Software Demo and Admin utilities www.citiprogram.orgwww.citiprogram.org Evaluation of the web based program Evaluation of the web based program

2-23-06

June 2000 DHHS June 2000 DHHS Mandates Training in Mandates Training in Human Subjects Human Subjects ProtectionProtection

N=6559

0%0%5%5%

10%10%15%15%20%20%25%25%30%30%35%35%40%40%45%45%50%50%

11 22 33 44 55 66 77 88 99 1010TOTALLY UNNECESSARY TOTALLY UNNECESSARY ABSOLUTELY VITALABSOLUTELY VITAL

On a scale of 1 - 10, how necessary is the PHS requirement that all key On a scale of 1 - 10, how necessary is the PHS requirement that all key personnel engaged in human subjects research must participate in a personnel engaged in human subjects research must participate in a

human subjects protection educational program? human subjects protection educational program?

MeanMean

2-23-06

Strongly disagree strongly agree

0%0%

5%5%

10%10%

15%15%

20%20%

25%25%

30%30%

35%35%

40%40%

45%45%

% o

f R

es

po

nd

ers

(7

29

)%

of

Re

sp

on

de

rs (

72

9)

11 22 33 44 55 66 77 88 99 1010

In my opinion, the June 2000 Federal education mandate HAS In my opinion, the June 2000 Federal education mandate HAS raised the level of awareness of investigators and staff at my raised the level of awareness of investigators and staff at my institution to the vulnerabilities of human subjects in research. institution to the vulnerabilities of human subjects in research.

N=764, 22 months

June 2000 DHHS Mandates Training in Human June 2000 DHHS Mandates Training in Human Subjects ProtectionSubjects Protection

MeanMean

2-23-06

CITI – CITI – Startup OrganizationsStartup Organizations 9-20009-2000

University of MiamiUniversity of Miami– Paul Braunschweiger,Ph.D., Ken Goodman, Ph.D.Paul Braunschweiger,Ph.D., Ken Goodman, Ph.D.

Fred Hutchinson Cancer Research CenterFred Hutchinson Cancer Research Center– Karen Hansen, BAKaren Hansen, BA

Albany Medical Center Albany Medical Center – Jeff Cooper, MDJeff Cooper, MD

Children's Hospital-Boston Children's Hospital-Boston – Susan Kornetsky, MPHSusan Kornetsky, MPH

Dartmouth CollegeDartmouth College– Liz Bankert, MALiz Bankert, MA

University of Kentucky University of Kentucky – Ada Sue Selwitz, MA, Norma Epley, MSAda Sue Selwitz, MA, Norma Epley, MS

The National Comprehensive Cancer NetworkThe National Comprehensive Cancer Network– Diane Paul, MSDiane Paul, MS

University of NebraskaUniversity of Nebraska– Bruce Gordon, MD, Ernie Prentice, Ph.D.Bruce Gordon, MD, Ernie Prentice, Ph.D.

Group Health Cooperative Group Health Cooperative – Barbara Young, Ph.D., Janelle Erickson, MPHBarbara Young, Ph.D., Janelle Erickson, MPH

University of Washington University of Washington – Helen McGough, MSHelen McGough, MS

2-23-06

Web based instruction Web based instruction Basic instructionBasic instruction Refresher courses.Refresher courses.

– Economy of time for investigatorsEconomy of time for investigators– Economy of time for trainers.Economy of time for trainers.– Economy of time for administrators.Economy of time for administrators.– Cost efficient for institutions.Cost efficient for institutions.

• 20 clinicians @ 2 hrs/session >$10,000/session20 clinicians @ 2 hrs/session >$10,000/session• $50,000 for a 4hr tutorial – Thousands of learners.$50,000 for a 4hr tutorial – Thousands of learners.• CITI is $1,000 / institution for unlimited usage.CITI is $1,000 / institution for unlimited usage.• NCI tutorial is “Free”NCI tutorial is “Free”

FlexibleFlexible– Rather easy to update and modifyRather easy to update and modify– CITI reviews and updates semiannuallyCITI reviews and updates semiannually

Learners like using the web for instruction.Learners like using the web for instruction.– Some like it lessSome like it less

2-23-06

0.0%0.0%

10.0%10.0%

20.0%20.0%

30.0%30.0%

40.0%40.0%

50.0%50.0%

60.0%60.0%

70.0%70.0%

80.0%80.0%

% o

f Res

pon

der

s%

of R

esp

ond

ers

AudioAudio VideoVideo ClassroomClassroom PaperPaper No ChangeNo Change

Assuming an equal time commitment, would you prefer any of the following to this web-based program?

N=1132

2-23-06

Web based instructionWeb based instructionWhat do you need to get started from scratch.What do you need to get started from scratch.

Content (html) and content providersContent (html) and content providers– Content reviewersContent reviewers

Distance Learning Software and supportDistance Learning Software and support Servers and IT support.Servers and IT support. Administrative supportAdministrative support

– Help DeskHelp Desk

2-23-06

Useful Features of most DLS Useful Features of most DLS PackagesPackages

User ID and Password protected.User ID and Password protected. HTML, slideshow +/- audio, video presentationsHTML, slideshow +/- audio, video presentations Linear presentation of course contentLinear presentation of course content Hyperlinks to other information sources on the webHyperlinks to other information sources on the web Assessment of achievement = quizAssessment of achievement = quiz

– Feedback Feedback – Minimum level of understandingMinimum level of understanding

Records and stores data to document compliance.Records and stores data to document compliance. Administrator access to the database to track usage Administrator access to the database to track usage

and compliance.and compliance. Certificate of achievement Certificate of achievement

2-23-06

Is there a downside to web based Is there a downside to web based instruction?instruction?

Learning styles vary.Learning styles vary.– Some people would rather listen than read.Some people would rather listen than read.

– Little or no opportunity for real time question and answer period.Little or no opportunity for real time question and answer period. Web Navigation can be frustrating for some users. Web Navigation can be frustrating for some users.

– browsers, ISPs, internet, bandwidth, software issues.browsers, ISPs, internet, bandwidth, software issues. ““Gaming the software” – CheatingGaming the software” – Cheating

– Hard to prevent but, not impossible.Hard to prevent but, not impossible. Development costs might be prohibitive for small Development costs might be prohibitive for small

organizations.organizations.– Multi-institutional agreementsMulti-institutional agreements

A web based modelA web based modelThe Collaborative Institutional Training Initiative (CITI)The Collaborative Institutional Training Initiative (CITI)

www.citiprogram.orgwww.citiprogram.org

2-23-06

What is the CITI Program?What is the CITI Program?

CITI is a volunteer organization whose goal is to develop CITI is a volunteer organization whose goal is to develop and distribute high quality, peer reviewed educational and distribute high quality, peer reviewed educational resources designed to raise awareness to human subjects resources designed to raise awareness to human subjects protection issues for all members of the research team.protection issues for all members of the research team.

CITI maintains an educational website where participating CITI maintains an educational website where participating organizations can design and roll out a customized organizations can design and roll out a customized curriculum for their investigators, staff and administrators. curriculum for their investigators, staff and administrators.

www.citiprogram.org

2-23-06

CITI – CITI – OrganizationOrganization

Participating Institutions Participating Institutions and Organizations (~580)and Organizations (~580)

CITI Developer Group (~40)CITI Developer Group (~40)

CITI Editorial Board (15)CITI Editorial Board (15)

CITI Administration – Office of Research Education, University of Miami CITI Administration – Office of Research Education, University of Miami

2-23-06

Semi-annual Developer MeetingsSemi-annual Developer Meetings

Seattle FloridaSeattle Florida

2-23-06

CITI CITI Institutional Members Institutional Members (578)(578)

Member CategoryMember Category No.No. %%

VAVA 157157 27%27%

AcademiaAcademia 193193 33%33%

Healthcare OrganizationsHealthcare Organizations 5555 10%10%

Hospitals / Med CentersHospitals / Med Centers 4646 8%8%

Research InstitutesResearch Institutes 3838 7%7%

D of EnergyD of Energy 2020 3%3%

Military / DoDMilitary / DoD 2020 3%3%

IndustryIndustry 1313 2%2%

Independent IRBsIndependent IRBs 99 2%2%

InternationalInternational 1515 2%2%

OthersOthers 1212 2%2%

International ParticipantsInternational Participants

2-23-06

80% of AAHRPP accredited institutions utilize the CITI 80% of AAHRPP accredited institutions utilize the CITI program as a component of their HSR educational program as a component of their HSR educational

Cedars-Sinai Medical Center Cedars-Sinai Medical Center Chesapeake Research Review Inc. Chesapeake Research Review Inc.

Children's Hospital Boston Children's Hospital Boston Copernicus Group IRB Copernicus Group IRB Dana-Farber/Harvard Cancer Dana-Farber/Harvard Cancer

Center Center East Tennessee State University East Tennessee State University Fox Chase Cancer Center Fox Chase Cancer Center Hunter Holmes McGuire Veterans Hunter Holmes McGuire Veterans

Affairs Medical Center Affairs Medical Center Iowa Health - Des Moines Iowa Health - Des Moines Massachusetts General Hospital Massachusetts General Hospital Brigham & Women's HospitalBrigham & Women's Hospital Michigan State University Michigan State University New England Institutional Review New England Institutional Review

Board Board

Newton-Wellesley HospitalNewton-Wellesley Hospital Saint Francis Hospital and Medical Saint Francis Hospital and Medical

Center Center The University of Iowa The University of Iowa The University of New Mexico The University of New Mexico

Health Sciences Center Health Sciences Center The University of South Dakota The University of South Dakota University of Arkansas for Medical University of Arkansas for Medical

Sciences Sciences University of Louisville University of Louisville University of Maryland, Baltimore University of Maryland, Baltimore University of Minnesota University of Minnesota University of Pittsburgh University of Pittsburgh Vanderbilt University Vanderbilt University Washington University in St. Louis Washington University in St. Louis

Western Institutional Review Western Institutional Review

Board Board

2-23-06

CITI program and the VACITI program and the VA

Financial supportFinancial support Software Software Software developmentSoftware development Software and IT support.Software and IT support.

Special Thanks to Drs. Joan Porter and Mike FallonSpecial Thanks to Drs. Joan Porter and Mike Fallon

2-23-06

CITI - CITI - The ContentThe Content

CITI Basic CourseCITI Basic Course www.citiprogram.orgwww.citiprogram.org– 12 Biomedical Modules12 Biomedical Modules– 11 Social / Behavioral Research Modules11 Social / Behavioral Research Modules– 5 General Interest Modules5 General Interest Modules– Quizzes Quizzes

CITI Refresher CourseCITI Refresher Course www.citiprogram.orgwww.citiprogram.org – Case studies and scenariosCase studies and scenarios– Biomed Refresher I, Biomed Refresher I, – Biomed Refresher II Biomed Refresher II – SBR Refresher I SBR Refresher I – SBR Refresher II (6-2006) SBR Refresher II (6-2006)

CITI International Research CourseCITI International Research Course www.irbtraining.orgwww.irbtraining.org – Chinese, Spanish, Chinese, Spanish, – Arabic, Portuguese, FrenchArabic, Portuguese, French

Good Clinical Practice (GCP)Good Clinical Practice (GCP) Spring 2006 Spring 2006 HIPAA SummerHIPAA Summer 2006 2006 RCRRCR – Summer 2006 – Summer 2006 Lab Animal WelfareLab Animal Welfare - Summer 2006 - Summer 2006 www.researchtraining.orgwww.researchtraining.org

2-23-06

CITI – CITI – Program ReviewProgram Review

The CITI Content is extensively reviewed and The CITI Content is extensively reviewed and critiqued. critiqued. – CITI DevelopersCITI Developers– CITI Editorial BoardCITI Editorial Board– External peer reviewers External peer reviewers – Professional editorial services (LERA Group, PTC of NY)Professional editorial services (LERA Group, PTC of NY)– Our Participants via user satisfaction surveys Our Participants via user satisfaction surveys

• Quiz questionsQuiz questions• Recommendations for new course contentRecommendations for new course content• RCRRCR

2-23-06

CITI Program CITI Program The Participating Institution:The Participating Institution:

Registers with CITI Registers with CITI – Institutional user fee, unlimited accessInstitutional user fee, unlimited access

Establishes Learner GroupsEstablishes Learner Groups– Biomedical ResearchBiomedical Research

• PI’s, IRB Members, Administrators, Lab Researchers, StudentsPI’s, IRB Members, Administrators, Lab Researchers, Students– Social & Behavioral ResearchSocial & Behavioral Research

• PI’s, IRB Members, Administrators, Interviewers, StudentsPI’s, IRB Members, Administrators, Interviewers, Students Designs a specific curriculum for each Learner GroupDesigns a specific curriculum for each Learner Group

– Choose “required” and “optional” modulesChoose “required” and “optional” modules– Set passing scoreSet passing score– Set recertification requirement timeline.Set recertification requirement timeline.

Provides additional organization specific instructional Provides additional organization specific instructional materials. materials. – The Institutional PageThe Institutional Page

2-23-06

How do member organizations use How do member organizations use the CITI Program?the CITI Program?

Investigator certification.Investigator certification. IRB member training.IRB member training. IRB coordinator trainingIRB coordinator training Refresher course, re-certification, continuing education.Refresher course, re-certification, continuing education. Web based alternative to seminars or lectures.Web based alternative to seminars or lectures. Undergraduate / graduate bioethics or research methods Undergraduate / graduate bioethics or research methods

courses.courses.

2-23-06

Coordinator “Survey says”:Coordinator “Survey says”:

http://www.surveymonkey.com/Report.asp?U=133513842908

N=248, 64%

2-23-06

CITI “Help Desk”CITI “Help Desk”

Resolve learner problemsResolve learner problems– RegistrationsRegistrations– Technical IssuesTechnical Issues

Institution set up supportInstitution set up support– Learner GroupsLearner Groups– Program CurriculumProgram Curriculum– Institutional page setupInstitutional page setup– Adding institutional contentAdding institutional content

Coordinator SupportCoordinator Support– Changes in contact informationChanges in contact information– Implement Modifications.Implement Modifications.– Report UtilitiesReport Utilities

The Target AudienceThe Target Audience

The Clinical InvestigatorThe Clinical Investigator

The Anatomy Lecture of Dr. Nicolaes Tulp, Rembrandt 1632

The Social & Behavioral ScientistThe Social & Behavioral Scientist

“The Psychologist” by Charles Bragg www.charlesbragg.com

2-23-06

The Basic Science InvestigatorThe Basic Science Investigator

Florescence In Situ Hybridization

StudentsStudents

“The Student” by Rembrandt

The IRB The IRB

Frans Hals, Regents of the St Elizabeth Hospital of Haarlem - 1641

The IRB CoordinatorThe IRB Coordinator

The Tribunal of the Brabant Mint in Antwerp 1594, by Marten de Vos 1523 – 1603Justice triumphs over deceit and violence

Chair of the IRBChair of the IRB

Hans HOLBEIN, the Younger, Henry VIII and the Barber Surgeons c. 1543

2-23-06

The Institutional OfficialThe Institutional Official

0.0%0.0% 5.0%5.0% 10.0%10.0% 15.0%15.0% 20.0%20.0% 25.0%25.0% 30.0%30.0%

% of The Survey Responders% of The Survey Responders

StudentStudent

PIPIClinical InvestigatorClinical Investigator

OtherOther

Research AssistantResearch Assistant

Study coordinator or Study nurseStudy coordinator or Study nurseInterviewerInterviewer

IRB Member or CoordinatorIRB Member or Coordinator

Lab ResearcherLab Researcher

AdministratorAdministratorData managerData manager

Social WorkerSocial Worker

Role in Human Subjects ResearchRole in Human Subjects Research

n=1130

CITI Learner CharacteristicsCITI Learner Characteristics

Registers

Chooses Learner Group

User Survey

Institutional Trainer

Completes the Required modulesMultiple sessions

Prints / Saves Completion Report

Completes any Optional modules

CME / CEU

Log Off

Refresher Notification

DatabaseArchive

Username & PasswordUsername & Password

www.citiprogram.org

Software must be user friendly Software must be user friendly The LearnerThe Learner

2-23-06

Software must be user friendlySoftware must be user friendly

Institutional Coordinator / AdministratorInstitutional Coordinator / Administrator

Has access to the databaseHas access to the database– Download aggregate training data.Download aggregate training data.– Retrieve individual learner’s record.Retrieve individual learner’s record.– Review user comments.Review user comments.

Receives individual completion report by email.Receives individual completion report by email.

CITI Course Site UsageCITI Course Site Usage

2-23-06

CITI Program GrowthCITI Program Growth

00

100100

200200

300300

400400

500500

600600

PA

RT

ICIP

AT

ING

OR

GA

NIZ

AT

ION

SP

AR

TIC

IPA

TIN

G O

RG

AN

IZA

TIO

NS

1-Aug-00

17-Apr-01

15-Aug-02

13-Aug-03

15-Aug-04

1-Aug-05

CITI PARTICIPANTS 2000 TO 2005CITI PARTICIPANTS 2000 TO 2005

1-jJan-06

2-23-06

Recent Monthly UsageRecent Monthly Usage

00

20002000

40004000

60006000

80008000

1000010000

1200012000

1400014000

1600016000

1800018000

2000020000

January 2005

MarchMay

July September

November

January 2006

NE

W L

EA

RN

ER

SN

EW

LE

AR

NE

RS

Basic course = 88%Basic course = 88%Refresher course = 12%Refresher course = 12%

RegistrantsRegistrantsCompletion ReportsCompletion Reports

87% Completion Rate87% Completion Rate

International AccessInternational AccessData from January 2006

2-23-06

CITI Cumulative RegistrationsCITI Cumulative Registrations

0

50

100

150

200

250

300

Aug-00

Oct-0

0

Dec-0

0

Feb-0

1

Apr-01

Jun-0

1

Aug-01

Oct-0

1

Dec-0

1

Feb-0

2

Apr-02

Jun-0

2

Aug-02

Oct-0

2

Dec-0

2

Feb-0

3

Apr-03

Jun-0

3

Aug-03

Oct-0

3

Dec-0

3

Feb-0

4

Apr-04

Jun-0

4

Aug-04

Oct-0

4

Dec-0

4

Feb-0

5

Apr-05

Jun-0

5

Aug-05

Th

ou

sa

nd

sR

egis

trat

ion

s

nov-05

Jang-0

6

350

Version 6Version 6

Version 5Version 5

Version 4Version 4Version 3Version 3

Ver 1 & 2Ver 1 & 2

2-23-06

0.0%0.0%

5.0%5.0%

10.0%10.0%

15.0%15.0%

20.0%20.0%

25.0%25.0%

30.0%30.0%

35.0%35.0%

40.0%40.0%

45.0%45.0%

50.0%50.0%

BiomedicalBiomedical Social BehavioralSocial Behavioral BothBoth

Which "Basic" Course did you complete?Which "Basic" Course did you complete?%

of

the R

esp

onders

CITI Program Course SiteCITI Program Course SiteAdministrator toolsAdministrator tools

www.citiprogram.orgwww.citiprogram.org

2-23-06

Program EvaluationProgram EvaluationWhat is the effectiveness of web based What is the effectiveness of web based

instruction.instruction.

IRB Coordinator PerspectiveIRB Coordinator Perspectivehttp://www.surveymonkey.com/Report.asp?U=133513842908

2-23-06

Utility: Utility: Coordinator “Survey says”:Coordinator “Survey says”:

http://www.surveymonkey.com/Report.asp?U=133513842908

2-23-06

Acceptance by learnersAcceptance by learnersCoordinator “Survey says”:Coordinator “Survey says”:

http://www.surveymonkey.com/Report.asp?U=133513842908

2-23-06

Impact:Impact:Coordinator “Survey says”:Coordinator “Survey says”:

http://www.surveymonkey.com/Report.asp?U=133513842908

70%

2-23-06

http://www.surveymonkey.com/Report.asp?U=133513842908

Value:Value:Coordinator “Survey says”:Coordinator “Survey says”:

2-23-06

Value:Value:Coordinator “Survey says”:Coordinator “Survey says”:

http://www.surveymonkey.com/Report.asp?U=133513842908

Program EvaluationProgram Evaluation What is the effectiveness of web based What is the effectiveness of web based

instruction.instruction.

Learner’s PerspectiveLearner’s Perspective

0.0%0.0%

10.0%10.0%

20.0%20.0%

30.0%30.0%

40.0%40.0%

50.0%50.0%

60.0%60.0%

70.0%70.0%

% o

f R

esp

on

der

s%

of

Res

po

nd

ers

1010 99 88 77 66 55 44 33 22 11Easy DifficultEasy Difficult

Rate the ease of the registration procedure. Rate the ease of the registration procedure.

Utility

0.0%0.0%

10.0%10.0%

20.0%20.0%

30.0%30.0%

40.0%40.0%

50.0%50.0%

60.0%60.0%

70.0%70.0%

% o

f R

es

po

nd

ers

% o

f R

es

po

nd

ers

1010 99 88 77 66 55 44 33 22 11Easy DifficultEasy Difficult

Rate the ease of navigating the course. Rate the ease of navigating the course.

Utility

0.0%0.0%

2.0%2.0%

4.0%4.0%

6.0%6.0%

8.0%8.0%

10.0%10.0%

12.0%12.0%

14.0%14.0%

16.0%16.0%

% o

f R

esp

on

der

s%

of

Res

po

nd

ers

22 33 44 55 66 77 88 99 1010 1111 1212 1313 1414 1515 1616 1717 1818 1919 2020

Hrs spent completing the courseHrs spent completing the course

Approximately how many hours did Approximately how many hours did you you

need to complete the Course? need to complete the Course?

UsageUsage

0%0%

5%5%

10%10%

15%15%

20%20%

25%25%

30%30%

35%35%

% o

f R

esp

on

der

s%

of

Res

po

nd

ers

1010 99 88 77 66 55 44 33 22 11

Strongly Agree Strongly DisagreeStrongly Agree Strongly Disagree

Having completed the course, I now have a clearerHaving completed the course, I now have a clearerunderstanding of human understanding of human research subjects protection. research subjects protection.

Mean = 8.5Mean = 8.5

N= 1176

ImpactImpact

ImpactImpact

0%0%

5%5%

10%10%

15%15%

20%20%

25%25%

% R

esp

on

der

s%

Res

po

nd

ers

1010 99 88 77 66 55 44 33 22 11Strongly agree Strongly disagreeStrongly agree Strongly disagree

Having completed the CITI course, I am now more Having completed the CITI course, I am now more capable of advising a colleague on issues regardingcapable of advising a colleague on issues regarding

human subjects protection.human subjects protection.

Mean = 7.7Mean = 7.7

N=1170

Since completing the CITI Course, I may be more willing to join an Since completing the CITI Course, I may be more willing to join an IRB or a Data Safety Monitoring Board to take a more active role in IRB or a Data Safety Monitoring Board to take a more active role in

protecting human subjects.protecting human subjects.

0%0%

2%2%

4%4%

6%6%

8%8%

10%10%

12%12%

14%14%

16%16%

% o

f R

esp

on

der

s%

of

Res

po

nd

ers

1010 99 88 77 66 55 44 33 22 11

Strongly Agree Strongly DisagreeStrongly Agree Strongly Disagree

34%34%

N=1146

Mean = 6.1Mean = 6.1

ImpactImpact

6.906.90

7.007.00

7.107.10

7.207.20

7.307.30

7.407.40

7.507.50

7.607.60

7.707.70

7.807.80

Ave

rag

e R

esp

on

se 1

= S

tro

ng

ly d

isag

ree;

A

vera

ge

Res

po

nse

1 =

Str

on

gly

dis

agre

e;

10 =

str

on

gly

ag

ree

10 =

str

on

gly

ag

ree

Both Biomed and SBR

Both Biomed and SBR

Biomed Only

Biomed Only

SBR Only

SBR Only

Students

Students

Clinical researcher

Clinical researcher

PIPI IRB Types

IRB Types

Males

Males

Females

Females

Overall

Overall

The time spent doing the course was well justified.The time spent doing the course was well justified.

****

****

**

*540 1137 1673 926 448 700 104 1119 2232 3365

valuevalue

7.60

7.70

7.80

7.90

8.00

8.10

8.20

8.30

Rat

ing

1-1

0

Both Biomed and SBR

Biomed O

nlySBR O

nly

StudentsClinical researcher

PI IRB TypesM

ales

Females

Overall

*

*

**

**

On a scale of 1 to 10, how would you rate the CITI Course, overall?

value

Program EvaluationProgram Evaluation

Learner Perspectives at 22 monthsLearner Perspectives at 22 months

2-23-06

REFRESHER COURSE SURVEYREFRESHER COURSE SURVEY

no0%0%

20%20%

40%40%

60%60%

80%80%

100%100%

% O

F T

HE

RE

SP

ON

DE

RS

% O

F T

HE

RE

SP

ON

DE

RS

yesyes

If you did complete the CITI "Basic Course", do you think the If you did complete the CITI "Basic Course", do you think the information in the Basic Course had a positive impact on how information in the Basic Course had a positive impact on how you have been conducting your human subjects research?you have been conducting your human subjects research?

N=549, AVERAGE TIME AFTER COPLETING BASIC COURSE = 22 MONTHS

2-23-06

REFRESHER COURSE SURVEYREFRESHER COURSE SURVEY

0%0%

5%5%

10%10%

15%15%

20%20%

25%25%

30%30%%

OF

TH

E R

ES

PN

DE

RS

% O

F T

HE

RE

SP

ND

ER

S

11 22 33 44 55 66 77 88 99 1010

The CITI Basic Courses, provided me with a knowledge base to The CITI Basic Courses, provided me with a knowledge base to protect human subjects from harms associated with being in a protect human subjects from harms associated with being in a

research project?research project?

STRONGLY DISAGREE STRONGLY AGREE

N=549, AVERAGE TIME AFTER COMPLETING BASIC COURSE = 22 MONTHS

2-23-06

REFRESHER COURSEREFRESHER COURSE

0%0%

5%5%

10%10%

15%15%

20%20%

25%25%

30%30%

% O

F R

ES

PO

ND

ER

S%

OF

RE

SP

ON

DE

RS

11 22 33 44 55 66 77 88 99 1010

The CITI Basic Course, helped me to a clearer The CITI Basic Course, helped me to a clearer understanding of the elements of consent. understanding of the elements of consent.

STRONGLY DISAGREE STRONGLY AGREESTRONGLY DISAGREE STRONGLY AGREE

N=549, AVERAGE TIME AFTER COPLETING BASIC COURSE = 22 MONTHS

2-23-06

Refresher Course SurveyRefresher Course Survey

0%0%

2%2%

4%4%

6%6%

8%8%

10%10%

12%12%

14%14%

16%16%

18%18%

% o

f R

es

po

nd

ers

(7

29

)%

of

Re

sp

on

de

rs (

72

9)

11 22 33 44 55 66 77 88 99 1010

I am now reading more literature or going to more seminars onI am now reading more literature or going to more seminars on topics related to human subjects protection topics related to human subjects protection

STRONGLY DISAGREE STRONGLY AGREESTRONGLY DISAGREE STRONGLY AGREE

40%40%

2-23-06

Refresher Course SurveyRefresher Course Survey

0%0%

10%10%

20%20%

30%30%

40%40%

50%50%

60%60%

70%70%

80%80%

YESYES NONO

I have taken a more active role in protecting human research I have taken a more active role in protecting human research subjects by joining an IRB, a DSMB or by pursuing IRB subjects by joining an IRB, a DSMB or by pursuing IRB

professional certification.professional certification.%

of

Res

pond

ers

% o

f R

espo

nder

s

N=766N=766

2-23-06

REFRESHER COURSE SURVEYREFRESHER COURSE SURVEY

Strongly disagree strongly agreeStrongly disagree strongly agree

0%0%

5%5%

10%10%

15%15%

20%20%

25%25%

30%30%

35%35%%

OF

RE

SP

ON

DE

RS

% O

F R

ES

PO

ND

ER

S

11 22 33 44 55 66 77 88 99 1010

The CITI Refresher Course is a worthwhile component of my The CITI Refresher Course is a worthwhile component of my institution’s program to protect human research subjects.institution’s program to protect human research subjects.

N=549

2-23-06

In SummaryIn Summary

What have we learned about web based instruction from the What have we learned about web based instruction from the CITI program experience. CITI program experience.

The www as a toolThe www as a tool– Web instruction can provide rich content in a time and cost effective Web instruction can provide rich content in a time and cost effective

manner. manner. – A web program can play a central role in an organization’s HSR A web program can play a central role in an organization’s HSR

education program.education program.– A web program can be utilized for both basic and refresher A web program can be utilized for both basic and refresher

instruction.instruction.– A web platform is overwhelmingly preferred by investigators and A web platform is overwhelmingly preferred by investigators and

training coordinators.training coordinators.– But, alas, the web is not for everyone.. and alternatives are But, alas, the web is not for everyone.. and alternatives are

advisable.advisable.

2-23-06

In SummaryIn Summary

What have we learned from the CITI program What have we learned from the CITI program experience.experience.

OrganizationOrganization– A dynamic committed organization can keep the instruction fresh A dynamic committed organization can keep the instruction fresh

and current.and current.• Frequent updates.Frequent updates.• Learner satisfaction surveys are very important to obtain feedback and Learner satisfaction surveys are very important to obtain feedback and

guidance to improve the program.guidance to improve the program.– Help deskHelp desk

2-23-06

In SummaryIn Summary

What have we learned from the CITI program What have we learned from the CITI program experience.experience.

UtilityUtility

– World wide training resource can be delivered in a cost effective World wide training resource can be delivered in a cost effective and value added manner. and value added manner.

– Customization capabilities are a key.Customization capabilities are a key.– Institutional page to provide institution specific materials is VERY Institutional page to provide institution specific materials is VERY

well received and a extensively used by some participating well received and a extensively used by some participating institutions. institutions.

2-23-06

In SummaryIn Summary

What have we learned about web based instruction What have we learned about web based instruction from the CITI program experience.from the CITI program experience.

EvaluationsEvaluations– Learners indicate the course helped them understand human Learners indicate the course helped them understand human

subjects protection issues and that the time required to complete subjects protection issues and that the time required to complete the course is time well spent.the course is time well spent.

– Learners indicate that the instruction has provided the confidence to Learners indicate that the instruction has provided the confidence to take a more active roll in the protection of human subjects. take a more active roll in the protection of human subjects.

– Institutional coordinators exhibit a high degree of satisfaction with Institutional coordinators exhibit a high degree of satisfaction with the web based program.the web based program.

Instruction in The Protection of Instruction in The Protection of Human Research SubjectsHuman Research Subjects

www.citiprogram/orgwww.citiprogram/org

Paul Braunschweiger Ph.D.Paul Braunschweiger Ph.D.pbraunsc@med.miami.edupbraunsc@med.miami.edu