Extent and determinants of agreement between Medical Assessors in the

assessment of seafarer fitness

Dr Isam RustomSpecialist Registrar in Occupational Medicine

22nd November 2011

RESEARCH QUESTIONS

1. What is the extent of agreement between Approved Doctors (ADs) and Medical Referees (MRs) in assessing the fitness of seafarers?

2. What is the extent of agreement between the MRs in assessing the fitness of seafarers?

3. What are the determinants of agreement between the MRs in assessing the fitness of seafarers?

METHODS

• Seafarer Medical Review- Medical Referee’s Report (MSF 4108) forms were collected for the period between May 2003 and May 2009.

• MRs were assigned individual codes (1-12) to ensure confidentiality.

• The medical categories of those seeking review were coded using the 16 categories on the (MSF 4108) forms.

METHODS

• Type of work has been categorised as Deck, Engine, Catering and Others.

• The AD’s and MR’s decisions were categorised into one of the four fitness categories.

• The reasons of grading or restrictions change were coded into four categories:

-More Information -New Event -GP/Specialist Report -Other (which includes experience, knowledge and

different interpretation of the standards by the MRs)

STATISTICAL ANALYSIS• Standard statistical tools were used: Chi-square, P values

< 0.05, inter-rater reliability Kappa statistic and the ration between upgraded/downgraded cases for each MR.

• The outcome of the appeal process (result of appeal) for the purposes of analysis is taken to be the difference between the AD’s decision and the MR’s decision which were classified as:

- No change (same category, but could alter the imposed restrictions);

- Upgrade (MR’s decision gave the seafarer more flexibility to work at sea);

- Downgrade (MR’s decision was more strict than the AD’s decision, further limiting the seafarer work at sea).

MR’s referrals - age profile

Medical categories and mean age

Medical category

Number n

Percentage %

Mean age years

Cardiovascular 143 30.8 53.1 Endocrine 66 14.2 40.0 Sensory 60 12.9 35.3 Neurological 44 9.5 41.7 Respiratory 42 9.0 29.5 Mental 38 8.2 37.8 Musculoskeletal 36 7.7 51.9 Genitourinary 13 2.8 45.9 Digestive 9 1.9 40.8 Blood 8 1.7 46.0 Cancer 6 1.3 53.7 Total 465 100.0 43.8

AD’s decision vs. MR’s decision

AD’s decision Number Percentage MR’s decision Number Percentage

Fit 0 0.0 Fit 66 14.2

Fit with restrictions 111 23.9 Fit with restrictions 252 54.2

Temporarily unfit 86 18.5 Temporarily unfit 62 13.3

Permanently unfit 268 57.6 Permanently unfit 85 18.3

Total 465 100.0 Total 465 100.0

Reasons for grading and restrictions change

Reasons for grading / restrictions change n (%)

Medical

category

More

information

New event GP/Specialist

report

Other Total

Cardiovascular 65 (45.4) 18 (12.6) 27 (18.9) 33 (23.1) 143

Endocrine 20 (30.3) 8 (12.1) 5 (7.6) 33 (50.0) 66

Sensory 10 (16.7) 10 (16.7) 3 (5.0) 37 (61.6) 60

Neurological 15 (34.1) 1 (2.2) 5 (11.4) 23(52.3) 44

Respiratory 10 (23.8) 1 (2.4) 3 (7.1) 28 (66.7) 42

Mental 17 (44.7) 0 (0.0) 7 (18.5) 14 (36.8) 38

Musculoskeletal 12 (33.4) 4 (11.1) 3 (8.3) 17 (47.2) 36

Total 149 (34.7) 42 (9.8) 53 (12.4) 185 (43.1) 429

*Genitourinary, digestive, blood and cancer (36) cases were excluded from analysis, see Methods section

Extent of agreement between the MRs

• The workload varied considerably between MRs as they were distributed across the UK. – Referee 9 saw (95 cases), – Referee 6 (68 cases), – Referee 10 (61 cases), – Referee 7 (46 cases),

• These Referees dealt with more than half (270 cases; 58%) of all appeal cases over the 6-year period of the study.

Ratio of upgraded/downgraded cases by MRs

Dove trait Hawk trait

Referee code a,b

6 10 9 1 3 5

Ratio

(upgraded/downgraded)

38 30 20 18 13.5 12.5

a Referee 2,4,7 and 11 had no downgraded cases, so their ratios were 0. b Referee 8 and 12 were excluded from the analysis as they only considered 5 and 1 respectively and had no downgraded cases, so their ratios were 0.

Determinants of agreement between the MRs

• There was significant association between age group and “Fit with restrictions” fitness category except under 20s age group.

• There was significant association between medical category and “Fit with restrictions” fitness category except Mental ill health cases.

• There was No significant association between MR’s decision and Referee code.

• There was No significant association between MR’s decision and type of work.

Discussion of extent of agreement between ADs and MRs.

• This study identified poor agreement between the MR’s decision and the AD’s decision among those seafarers who decided to appeal against the AD’s initial decision [kappa=0.18; poor].

• The most frequent category for a grading or restrictions change was “Other reasons” (43.1%), in all medical categories apart from Cardiovascular 65 (45.4%) and Mental ill health 17 (44.7%) cases which "More Information" was the most frequent category when reviewing fitness decisions.

Outcome of the appeal process• 58% of the appellant seafarers benefited from the

process and were granted an upgrade in their fitness category.

• 40% of the appellant seafarers who had no change in their fitness category often benefited from less or different restrictions which allowed them to continue in employment and working at sea at some capacity.

• Only the 2% of the appellant seafarers who had been downgraded were disadvantaged by the process.

• This study attempted to determine the factors that may have affected the outcome of the appeal process. The relationships with measurable variables like age group, medical category, Referee code, and type of work were studied.

• It appears that the outcome of the appeal process was multi-factorial and was affected by a combination of the above variables.

CONCLUSIONS• First research question: this study found that there was poor

agreement between the MR’s decision and the AD’s decision regarding the fitness of those seafarers who decided to appeal against the AD’s initial assessment.

• Second research question: MRs tended to agree with each other as they upgraded the fitness category of seafarers to allow more of them to work at sea.

• Third research question: MRs decisions on the fitness of seafarers were affected by the age group and the medical category and MRs preferred to use “Fit with restrictions” fitness category more than any other fitness categories in order to impose more or less restrictions whilst keeping the seafarer in employment.

THE WAY FORWARD

• In 2008 the MCA decided to review the 2002 fitness standards in the light of changes in medical diagnosis, treatment, disability legislation and decision taking process especially apparent in the reviews undertaken by MRs. As a result more seafarers would be allowed to work at sea with restrictions and reasonable adjustments which were taken into consideration in the new 2010 fitness standards.

FURTHER RESEARCH

• There is a need to explore other factors which might affect the ADs and MRs decisions when assessing the fitness of seafarers. For example, the hypothesis being that ADs with occupational medicine qualifications would be less likely to have their verdicts set aside on appeal. There might be a gradient, with higher levels of qualifications more likely to produce a supported verdict.

Thank You