Prompt Evaluation of the follow-up IRRS Mission to the USA

Table of Contents1.Introduction22.Basic Mission Data23.Feedback from the Team Members on the Advance Reference Material of the IRRS Mission2Characteristic additional comments3Further specific remarks3General conclusions34.Feedback from the Team Members on the Effectiveness of the Mission3Characteristic additional comments3Further specific remarks4General conclusion45.Feedback from the Host Country on the Effectiveness of the Follow-up Mission46.Overall Evaluation of the Effectiveness and Efficiency of the Mission5Discussion and conclusions77.Summary7

1. Introduction

The Contribution Agreement ENER/11/NUCL/SI2.588650 between the European Atomic Energy Community (represented by the European Commission) and the IAEA among others foresees as an expected result a performance monitoring based on the evaluation of some key performance indicators of the IRRS missions.

The Nuclear Safety Action Plan of the IAEA has as one of its main objectives the requirement to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States” and in specific, it calls the IAEA Secretariat to review the effectiveness of the IRRS peer reviews.

In reply to these requirements a system of performance indicators have been elaborated in order to measure the efficiency and effectiveness of the IRRS missions. Some of the performance indicators are based on direct feedbacks from the IRRS team members and from the representatives of the country hosting the IRRS mission.

Results of and conclusions from the feedbacks and efficiency and effectiveness evaluations are given in the present Prompt Evaluation Report.

2. Basic Mission Data

Host country: United States of America

Host organization: US Nuclear Regulatory Commission (USNRC)

Mission date: 3 – 11 February 2014

Team Leader: Colin PATCHETT (UK)

Deputy Team Leader: Isabel MELLADO (SPA)

Team Coordinator: Adriana NICIC (IAEA NSNI)

Number of external experts: 5

Number of IAEA staff: 3

Number of observers: 0

Mission type: follow-up

Scope of the mission:

· Core modules (No. 1 through 10)

· Facilities and activities: NPP

· Fukushima module

3. Feedback from the Team Members on the Advance Reference Material of the IRRS Mission

The Advance Reference Material (ARM) normally includes the results of the self-assessment of the host country (usually performed with the aid of the Self-Assessment Tool – SARIS); the Action Plan for improvement in issues found in the self-assessment, a module-wise summary of the status and activity of the regulatory body reviewed and a number of other documents needed for an objective and well informed peer review.

The team members are requested to offer their opinions on the quality of the ARM by rating from 1 to 5 (5 reflecting the highest satisfaction) three questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No.

Question

Average mark

Qr1

How complete do you consider the ARM?

3.9

Qr2

How realistic picture could you obtain on the area you will be reviewing from the ARM?

3.8

Qr3

What is your overall evaluation on the quality of the ARM?

3.4

Total average

3.7

Characteristic additional comments

Qr1:

· The ARM was considered generally complete, yet for some suggestions it was not fully adequate;

Qr2:

· The ARM gave a reasonable realistic picture in a concise manner, especially where references were made to further documents;

Qr3:

· Some parts of the ARM were of high quality, some others were too brief and not sufficiently concentrating on the suggestions of the initial mission;

· The ARM should have been better structured and more clearly focussed on the findings;

· A final comprehensive quality control would have been beneficial.

Further specific remarks

· The reviewers received the ARM too late, it should have been available at least one month before the mission;

· Splitting of the ARM to many, often brief, parts was not very practical.

General conclusions

1) The ARM was fairly complete and realistic, although not balanced in length and quality;

2) The ARM should have been submitted earlier.

4. Feedback from the Team Members on the Effectiveness of the Mission

The team members are requested to offer their opinions on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) five questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No.

Question

Average mark

Qt1

How effective do you consider the activity of the expert team during the mission?

4.3

Qt2

How effective do you consider the activity of the IAEA staff in the team during the mission

4.7

Qt3

How effective do you consider the activity of the Team Leader?

4.4

Qt4

How effective do you consider the activity of the Deputy Team Leader?

4.2

Qt5

How satisfied are you with the preparations of the mission (enough time for preparing yourself, information provided by the IAEA, etc.)?

4.4

Total average

4.4

Characteristic additional comments

Qt1:

· The team was small but high level and worked very well together;

Qt2:

· The experienced IAEA staff supported the mission exemplary;

· The IAEA team provided clear guidance;

Qt3:

· The TL provided direction, although the experienced team needed only limited guidance;

· The team was lead in a democratic way, decisions were made as a team;

· The TL activity was effective, smooth and created a collaborative and friendly atmosphere.

Qt4:

· The DTL was effective and cooperated well with the TL;

· In such a small team the role of DTL is limited, she acted more like one of the reviewers

Qt5:

· The ARM was available too late;

· Continuity of team members made the preparations easier;

Further specific remarks

· Putting status assessment of findings right after the description of changes would be more practical;

· In assessing effectiveness, distinction should be made between initial and follow-up missions.

General conclusion

1) The team was small, experienced, effective and well composed;

2) The IAEA staff was well prepared, effective, supportive;

3) Both the TL and the DTL acted very well;

4) The structure of the follow-up missions report might be reconsidered;

5. Feedback from the Host Country on the Effectiveness of the Follow-up Mission

The Liaison Officer of the host country is requested to offer the opinion of the host on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) seven questions and also expanding in free text their comments on the questions. The questions, the host country’s ratings and the associated comments are given below:

No.

Question

Mark

Qh1

How effective do you consider the mission in assisting the continuous improvement of nuclear safety in your country?

5

Qh2

How objective was the peer review?

5

Qh3

How has the mission helped the exchange of information, experience and good practice with other countries?

4

Qh4

How consistent was the use of the IAEA safety requirements and guides in the mission?

4

Qh5

How justified are the findings of the peer review?

5

Qh6

How relevant are the findings of the peer review for the future development of your regulatory body?

5

Qh7

How competent were the reviewers in their review and findings?

5

Average

4.7

Further comments:

· The mission was very effective in challenging the NRC on how it regulates nuclear power plants. The program of performing a critical self-assessment, having an in-situ independent peer review, and receiving report findings including good practices provides an excellent framework for identifying improvement opportunities;

· The members of the peer review team were very knowledgeable and objective in their active listening, questioning, and assessment of NRC approaches;

· A valuable part of the mission was exchanging information and perspectives on IAEA guides and international practice with senior international regulators;

· Most discussions were put into the context of the IAEA requirements and safety guides, and provided a useful foundation for the comparisons and discussions of NRC practices;

· The insights and findings from the peer review were well founded and showed a good understanding and appreciation of NRC’s accomplishments and plans;

· The findings from the peer review along with other ongoing interactions are very useful to the NRC as it develops regulatory improvements;

· The members of the peer review were extremely knowledgeable and skilled in their assessments and in communicating questions and insights.

6. Overall Evaluation of the Effectiveness and Efficiency of the Mission

The Performance Indicators developed for the measurement of the effectiveness and efficiency of an IRRS mission were evaluated in the extent as it was made possible by the data available at the time of the present evaluation.

In the next figure the values of the performance indicators as they follow from the evaluation of the mission data[footnoteRef:1], as well as the overall effectiveness of the mission are presented. The rightmost columns (EFF. INDICATION) present the ranges where the particular PIs fall (green – optimum, yellow – acceptable, red – needing attention), whereas the frame in the right lower part summarizes the overall effectiveness of the mission (green – optimum, white – effective, yellow – acceptable, red – to analyse). [1: On the Efficiency and Effectiveness of IRRS Missions, Draft v6, Working Material, IAEA, Vienna, 2013]

Prompt Evaluation of the follow-up IRRS mission to the USAPage | 5

Prompt Evaluation of the follow-up IRRS mission to the UK Page | 7

Discussion and conclusions

The values of the effectiveness and efficiency Performance Indicators (upper right table in the figure headed by EFF. INDICATION) suggest the following conclusions:

1) The size of the team was smaller than the theoretical optimum, although this was not reflected by the results;

2) The length of the report was in the optimum range;

3) The complete ARM was provided definitely too late;

4) Most of the team members provided advance comments in time, the yellow range of this PI is due to the small team size;

5) All experts in the team have prior IRRS experience as reviewers;

6) The feedback on the ARM was in the yellow range, the actual comments explain the reasons;

7) The team in general was highly satisfied with the effectiveness of the mission, yet provided suggestion on further possible developments;

8) The host appreciated rather high the mission effectiveness;

9) The final mission report was issued rather late for it took long time for the IAEA to finalize the report.

10) The overall effectiveness of the mission is in the white (effective range) and – mainly for the controversial PI values because of the small, yet very effective team – its value (0.091) is somewhat inferior to the average in recent missions.

7. Summary

The data of and feedback from the extended follow-up IRRS mission to the United Kingdom have been analysed. The following conclusions are drawn:

· The ARM was fairly complete and realistic, although not balanced in length and quality;

· The ARM should have been submitted earlier.

· The team was small, experienced, effective and well composed;

· The IAEA staff was well prepared, effective, supportive;

· Both the TL and the DTL acted very well;

· The team size was smaller than expected yet it consisted of highly experienced reviewers and therefore it was very efficient;

· The host appreciated the results of the mission;

· Three Performance Indicators of the mission were in the red range indicating that attention is needed to certain organizational issues.

· The mission was rated effective by the evaluated Performance Indicators.

Efficiency and Effectiveness of an IRRS Mission

follow-up

INPUT QUANTITIES value

A priory mission data

No(id)UnitValue

min.(>)max.(≤)min.(>)max.(≤)

col

PI

Type of mission (1 - initial or 2 - follow-up) 2optimum0.0C1(T)-89118120.11Team size

Host country

USA

effective0.1C2(P)-67577051770.00Report length

Stating time of the mission (yyyy-mm-dd) 2014-02-03acceptable0.2C3(t

ARM

)day2345-30450.30ARM review time

Ending time of the mission (yyyy-mm-dd) 2014-02-11needs attention

C4(N

AWC

)

-78-680.13Advance comments

Number of modules (n

M

) 10 C5(fb

ARM

)-3.704.005.003.004.000.08Feedback on ARM

Number of supplementary modules (in follow-up) ( n

M+

)0T

0-first

-0.96C6(Exp)-1.000.661.000.500.660.00Team experience

Number of facility and activity types ( n

x

) 1λ

T-first

1.48C7(fb

Host

)-4.714.005.003.004.000.00Host feedback

Number of additional technical areas ( n

T

) 0T

0-fu

3.09C8(fb

Team

)-4.404.005.003.004.000.00Team feedback

Number of policy discussion issues ( n

P

) 0λ

T-fu

0.71C9(β

0

)--1.00-0.801.00-

-

Action Plan Extent

Number of NPP units in the host country ( ν)

100

C10(β

1

)

--1.00-0.801.00-

-

Action Plan coverage

Pre-mission data

P

0-first

17.60

C11(β

2

)

--1.00-0.801.00-

-

Beyond AP coverage

Number of Team Members (including IAEA) ( T)

8λ

M-first

5.83C12(ρ

1

)--0.751.250.501.50-

-

Balance of R & GP

Number of Experts in the Team (without IAEA) ( T

e

) 6λ

F-first

0.63C13(ρ

2

)--0.751.250.501.50-

-

Balance of R & S

Number of Experts with IRRS experience ( T

x

) 6P

0-fu

54.20C14(R

Cont

)--0.901.000.800.90-

-

Report conciseness

ARM distribution date (yyy-mm-dd) 2014-01-10

λ

M-fu

1.55

C15(R

Time

)

day139090901200.30Report completion

Number of issues in the Action Plan ( N

AP

)

0

λ

F-fu

0.64C16(OpenI)-0.090.000.200.200.400.00Open issues left

Number of advance ARM comments from TM's ( N

AWC

)

7μ0.90G1 (InitEff)--4.005.003.004.00-

-

Init. miss. effectiveness

Number of findings (R+S) in the initial mission (for follow-up) 22

Mission data

(R+S)

av

39

Number of Recommendations ( R)

0R

av

15.1

Number of Suggestions (S)

1G

av

12.3

Number of Good Practices (G)

1

(R*G)

av

157.4

Number of findings also in the Action Plan ( N

=

)

0

(R/S)

av

0.89

Number of issues remained open (in follow-up) 2

Number of Report Pages (P)

67Size of nucl.pr.1

Conciseness of the Mission Report ( ρ)

0.00

Mission size10

Feedback data (average marks)

Opt.team size10.2

By the Team on ARM quality

3.70

Opt.rep.length64

By the Host on mission effectiveness

4.71

EFF.& EFF. OF THE MISSION TO

By the Team on mission effectiveness

4.40

Mission type:

By the Host on initial mission effectiveness (for follow-up)

0.00

Missions duration:

Post-mission data

Average Measure of Deviation ( Δ):

Date of isuance of mission report (yyyy-mm-dd) 2014-06-30fields to input!

2014-02-03

effective

Overall Effectiveness:

November 2013

0.091

2014-02-11

follow-up

CALCULATED DATA

IAEA, Version 5

EFF. INDICATION

Rel.Dist

from Opt.

Optimum rangePERFORMANCE INDICATORS Acceptable range

USA

USA

LIMITS & PARAMETERS

For average deviation

For team size

For report length

For findings

0.01.02.03.04.05.0Qh1Qh2Qh3Qh4Qh5Qh6Qh7

Host feedback