Job Analysis for U.S. Property And Casualty Qualified...

Job Analysis for U.S. Property And Casualty

Qualified Actuary

PRESENTED TO:

National Association of Insurance Commissioners (NAIC) 1100 Walnut Street, Suite 1500

Kansas City, KS 64106-2197

November 2017

2

TABLE OF CONTENTS

TABLE OF CONTENTS ..........................................................................................................................2

INTRODUCTION .................................................................................................................................3

KEY FINDINGS ....................................................................................................................................4 Number of Included Statements.............................................................................................................. 4 Task and Knowledge Statement Survey Ratings: Mean importance ...................................................... 4 Task and Knowledge Statement Survey Ratings: Mean Criticality .......................................................... 5 Test Specifications ................................................................................................................................... 5

DEFINING A JOB ANALYSIS…………………………………………………………………………………………………………………. 6

RELATIONSHIP TO STANDARDS AND COMMON PRACTICES .................................................................6

METHOD ............................................................................................................................................7 1. Kickoff Meeting ................................................................................................................................... 7 2. Identify Task and Knowledge Statements ........................................................................................... 7 3. Task Force Meeting ............................................................................................................................. 7 4. Survey Development and Administration ............................................................................................ 8 5. Data Analysis ....................................................................................................................................... 8 6. Test Specifications Meeting ................................................................................................................. 9

SURVEY RESULTS ................................................................................................................................9 Response Rate ......................................................................................................................................... 9 Demographic Results ............................................................................................................................... 9 Task and Knowledge Importance Rating Results .................................................................................. 10 Task and Knowledge Criticality Rating Results...................................................................................... 11

TEST SPECIFICATIONS ....................................................................................................................... 12 Linking Task Statements to Knowledge Statements ............................................................................. 14 Definition of U.S. Appointed Actuary .................................................................................................... 14

CONCLUSION ................................................................................................................................... 16

APPENDICES………………………………………………………………………………………………………………………………………17

3

INTRODUCTION

The National Association of Insurance Commissioners (NAIC) is the U.S. standard-setting and regulatory support

organization created and governed by the chief insurance regulators from the 50 states, the District of Columbia,

and five U.S. territories. Through the NAIC, state insurance regulators establish standards and best practices,

conduct peer review, and coordinate their regulatory oversight. The NAIC staff supports these efforts and

represents the collective views of state regulators domestically and internationally. The NAIC members, together

with the central resources of the NAIC, form the national system of state-based insurance regulation in the U.S.

The mission of the NAIC is to assist state insurance regulators, individually and collectively, in serving the public

interest and achieving the following fundamental insurance regulatory goals in a responsive, efficient, and cost

effective manner, consistent with the wishes of its members:

Protect the public interest;

Promote competitive markets;

Facilitate the fair and equitable treatment of insurance consumers;

Promote the reliability, solvency, and financial solidity of insurance institutions; and

Support and improve state regulation of insurance.

To align with the mission and recommendations to expand and revise the current definition of a qualified actuary

as defined by the American Academy of Actuaries,1 the NAIC engaged Workcred to perform a job analysis to

outline the body of knowledge and expand and review the definition of a "qualified actuary." The key output of

this job analysis is a psychometrically-validated Test Specifications document (Appendix N), which includes a

comprehensive list of the important and critical knowledge and task statements for a qualified actuary, which

functions as an outline of the body of knowledge. In addition, an evidence-based definition of a qualified actuary

was produced. The NAIC can use these outputs to guide future assessment, education, and training endeavors.

1 https://www.actuary.org/files/imce/qualification_standards.pdf

Test Specificationsof a "Qualifed Actuary"

KNOWLEDGE STATEMENTS

TASKS BODY OF KNOWLEDGE

DEFINITION OF QUALIFED

ACTUARY

https://www.actuary.org/files/imce/qualification_standards.pdf

4

KEY F INDINGS

The following is a summary of the job analysis results.

NUMBER OF INCLUDED STATEMENTS

of 59 (98%) tasks from the job analysis survey were included by subject matter experts (SMEs)

at the test specifications meeting. One new task statement was added by the SMEs.

of 106 (97%) knowledge statements from the job analysis survey were included by SMEs at

the test specifications meeting. No new knowledge statements were added by the SMEs.

TASK AND KNOWLEDGE STATEMENT SURVEY RATINGS: MEAN IMPORTANCE

• Of the task statements resulted in a mean importancerating of at least 2.00 ("moderately important") out of 4.00 ("very important")

100%

• Of the knowledge statements resulted in a mean importance rating of at least 2.00 ("moderately important") out of 4.00 ("very important")

100%

58

103

5

TASK AND KNOWLEDGE STATEMENT SURVEY RATINGS: MEAN CRITICALITY

TEST SPECIFICATIONS

Finally, the primary outcome of the job analysis is the test specifications, which is summarized below. It should be

noted that the SMEs decided to merge the "Principles of Ratemaking" domain into the "Premium, Loss, and

Expense Reserves" domain.

• Of the task statements resulted in a mean criticalityrating of at least 2.00 ("moderate harm") out of 4.00 ("extreme harm")

97%

• Of the knowledge statements resulted in a mean criticality rating of at least 2.00 ("moderate harm") out of 4.00 ("extreme harm")

100%

Domain

A. Law

B. Policy Forms and Coverages, Underwriting, and Marketing

C. Reinsurance

D. Premium, Loss, and Expense Reserves

E. Statutory Insurance Accounting and Expense Analysis

F. Professionalism and Business Skills

Total

Number of Knowledge Statements

8

9

13

43

27

3

103

Percent Allocated to Domain

5%

15%

10%

30%

25%

15%

100%

6

DEFINING A JOB ANALYSIS

A job analysis can be summarized as a description of what people

do and know when completing a job or task. Often, researchers

classify the outputs of a job analysis as the skills, knowledge, and

abilities (KSAs) needed for proficiency in a job role.

One of the uses of a job analysis is to create an exam specification

document that delineates the number and percentage of test

questions representing content areas or domains of a job role. The

exam specification document serves as a blueprint for creating the

appropriate amount and types of exam questions (and in some

cases, cognitive levels – knowledge, comprehension, application,

analysis, synthesis, and evaluation). The exam specification,

including the tasks of the job analysis, can be used to define and

expand upon job roles and outline a body of knowledge.

While the main purpose of a job analysis is to develop the blueprint for credentialing exams, such as those used by

the Casualty Actuarial Society (CAS) and Society of Actuaries (SOA), it can also be useful in developing body-of-

knowledge studies, educational content and offerings, professional development activities, and candidate criteria.

Workcred’s job analysis methodology is aligned to Section 8 of the international standard, ISO/IEC 17024:2012,

Conformity assessment -- General requirements for bodies operating certification of persons, and other evidence-

based methods in developing competencies, as outlined below.

RELATIONSHIP TO STANDARDS AND COMMON PRACTICES

Workcred follows industry best practices by aligning all test development activities to various testing standards

and common test development and psychometric practices, including ISO/IEC 17024:2012, Conformity assessment

-- General requirements for bodies operating certification of persons; ASTM E2659-09, Standard Practice for

Certificate Programs; The Standards for Educational and Psychological Testing (APA, AERA, NCME); The Society for

Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures;

and evidence-centered design (ECD). In many cases, the testing standards do not address practices that are

common to certain test development activities such as interviewing, conducting focus groups, survey research,

sampling, data management and analysis, statistical analysis, and reporting. In such cases, Workcred employs

other standards from organizations such as the guidelines drafted by American Association for Public Opinion

Research on survey sampling. When official standards are unavailable, Workcred uses common practices published

in scholarly journals and books.

A job analysis is: “the process of discovery of the nature of a job” (p. 7). “the systematic process of discovery of the nature of a job by dividing it into smaller units, where the process results in one or more written products with the goal of describing what is done in the job or what capabilities are needed to effectively perform the job” (p. 8). -Michael Brannick (2006)

7

METHOD

The figure below summarizes the steps taken by Workcred to perform this job analysis. Explanations and results of

each step are further defined in this section.

1. KICKOFF MEETING

The kickoff meeting occurred on August 15, 2017, via a conference call. The goal of the meeting was to

discuss and finalize the project plan (e.g., SME recruitment methods and logistics, meeting needs,

confirmation of project dates, and assigning roles and responsibilities).

2. IDENTIFY TASK AND KNOWLEDGE STATEMENTS

The NAIC provided Workcred with a draft list of task and knowledge statements. This list was initially

reviewed and revised by six SMEs who later participated in the September 13-14, 2017, task force

meeting (see step 3 below), at which the statements were also reviewed by a larger group.

3. TASK FORCE MEETING

A task force consisting of 13 property and casualty (P&C) actuaries from the U.S. was formed, and the

group convened on September 13 and 14, 2017, in Philadelphia (see Appendix A2 for a list of the SMEs).

Prior to the meeting, the task force was given an orientation kit which provided background information

about the purpose of the meeting, logistical information, definition and uses of a job analysis,

expectations of a job analysis meeting, and best practices in writing and reviewing task and knowledge

statements (see Appendix A1).

At the meeting, Workcred staff provided a 20-minute overview presentation, which included the meeting

agenda and logistics, an introduction to the job analysis, as well as information on writing and reviewing

the task and knowledge statements. After the presentation, the task force reviewed and revised the draft

list of task and knowledge statements provided by the NAIC. A total of 59 task statements and 106

knowledge statements were identified.

1. KICKOFF

MEETING

2. IDENTIFY TASK

AND KNOWLEDGE

STATEMENTS

3. TASK FORCE

MEETING

4. SURVEY

DEVELOPMENT & ADMINISTRATION

5. DATA ANALYSIS6. TEST

SPECIFICATIONS

MEETING

8

4. SURVEY DEVELOPMENT AND ADMINISTRATION

Using the task and knowledge statements identified from the task force meeting, a survey was

constructed and piloted. The pilot version of the survey was completed by a total of 27 SMEs who

provided comments on the content (e.g., suggestions to revise, omit, or add task or knowledge

statements) and functionality of the survey. The task force also reviewed the pilot survey comments to

determine if any revisions were necessary.

Once all revisions were completed, the final version of the survey incorporated the sections shown below:

• Welcome Page

• Demographics

• Task Statements - Importance and Criticality Ratings

• Knowledge Statements - Importance and Criticality Ratings

• Weighting

• Incentive

• Thank you

From October 2-20, 2017, the job analysis survey was distributed to a total of 463 2016 U.S. P&C

appointed actuaries. The survey participants were asked to rate the task and knowledge statements on

importance and criticality for each domain area.

The importance and criticality rating scales are outlined below:

Next, survey participants were asked to provide feedback on the statements identified as needing to be

added, omitted, or revised. That input was then reviewed by SMEs at the November 15-16, 2017, test

specifications meeting (see step 6 on page 9). The results are reported in Appendices I and J.

5. DATA ANALYSIS

The data was then exported from SurveyGizmo, an online survey program, and analyzed. The analysis

included frequency, mean, and variance analyses on demographic variables, as well as importance and

criticality ratings. In addition, mean comparative analyses were conducted to provide mean importance

rating results from a set of demographic questions as outlined in the "Survey Results" section (page 9)

below.

Importance Rating Scale:

Please rate how IMPORTANT the following tasks are for a P/C Appointed Actuary writing a statutory P/C Statement of Actuarial Opinion after having three years of responsible experience.

•0 = Of No Importance

•1 = Of Little Importance

•2 = Moderately Important

•3 = Important

•4 = Very Important

Criticality Rating Scale:

Please rate the degree to which a P/C Appointed Actuary with three years of responsible experience who does NOT perform the following tasks would be seen as causing harm to stakeholders. (Harm may be legal, financial, etc.).

•0 = No Harm

•1 = Minimal Harm

•2 = Moderate Harm

•3 = Substantial Harm

•4 = Extreme Harm

9

6. TEST SPECIFICATIONS MEETING

On November 15-16, 2017, 12 SMEs participated in a two-day meeting in Chicago. Using the survey

analysis results, the SMEs finalized a list of task and knowledge statements to be included across each

domain area. The SMEs also reviewed the survey comments regarding the statements that should be

added, revised, or omitted, and also provided definitions for each of the domain statements. In addition, a

weighting activity was conducted, in which SMEs suggested the number of examination items across each

of the domains. The final meeting task included an exercise in which SMEs linked at least one knowledge

statement to a task statement. See the “Test Specifications" section (page 12) for further information.

SURVEY RESULTS

RESPONSE RATE

The response rates for the job analysis surveys are summarized in the figure below.

DEMOGRAPHIC RESULTS

Below is a list of the most frequent responses for each of the demographic questions asked on the survey.

Demographic Question Most Frequent Response Percent

1. What is your PRIMARY work/practice setting? Reinsurance 57.4%

2. What U.S. state do you PRIMARILY work in? California 9.0%

3. For how many years have you served as a U.S. Appointed

Actuary?

20 or more years 31.1%

4. What is your HIGHEST level of education completed? University/college graduate 64.5%

5. Which of the following is the highest CAS designation you

CURRENTLY hold from taking examinations (i.e., not from mutual

recognition)?

FCAS

73.0%

6. Which of the following is the highest SOA designation you

CURRENTLY hold from taking examinations (i.e., not from mutual

recognition)?

Neither 94.70%

7. Please list any other actuarial credentials that you CURRENTLY

hold. Include any granted through mutual recognition.

None;

CERA

2.5%;

1.6%

8. Indicate the last NAIC P&C annual statement year in which you

were a U.S. P&C Appointed Actuary (enter 2017 if you are currently

a U.S. P&C Appointed Actuary).

2017 88.5%

• 463 SURVEY INVITATIONS

• 122 RESPONSES (26.35%)Survey Response Rates

10

Demographic Question Most Frequent Response Percent

9. Please select all of the annual statement lines for which you have

experience as a U.S. P&C Appointed Actuary (select all that apply).

Other Liability – Occurrence or

Claims-Made*

78.7%

10. What is your gender? Male 76.0%

11. What is your PRIMARY race/ethnicity? White 82.0%

*The second most frequent response was “Private Passenger or Commercial Auto/Truck Liability/Medical/Physical Damage.”

See Appendix B for additional demographic information.

TASK AND KNOWLEDGE IMPORTANCE RATING RESULTS

The following figure shows the importance rating results of the task statements across the domains (see Appendix

C for more details). It includes passing and failing statements based on mean importance rating results.

Similarly, the figure below shows the importance rating results of the knowledge statements across domains (see

Appendix D for more details).

TASK DOMAINS

A. Law

B. Policy Forms

C. Principles of Rate Making

D. Reinsurance

E. Premium, Loss, and Expense Reserves

F. Statutory Insurance Accounting and Expense Analysis

G. Professionalism and Business Skills

NO. PASSING(MEAN ≥

2.50)

2

2

1

1

15

33

4

NO. FAILING(MEAN <

2.50)

0

1

0

0

0

0

0

TOTAL

NO.

2

3

1

1

15

33

4

KNOWLEDGE DOMAINS

A. Law

B. Policy Forms


D. Reinsurance





2.50)

8

9

6

13

35

29

3

NO. FAILING(MEAN <

2.50)

0

0

0

0

2

1

0

TOTAL

NO.

8

9

6

13

37

30

3

11

The failing task and knowledge statements based on the importance ratings are included in the figure below.

Mean importance rating analyses across demographic questions are included in Appendices E1 to E7 and F1 to F7.

TASK AND KNOWLEDGE CRITICALITY RATING RESULTS

The figure below shows the criticality rating results of the task statements across the domains (see Appendix C for

more details). It includes passing and failing statements based on mean criticality rating results as well.

Similarly, the figure below shows the criticality rating results of the knowledge statements across domains (see

Appendix D for more details).

Failing Task

•Obtain information related to marketing (e.g., target markets, distribution channels, sales incentives, competitors)

Failing Knowledges

•Premium asset for retrospectively rated policies

•Premium deficiency reserves

•Risk-Based Capital (RBC) formulas

TASK DOMAINS

A. Law

B. Policy Forms


D. Reinsurance





2.50)

2

2

0

1

13

26

3

NO. FAILING(MEAN <

2.50)

0

1

1

0

2

7

1

TOTAL

NO.

2

3

1

1

15

33

4

KNOWLEDGE DOMAINS

A. Law

B. Policy Forms


D. Reinsurance





2.50)

8

8

6

12

32

27

3

NO. FAILING(MEAN <

2.50)

0

1

0

1

5

3

0

TOTAL

NO.

8

9

6

13

37

30

3

12

The failing task and knowledge statements based on the criticality ratings are included in the figure below.

Mean criticality rating analyses across demographic questions are included in Appendices G1 to G7 and H1 to H7.

TEST SPECIFICATIONS

A two-day in-person test specifications meeting occurred in Chicago on November 15-16, 2017, to determine the

final task and knowledge statements that should be included and weighted across the domains. Test specifications

can be used to emphasize areas on an assessment or when developing educational or training content. During the

meeting, 12 SMEs provided a definition for each domain and linked each task with at least one knowledge

statement (see Appendix A2 for a list of the SMEs). Prior to the meeting, the SMEs were provided the survey

results.

Failing Tasks

•Obtain information related to marketing (e.g., target markets, distribution channels, sales incentives, competitors)

•Utilize principles and techniques of ratemaking as may be applicable for estimation of unpaid claims (e.g., onlevel premiums)

•Calculate discounted reserves if appropriate and permitted

•Review all work products of the previous Appointed Actuary

•Document intercompany pools' participation

•Document Appointed Actuary's relationship to the company

•Provide relevant commentary regarding any retroactive or financial reinsurance

•Test whether the adverse development exceeded 5% of surplus in at least three of the last five years

•Create an exhibit comparing the Appointed Actuary's current estimates to comparable prior estimates

•Determine the data elements that the Appointed Actuary deems significant to the analysis in compliance with instructions for "NAIC Annual Audited Financial Reports"

•Communicate to the company the data elements that the Appointed Actuary deems significant to the analysis so the company can inform external auditors

•Apply business skills

Failing Knowledges

•Rating plans (e.g., individual risk rating, class rating, experience rating, retrospective rating, participating policies, judgment rating, schedule rating)

•Reinsurance risk transfer testing

•IT and accounting procedures and processes

•Industry benchmarks to reserve analysis applications, composition, and applicability of industry data

•Approaches and considerations in discounting reserves (e.g., interest rates)

•Premium asset for retrospectively rated policies

•Premium deficiency reserves

•Risk-Based Capital (RBC) formulas

•Company’s financial health evaluation methods

•IRIS Property/Casualty Ratios

13

At the meeting, Workcred staff presented the agenda, expectations of the meeting, and the description of a U.S.

P&C Appointed Actuary. In addition, Workcred staff facilitated introductions for the attending SMEs and the NAIC

staff.

Following the presentations, Workcred staff presented the job analysis survey results, which included:

demographic responses

mean and standard deviation and frequency responses of the importance and criticality ratings of

task and knowledge statements

comparative mean analysis (importance and criticality ratings by selected demographic questions)

test weight survey recommendations

Using the survey results, SMEs were asked to determine which task and knowledge statements should be included.

In addition, a list of statements that survey respondents recommended be revised, added, or omitted were shared

with SMEs (see Appendices I and J) to determine if any of those statements should be included, omitted, or

revised. A final list of their inclusion decisions, revisions, and rationales for each statement are included in

Appendices K and L.

The figure below summarizes the SMEs’ selection of the task and knowledge statements.

The next stage of test specifications requires a weighting activity, and SMEs were asked to assign a weight to each

knowledge domain during the meeting. The individual weights were collected and documented in a spreadsheet.

The spreadsheet results – the mean, mode, median and variance measures, and the weightings provided by the

survey participants (see Appendix M) – were shared with the SMEs, who then discussed these outcomes and

finalized the weightings. During the test specifications meeting, the SMEs merged the domain of “Principles of

Ratemaking” into the “Premium, Loss, and Expense Reserves” domain. In addition, “Statutory Insurance

59NO. OF TASK

STATEMENTS ON

PRACTICE ANALYSIS

SURVEY

1NO. OF TASK

STATEMENTS

EXCLUDED BY SMES

1NO. OF TASK

STATEMENTS ADDED

BY SMES

59NO. OF TOTAL TASK

STATEMENTS

INCLUDED BY SMES

106NO. OF KNOWLEDGE

STATEMENTS ON PRACTICE

ANALYSIS SURVEY

3NO. OF KNOWLEDGE

STATEMENTS EXCLUDED BY

SMES

0NO. OF KNOWLEDGE

STATEMENTS ADDED BY

SMES

103NO. OF KNOWLEDGE

STATEMENTS INCLUDED BY

SMES

14

Accounting and Expense Analysis” was renamed to “Statutory Insurance Accounting and Role of The Appointed

Actuary.” Results of the final weighting can be seen below.

LINKING TASK STATEMENTS TO KNOWLEDGE STATEMENTS

Through a linking exercise, the SMEs were asked to provide at least one knowledge statement for each of the task

statements, linking the most relevant knowledge statements to the task statements. Results of this linking are

presented in Appendix O.

DEFINITION OF U.S. APPOINTED ACTUARY

Using the list of accepted task and knowledge statements, SMEs were asked to define each of the domains in order

to provide a high level definition of a U.S. Appointed Actuary for each domain; the results of which are depicted in

the table below. More elaborate definitions, along with the task and knowledge statements, are located in

Appendix P.

Domain Definition

A. Law The Appointed Actuary (AA) should understand relevant

U.S. and state insurance law, regulatory authority, and

regulations. The AA should be aware of tort law,

relevant types of litigation, including class actions and

mass torts, as well as precedent case law and changes

therein.

The knowledge will enable the AA to assess the effect of

the legal environment on the reserves for which the AA

is opining and associated risks and uncertainties.

B. Policy Forms and Coverages, Underwriting, and

Marketing

The AA should understand types of insurable exposures,

related insurance policy products, and how insurance

companies assume risk through marketing and

underwriting.

Domain

A. Law

B. Policy Forms and Coverages, Underwriting, and Marketing

C. Reinsurance

D. Premium, Loss, and Expense Reserves

E. Statutory Insurance Accounting and Role of The Appointed Actuary

F. Professionalism and Business Skills

Total

Percentage

5%

15%

10%

30%

25%

15%

100%

15

Domain Definition


insurance coverages, underwriting and marketing, and

changes therein on the reserves for which the AA is

opining and associated risks and uncertainties.

C. Reinsurance The AA should understand the functions and types of

reinsurance including relevant contract features and risk

transfer principles. The AA should understand

reinsurance accounting, recognition, and collectability

issues.


reinsurance on the reserves for which the AA is opining

and associated risks and uncertainties.

D. Premium, Loss, and Expense Reserves The AA should understand and apply reserving methods,

analysis, and diagnostics to derive actuarial reserve

estimates in the context of the requirements of the SAO.

The AA should understand the company’s operations

and data, external environment, and relevant changes

therein.

The knowledge will enable the AA to derive estimates

that recognize the internal and external environment

and the associated risks and uncertainties.

E. Statutory Insurance Accounting and Role of The

Appointed Actuary

The AA should understand relevant statutory accounting

principles and reporting requirements including the U.S.

NAIC P&C Annual Statement. The AA should understand

the responsibilities of an AA, including key deliverables.

The knowledge will enable the AA to produce an SOA,

AOS, and Actuarial Report in accordance with the NAIC

Annual Statement Instructions.

F. Professionalism and Business Skills The AA should know and adhere to the Code of

Professional Conduct as well as relevant Actuarial

Standards of Practice and must meet the U.S.

Qualification Standards. The AA should have the

professional and business skills to manage the tasks,

make informed decisions, communicate effectively with

users of the actuary’s work products, resolve

disagreements, and seek guidance as necessary.

The skills will enable the AA to perform the required

actuarial services in an ethical manner that upholds the

reputation of the actuarial profession.

16

CONCLUSION

This report summarizes the methodology used to conduct a job analysis aligning with common and evidence-based

practices. The results of the job analysis provide the NAIC with a Test Specifications document (Appendix N) that

can be used to develop assessments, training, and educational content.

An evidence-based definition of a U.S. Appointed Actuary has been developed (Appendix P) using a mixed-method

research approach, where the job analysis used P&C Appointed Actuary SMEs who provided data and input. The

data and input was collected through background information, focus groups, and a survey, and was analyzed using

quantitative methods.

The NAIC can use these job analysis results to develop a body of knowledge, assess, train, and/or educate U.S. P&C

Appointed Actuaries with three years of responsible experience.

Job Analysis for U.S. Property And Casualty Qualified...

Documents

Transcript of Job Analysis for U.S. Property And Casualty Qualified...