January 10, 2017

Frank Pokrop

BD – Becton Dickinson and Co.

AGENDA

1. Welcome –

2. Introductions

3. As We Begin

4. Goals for Today’s Talk

5. Why ASQ is Important

6. What is Auditing and Basics?

7. When is an Audit Effective?

8. Measuring Audit Criteria

9. How Many Types of Audits are

There?

10. The Science

11. The Art

12. The Value of an Independent

Eye, or How to Use Your

Auditors

13. Risk

14. Becoming an auditor

15. The Value of a Corporate Audit

Function

W h a t i s Yo u r V i e w o f A u d i t i n g ?

3. As We Begin:

• Thank You!

• Today’s Presentation

• ASQ Test Prep?

• About me

• “Q&A” At Any Time

• About You – A Few Questions

S AN D I EGO ASQ - Auditing – Science, Art and More

4. Goals for Today’s Talk:

a. Education, hopefully

b. “Frank” discussion

c. Identify creative and constructive uses of audits

and auditor assignments

d. See the role that risk now plays in our lives:

i. Cost and product impact

ii. Patient and user

iii. Selection of audit sites and target

S AN D I EGO ASQ - Auditing – Science, Art and More

4. Goals for Today’s Talk:

e. Understanding:

i. VUCA (Volatility, Uncertainty, Complexity, and Ambiguity)

ii. Risk Based Thinking

f. Risk-Based Thinking

i. New Topic, or,

• ISO 9001: 2015

• ISO 13485: 2016

• ?

S AN D I EGO ASQ - Auditing – Science, Art and More

4. Goals for Today’s Talk:

g) The value of Corporate auditing – or just independence?

S AN D I EGO A SQ - Auditing – Science, Art and More

5. Why ASQ is Important

a. Personal and professional growth

b. Value to you, your family and profession

c. Learning and insights into many things,

processes, companies, issues, etc.

6. What is Auditing, and basics?

http://www.theiia.org

"Internal auditing is an independent, objective assurance and

consulting activity designed to add value and improve an

organization's operations.

It helps an organization accomplish its objectives by bringing a

systematic, disciplined approach to evaluate and improve the

effectiveness of risk management, control, and governance

processes."

S AN D I EGO A SQ - Auditing – Science, Art and More

6. What is Auditing?

a. Many, many good definitions

b. Walks like a duck….

c. Determines state of compliance with rules,

regulations, standards, industry-wide tests….

An expected level of performance

d. ASQ / ISO defined elements… formal,

announced, sponsor, etc.

e. Formal comparison against internal

expectations, external requirements and

customer needs

f. Non-standard, special request type of

examination…

S AN D I EGO A SQ - Auditing – Science, Art and More

The Goal Of An Audit:

- To Collect Objective Evidence

- To Permit An Informed Judgment About The

Status Of The Systems or Product Being Audited

Remember that Audits Have Formal Parts, Formal Requirements and a Formal Process

ISO 19011:2011—Guidelines for auditing management systems(see handouts)

Phases of an audit:

1. Audit preparation

2. Audit performance

3. Audit reporting

4. Audit follow-up and closure

http://asq.org/learn-about-quality/auditing/

A u d i t P r o g r a m / A u d i t L i f e c y c l e F l o w C h a r t

Develop resource plan

Write discrete audit plan

Perform audit: report to functional mgmt.

Enter non-conformities into CAPA system

Functional mgmt.takes ownership of CAPA response

Measure audit effectiveness

Develop and approve audit schedule

Audit Findings: (6.4.7)

Audit evidence should be evaluated against the audit criteria in order to determine audit findings. Audit findings can indicate conformity or nonconformity with audit criteria. When specified by the audit plan, individual audit findings should include conformity and good practices along with their supporting evidence, opportunities for improvement, and any recommendations to the auditee.

Nonconformities and their supporting audit evidence should be recorded. Nonconformities may be graded. They should be reviewed with the auditee in order to obtain acknowledgement that the audit evidence is accurate, and that the nonconformities are understood. Every attempt should be made to resolve any diverging opinions concerning the audit evidence or findings, and unresolved points should be recorded.

Audit Findings:

• A well-written audit finding provides the auditee

with enough information to act on and to correct

or prevent a reoccurrence.

The Audit Report:

1. Cover page or logistics header: contextual information that characterizes the report.

2. Introduction and background: information regarding the purpose and scope that was contained in either the audit plan or notification letter.

3. Results and summary: the overall conclusions and recommendations.

4. Detailed results: a listing of all findings.

5. Appendix: supplemental information that might explain or clarify the information contained in the report.

http://asq.org/quality-progress/2007/06/auditing/preparing-and-structuring-audit-report.html

Writing an audit report + ranking

a. Clear, consistent, spell-checked, formatted,

quantitative, qualitative, respectful, thankful,

reviewed (hint: independent eye)

b. Lists the site, location, dates, auditors, type of audit,

scope and other audit formalities

c. Show the conclusions, ranking, number of

observations by different severities, due date for

responses and corrective actions

Writing an audit report + ranking

d. Cites the as-found condition including qualitative and

quantitative observations and supporting data

e. Compares the finding against one or more standards,

rules or regulations

f. Ranks the severity

Risk might be used in (see handouts):

• Audit observations

• Audit reports

• Site conclusions

• Scheduling

• Priority

QS ASSESSMENT SCORECARD (Risk) High LOW

Criteria 1 2 3 4 5

Quality System:

Change X

Influence & Deployment X

Harmonization Status (BQSR Matrix) X

Product:

Risk Management (Clinical Significance) X

Changes to Product X

Regulatory Status:

Recall Classification

Functional Product Issues Not Addressed: (Production Processes and Design Controls)

Organizational Effectiveness:

Organizational Support X

Quality Organization X

Capabilities:

Manufacturing (N/A)

Engineering (N/A)

Supplier Management (N/A)

Business Influences:

Growth X

New Products X

Relocating X

Expansion X

Low Profit X

Improvement Process:

Quality Organization Capability X

Improvement Since Last Audit (N/A)

Compliance Culture X

7. When is an Audit Effective?

a. When the president gets the report

i. Who decides who sees the report?

b. When change takes place?

c. When compliance is increased?

d. When enforcement takes place?

e. When recalls decrease?

f. Role of the customer?

g. Remember to measure every aspect of an audit

When is an Audit Effective?

a. When the president gets the

report

i. Who decides who sees the

report?

b. When change takes place?

c. When compliance is increased?

d. When enforcement takes place?

e. When recalls decrease?

f. Role of the customer?

g. Remember to measure every

aspect of an audit

- Dynamic views….

- Changes with leadership

- Changes with time

- Cost measures

- Compliance status

- Audit frequency changes

- Company goals

- Insurance and liability

- Specific observations

are resolved, or……..

8. Remember to measure every aspect of an audit – see handouts

• Audits / year

• Observations: number – by site, by product, by

category, by severity, type, trends, time to close,

number- corporate, year-to-year comparisons

• Sites audited / Sites not audited

• Frequency

• Re-scheduled or cancelled

• Risk rankings by………….

SAN D IEGO ASQ

• Product

• Process

• Management

• Agency Approval

• Agency Approval – spec limits

• Directed

• Requested

• Lifecycle

• Utility

• Product returns

• Gov’t document

9. How Many Types of Audits are There?

Internal

External

Compliance

Enforcement (FDA, DEA)

Complaints

R&D

Project

System

Bottom-up

Top-down

System

• Country

• Process Deviation

• Aviation failure

• IT

• Installation and Repair

• Field Service

• Training

• Vendor

• Service

• Third-party contractor

• Raw material

• Laboratory

• ?

10. The Science (consistent, scientific approach)

Phases of Auditing

• Planning and Preparing for the audit

• Execution of the audit plan

• Reporting the audit results

• Close out of corrective actions

The Visible Part of Auditing:

• Opening Meeting

• Collection of Information

• Record and Grade Nonconformances

• Evaluation of Number and Significance of

Nonconformances

• Assessment of Compliance to Requirements

• Preparation of Findings

• Closing Meeting Review

Scheduling:

• Site

• Product

• Division

• Corporate

10. The Science – as an Auditor:

a. Scientific background

b. Detailed

c. Inquisitive nature

d. Consistent approach

e. Understands technology, organizations, documentation,

standards and how the product works

f. Preparation is the key to an effective audit

11. The Art:

a. Question: how to be effective in every audit?

b. Answers:

i. Preparation + team input + consistency

ii. Not using the same approach too often

iii. Team and individual review before starting an audit

iv. Alternating duties and roles (leader, auditor, site)

v. Changing the focus

vi. Changing the makeup of the auditing team

i. Guest auditor (development role)

ii. Guest expert

11. The Art – questions:

a. What hasn’t been audited at this site?

b. Changing your starting points

c. Reviewing external data

d. Watching the practices and processes

e. Developing new or additional questions for a specific audit

f. Value in the writing the “other audit report”

12. The Value of an Independent Eye, or How

to Use Your Auditors:

12. The Value of an Independent Eye, or How

to Use Your Auditors:

a. Standards – numerous examples

b. System:

a. Hardware/electrical - reviewing

engineering runs

b. Paper-based system examinations

c. Benchmarking – evaluating weaknesses

d. Site audit preparation

The Growth of Standards

• “audit” = 757 standards

• “audit checklist” = 12 standards

• “risk” = 2, 525 standards

• “quality audit” = 108 standards

• “compliance” = 3,533 standards

• “compliance audit” = 14 standards

12. The Value of an Independent Eye, or How

to Use Your Auditors:

a. Aviation examples

b. Medical device examples

c. Examples of cars

d. Your examples

13. Risk:

a. See handouts for examples

b. ISO 9001/ ISO 13485 (RBT)

i. Risk in every activity

ii. Risk of not performing an activity

correctly must be known (documented

too?)

iii. Risk in various audit activities: site, plant

corporate??? What else?

iv. How have you documented this?

• Curious

• Disciplined

• Understands science

• Structured

• Can shift between qualitative

and quantitative topics

14. Becoming an auditor

Some industry experience

Can build rapport but not

friendship

Familiar with auditing tools

and can shift gears

Schedule

From the FDA’s “Case for

Quality” report….

Relevance & value:

- Significant industry input

- Moderated by McKinsey

- Qualitative and quantitative

15. The Value of a Corporate Audit Function

Firms with an independent

corporate auditing function

have a better compliance

profile than those

companies that don’t

SAN D IEGO ASQ

T h a n k Yo u !

Frank Pokrop

CareFusion, Inc.

frank.pokrop@bd.com

(858) 617-4364