USING SURVEYS IN INTERNAL AUDITSHernan Murdock is Vice President, Audit Division, with MIS Training...

USING SURVEYS IN INTERNAL AUDITS

Dr. Hernan Murdock, CIA, CRMA

MISTI

© 2016 MIS Training Institute Holdings, Inc.

All rights reserved.Slide 2

Welcome!

Webinar runs 12-2 PM (ET)

LINK TO MATERIALS: this was included in the email you

received with your credentials

Breaks: 5 minutes each hour

Polling: You MUST answer 90% to qualify for NASBA CPEs!

Please fill out the survey following the webinar!

Questions during Webinar--Use the Chat Function

Content: Direct to Dr. Hernan Murdock

Access, Audio or Technical:

Direct to MISTI Webinar Series

Welcome, Introduction, Administrative Matters



Dr. Hernan Murdock, CIA, CRMA

Hernan Murdock is Vice President, Audit Division, with MIS

Training Institute. Previously he was the Director of Training at a

global internal audit firm, where he oversaw the company’s

training and employee development program. Prior to that, he

was a Senior Project Manager leading audit and consulting

projects for clients in the manufacturing, transportation, high tech,

education, insurance and power generation industries.

Dr. Murdock is a Senior Lecturer at Northeastern University where he teaches

management, leadership and ethics. He is the author of 10 Key Techniques to

Improve Team Productivity and Using Surveys in Internal Audits. He has also

written chapters for two books, and articles on internal auditing, whistleblowing

programs, fraud, deception and behavioral profiling and has delivered

numerous invited talks and conference presentations at internal audit,

academic and government functions in the United States, Latin America,

Europe and Africa.



Table of Contents

1. What are surveys?

2. Types of questionnaires

3. Designing and conducting surveys

4. Preparing and running surveys

5. Analyzing survey results

6. Reporting Results

7. Best practices



SECTION I

WHAT ARE SURVEYS?



What Are Surveys?

A common data collection instrument

Used to collect information from or about people to

describe, compare or explain knowledge, opinions

and behaviors

Internal auditors need sufficient, reliable, relevant

and useful information to achieve the engagement

objectives

Surveys are useful to assess objective and

subjective information

Increases the persuasiveness of auditor findings and

comments



Types of Projects Where Surveys Can be Used

Risk Assessment/ERM

Entity-Level Controls

Corporate Governance

Ethics

Sarbanes-Oxley

Fraud

Human Resources

Whistleblowing Programs

Operational Audits

Quality Assurance Reviews

IT Reviews

Others



1. Setting objectives

2. Designing the survey

3. Preparing a reliable and valid questionnaire

4. Conducting the survey

5. Analyzing the data

6. Reporting the results

Stages of Successful Surveys



1. Setting Objectives

Clear and concise

Baseline for resources, timeline and population

Affect survey design and types of questions asked

Affects data collection and analysis




Examples of Objectives

Assess the control environment

Gather feedback on the audit team’s performance

Determine the degree of agreement with management’s operating

philosophy and ethics

Gather operational information

Solicit feedback on the perceptions regarding ethical environment

among Grade 10 or lower employees in US operations




Key Questions

What is the objective of the survey?

What do I hope to achieve with the survey?

Is this the best way to collect the desired information?

How will the requested information help me achieve my goals?

How much detail do I need?




Key Questions

What decisions do I hope to impact with the results of the survey?

Do we want to compare our results to others to do benchmarking

or against best practices?

How will I use the data I collect?

Who else will use the results and conclusions drawn from the data

analysis?




2. Designing the Survey

Align objectives with survey design. Ask:

What is the population and what sampling technique should we

use?

How many people should we poll to feel comfortable acting on

their responses?

Who will be included and who will be excluded from the survey?

How are we going to collect the necessary data?

How are we going to follow up with those that are late responding

to the questionnaire?




2. Designing the Survey

Key questions:

What resources (e.g., money, time, skills and tools) are available

to pursue our objectives?

Do we have the support from senior management to do this?

What are we going to do with the data after we get it?

What can go wrong?

What needs to go right?




3. Preparing a Reliable and Valid Questionnaire

Clarify the terminology: Terms like “ethical behavior”,

“effectiveness” and “job satisfaction” are prone to multiple

interpretations

Know the respondents: Important considerations include reading

ability and knowledge of the subject matter

Carefully match what knowledge is sought against the amount of

time available to find out




3. Preparing a Reliable and Valid Questionnaire

You should not conduct a larger-than-necessary survey asking a

myriad of questions just because you received management’s

approval to run a survey

Standardize the response format by developing an easy-to-use

questionnaire or data sheet




4. Conducting the Survey

Distribute to the target population

Make sure the staff involved is knowledgeable about the survey.

Make sure they know the objectives of the study, the timeline and

milestones and the importance to the organization

Focus on moving things along, looking out for unusual and

unexpected situations developing and being receptive to informal

feedback from participants




4. Conducting the Survey

Treat the survey with a sense of urgency, giving reasonable but

short deadlines for completion

Have a high-ranking officer in the organization write the cover letter

or announcement

Have a process in place to send reminder notices

Use blanket reminders if anonymity is more important than response

rate

Otherwise, you can target your reminders




5. Analyzing the Data

Enter the data into analytics tool as soon as possible

Look for issues, anomalies in the data for immediate attention

Don’t overuse statistics




6. Reporting the Results

Be flexible with presentation and format

Consider visual presentation over heavy text content

Format:

Methodology then results or

Results then methodology

Report as quickly as possible

Share results with as broad an audience as practical, possible,

reasonable and appropriate




SECTION II

TYPES OF QUESTIONNAIRES



Types of Questionnaires

Oral Surveys

Individual or group setting

Personally or over the telephone

More personal than written or electronic surveys

Group or audience polling technology

Turning Technologies

Option Technologies Interactive

Elway Research

ThinkTank




Oral Surveys – Advantages

Personal contact

Quality of the data

Body language

Response rate

Oral Surveys – Disadvantages

Cost and time

Types of questions possible

Bias

Negative attitude




Group Questionnaires – Advantages

Response Rate

Group Questionnaires – Disadvantages

Sample size

Scheduling




Drop-Off Surveys – Advantages

Convenience

Response rate

Drop-Off Surveys – Disadvantages

Sampling

Time

Response rate




Mail Surveys – Advantages

Sample size

Convenience

Cost

Personal bias

Mail Surveys – Disadvantages

Low response rate

Limits on respondent abilities




Electronic Surveys - Advantages

Cost savings

Easier data collection and initial analysis

Higher response rate

More candid responses

Faster distribution

Potentially faster response time and wider coverage

Interactivity




Electronic Surveys – Disadvantages

Layout and formatting

Additional instructions

Sample limitations

Level of confidentiality

Potential technical problems




Electronic Surveys – Providers

Zoomerang

SurveyMonkey

SurveyGizmo

QuestionPro

SharePoint

FreeOnlineSurveys

Lotus Notes

Others



SECTION III

DESIGNING AND CONDUCTING

SURVEYS



Survey Design

Correlational Study

Example: Relationship between

1. Employee morale and customer satisfaction

2. Trust in management and willingness to use

whistleblowing program

3. Damage claims, complaints, lost-time injuries,

absenteeism and sick leave



Survey Design

Time Studies

1. Prospective study: Study data over time into the

future. Capture present state, then examine what

happens subsequently

Cohort study of Accounting grads

2. Retrospective study: Study events that occurred

before survey.

Study of academic or professional backgrounds of

internal auditors

Number and severity of findings in departments where

manager participated in I.A. Rotation Program



Survey Design

Time Studies

Control Group with Treatment

Control (group live training)

Experimental (CBT)

Self-Control / Longitudinal / Pretest-Posttest: Study

members of group two or more times.

Take a test, take ethics class, then retake test



SECTION IV

PREPARING AND RUNNING SURVEYS



Designing Questions

Closed-ended vs. Open-ended Questions

Likert scale

Use Open-Ended Questions Use Closed-Ended Questions

The respondents’ own words are essential

You want or prefer to rate or rank responses

Respondents are willing and able to provide answers

You know what response alternatives are beforehand

The response alternatives are unknown

You prefer to count and tabulate responses

You have the skills to categorize and analyze comments

You will use charts, graphs and other statistical tools to report results

More time available Less time available and have tool to analyze results



Closed-Ended Questions

Advantages

Ease of analysis

More specific

Faster completion time

Disadvantages

Limited answers

Limited information

Discourages disclosure



Open-Ended Questions

Advantages

More flexibility

Avoid rote answers

Develop trust

Disadvantages

Time to complete

Unnecessary information and additional effort

Greater subjectivity

Confidentiality



Key Considerations

Avoid ambiguity

Phrase questions carefully

Consider level of language proficiency

Use focus groups or series of individual interviews

Changing from open-ended to close ended

Be careful when translating questionnaire



Key Considerations - Administration

Endorsement by senior management

Set reasonable but tight deadline

Explain importance if survey is optional

Let participants know how long it will take to complete

the survey

Partner with management about the survey

Consider and pre-establish roll-up levels to protect

confidentiality and anonymity

Keep the questionnaire short, questions short and

language simple



SECTION V

ANALYZING RESULTS



Analyzing Results

Implications on survey design

Software for Statistical Analysis

Excel SAS Snap Survey

SPSS Minitab Statgraphics

Winstat

Variables

Dependent

Independent



Processing Results

Edit the data to make sure it is correct and complete

Descriptive vs. Inferential Statistics

Percentages, totals, anticipating, explaining

Measures of Central Tendency

Mean, Median, Mode

Measures of Variability

Range, Variance, Standard Deviation



Presenting Results

Graphs and charts

Tables

Summary statistics



SECTION VI

REPORTING RESULTS



The purpose or objective of the study

The population, sample surveyed and criteria for inclusion

The methodology used to develop the survey and if a pilot study

was conducted

The response rate and demographic background of participants

The analytical methodology applied to the data

A discussion of the results, conclusions and recommendations

A copy of the questionnaire

Report Contents



Reports often show:

Each question as it appeared on the questionnaire

The list of response alternatives (if applicable)

A column for the frequency with which this option appeared

(as a percentage)

A count showing the number of times that response

alternative was selected (as a whole number)

Descriptive statistics (mean, median, mode, standard

deviation, etc.) applied to the results

Report Contents



Use tables, charts, graphs, histograms

Use colors, shapes and textures to

draw attention

Cross-reference the information to

locations, titles, genders, years of

experience, etc.

Group Strongly Disagree with Disagree

and Agree with Strongly Agree

Report Contents



SECTION VII

BEST PRACTICES



Anonymity is often key, so tell participants and follow through. It is

particularly important when studying highly personal, sensitive and

controversial topics and to obtain honest answers

Design the questionnaire carefully, balancing close and open-ended

questions

Ask clearly for exact information, especially when using open-ended

questions

Make questionnaire as short as possible

Engage a focus group and field test for quality control

Limit number of response alternatives

Orient response alternatives consistently

Best Practices



If you need to translate the questionnaire, use native speakers to do

so

Keep good notes about the process and procedures followed

Decide if the survey might be repeated in the future, it would affect the

design and procedures followed for subsequent comparisons and

trend analysis

Keep the questions simple and the response options logical

Avoid jargon, abbreviations, very technical, colloquial or slang terms

Simplify the language

Best Practices



Avoid words that carry a connotation that could bias the

participants’ responses

Define specific time periods and components

Use complete sentences, whether as statements or

questions

Avoid negative questions such as “Did the Auditor in

Charge fail to give you a copy of the exit meeting notes?”

Avoid confusion with Agree/Disagree items, such as

“Employees should not be required to work excessive

overtime or travel on weekends”

Best Practices



Avoid two-edged questions, which have two separate ideas. For

example: “Would you inform your manager or the ethics hotline if you

became aware of an unethical act?”

Enter data into analytics tool ASAP to check procedures

Questions that rely on time periods should not be vague or undefined

Train all data collectors

Run the survey with the endorsement of senior management

Report the results clearly, promptly, respectfully and anonymously

Best Practices



Entity Level Controls

IT

Operational

Quality Assurance – Customer Satisfaction

Representation

Sarbanes-Oxley Compliance

Risk Assessment

Best Practices - Examples



Summary

Surveys are a flexible tool, increasingly

being used by auditors to assess multiple

aspects of their organizations

Design it carefully, keeping in mind what

your objectives are and who the best

participants are

Use the results to present objectively what

many have historically considered

subjective topics

Use the results to enhance the control

environment of your organization



Please fill out the survey

following this webinar!

Thank You!

USING SURVEYS IN INTERNAL AUDITSHernan Murdock is Vice President, Audit Division, with MIS Training...

Documents

Transcript of USING SURVEYS IN INTERNAL AUDITSHernan Murdock is Vice President, Audit Division, with MIS Training...