Developing A Cognitive Assistant For Audit Plan Brainstorming Sessions

Qiao Li

Miklos Vasarhelyi

Agenda

Introduction

Research Problem

Proposed Solution - Framework

Experiment and Demo

Limitation and Future

2

Introduction

Mandated in two auditing standards: SAS No. 99 and SAS No. 109

Purpose: Discuss the susceptibility of the entity’s financial statements to material misstatement

due to fraud (Landis 2008, Hammersley et al. 2010, Beasley and Jenkins 2003)

Common procedure: Checklist in open-ended form

Importance: Generate more high-quality ideas on fraud risks as a team (Hammersley et al. 2010,

Carpenter 2007)

Affect auditors’ subsequent performance of audit

Current Issues:

Limitations with Checklist

Ineffective new ideas generation

Uncollected experience and expertise

Unreached additional external information

Audit Plan Brainstorming Session:

3

What we can do??

Audit Cognitive Assistant

Cognitive Assistant (Intelligent Personal Assistant)

Interact with users with speech-enabled technologies

Provide assistance by answering questions, making recommendations, and performing

actions (Canbek and Mutlu 2016, Hauswald et al. 2015, Myers et al. 2007, Garrido et al.

2010, Chen, 2015)

Commercial personal assistants (Mehrez, 2013): Apple Siri, Google Now, Microsoft’s

Cortana. Amazon’s Echo/Alexa IBM Watson

Cognitive Assistant (Intelligent Personal Assistant)

4

Advantages Of Cognitive Assistant In Audit Decision Plan Brainstorming:

Technically feasible

Advanced AI technologies

Big data support

Solution to a success of brainstorming meeting

Knowledge collection and organization (Knowledge Base)

Audit information retrieval and decision making support (Cognitive Computing)

(IBM 2016)

Recommendation on discussion procedure and topics (Adaptive Learning) (Myers

et al., 2007)

Audit applications / program connection (Invoking Apps) (Canbek and Mutlu

2016, Ebling 2016)

Quick interactions with auditors (Spoken Dialogue System )

Research Motivation5

QA

Proposed Method: Architecture of the Proposed Audit Cognitive Assistant

Automatic

Speech

Recognition

Query

Classifier

Question or

Action

text

Answer ShowAnswer

ExecuteAction

Action

Lucaindustry

Client

Position

Luca Luca

LucaRecommended Topics:General understanding, new events, business risks…

Processing…

You may also interested in:…

Query

Open an application

Inte

rface

Arc

hitectu

re

Modules:

• Automatic Speech

Recognition (ASR)

• Language Understanding

• Dialogue Management

• Natural Language

Generation

• Text-to-Speech synthesis

Audit Related Applications It Can Access

Web

SearchOpen (ACL,

IDEA…)Calculator

Open

standards

Open

templatesAudit

workpaper

Calendar …….backsta

ge

support

er

Knowledge Database

Knowledge

about users

Knowledge Base

DBMS

Unstructu

red data

Domain

Knowledge

6

Luca

industry

Client

Position

Luca Luca

Luca

Brainstorming

Discussion Topics

Recommended Topics:General understanding, new events, business risks…

Business risks

Processing…

You may also interested in:…

Query

Inte

rface

• General understanding

• New events/ areas

• Significant accounts

• Business Risks

• Financial statement

level risks

• Fraud Risks

• Going Concern

• Accounting policies

• IT controls

• Related Parties

• Internal Control

• Professional skepticism

• Materiality …

Risk Assessment Areas

General understanding

New events/ areas

Significant accounts

Business Risks

Fraud Risks

……

Recommended Risk

Assessment Areas

Based on Industry and Client

Examples of Recommended Sub-topics

General understanding

You may be interested in…

Company information

Management

Business strategy

Industry environment

Market competition

Economic factors

New areas/events

You may be interested in…

foreign operations

joint ventures

new agreements with

covenants

new acquisitions

new stores in other

countries (branches)

new general counsel

Cyber security

Significant Accounts

You may be interested

in…

Recom

mendation

Syste

m

Open an application

Recommender System of the ProposedCognitive Assistant7

Architecture of the Proposed Audit Cognitive Assistant

Lucaindustry

Client

Position

Luca Luca

LucaRecommended Topics:General understanding, new events, business risks…

Processing…

You may also interested in:…

Query

Open an application

Inte

rface

QAAutomatic

Speech

Recognition

Query

Classifier

Question or

Action

text

Answer ShowAnswer

ExecuteAction

Action

Arc

hitectu

re

Modules:

• Automatic Speech

Recognition (ASR)

• Language Understanding

• Dialogue Management

• Natural Language

Generation

• Text-to-Speech synthesis

Knowledge Base

DBMS

Audit Related Applications It Can Access

Web

SearchOpen (ACL,

IDEA…)Calculator

Open

standards

Open

templatesAudit

workpaper

Calendar …….backsta

ge

support

er

Knowledge Database

Knowledge

about users

Unstructu

red data

Domain

Knowledge

8

Question

Analysis

Query

Generation

Question

Normalized Question

Generated Query Answer

Matching

Answer

Extraction

Knowledge

base

Doc. With Candidates

Answer with highest score

Answer

Selection

Answer

NL Answer

Generating

Paired QA DBMS

Machine Learning Models

Answer

①

②

Question Answering System of theProposed Cognitive Assistant

① +②IR-based (Domain) QA

+ Knowledge-based

(Cognitive) QA

9

Keyword

extraction;

Punctuations,

abbreviations,

nouns and verbs

Question Answering System of theProposed Cognitive Assistant

Specialists

Materiality

Related parties

Going Concern

Client acceptanc

e

Things to be careful

Business understan

ding

Company informatio

n

history

management

executives

employee

compensation plan

events

Business

franchise/store

number

location

performance

business lines

main

new

products

competitor

market

strategy

strategy

old

new

risks

sales performan

ce

Economic factors

Weather

Season

IT security

Financial Reporting

Regulation/law

Controls

Political environme

nt

Proposed Questions and Answers Categories

Example: IT security

Q A

Do they have any security issue

before?

Customer‘s credit card

information leakage issue

What IT controls do they have? New head of IT, control over

POS systems in the stores;

control from store POS system

to servers at IT center

10

Experiment

Experiment Objective:

• Develop knowledge base for the proposed cognitive assistant

System supporting data:

• Recordings of four typical brainstorming discussions conducted in a big audit firm

• Additional cases can be integrated into the system

(Brian H. manager)

53And you mentioned that they're still trying to figure out the right strategy there

54the document also talks about a new strategy

55and transforming the business

56and I think they are trying to go from maybe a perception of a little bit more upscale,

57to more of the bar and grill.

(Brian F.Partner)

58It is interesting

59because when they started they were clearly a bar and grill restaurant

11

Question

Analysis

Query

Generation

Question

Normalized Question

Generated Query Answer

Matching

Answer

Extraction

Knowledge

base

Doc. With Candidates

Answer with highest score

Answer

Selection

Answer

NL Answer

Generating

Paired QA DBMS

Machine Learning Models

Answer

①

②

Question Answering System of theProposed Cognitive Assistant

① Domain QA:

PandoraBot②Cognitive QA:

IBM Watson

12

System Testing

Testing input: question asked by user

Testing output: generated answer

Using developed QA pairs to build Domain QA part:

Examples from Pandorabot (written with AIML):

Sample Q A

what is the number of stores and

franchise

700 stores: 650 company-owned franchise and 50

franchises

what are the main businesses

Main businesses include bar and grill, and Mexican

restaurants.

13

Limitation and Future Work

Limitation

Need large number of brainstorming meeting cases

Continue Work

Develop industry based recommendation system

Develop audit knowledge QA system

Design an “Audit Watson” for predictive audit risk assessment: case of

medical industry

14

Thank You !

15