ISAE 3402 - VUrORE · The ISAE 3402 framework is used to provide comfort to user entities and their...

........

Authors: A. Brugge MSc (PwC) and S.P.J. Vuong MSc (PwC)

ISAE 3402

ISAE 3402 Additions for future operating effectiveness of controls

I

Preface In one of our professional debates, we often discussed how the ISAE 3402 framework could be made more useful. A recurring subject was the limitation of information on the future operating effectiveness of controls. With this idea in mind, we noted in many discussions with colleagues and fellow students that this subject is easily recognizable and people were curious in finding the solution. After this, it seemed clear to us that this would become the subject of our thesis.

By writing this thesis, we would like to contribute to the profession of IT auditing and to NOREA. Within the period this thesis was written, we couldn’t create a completely approved and formalized framework to be used internationally. However, we believe this thesis will provide in the knowledge needed to make the first steps to enhance the current set of assurance frameworks (ISAE) to elaborate on the future operating effectiveness of controls in order to address the changing assurance needs.

We could not have written this thesis without the guidance and feedback of our supervisors René Matthijsse from VU University and Tom Ooms from PwC. We would also like to thank Arnold’s wife Maaike Brugge-Cobelens (who is pregnant at the time of writing) for her support and understanding.

October 2014

Arnold Brugge and Johnny Vuong

II

Executive summary To gain assurance about a process executed by a third party, independent auditors issue an opinion about the way a process is performed by the service providing organization, using for instance the International Standard on Assurance Engagements (ISAE) 3402. Different developments are discerned, such as continuous auditing and monitoring, with the focus on more insights in the continuity aspects of an organization. Currently, the ISAE 3402 framework does not encompass information about future operating effectiveness of controls and therefore, the continuity of controls. Given these changes, clients and auditors do not only need assurance of a process performed in the past, but also need more information about how the business and controls will operate in the future. This thesis investigates which additions should be made in the current ISAE 3402 approach to give the user of the ISAE 3402 the ability to report more insights in the future operating effectiveness of the controls at the service provider.

The ISAE 3402 framework is used to provide comfort to user entities and their auditors about the internal control components related to financial reporting of the service organization covering a specified period in which controls; designed and implemented, suitably designed throughout the specified period or as at a specified date and operated effectively throughout the specified period. This leads us to the most significant limitation of the ISAE 3402 framework within the context of this thesis research; the lack of information on future operating effectiveness of controls. The most important reasons why this absence of information is essential are effective operation of primary processes, more control over the processes (contributing to continuous monitoring) and transparency regarding continuity, as it is also a necessity within financial statement audits.

Based on the analysis of similar frameworks, such as ISAE 3000, combined with interviews with stakeholders, the following conceptual additions on the audit approach are suggested to contribute the future operating effectiveness of controls and are proven in practice by the use of case studies:

1) Select the right assurance framework to address the assurance need by choosing the ISAE 3000 framework or one of the SOC2/3 related frameworks. Maintain at least the scope of ISAE 3402 to cover the essential and obligatory assurance needs and expand this scope with the additional audit work to address the future operating effectiveness aspects.

2) Understanding the client and engagement by gaining an update of knowledge, and review the effects of changes regarding applicable industry and regulatory standards. Verify if an approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system and to verify if an Internal Audit function is established and actively involved in managing the achievement of control objectives related to the ISAE 3402 scope.

3) While execution of the audit, more attention is dedicated to the amount of Meta controls, monitoring the key controls and the amount of automated controls.

4) Ensure that the report covers subsequent events, a statement of the limitations of controls and the risk of projecting to future periods, a statement of direction by management.

III

Table of contents Preface ........................................................................................................... I

Executive summary ....................................................................................... II

Table of contents ......................................................................................... III

List of Figures ................................................................................................. V

List of Tables .................................................................................................. V

1 Introduction ............................................................................................. 1

1.1 Problem statement and decomposition ....................................................... 5

1.2 Research methodology ............................................................................. 5

1.3 Scope ..................................................................................................... 6

1.4 Relevance ............................................................................................... 6

1.5 Outline .................................................................................................... 7

2 The ISAE 3402 framework and its limitations .......................................... 8

2.1 Background ............................................................................................. 8

2.2 The scope of the ISAE 3402 framework ..................................................... 9

2.3 Objectives of the ISAE 3402 framework .................................................... 10

2.4 Usage of the ISAE 3402 framework in practice .......................................... 10

2.5 Limitations of the ISAE 3402 framework ................................................... 11

3 Analysis of the ISAE 3402 framework and other relevant frameworks .. 14

3.1 Elements within the 3402 framework ........................................................ 14

3.2 Analysis of frameworks similar to ISAE 3402 ............................................. 16

3.2.1 ISAE 3000 ........................................................................................ 17

3.2.2 ISO 27001 ....................................................................................... 18

3.2.3 SOC1, SOC2 and SOC3 ..................................................................... 18

3.2.4 PCI-DSS........................................................................................... 20

3.2.5 ISA 520 – Going Concern .................................................................. 21

4 Exploratory interviews and results ......................................................... 23

4.1 Interview approach ................................................................................. 23

4.2 Interview results ..................................................................................... 25

IV

4.3 Additions in the regular ISAE 3402 audit approach as derived from research 27

4.3.1 Choose the assurance framework to address the assurance need ......... 27

4.3.2 Planning and understanding the client ................................................ 27

4.3.3 Execution of the audit ....................................................................... 28

4.3.4 Reporting ......................................................................................... 28

5 Case study research ............................................................................... 29

5.1 Approach ............................................................................................... 29

5.2 Case study A .......................................................................................... 29

5.2.1 Context ........................................................................................... 29

5.2.2 Case study findings and analysis ........................................................ 30

5.2.3 Summary ......................................................................................... 35

5.3 Case study B .......................................................................................... 36

5.3.1 Context ........................................................................................... 36

5.3.2 Case study findings and analysis ........................................................ 37

5.3.3 Summary ......................................................................................... 42

5.4 Case research outcomes and analysis ....................................................... 43

6 Research question and conclusion ......................................................... 45

6.1 Research question ................................................................................... 45

6.2 Additions in the regular ISAE 3402 audit approach ..................................... 46

6.3 Limitations of this research ...................................................................... 47

6.4 Further research ..................................................................................... 48

7 Bibliography ........................................................................................... 49

Appendix ...................................................................................................... 51

A Exploratory interview: Domain Expert .......................................................... 51

B Exploratory interview: Service Provider ........................................................ 54

C Exploratory interview: Client of Service provider ........................................... 56

D Exploratory interview: External auditor ......................................................... 58

V

List of Figures Figure 1: Outline ................................................................................................. 7

Figure 2: Standards (source: AICPA, 2010) ........................................................... 19

Figure 3: Meta controls ....................................................................................... 26

List of Tables Table 1: ISAE 3402 requirements analysis ............................................................. 14

Table 2: Case A analysis of additions .................................................................... 35

Table 3: Case B analysis of additions .................................................................... 42

ISAE 3402 - Additions for future operating effectiveness Page 1 of 59

1 Introduction Throughout the years, IT has become more and more Service-Oriented by which IT processes are outsourced to third parties. However, by outsourcing a process one does not outsource its accountability. To gain assurance about a process executed by a third party, independent auditors give an opinion about the way a process is performed by the service providing organization.

Until 2011, SAS70 was the reporting standard regarding service-providing organizations. The International Standard on Assurance Engagements (ISAE) developed by the International Auditing and Assurance Standards Board (IAASB) is a standard now used for an assurance opinion about the work performed by a Service Organization over a historic period in time, the successor of SAS70.

At the moment of writing, different developments are discerned such as Continuous auditing and monitoring. The developments are focussed on having more insight in the continuity aspects of an organization. For instance, regarding the annual financial statements reports, many discussions are held regarding the unavailability of continuity aspects of the audited organization in the annual financial statements report. The readers of the annual financial statements report, shareholders and other stakeholders, cannot form a grounded opinion and/or get insight in the future operating effectiveness of an organization as in the report the continuity aspects of an organization is not clearly explained (Mertens, Meliefste MSc, & Blij CFA, 2013). Especially with the current uncertainty in the economic developments, special attention is dedicated to the continuity of organizations. However, why is it important to look into aspects regarding continuity and future operating effectiveness? Before we look into depth why it is important to consider future operating effectiveness, allow us to first introduce our definition of future operating effectiveness.

Definition of future operating effectiveness

When we mention operating effectiveness, we refer to the effectiveness of the operation of a control. In nearly all audit standards, when performing an audit on the operating effectiveness of the controls (i.e. a Type II report), the historical information regarding the operation of the control are assessed and tested. This way the auditor gains reasonable assurance that the control has worked as it should be in a certain period in the past.

When we refer to the future operating effectiveness in this thesis, we mean the operating effectiveness of the specific control in the future. Based on the information acquired during the audit regarding the past and current operation of the control, a high-level opinion – with limited degree of assurance – on the future operation of the control can be formed. Future operating effectiveness is the operating effectiveness of the control in the future. By future, we mean any point in time after the audit has been performed.

With this definition being clear, the most important and relevant reasons why a stakeholder is interested in the future operating effectiveness of the key processes at a service organization are mentioned below.


Effective operation of primary processes

With the recent financial crisis, it is important for organizations to be more aware of the effectiveness of their processes and the related controls. This way, more or better goods and / or service can be delivered while saving costs. When (partly) outsourcing processes, it is of essence to have insight in the operating processes at the service organization. This way the user entity knows what processes are implemented at the service organization, and can use that to connect and improve their own processes. In the total value chain, more can be achieved in a more efficient matter (Holcomb & Hitt, 2007).

Nevertheless, with this integration of the user entity’s processes and the service organization’s processes, it is important to have information about whether the relevant processes at the service organization will continue to operate effectively in the near future. With this information, the user entity can anticipate its dependencies in their own primary processes.

When illustrating this perspective with an example; for a trading company it is essential to know how the logistic processes at a service providing logistics company operates. With this information, the user entity can connect their own processes to the ones of the logistics company and increase the total added value in the value chain. The trading company can now for instance inform their customer more accurate on the delivery times. With the acquired information on the operating effectiveness of the logistic company’s processes, the trading company can implement additional processes and / or controls to maintain their level of service quality when a calamity or exception occurs in the processes of the logistics company.

More control over the processes

As stated by (Roozendaal, 2011), stakeholders hold directors accountable regarding the reliability of (financial) processes, as governance becomes more and more important these days. Developments such as continuous monitoring and continuous audit become more relevant and can help organizations to map and accurately estimate the process risks. With the implementation of continuous monitoring and / or continuous auditing, it becomes possible to have insight in the operating effectiveness of the primary processes at all times, and therefore gain continuous assurance about the specific processes.

Continuous monitoring and auditing enables the organization to quickly relate the process output to the corresponding risk profile. This way the organization can instantly identify exceptions in the process output or changing risks and take corrective actions accordingly (Roozendaal, 2011).

As we have identified before in this thesis an increasing amount of organizations outsource (parts of) their processes to service organizations. In order of fulfilling the full potential of the benefits of continuous monitoring / auditing, it is of great importance to gain assurance on the processes at the service provider. An ISAE 3402 report gives assurance regarding the operating effectiveness of the controls of the service organization in the past period but it does not provide any information regarding the operating effectiveness in the near future. When relying on service organizations and willing to utilize the full potential of continuous monitoring, it is important to gain insight in the future operating effectiveness of the relevant process at the service provider.


With this information the process output can be more controlled, in a continuous assurance matter, and be corrected within the own (user entity’s) processes and controls if necessary. For example, a retail organization has outsourced its payment processes to a payment service provider (PSP). With continuous monitoring, the directors of the retail organization can monitor the most important processes related to sales and logistics. The payment process lies with the PSP, which provides not only information regarding the operating effectiveness of the payment process for the retail organization, but information on future operating processes as well. When the process impends to miss their process / control objective, the retail organization will know this soon enough to implement corrective measures and create their own workaround(s) to maintain their level of control and quality of its processes. Transparency regarding continuity also a necessity within financial statement audits

If we relate the need for information and transparency regarding continuity to the annual financial statements audit, the same can be concluded. Most financial statement audit reports lack a foundation or elaboration of the so-called “continuïteitsveronderstelling” (in English the assumption of continuity). Information regarding the continuity of a company is not transparent, implicit and / or spread across the report (Mertens, Meliefste MSc, & Blij CFA, 2013). For an outsider, the stakeholder it is hard to determine the continuity chances of an organization.

In order of fulfilling the information need regarding continuity (NBA, 2013), the accountant of the user entity needs to assess what processes are of key essence for the continuity of the organization. These key processes can be dependent on one or more service providers. Therefore, the accountant of the user entity needs insight in not only the operating effectiveness of the processes and controls at the service provider in the past, but in the future as well. This way the accountant of the user entity can include this information in his considerations regarding the assessment of the continuity of its auditee organization.

As you can imagine, a datacentre is very dependable on the controls and processes at the telecommunications company. When the controls at the telecommunications company are (partly) failing, this will have a significant impact on the continuity of the datacentre.

Additionally, with the control-based approach within the financial statement audits it is important to gain insight in the operating effectiveness of the controls in the service organization’s processes as these can affect the financial statements.

In practice, assessing the ISAE 3402 report on the service organization’s processes happens during the interim work of the audit. At this moment of the audit only an ISAE 3402 report of the previous year / period is available, in which it states that the controls in scope has or has not worked properly in the past period. This past period does not match with the time scope of the financial audit, in which we actually should conclude that limited assurance could be derived from the ISAE 3402 report.


For example, in the FY14 financial statements audit during the interim only ISAE 3402 FY13 reports are available, although as an accountant you would like assurance over the operating effectiveness of the controls in FY14, as these affects the financial statements of FY14. In some cases, the ISAE 3402 report over the period FY14 arrives at the very last moment of the financial statements audit, which is not an ideal situation.

It would be useful if the accountant of the user entity could gain more information about the continued effectively operating controls at the service organization, so he can anticipate on possible qualifiers (if present) in its audit approach for the financial statement audit.

We have defined future operating effectiveness as the operating effectiveness of the specific control in the future. Based on the information acquired during the audit regarding the past and current operation of the control, a high-level opinion – with limited degree of assurance – on the future operation of the control can be formed.

Goal

Based on the need for assurance going further than the past, this master thesis investigates which additions are needed to enhance the value of the current ISAE 3402 standard such that it is able to give insight about the operating effectiveness of a service organization in the near future.

The research goal of this thesis can be categorized as Understanding and “Guidance for Action: Design”. This thesis explains the characteristics of an ISAE 3402 audit. After the setting out the context of the ISAE 3402 standard, the ISAE 3402 standard is compared with other Assurance standards, in order to amplify the key differences and assess the added value and potential of the ISAE 3402 standard.

Once the current situation is defined, we focus on research to gain insight on how the current ISAE 3402 report is used and which information regarding the future operating effectiveness of the service organization is missing in the reports issued. Additionally we aim to gain insight in how the gap is overcome or accepted by the users of the ISAE 3402 report.

After the research performed to identify the need for information about the future operating effectiveness of a service organization, additions to a standard based on ISAE 3402 are proposed, which does not only provide assurance over the past operating effectiveness, but also provide information over the (near) future operating effectiveness of the controls at the service provider concerned. The concept additions to the regular audit approach are applied in two case studies as a proof of concept, to verify the added value.

As a result of this thesis research, not only an understanding of ISAE 3402 is provided but also additions are proposed for the ISAE 3402 approach. When these additions are performed in addition to (or integrated with) the current ISAE 3402 audit approach, the value of the ISAE 3402 audit will be enhanced, providing information about the operating effectiveness of a service organization in the near future.


1.1 Problem statement and decomposition Based on the context outlined in the paragraph above, the main question of this thesis is:

What additions should be made in the current ISAE 3402 audit approach to give the user of the ISAE 3402 report more assurance regarding the future

operating effectiveness of the service provider?

The main question can be decomposed in the following sub questions:

1) What are the main elements and characteristics of the current ISAE 3402 audit?

2) How is the current ISAE 3402 report used by stakeholders and what information is missing in the report regarding the future operating effectiveness of the service organization?

3) Which additions to the ISAE 3402 audit approach can be defined in order of assessing a service provider regarding the future operating effectiveness of controls?

1.2 Research methodology The method of this thesis research is a combination of literature study, case study and semi structured interviews with domain experts/ stakeholders regarding an ISAE 3402 audit.

To answer the sub questions, the following methods are used:


Literature study Using literature study we can build a solid base for the thesis research.


Semi-structured interviews The results from the literature study are then assessed with stakeholders who have experience in the execution and/or undergoing an ISAE 3402 audit. This is done by performing semi-structured interviews with the relevant stakeholders. The selected stakeholders are persons from an auditing firm, service provider (auditee), domain expert and a firm that uses the services of the auditee. With these four perspectives, adequate insight is acquired on the use and view of ISAE 3402 reports. The number of interviews and combination of different perspectives validates the output of the interviews.

3) Which additions to the ISAE 3402 audit approach can be proposed in order of assessing a service provider regarding the future operating effectiveness of the service organization?


Case studies The results of the thesis research are validated with two case studies and discussed with relevant stakeholders/domain experts to ensure validation.

1.3 Scope We limit our research to the standard ISAE 3402. This standard is chosen because it is widely used in the area of financial reporting. However, other standards might exist that are comparable to the ISAE 3402 standard.

Furthermore, we limit our research to perform two case studies in two different environments therefore resulting in a qualitative research.

1.4 Relevance The relevance of this research can be decomposed into two perspectives.

Firstly, currently the society requires special attention1 on the continuity of organizations. In the past years organizations encounter problems with continuity, which affect many other organizations up/ down the supply chain, employees, government and / or regular civilians. The society requires having more insight in the management of continuity risks in order of being able to anticipate on the possible consequences. In the modern world, many organizations work tightly with service providers to manage the whole process chain as efficient as possible. For this reason, it is of importance to be clear about the continuity of the operating effectiveness of the processes at the service provider related to the audittee organization, as this affects (partly) the continuity of the audittee organization. Assessing the need and implementation of continuity aspects in the ISAE 3402 standard helps to plot all the relevant continuity risks of an organization.

Secondly and partly related to the first described perspective, the current developments regarding continuous assurance, as stated in Spotlight (openly published company literature (Roozendaal, 2011), enables organizations to have more insight the effectiveness and efficiency of processes. This should include the processes that are (partly) outsourced to service providers. The current ISAE 3402 framework provides assurance based on historical information regarding the processes in scope. By adjusting the work performed it is possible give more insight in the operating effectiveness in the (near) future and therefore for the user organizations to include it in their monitoring processes in the light of continuous assurance.

1 http://www.accountancynieuws.nl/actueel/accountancymarkt/risicorapportage-in-jaarverslag-te-algemeen-voor.125662.lynkx

http://www.accountancynieuws.nl/actueel/accountancymarkt/risicorapportage-in-jaarverslag-te-algemeen-voor.125662.lynkx

http://www.accountancynieuws.nl/actueel/accountancymarkt/risicorapportage-in-jaarverslag-te-algemeen-voor.125662.lynkx


1.5 Outline This thesis is structured as described below. The previously described sub questions form broadly the structure of this thesis:

• Introduction, decomposition of the main question and its sub question with the description of the research methods

• Theoretical research regarding the ISAE 3402 standard, its limitations and the relevant developments regarding the stakeholders of the standard. Also analysis on similar frameworks on aspects that might point to future operating effectiveness

• Exploratory interview results and comparison with other relevant frameworks regarding the limitations noted

• Conceptual additions to the ISAE 3402 audit approach and case studies to proving the conceptual additions

• Conclusion with proposed additions to the regular audit approach to provide more information on future operating effectiveness

Chapter 3 Relevant

frameworks

Chapter 5

Validation by case study

Chapter 4 Practice (interviews)

Chapter 6 Conclusion

Chapter 1 Introduction

Figure 1: Outline

Chapter 2 ISAE 3402 framework


2 The ISAE 3402 framework and its limitations To gain a good understanding of the ISAE 3402 framework and its limitations, this chapter will first go into the background and the organization behind the standard. With the background in mind, the objectives and usage of the ISAE 3402 is elaborated. Based on this understanding, the limitations of the ISAE 3402 are explored and analysed. From the limitations this thesis research focuses particularly on the limitation of future operating effectiveness of controls, as in the last part of this chapter is described why this limitation is important.

2.1 Background Until 2011, Statement on Auditing Standards No. 70 (SAS 70) was the reporting standard regarding service-providing organizations. SAS 70 was a widely recognized American audit standard issued by the American Institute of Certified Public Accountants. SAS 70 provides guidance to service auditors when assessing the internal control of a service organization on behalf of a user organization. SAS 70 is applied in situations where outsourcing is in place. SAS 70 provides information on the service organization’s internal control on behalf of the user organization’s financial statement. SAS 70 is developed by accountants for accountants (Ewals, 2009). The scope of SAS 70 covers the integrity of financial reporting and may include specific controls determined by the client, who has engaged the service auditor.

A distinction in two types of SAS 70 can be made: type I and II (Ewals, 2009). A SAS 70 type I report states whether the service organization’s description of its controls are fairly presented and implemented on a certain date. A SAS 70 type II report provides the same information as a SAS 70 type I report and adds another part that reports on whether the controls that were tested were operating with sufficient effectiveness to provide reasonable assurance that the related control objectives were achieved during a specified period.

The main reason for the replacement of SAS70 was the need for an international standard. As SAS70 is an American standard, it complicates engagements that cross borders. There was a demand for a new single auditing standard that provides consistency to customers around the world. Global service organizations often issued assurance reports under various country specific standards, thereby creating more inconsistencies and confusion. Another reason was that SAS 70 did not maintain a risk based approach, its scope being limited to integrity of financial reports and management did not explicitly take the responsibility regarding internal control (Ernst & Young, 2009).

The International Standard on Assurance Engagements (ISAE), developed by the International Auditing and Assurance Standards Board (IAASB), is a standard now used for an assurance opinion about the work performed by a Service Organization over a historic period in time, the successor of SAS 70 mitigating the shortcomings noted above.


2.2 The scope of the ISAE 3402 framework To understand the scope of the ISAE 3402 framework, relevant scoping paragraphs of the framework are noted and analysed below.

Scope

According to the report issued by IFAC (IAASB, 2009).,

“The International Standard on Assurance Engagements (ISAE) deals with assurance engagements undertaken by a professional accountant in public practice to provide a report for use by user entities and their auditors on the controls at a service organization that provides a service to user entities that is likely to be relevant to user entities’ internal control as it relates to financial reporting.”

This means that the framework is used to provide comfort to user entities and their auditors about the internal control components related to financial reporting of the service organization.

“This ISAE applies only when the service organization is responsible for, or otherwise able to make an assertion about, the suitable design of controls. This ISAE does not deal with assurance engagements:

(a) To report only on whether controls at a service organization operated as described, or

(b) To report on controls at a service organization other than those related to a service that is likely to be relevant to user entities’ internal control as it relates to financial reporting (for example, controls that affect user entities’ production or quality control).

This ISAE, however, provides some guidance for such engagements carried out under ISAE 3000.” ( (IAASB, 2009)

This means that the framework only applies to controls related to financial reporting. Additionally, ISAE 3402 provides some guidance to a related framework ISAE 3000 but does not cover all.

“The performance of assurance engagements other than audits or reviews of historical financial information requires the service auditor to comply with ISAE 3000.” (IAASB, 2009)

Although our scope is set to the ISAE 3402 framework, because of the relation between both frameworks, a comparison between the two frameworks is included in chapter three to ensure that relevant information is encompassed in this research.

Based on the above, we consider the scope of the ISAE 3402 framework to be a framework used to provide comfort to user entities and their auditors about the internal control components related to financial reporting of the service organization relating to the ISAE 3000 framework, which covers internal control components other than audits or reviews of historical financial information.


2.3 Objectives of the ISAE 3402 framework According to (IAASB, 2009) the objectives of the service auditor are:

a) To obtain reasonable assurance about whether, in all material respects, based on suitable criteria:

(i) The service organization’s description of its system fairly presents the system as designed and implemented throughout the specified period (or in the case of a type 1 report, as at a specified date);

(ii) The controls related to the control objectives stated in the service organization’s description of its system were suitably designed throughout the specified period (or in the case of a type 1 report, as at a specified date);

(iii) Where included in the scope of the engagement, the controls operated effectively to provide reasonable assurance that the control objectives stated in the service organization’s description of its system were achieved throughout the specified period.

b) To report on the matters in (a) above in accordance with the service auditor’s findings

Based on the above, we consider the objectives of the ISAE 3402 framework to be:

A framework used to provide comfort to user entities and their auditors about the internal control components related to financial reporting of the service organization relating to the ISAE 3000 framework, which covers internal control components other than audits or reviews of historical financial information covering a specified period in which controls:

• Designed and implemented • Suitably designed throughout the specified period or as at a specified date • Operated effectively throughout the specified period

2.4 Usage of the ISAE 3402 framework in practice The ISAE 3402 framework is used in practice for different reasons than the intended purpose (refer to chapter 2.3. According to the interview with Domain Expert and (Leenders RA & Nagy RO, 2013), the following three reasons can be distinguished:

• Mandatory because of external requirements (law and regulations) • As a trigger to improve a company’s internal control framework • As a unique selling point to prove to their customers that they are in control

Depending on the reason, one is more eager to cover more processes and controls. Mainly, the ISAE 3402 is used as an auditor to auditor’s report (reason one of the above) to cover the risk of material misstatement in processes that are performed by the service organization.


When performing the audit of the annual financial statement of a company, the report as stated above is needed to cover all financial statement line items. As an auditor, the audit approach for the coming year relies on the ISAE 3402 audit report to be present for service organizations, especially, when this was to be true in the previous year. To help the auditor in the process of determining the audit approach, more insight in the quality of the service organization is needed to assume that a report without a qualified opinion can be issued the next year. Currently, this is not part of the ISAE 3402 framework and report. More details will be provided to this limitation in paragraph 2.5.

Regarding the second and third reason, the emphasis lies on proving or improving a company’s internal control system. Therefore, a company’s goal is to embed as many processes and controls as possible (within reasonableness).

The ISAE 3402 framework does however not support:

• All types of assurance • All objects of research • All types of scope • All periods of time

In practice, according to the interviewed domain expert, (Leenders RA & Nagy RO, 2013) and our own experiences, the ISAE 3402 framework is sometimes used to report on more than the framework was intended to provide. This leads us to the limitations of the ISAE 3402 framework in the next paragraph.

2.5 Limitations of the ISAE 3402 framework With the current use of the ISAE 3402 framework, we gain insight in the design of the controls in place at the service organization and whether controls operate effective over the period in scope.

However, there are limitations on the ISAE 3402 framework as we encountered during our audit work. From literature study and the interviews held the same limitations are observed.

From all the mentioned limitations in our daily work, interviews held and literature study, it comes down to the limitations as mentioned below in this paragraph, including the impact that these limitations have on the assessment of the external auditor and/or user entity.

Considering the use and appliance of the results of the ISAE 3402 audit in other audits as described in the previous chapter, we can observe several limitations in the framework. Out of these limitations, we look further into the relevance and importance of the limitations.

1) The ISAE 3402 framework requires a risk-based approach. Based on the risk management procedures of the service organization the most relevant controls are considered and included in the scope of the ISAE 3402 audit. These controls are the controls related to a service organization’s operations and compliance objectives, which is relevant to a user entity’s internal control as it related to financial reporting (IAASB, 2009). Defining which controls at a


service organization are likely to be relevant to user entities’ internal control is dependable on the defined control objectives and the suitability of the criteria as set by the service organization. This entails that the risk management procedures are adequately implemented. When not properly implemented, there is a risk that one or more relevant controls are not taken into account. For this reason, it is important to assess the controls in scope of a performed ISAE 3402 audit in order of adequately estimating the impact on the user entities’ internal control. The ISAE 3402 framework does dictate certain controls to be in scope.

2) An ISAE 3402 report describes whether the controls in design, implementation and operating effectiveness have met the related control objectives. However, this does not give information whether the controls will meet the related control objectives in the future (Buitendijk & van Gerner, 2011). The report only describes merely what controls have operated effectively and which ones encountered exceptions in the past period; it does not give any direct insight in the operating effectiveness of the controls in the (near) future.

3) The ISAE 3402 framework is not designed to cover all possible scope, types of assurance, objects of research and periods (please refer to point two above). For different (commercial) reasons, companies would like to fit as much as possible in the report which is in conflict with the original goal of the framework (Leung, 2011)

If we look into the limitations above, the limitation of not providing information about the reasonableness of the future operating effectiveness of the controls in scope is considered the most important one. Especially with the current need, in the light of the recent financial crisis, accounting scandals, for more transparency and control of one’s processes; we determined that organization require more insight the operation effectiveness of their internal controls, including the related controls at the service organization.

As described in detail in chapter one, the most important reasons why information on future operating effectiveness is relevant for the different stakeholders of the service provider can be summarized in three points:

• Effective operation of primary processes: as many processes are (partly) outsources, it is important to have insight in the operating processes at the service organization and its dependencies with one's own primary processes. With future operating effectiveness more can be said over the output of the outsource process (parts) over the upcoming period and therefore strengthen the control on the process output over time. This way the output of the primary processes remains controllable over time.

• More control over the processes: with the current development towards continuous monitoring, it enables organizations to relate the process output to the corresponding risk profile. This way the organization can instantly identify exceptions in the process output or changing risks and take corrective actions accordingly. To be able to be ahead of upcoming exceptions and/or risks, it is important to have insight in the future operating effectiveness of outsourced (parts of) processes.


• Transparency regarding continuity also a necessity within financial statement

audits: when a user organization is very dependable on a service provider, it is important to assess the future operating effectiveness of the controls. When these controls, mainly the ones that are important for the continuity of the user organization, are likely to (partly) fail the user organization can act timely upon to secure its continuity.

In the next chapter, we look into the conceptual additions to overcome the identified limitations.


3 Analysis of the ISAE 3402 framework and other relevant frameworks

In order to suggest additions to expand the scope of the regular ISAE 3402 audit approach, the framework itself is analysed, similar frameworks and other industry standards are reviewed on aspects, which might point to information on future operating effectiveness.

3.1 Elements w ithin the 3402 framework Because of the goal of the ISAE 3402 framework, there are no elements specified that support statements about future operating effectiveness of controls. However, keeping the concept of future operating effectiveness (from chapter one) in mind, we are able to define current elements that might be able to address the subject.

The framework consists of a list of requirements. Those requirements need to be addressed in the audit and or in the report. Based on our experience with ISAE 3402 assignments, we have indicated which requirements are likely to be useful to gain insight for future operating effectiveness of controls.

Table 1: ISAE 3402 requirements analysis

Requirements Likely to be usable

Unlikely to be usable

ISAE 3000 X

Ethical requirements X

Management and Those Charged with Governance X

Acceptance and Continuance X

Assessing the Suitability of the Criteria X

Materiality X

Obtaining an Understanding of the Service Organization’s System X

Obtaining Evidence Regarding the Description X

Obtaining Evidence Regarding Design of Controls X

Obtaining Evidence Regarding Operating Effectiveness of Controls X

The Work of an Internal Audit Function X

Written Representations X

Other Information X

Subsequent Events X


Requirements Likely to be usable

Unlikely to be usable

Documentation X

Preparing the Service Auditor’s Assurance Report X

Obtaining an Understanding of the Service Organization’s System

Regarding Obtaining an Understanding of the Service Organization’s System (IAASB, 2009), the following is defined:

“ 20. The service auditor shall obtain an understanding of the service organization’s system, including controls that are included in the scope of the engagement.”

In practice, using for instance PwC working papers PwC ISAE 3402 library, 2012 (PricewaterhouseCoopers, 2012), we gain an update of knowledge of, and review the effects of applicable industry and regulatory standards with a focus on significant changes affecting the current period or future periods. Therefore, this requirement already has insight in significant changes that would affect future periods. Based on this insight, it is likely that we can assess the impact on controls and their future operating effectiveness.

Obtain evidence

While performing procedures regarding Obtaining Evidence Regarding Design of Controls and Obtaining Evidence Regarding Operating Effectiveness of Controls, information is gathered from employee’s carrying out the day-to-day activities. This information might be relevant for next year’s audit, which will be documented in the working papers, but is not part of the final report due to the reporting period agreed upon.

Subsequent Events

Regarding Subsequent Events (IAASB, 2009), the following is defined:

“ 43. The service auditor shall inquire whether the service organization is aware of any events subsequent to the period covered by the service organization’s description of its system up to the date of the service auditor’s assurance report that could have a significant effect on the service auditor’s assurance report. If the service auditor is aware of such an event, and information about that event is not disclosed by the service organization, the service auditor shall disclose it in the service auditor’s assurance report.

As states above, Subsequent Events is part of the final stage before preparing the Service Auditor’s Assurance Report. These events cover the period between test work performed regarding the reporting period agreed upon to the moment that the report would be issued.


Preparing the Service Auditor’s Assurance Report

Regarding Preparing the Service Auditor’s Assurance Report (IAASB, 2009), the following is defined:

“j) A statement of the limitations of controls and, in the case of a type 2 report, of the risk of projecting to future periods any evaluation of the operating effectiveness of controls.”

Although the above might be seen as a limitation, the fact that the auditor needs to make a statement about the limitations might be usable for mentioning relevant information about future operating effectiveness of controls.

Bridge letter In practice, as is done for ADP, a so-called Bridge letter is issued based on inquiry with Management and those charged with governance. This is a solution to mitigate the limitation as set out in chapter one. However, inquiry is the lowest level of evidence (out of inquiry, observation, inspection and re-performance) and might not be sufficient using the current scope of the ISAE 3402 framework. The Bridge letter itself however might be usable to reflect on the proposed additions as set out in paragraph 5.1.

Based on the analysis performed in paragraph 3.1, the following items of the ISAE 3402 framework might contribute to elaborate on the future operating effectiveness of controls:

• Obtaining an Understanding of the Service Organization’s System • Obtain evidence • Subsequent Events • Preparing the Service Auditor’s Assurance Report • Bridge letter

3.2 Analysis of frameworks similar to ISAE 3402 In this sub chapter, several frameworks related and/or similar to ISAE 3402 are analysed. We look into the frameworks, identifying points in the current frameworks, which provide information on the future operating effectiveness within the organizations.

We expect these points to be not explicit regarding future operating effectiveness as we identified that the focus is very limited on this matter. Therefore, we are looking for starting points for gathering information on future operating effectiveness in the current frameworks. If we can incorporate these identified starting points in the ISAE 3402 audit, we are able to give more information on future operating effectiveness.

These identified points are summarized below.


3.2.1 ISAE 3000 A framework that is closely related to the ISAE 3402 framework is the ISAE 3000 framework (IAASB, 2008). “This International Standard on Assurance Engagements (ISAE) deals with assurance engagements other than audits or reviews of historical financial information, which are dealt with in International Standards on Auditing (ISAs) and International Standards on Review Engagements (ISREs), respectively.” When we examine the objective from a practitioner point of view: “6. In conducting an assurance engagement, the objectives of the practitioner are: (a) To obtain either reasonable assurance or limited assurance, as appropriate, about whether the subject matter information (that is, the reported outcome of the measurement or evaluation of the underlying subject matter) is free from material misstatement; (b) To express a conclusion regarding the outcome of the measurement or evaluation of the underlying subject matter through a written report that clearly conveys either reasonable or limited assurance and describes the basis for the conclusion; (Ref: Para. A1) and (c) To communicate further as required by relevant ISAEs.” Results from the analysis related to identify points, which might refer to future operating effectiveness, are summarized below:

• The objective defers from the ISAE 3402 standard leaving more room for professional judgment of the auditor.

When we examine the phase Preparing the Service Auditor’s Assurance Report:

“For example, in an assurance report related to the effectiveness of internal control, it may be appropriate to note that the historic evaluation of effectiveness is not relevant to future periods due to the risk that internal control may become inadequate because of changes in conditions, or that the degree of compliance with policies or procedures may deteriorate.” Results from the analysis related to identify points, which might refer to future operating effectiveness, are summarized below:

• Like the ISAE 3402 standard, ISAE 3000 states that a remark needs to be made in the report regarding future periods and effectiveness of controls.


3.2.2 ISO 27001 The standard describes itself as (British Standard Institute, 2005):

ISO 27001 certification is used by the service provider to show the outside world (i.e. their clients) that their information security is in control. Results from the analysis related to identify points, which might refer to future operating effectiveness, are summarized below:

• The standard refers to a comprehensive Information Security Management System (ISMS) in which changes to the ISMS is included in the standard. The standard takes into account that the ISMS and their supporting systems are subject to change over time. The standard adopts a process approach for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's ISMS. This approach is based on the “plan-do-check-act” (PDCA) model. Assessment of how the organization handles changes in ISMS, and therefore its processes/controls is useful in the light of future operating effectiveness.

If the controls are tested for a certain period of time, one wants to know whether the controls will work in the future and are limited affected by organizational and or process changes. If the user of the service provider is able to gain insight in the management regarding process and/or organization changes, more information can be gathered on the future operating effectiveness of the controls in scope. 3.2.3 SOC1, SOC2 and SOC3 Service Organization Controls (SOC) is a term used in US standards to refer to audit reports giving an attestation regarding controls at a company providing services (AICPA, Service Organization Controls, managing risks by obtaining a Service Auditor's Report, 2010).

SOC 1 engagements are performed in accordance with Statement on Standards for Attestation Engagements (SSAE) 16, Reporting on Controls at a Service Organization. SOC 1 reports focus solely on controls at a services organization that are likely to be relevant to an audit if a user entity’s financial statements. SOC 2 and 3 reports represent significant changes in service organization reporting approaches brought about as a result of several important changes.

“This International Standard has been prepared to provide a model for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an Information Security Management System (ISMS). The adoption of an ISMS should be a strategic decision for an organization. The design and implementation of an organization’s ISMS is influenced by their needs and objectives, security requirements, the processes employed and the size and structure of the organization. These and their supporting systems are expected to change over time. It is expected that an ISMS implementation will be scaled in accordance with the needs of the organization, e.g. a simple situation requires a simple ISMS solution.”


SOC 1 (SSAE 16, AT Section 801)

SOC 1 is based on SSAE 16 or AT Section 801 (AICPA, Reporting on Controls at a Service Organization, 2011).

Results from the analysis related to identify points, which might refer to future operating effectiveness, are summarized below:

• SOC 1 and SSAE 16 standards are much like the ISAE 3402 framework. They cover the same goal and it is clearly noted in AT Section 801 that the objectives of the service auditor is to: a. obtain reasonable assurance about whether, in all material respects, based on suitable criteria,

i. management's description of the service organization's system fairly presents the system that was designed and implemented throughout the specified period (or in the case of a type 1 report, as of a specified date). ii. the controls related to the control objectives stated in management's description of the service organization's system were suitably designed throughout the specified period (or in the case of a type 1 report, as of a specified date). iii. when included in the scope of the engagement, the controls operated

effectively to provide reasonable assurance that the control objectives stated in management's description of the service organization's system were achieved throughout the specified period. b. report on the matters in 6(a) in accordance with the service auditor's findings. (AICPA, Reporting on Controls at a Service Organization, 2011)

Therefore, SOC 1 does not provide in handles to be used for our research other than already mentioned in paragraph 3.1.

Figure 2: Standards (source: AICPA, 2010)


SOC 2 and 3 (AT Section 101)

SOC 2 and 3 are based on AT Section 101 (AICPA, Attest Engagements, 2001) and Trusted Service Principles (AICPA, TRUST SERVICES PRINCIPLES AND CRITERIA, 2014). Important note is that SOC 1 / 2 / 3 are terms on which assurance frameworks can be related to, SOC 1 / 2 / 3 are no assurance frameworks. Furthermore, as noted in the mentioned source above SOC 2 / 3 requires that the audit approach should address, besides the objectives on the financial aspects, the obligated objectives of the Trust Service Principles as well.


• The AT section 101 does not support including remarks about future operating effectiveness of controls but does also not clearly state that the report should only cover historical data. Instead, the following is mentioned regarding the subject matter: Historical or prospective performance or condition. Therefore, it should be possible to report on future operating effectiveness of controls using SOC 2 or 3.

3.2.4 PCI-DSS The standard describes itself as:

This PCI-DSS framework is concerned with service providers, and therefore relevant in this thesis research for further analysis.


• In the scope of the PCI-DSS standard, there is no special attention on operating effectiveness of controls. However, the standard does mention that changes to the organizational structure should be appropriately addressed and mapped to the impact on PCI DSS scope and requirements. The periodic (audit) reviews should verify that the PCI DSS requirements continue to be in place at the organization. It does not mention that auditors should provide information on future organizational changes and its impact on the PCI DSS scope and requirements.

• In the standard itself, no references to future operating effectiveness are present.

“The Payment Card Industry Data Security Standard (PCI DSS) was developed to encourage and enhance cardholder data security and facilitate the broad adoption of consistent data security measures globally. PCI DSS provides a baseline of technical and operational requirements designed to protect cardholder data. PCI DSS applies to all entities involved in payment card processing—including merchants, processors, acquirers, issuers, and service providers, as well as all other entities that store, process or transmit cardholder data and/or sensitive authentication data” (PCI Security Standards Council, 2013).


3.2.5 ISA 520 – Going Concern As we understand from the International Standard on Auditing 520, as published on the IFAC website (IFAC, 2009) and described in the PwC audit guide (PricewaterhouseCoopers, 2014), this standard describes the auditor’s responsibilities in the audit (of financial statements) relating to management’s use of the going concern assumption (in the preparation of the financial statements).

Under the going concern assumption, an entity is viewed as continuing in business for the near future. This means that the results of the audit (in this case of the ISA 520 the general purpose financial statements) are prepared on a going concern base. In some of the financial reporting frameworks, require the management to make a specific assessment of the entity’s ability to continue as a going concern. The auditor’s responsibility is to obtain sufficient appropriate audit evidence about the appropriateness of management’s use of the going concern assumption.


• As the going concern is focused on the overall continuity of the business of the entity, we believe that we can use the same responsibility outlines, as described in the ISA 520, to enforce the auditor and management to assess the effect of current or near future developments within the entity on the (future) operating effectiveness of the entity’s controls. For instance, major IT system migrations or reorganizations can have an impact on the operating effectiveness of controls. With the responsibility, outlines similar to those in ISA 520, management and the auditor are required to assess the entity’s ability to maintain effectively operating controls.

Conclusion

Based on the analysis of the different frameworks above, we conclude that:

• The objective of ISAE 3000 defers from the ISAE 3402 standard leaving more room for professional judgment of the auditor and it states that a remark needs to be made in the report regarding future periods and effectiveness of controls.

• The ISO27001 standard refers to a comprehensive Information Security Management System (ISMS) in which changes to the ISMS is included in the standard. The standard adopts a process approach for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's ISMS. This approach is based on the “plan-do-check-act” (PDCA) model.

• The AT section 101 does not clearly state that the report should only cover historical data. Instead, the following is mentioned regarding the subject matter: Historical or prospective performance or condition. Therefore, it should be possible to report on future operating effectiveness of controls using SOC 2 or 3.

• PCI-DSS mentions that changes to the organizational structure should be appropriately addressed and mapped to the impact on the scope and requirements.


• The responsibility outlines, as described in the ISA 520, can be used to enforce

the auditor and management to assess the effect of current or near future developments within the entity on the (future) operating effectiveness of the entity’s controls.


4 Exploratory interviews and results In this chapter, additional research, on top of the literature research, is performed on the actual reality of the ISAE 3402 standard. To get a good overview of the practice, use and performing the ISAE 3402 audit in reality, we have chosen to perform semi-structured interviews with the stakeholders of the ISAE 3402 audits. The interviews serve an explanatory goal within this thesis research on the appliance of the ISAE 3402 in practice. In this way, we can combine and merge the research results from both the theory as the reality, in order of coming to a realistic approach on giving more information regarding future operating effectiveness within the ISAE 3402 reports.

4.1 Interview approach The interviews are held with the stakeholders involved with an ISAE 3402 audit. To gain a complete overview of the use and view of the ISAE 3402 reports, we have selected four perspectives from which we arranged the interviews.

1) External auditor, performing ISAE 3402 audits 2) Service provider organization, the auditee 3) Domain expert on the subject of ISAE 3402 4) User organization (the firm that relies on the services of the service provider. In

this case, a user organization with a reasonably sized internal audit department)

For the four different perspectives, we have selected the persons / organizations, which are significantly involved in ISAE 3402 and have a strong opinion on the standard and its developments. This way we want to acquire as much information from the interviewees as possible.

With these four perspectives, adequate insight is acquired on the use and view of ISAE 3402 reports. Insight in the practical use of the ISAE 3402 is gained, in which we can also detect the limitations of the framework in practice. Interviewees, from their perspectives, experience and knowledge share their ideas on possible additions/solutions – which can lead us to broaden or deepen our research.

The number of interviews and combination of different perspectives validates the output of the interviews. The interviews are semi-structured, and based both the results from the literature study within this thesis research as well as our knowledge and experience as external auditors performing the ISAE 3402 audits.

Below the interviewee’s and their role are summarized. From each interview we have summarized the points that we have discussed, this can be found in the appendix of this thesis. We have anonymized the interviewee’s names, the names are known with the thesis supervisors.

Interviewee’s

Service provider

A Controller working for a fast growing Payment Service Provider, ISAE 3402 audits are performed annually in his organization.


Client of a service provider

Involved in many client consultations to intermediate on behalf of the user organization with the service organization(s).

External Auditor

Involved in the execution of many ISAE 3402 engagements by the (Big 4) firm.

Domain expert

Involved in many (global) developments regarding SAS70 and ISAE 3402. Currently in discussion with NBA/NOREA on SOC2 audits.

Question structure

Per interviewee, we have defined open and closed questions. For each person / role, dependable on their role in relation to ISAE 3402 audits, we have additional questions. Both the general as the additional questions based on the interviewee’s role are summarized below. Please note that we have not walk through the questions on a sequential manner, as the questions function as a guide for the interviews, but are not exhaustive. In this way, the interviewees have enough space to bring their own opinions and suggestions for the framework.

Questions (general)

• Can you describe your professional role and background • How does your profession relate to the ISAE 3402 framework • What do you like and dislike about the ISAE 3402 framework • What is your view regarding the limitation of future operating effectiveness of

controls • What is your experience in practice related to the limitations • What would be regarded as added value to the report when issued without the

mentioned limitations • What would you suggest to add or change to mitigate the limitations

Questions (domain specific)

Service Provider

• Do you have the insight in your processes to assess operating effectiveness of control objectives over the coming year

• Which conditions needs to be addressed (e.g. technology, people and processes)

Auditor

• What is needed to use a ISAE 3402 more efficiently / effectively in your audit • How would you embed the suggested additions in the ISAE 3402 audit

approach Client of service provider

• Would it give you more assurance if a report is issued without the mentioned limitations


Domain expert

• Is it possible within the boundaries of the audit standard to mention future operating effectiveness in the report, if the information is available to the auditor

• Which developments do you see in the audit profession regarding third party assurance that would affect our object of research

4.2 Interview results The complete results of the interviews are part of the appendix. The most important results are documented below.

Choosing the right framework covering the need for assurance Both the Domain Expert as Client of the Service Provider mentioned in their interviews that the ISAE 3402 framework is sometimes stretched to cover more needs than originally intended. To some stakeholders, a Service Level Agreement and Reporting is sufficient to the need for assurance. Therefore, there is no need to use the ISAE 3402 framework for such assurance when other formats and frameworks are in place.

Based on our review of the ISAE 3402 framework, the SOC frameworks and the ISAE 3000 framework, the following additions are suggested:

• The ISAE 3000 framework covering the 3402 format enables the auditor to extend the scope of the audit beyond the limitations of 3402. The auditor can base its audit on the principles of the 3402 standard, but is not bound to its limitations.

• SOC 2/3 enables the auditor to extend the scope of SOC1 beyond its limitations (similar to the ISAE 3402 framework)

Organizational change management Both the Domain Expert and Service Provider indicated in their interviews that there is a need to continuously monitor organizational changes and its impact on the controls in scope of the ISAE 3402 audit. The PCI-DSS framework mentions this aspect as well. The Service Provider mentioned in the interview that an Internal Audit function which is actively involved in managing the achievement of control objectives, related to the ISAE 3402 scope, is highly recommended and might contribute to ensure future operating effectiveness of controls. The Domain Expert suggested in the interview that we should assess the change management processes over the changes in primary process to determine how the organization estimates the impact of process changes and its impact on the control objectives. Theses assessments can be part of the description of COSO elements, as pointed out by interviewed External Auditor. With the combined view of the controls framework and the COSO elements in place the user of the report can form its own opinion on the future operating effectiveness of the controls and/or the control organization in total.

Based on our review of the PCI-DSS framework and ISO 27001 and the performed interviews, the following additions are suggested:


• An approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system.

• An Internal Audit function is established and actively involved in managing the achievement of control objectives related to the ISAE 3402 scope.

• Both additions can be part of the COSO assessment in the existing audit approach and report.

Meta controls and automated controls During the interviews with the Domain Expert and the External Auditor mentioned that when a company’s internal control system is more mature, Meta controls covering regular controls are implemented that contribute to a more mature control environment. Meta controls can be defined as controls implemented to monitor the key controls. Therefore, it is more likely that controls in mature environment with Meta controls implemented will continue to operate effectively in the future. These Meta controls can be part of an organization wide quality management system.

Examples of monitoring controls include monitoring controls over key controls but can also controls regarding the reliability of Service Level Reports, which the service organization sends to their user organizations. This way the user organization gets reliable insight in the performance of the Service Provider during the year, even after the audit report has been issued. This can be regarded as form of information on “future operating effectiveness” of controls, as the audit report is older than the information provided by the Service Level Report. Important is that the KPI’s internally within the service organization are aligned with the KPI’s mentioned towards the user organization, on which the latter relies on. The Meta controls can be included in the controls framework, so the recent audit approach is not required to be changed.

With reliable periodic Service Level Reports, the user organization can better anticipate on possible failure of controls. As these controls are included in the controls framework, no special adjustment on the audit approach is required.

The same holds for automated controls, which are unlikely to operate ineffectively working except when the IT General Controls are found to be inadequate. These IT General controls can be part of an organization wide quality management system as well.

Based on the interviews, the following additions are suggested:

• The amount of automated controls as a percentage of the total control measures per control given the presence of reliable IT General Controls.

• Include Meta controls, such as monitoring controls over key controls and controls over reliability of Service Level Reporting in the controls framework.

Figure 3: Meta controls

C C

MC


Directive Report According to the interviewed Service Provider, a Directive Report by management is always part of the annual report of a company. Such a report is not part of a standard ISAE 3402 report in which only a management’s assertion is included and the system description. This report can be setup because of similar guidelines as described in ISA 520. This means that management is required to assess the impact of current of near future developments within the entity on the operational effectiveness of its controls.

Based on the interviews, the following additions are suggested:

• A statement of direction by management is required and should be part of the report to be issued. This means that management is required to assess the impact of current of near future developments within the entity on the operational effectiveness of its controls.

The conceptual additions to the regular ISAE 3402 audit approach as derived from the interviews held, combined with the results from the literature study are described in chapter 4.2

4.3 Additions in the regular ISAE 3402 audit approach as derived from research

The results from the literature study and the interviews held combined and analysed, we have derived the following suggestions and / or conceptual additions on providing more information on future operating effectiveness.

4.3.1 Choose the assurance framework to address the assurance need

1) ISAE 3000 framework covering the 3402 format enables the auditor to extend the scope of the audit beyond the limitations of 3402.

2) SOC 2/3 enables the auditor to extend the scope of SOC1 beyond its limitations (similar to the ISAE 3402 framework)

4.3.2 Planning and understanding the client 1) Gain an update of knowledge of, and review the effects of applicable industry

and regulatory standards with a focus on significant changes affecting the current period or future periods.

2) An approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system. This approach can be included in the current COSO assessments. Note the changes within the organization and its impact on the control framework.

3) An Internal Audit function is established and actively involved in managing the achievement of control objectives related to the ISAE 3402 scope. This information can be included in the current COSO assessments


4.3.3 Execution of the audit

1) The amount of automated controls as a percentage of the total control measures per control given reliable IT General Controls.

2) Include Meta controls, such as monitoring controls over key controls and controls over reliability of Service Level Reporting in the controls framework.

4.3.4 Reporting 1) The service auditor shall inquire whether the service organization is aware of

any events subsequent to the period covered by the service organization’s description of its system up to the date of the service auditor’s assurance report that could have a significant effect on the service auditor’s assurance report.

2) A statement of the limitations of controls and, in the case of a type 2 report, of the risk of projecting to future periods any evaluation of the operating effectiveness of controls.

3) A statement of direction by management is required and should be part of the report to be issued.

4) The bridge letter needs to reflect on the proposed additions that are likely to contribute to the assessment of future operating effectiveness of controls.

As the conceptual additions are now identified, the added value is assessed in two case studies. Please refer to chapter five.


5 Case study research Using two case studies, the results of the chapters before are applied in practice. By doing so, we gain insight in the contribution to assess future operating effectiveness per suggested addition. This chapter describes the approach of the case studies, the results per case study and the overall conclusion based on the two case studies performed. The case studies are based on real organizations from ISAE 3402 engagements where we were involved, but the information is anonymized.

5.1 Approach To prove our theoretical additions in practice, two case studies are conducted at two different organizations.

First, we describe the context to which the additions are applied to gain an understanding about the as is situation.

Second, we apply the theoretical additions to the case in order to verify whether the suggested additions contribute to a better knowledge about future operating effectiveness of controls given the situation.

Last, we conclude with a conclusion per case study stating the results per addition and whether an addition is likely to contribute to the overall goal of future operating effectiveness or not.

5.2 Case study A Per case study, the context, the results and conclusions will be described in the paragraphs below.

5.2.1 Context Company description

The company used for case study A, is known for its highly digital platform that enables people in the Netherlands and abroad, to buy and sell personal belongings. Besides consumers, also businesses are allowed on the platform. Revenue by advertisement is the most important stream for this company.

ISAE 3402 description

One of the products related to the advertisement revenue stream, is the possibility to pay per click per advertisement. The scope of the ISAE 3402 engagement is from the moment a click is generated to the moment that the cost of this click is invoiced.

The following objectives are part of the scope of the engagement:

• Accuracy and completeness of clicks assigned to advertisers • Accuracy and completeness of invoices based on the usage data • Reliability of IT General Controls


5.2.2 Case study findings and analysis The results of the case study performed are noted below using the three stages of the audit engagement as we know them; client and engagement acceptance, planning and understanding, execution of the audit and reporting.

Choose the reporting framework to address the assurance need (engagement acceptance)

1) ISAE 3000 framework covering the ISAE 3402 format enables the auditor to extend the scope of the audit beyond the limitations of 3402.

Applied in practice Because ISAE 3402 is an extension of ISAE 3000 based on ISA402 regarding service organization, there is no problem in using the ISAE 3000 framework for the current scope as long as the ISAE 3402 format is used such that the report is usable as audit evidence for the annual audit of financial statements.

Result Based on the above, the current scope of the engagement can be executed using the ISAE 3000 standard.

2) SOC 2/3 enables the auditor to extend the scope of SOC1 beyond its limitations

(similar to the ISAE 3402 framework)

Applied in practice SOC 2, based on the Trusted Service Principles, does not have the limitations of SOC1/ISAE 3402. Therefore, the five pillars can be mapped to the current scope of the engagement. The pillars are security, privacy, process integrity, continuity and availability.

The objectives below can be mapped as follows:

Process integrity

• Accuracy and completeness of clicks assigned to advertisers • Accuracy and completeness of invoices based on the usage data

Security, privacy, continuity and availability

• Reliability of IT General Controls

Result Based on the above, the current scope of the engagement can be executed using the SOC2 standard.


Planning and understanding the client

1) Gain an update of knowledge of, and review the effects of applicable industry and regulatory standards with a focus on significant changes affecting the current period or future periods.

Applied in practice In the current situation, no control objectives are related to industry or regulatory standards such that a change the average laws and regulations would significantly impact the scope of the audit. Additionally, there are no specific (industry) standards related to click registration of websites.

Result Based on the current scope, an updated knowledge of industry and regulatory standards do not give more insight in the future operating effectiveness of controls.

2) An approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system. Note the changes within the organization and its impact on the control framework. Applied in practice There is no formal approach of managing the organization’s ISMS. Therefore, organizational changes and its possible impact to the ISAE 3402 scope are not noted as such. Therefore, there is no auditable procedure to gain insight in the organizations controls to manage changes (besides IT General Controls) in an controllable manner. As an example, the platform mentioned in the context is recently transformed from a local platform to an international platform without performing an impact assessment on the controls of the ISAE 3402 scope. Because of this, it is likely that control weaknesses will be noted in the coming audit. Result Based on this case, from the planning phase it is known that organizational changes with impact on the operating effectiveness of controls are possibly applied without assessing the impact on the control framework. If the approach on assessing the impact of organizational changes on the control framework is included in the COSO elements, the user of the report can have insight in what controllable manner the changes are applied (or will be applied in the case that no changes has occurred). Therefore, this addition is likely to contribute to the overall assessment of future operating effectiveness of controls.


3) An Internal Audit function is established and actively involved in managing the

achievement of control objectives related to the ISAE 3402 scope.

Applied in practice In this case, an Internal Audit function (IA) is established, however not actively involved in managing the achievement of all control objectives of the current ISAE 3402 scope. The part which is covered by the IA is managed without control deficiencies, the non-managed controls however are assigned to control owners but have proven to show more deficiencies which need to be followed-up to mitigate the risk of a qualified opinion.

Result Based on the arguments described above and our experience with entities with an Internal Audit department, an actively involved Internal Audit function contributes to reliable execution and achievement of control (objectives). When the information regarding the tasks and responsibilities of IA are included in the COSO elements, the user of the report can partly base its opinion on the future operating effectiveness of the relevant controls.

Execution of the audit


Applied in practice The majority of controls is considered automated for case A. As it is a recurring engagement, we found that the automated controls programmed in the kernel of the system were unchanged compared to the previous year. The few controls that were not automated showed deficiencies in design. However, those controls were not activated during the year but if it were so, they would have led to significant control weakness in operating effectiveness.

Result Based on the above, it would seem that a higher amount of automated controls would contribute to the assessment of future operating effectiveness. However, one should be aware of the possible failure of IT General Controls regarding change management affecting these controls.


Applied in practice In the current scope of case A, there are no Meta controls (controls covering controls) defined. If we would take the concept of Meta controls and map it to the current case, the following controls would be defined:

• Click registration monitoring for reliability • Interface monitoring for reliability


With these controls, the whole set of automated controls is monitored for operating effectiveness leaving a total set of four IT dependent controls. If we would add one more controls to cover the above, it would be:

• Monitoring the monthly business and finance review of invoices

With these three controls (besides the IT General Controls), a coverage of the whole scope is ensured. Result Based on the above reasoning, having Meta controls would contribute to a higher internal control system. Consequently, it would seem that including the Meta controls covering the control objectives in the existing control framework contributes to the assessment of future operating effectiveness. Also if the controls regarding reliability of the Service Level Reporting (e.g. in this case the success rate of invoice reviews) are included, the user organization can rely on the Service Level reports for operating effectiveness of the key controls after issue date of the audit report.

Reporting

1) The service auditor shall inquire whether the service organization is aware of any events subsequent to the period covered by the service organization’s description of its system up to the date of the service auditor’s assurance report that could have a significant effect on the service auditor’s assurance report.

Applied in practice In the context of case A, we were unaware of the timeliness in which changes would take place. We were informed that the platform would be used to support multiple countries; however, we did not know their approach and how it would affect the ISAE 3402 scope. As for the time between issuing the report and the period covered, there was no need to mention these developments as subsequent events as they did not yet occur. It would however be of interest to users of the report to extend the subsequent events to event that will happen shortly after the report is issued. With some additional work, the impact of the changes that are about to take place, could have been assessed and appropriate actions to ensure operating effectiveness could have been mentioned in the report giving the users of the report more insight.

Result Extending the assessment of subsequent event beyond the timeframe of report date to period date would contribute to the assessment of future operating effectiveness of controls.


2) A statement of the limitations of controls and, in the case of a type 2 report, of the

risk of projecting to future periods any evaluation of the operating effectiveness of controls.

Applied in practice A statement of the limitations of controls and de risk of projecting to future periods can be used to state the change of the platform which is planned to be implemented the coming year (as stated in point 1). With this statement, upcoming changes and their impact can already be assessed. Actions to uphold the control objectives can be determined and enclosed in the report.

Result Besides a statement of limitations and the risk of projecting to future periods, based on the above it can be of interest to extend the statement with directions to which management wants to move.

3) A statement of direction by management is required and should be part of the report to be issued.

Applied in practice Additional to point 2, a clear statement of direction would give the user of the report the insight (and possibly assurance) needed to ensure that developments the coming year will be addressed and no controls and control objectives will fail because of unmanaged events. As an example regarding case A:

Statement of direction With regard to the developments that concern the scope set out in this report, the following developments need to be addressed: 1) The platform used for click registration will be changed to support a multi country structure. Therefore, the complete set-up as is will be converted per June 201x. With this conversion, the impact on IT General Controls is assessed. As a result, one data centre is added to the scope and two application systems will be in scope; one from January to June and one from June to December 201x. The controls covering these changes will be monitored and if needed implemented by or with the support of our Internal Audit Service.

Result Based on the above, a statement of direction by management would contribute to the assessment of future operating effectiveness of controls.


Applied in practice Based on the audit schedule for case A, two testing periods (both of a year) are defined. As a result, one letter can be issued to cover the time between reports, i.e. between the two years. Using the bridge letter would inform the user about the progress of items in the proposed statement of direction or other relevant information regarding control objectives.


Result The bridge letter is still a good solution for the need of assurance between reports and can be extended with an evaluation of the statement of direction. It is in itself not part of the ISAE 3402 report and therefore not considered to contribute to the assessment of future operating effectiveness of controls.

5.2.3 Summary As a result, the following additions seem to contribute to the assessment of future operating effectiveness of controls:

Table 2: Case A analysis of additions

Additions Likely to contribute

Unlikely to contribute

Assurance framework

ISAE 3000 framework covering the 3402 format X

SOC 2/3 enables the auditor to extend the scope of SOC1 X


Gain an update of knowledge of, and review the effects of applicable industry and regulatory standards with a focus on significant changes affecting the current period or future periods.

X

An approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system

X

An Internal Audit function is established and actively involved in managing the achievement of control objectives related to the ISAE 3402 scope

X


The amount of automated controls as a percentage of the total control measures per control given reliable IT General Controls

X

Include Meta controls, such as monitoring controls over key controls and controls over reliability of Service Level Reporting in the controls framework.

X




Reporting

The service auditor shall inquire whether the service organization is aware of any events subsequent to the period covered by the service organization’s description of its system up to the date of the service auditor’s assurance report that could have a significant effect on the service auditor’s assurance report

X

A statement of the limitations of controls and, in the case of a type 2 report, of the risk of projecting to future periods any evaluation of the operating effectiveness of controls

X

A statement of direction by management is required and should be part of the report to be issued X

The bridge letter needs to reflect on the proposed additions that are likely to contribute to the assessment of future operating effectiveness of controls

X

5.3 Case study B Per case study, the context, the results and conclusions will be described in the paragraphs below.

5.3.1 Context Company description

The company selected organizes conference meetings, in a B2B business model. A client defines its requirements for the setup of an event, on which the company organizes the event by not only selecting the venue and hosting the actual event but also facilitating in the registration of invitees / attendees, collecting entrance fees beforehand, coordinating the suppliers and arranging keynote speakers.

ISAE 3402 description

User organizations of the described company are required to know that the company organizes events in an accurate and timely manner. Furthermore, it is important for the user organization to gain insight on the controls regarding the financial aspects of event organizing and the way this is invoiced to the user organization.

The following objectives are part of the scope of the engagement:

• Accuracy and timeliness of registering and managing events • Accuracy and timeliness of the financial processes before, during and after the

event • Reliability of IT General Controls


5.3.2 Case study findings and analysis The results of the case study performed are noted below using the three stages of the audit engagement as we know them; client and engagement acceptance, planning and understanding, execution of the audit and reporting.

Choose the assurance framework to address the assurance need (engagement acceptance)

1) ISAE 3000 framework covering the 3402 format enables the auditor to extend the scope of the audit beyond the limitations of 3402.

Applied in practice Because ISAE 3402 is an extension of ISAE 3000 based on ISA402 regarding service organization, there is no problem in using the ISAE 3000 framework for the current scope as long as the ISAE 3402 format is used such that the report is usable as audit evidence for the annual audit of financial statements.

Result Based on the above, the current scope of the engagement can be executed using the ISAE 3000 standard.

2) SOC 2/3 enables the auditor to extend the scope of SOC1 beyond its limitations

(similar to the ISAE 3402 framework)

Applied in practice If we look at the scope of the ISAE 3402 audit on company B, the controls in scope are related to process integrity and continuity. The controls in scope describe how the events / projects are managed and what controls exist to ensure that the process operates on an accurate and timely manner.

The IT general controls, in scope of the audit as well, cover the aspects of continuity. Result Based on the above, the current scope of the engagement can be executed using the SOC2 standard.


1) Gain an update of knowledge of, and review the effects of applicable industry and regulatory standards with a focus on significant changes affecting the current period or future periods.


Applied in practice In the current situation, no control objectives are related to industry or regulatory standards such that a change the average laws and regulations would significantly affect the scope of the audit. There are no specific (industry) standards related to hosting of conference meetings.

Result Based on the current scope, an updated knowledge of industry and regulatory standards do not give more insight in the future operating effectiveness of controls

2) An approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system. Note the changes within the organization and its impact on the control framework.

Applied in practice There is no formal approach of managing the organization’s ISMS. Therefore, organizational changes and its possible impact to the ISAE 3402 scope are not noted as such. Therefore, there is no auditable procedure to gain insight in the organizations controls to manage changes (besides IT General Controls) in a controllable manner. For example, during this year a significant update on the financial administration application will be implemented. This will have an impact on the operating effectiveness of the automated controls in the application. The company has not assessed this impact yet. Assessment will be performed in the design phase of the implementation process.

Result With the information from the impact assessment of the update on the financial administration application on the control framework, an estimate can be made on the impact on the operating effectiveness of the controls in in the control framework. This impact assessment is included in the report, combined with the described COSO elements, gives the user organization information on how the significant update in the system can affect the service organization’s services and maybe even the user organization itself. Based on this information the user organization can decide on what mitigating activities it might require to perform to limit the effects of the update on their way of working.

Therefore, this addition is likely to contribute to the overall assessment of future operating effectiveness of controls.

3) An Internal Audit function is established and actively involved in managing the

achievement of control objectives related to the ISAE 3402 scope.

Applied in practice There is no Internal Audit function established in this company.


Result With the information of the lack on an Internal Audit function in the service organization, the user organization can draw their own conclusions upon. This might vary from extending / reinforcing their own complementary user entity controls to implementing monitoring controls on the output of the service organization. Without this information, the user entity cannot estimate the impact of risks adequately and act upon it accordingly.



Applied in practice A significant amount of controls at case B can be categorized as automated controls. The controls will continue to operate effectively over time when there are no changes applied on the systems. When there are changes implemented, it is important to assess how the system changes are implemented and what its effect has been on operational effectiveness of the control framework. If the IT general control change management operates adequately, the company can conclude that the impact of the change has been correctly estimated, and the controls in the control framework continue to perform effectively.

Result With a higher amount of automated controls, the predictability of the operating effectiveness of controls becomes higher. An adequately implemented change management process secures the operating effectiveness of the automated controls. In conclusion, the information on the amount of automated controls gives the user organization indications on the predictability of the operating effectiveness of the (automated) controls.


Applied in practice The company in case B has Meta controls implemented on the process of reconciliation of project costs and revenue. These controls include monitoring of the controls implemented in the financial processing of projects. so when one of these controls impends to fail this will be timely detected and appropriate action can be taken upon. Within this company many more Meta or monitoring controls can be implemented to be assurance of the accurate and timely operation of the controls and processes.

Result With this information of Meta controls as included in the control framework in scope, the user organization gains insight the how the service organization manages to ensure operating effectiveness in its controls. If no Meta controls are implemented, the chance is significantly higher that failure of controls is not detected timely or not at all. Having Meta controls contribute to a higher internal control system. Consequently, it would seem that the amount and operating


effectiveness of Meta controls covering the (key) control objectives contributes to the assessment of future operating effectiveness. Also if the controls regarding reliability of the Service Level Reporting (e.g. on the success rate of reviews on financial processing of projects) are included, the user organization can rely on the Service Level reports for operating effectiveness of the key controls after issue date of the audit report.

Reporting

1) The service auditor shall inquire whether the service organization is aware of any events subsequent to the period covered by the service organization’s description of its system up to the date of the service auditor’s assurance report that could have a significant effect on the service auditor’s assurance report.

Applied in practice We were informed that several controls are about to change, right after the period of review. This regards changes to improve or strengthen the control so its output meets the control objective better. At this moment this information is included as management response to the findings, but no further details on the impact on the controls is not given in the report.

Result When more information is given on subsequent events in this case, the impact of the findings can be better estimated by the user organization. With this impact assessment, the user organization can either decide to accept the risk from the finding, because the finding is resolved on the short term, or temporarily implement complementary user controls. Without this information, the finding seems worse than it might be.

2) A statement of the limitations of controls and, in the case of a type 2 report, of the risk of projecting to future periods any evaluation of the operating effectiveness of controls.

Applied in practice Apart from the implementation of the update on the financial administration system, we have indicated no other developments that may impact the future operating effectiveness of the controls.

Result With the statement on the process of implementation the update on the financial administration system, upcoming changes and their impact can already be assessed. Actions to uphold the control objectives can be determined and enclosed in the report. Also with the notion of no further developments, the user acquires information on the chances on failure in the future operating effectiveness.


3) A statement of direction by management is required and should be part of the

report to be issued.

Applied in practice At this moment the developments on improving controls, and solving this year’s findings are included as management response in the table of the controls framework in the appendix of the report. The update of the financial administration application is shortly mentioned in the system description, but more attention can be given here on the process of implementation and its impact on the control objectives and the related control measures.

Result As the statement of direction lacks at this moment, no information is given on the developments or events within the organization that might impact the scope of the ISAE 3402 report, the operating effectiveness of the controls in scope and / or the complementary user controls.


Applied in practice Based on the audit schedule for case B, two testing periods (both of a half year) are defined. As a result, one letter can be issued to cover the time between reports, i.e. between the two years. Using the bridge letter would inform the user about the progress of items in the proposed statement of direction or other relevant information regarding control objectives.

Result The bridge letter is good solution for the need of assurance between reports and can be extended with an evaluation of the statement of direction. It gives the user indications on the operating effectiveness in the time after the last release of the report, and therefore future operating effectiveness if compared to the results from the most recent released report. But looking strictly at the purpose of the bridge letter, it does not give information on future operating effectiveness.


5.3.3 Summary As a result, the following additions seem to contribute to the assessment of future operating effectiveness of controls:

Table 3: Case B analysis of additions



Assurance framework

ISAE 3000 framework covering the 3402 format X

SOC 2/3 enables the auditor to extend the scope of SOC1 X


Gain an update of knowledge of, and review the effects of applicable industry and regulatory standards with a focus on significant changes affecting the current period or future periods.

X

An approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system

X

An Internal Audit function is established and actively involved in managing the achievement of control objectives related to the ISAE 3402 scope

X


The amount of automated controls as a percentage of the total control measures per control given reliable IT General Controls

X

Include Meta controls, such as monitoring controls over key controls and controls over reliability of Service Level Reporting in the controls framework.

X

Reporting

The service auditor shall inquire whether the service organization is aware of any events subsequent to the period covered by the service organization’s description of its system up to the date of the service auditor’s assurance report that could have a significant effect on the service auditor’s assurance report

X




A statement of the limitations of controls and, in the case of a type 2 report, of the risk of projecting to future periods any evaluation of the operating effectiveness of controls

X

A statement of direction by management is required and should be part of the report to be issued X

The bridge letter needs to reflect on the proposed additions that are likely to contribute to the assessment of future operating effectiveness of controls

X

5.4 Case research outcomes and analysis Based on the two case studies performed, all additions were found to be likely to contribute to the assessment of operating effectiveness of controls except for:

• Gain an update of knowledge of, and review the effects of applicable industry and regulatory standards with a focus on significant changes affecting the current period or future periods.

• The bridge letter needs to reflect on the proposed additions that are likely to contribute to the assessment of future operating effectiveness of controls

Although these additions were not found to be as relevant as the others, we would like to point out that the additions might be relevant to cases which are more subject to industry and regulatory standards. For instance, a case with controls in the pharmaceutical industry might be significantly impacted by such changes and would definitely affect the future operating effectiveness if not acted appropriate by the service organization.

The bridge letter is not part of the auditors section of the ISAE 3402 report and therefore not likely to contribute for the aim of our thesis. When a bridge letter is agreed to be issued, we recommend elaborating on the suggested additions such as the directive report by management.

SOC 2/3 practice

As noted in chapter three SOC 2 / 3 requires that the audit approach should address, besides the objectives on the financial aspects, the obligated objectives of the Trust Service Principles as well. As in this thesis research we are looking for additions on the regular ISAE 3402 audit approach, additional control objectives are not desirable.

Therefore, from this case study, aspects from SOC 2 / 3 can be used, but should be applied within the ISA3000 framework. The audit (and resulting report) can be executed within the ISA3000, in which the standard gives flexibility to implement aspects from for example SOC 2 / 3.


The desired level of assurance

This research provides suggestions on expanding the ISEA3402 audit in order of providing information on the future operating effectiveness of controls. Because it regards the future, it is very difficult to give information on the future with the desired rate of certainty.

We are aware that an accountant, auditor for this matter, always prefers tangible audit evidence. Tangible evidence is 1) directly related to control objective 2) precise and delivers high level of certainty and 3) can be independently acquired by the auditor. Stocktaking is an example of tangible audit evidence, audit evidence related to entity level controls is less tangible. This preference for tangible audit evidence is verified in research of (Buuren, van, Koch, Nieuw Amerongen, van, & Wright, 2011) and included in the ISA 500.a31.

Tangible audit evidence is hard to define when providing information on future operating effectiveness, but we have not researched how to overcome this limitation. Therefore, it is not possible that our conceptual additions give assurance.

The auditor can only facilitate in providing information on which the user of the report can form its opinion on the future operating effectiveness.


6 Research question and conclusion Based on the literature study the framework is explored to identify possibilities for additions on the framework. Combined with the semi-structured interviews with the stakeholders, from four perspectives, we have defined conceptual additions to the regular ISAE 3402 audit approach. By applying the conceptual additions in two case studies, we have verified the added value. In this chapter, the conceptual additions are summarized in a simplified manner.

6.1 Research question The main question, to which this thesis is dedicated, is:

What additions should be made in the current ISAE 3402 audit approach to give the user of the ISAE 3402 report more assurance regarding the future

operating effectiveness of the service provider?

To answer the main question, the question is divided into three sub questions.


The International Standard on Assurance Engagements (ISAE) developed by the International Auditing and Assurance Standards Board (IAASB) is a standard now used for an assurance opinion about the work performed by a Service Organization over a historic period in time, the successor of SAS70. The ISAE 3402 framework to found to be a framework used to provide comfort to user entities and their auditors about the internal control components related to financial reporting of the service organization relating to the ISAE 3000 framework, which covers internal control components other than audits or reviews of historical financial information covering a specified period in which controls:

• Designed and implemented • Suitably designed throughout the specified period or as at a specified date • Operated effectively throughout the specified period


The ISAE 3402 framework is not designed to cover all possible scope, types of assurance, objects of research and periods. For different (commercial) reasons, companies would like to fit as much as possible in the report, which is in conflict with the original goal of the framework. In practice, the ISAE 3402 framework is therefore sometimes used to report on more than the framework was intended to provide. Although companies would like to report on the future operating effectiveness of controls, the framework does not support this kind of statements. The few elements in the framework that might be of use to contribute to sub question three are:

• Obtaining an Understanding of the Service Organization’s System • Obtain evidence • Subsequent Events • Preparing the Service Auditor’s Assurance Report • Bridge letter


3) Which additions to the ISAE 3402 audit approach can be defined in order of assessing a service provider regarding the future operating effectiveness of controls?

The third and last sub question is covered in the next paragraph, 6.2.

6.2 Additions in the regular ISAE 3402 audit approach From the thesis research, it can be concluded that the desired level of assurance on future periods is not possible. The auditor can only provide insight and information related to future operating effectiveness of controls, with this information the user of the ISAE 3402 report is required to form its own opinion on the future operating effectiveness of controls.

As it is not possible to change the current ISAE 3402 framework to support reporting about future periods, the first step would be to select a framework to cover the assurance needs beyond the traditional financial statement. Based on the thesis research we have found that the ISA3000 framework with the 3402 content (structure, requirements, scope) is best fit for a regular ISAE 3402 audit, which additionally provides information on the future operating effectiveness of controls.

Based on the literature research, the exploratory interviews and empirical case research, the following additions are proposed to enhance the ISAE 3402 reporting to support an assessment of future operating effectiveness of controls.

To embed the additions in the audit approach, we have related the addition to the different stages of the audit:

• Planning and understanding • Execution • Reporting

Planning and understanding

To be able to make a first assessment of the control environment, the following additions needs to be addressed:

• Gain an update of knowledge of, and review the effects of applicable industry and regulatory standards with a focus on significant changes affecting the current period or future periods

• An Internal Audit function is established and actively involved in managing the achievement of control objectives related to the ISAE 3402 scope. This function can be described in the COSO elements

• An approach is implemented for establishing, implementing, operating, monitoring, reviewing, maintaining and improving an organization's internal control system; Note the changes within the organization and its impact on the control framework

The last item needs to be part of the scope of the execution of the audit since these are controls regarding organizational change management. This can be done by either including it in the controls framework or describing the approach in the COSO elements.



During the execution of the audit, controls are tested and an overview can be created covering:

• The amount of automated controls as a percentage of the total control measures per control given reliable IT General Controls

• Include Meta controls, such as monitoring controls over key controls and controls over reliability of Service Level Reporting in the controls framework

The ratios above contribute to an understanding of the maturity of the internal control system as well as the reliability on automation or monitoring controls. Besides the ratios, the auditor can express the importance to the service organization to include automated controls and Meta controls in the controls matrix.

Reporting The report as we know it should be extended with a section written by the management of the service organization in which they elaborate on:

• Subsequent events between the report date and period in scope • Foreseen events within the next year of the period in scope • A statement of the limitations of controls and a projection to future periods • A statement of direction with associated risks and planned mitigating actions

The purpose and structure of this section is similar to ISA 520, with the management description and the auditors’ assessment. However, the purpose of this section is focused on the future operating effectiveness.

When a bridge letter is issued, we suggest elaborating on the section above to inform the users of the report about the developments regarding the foreseen events.

6.3 Limitations of this research In our research, we have not dedicated special attention on the cost aspects of the concept additions on the ISAE 3402 audit. We are fully aware that this is an important aspect, as in the results of the research it remains unclear who is willing to pay the additional work to acquire more information on the future operational effectiveness. In this research, we wanted to explore the limitations of the ISAE 3402 framework on a technical way, although we have implicitly included the cost aspect (willing to pay for the conceptual additions) in the interviews held by measuring the neediness of the stakeholders on the additions to provide more information on future operating effectiveness.

Furthermore, we are aware that the financial continuity of a service provider has significant impact on the future operating effectiveness of the controls where the user organization relies on. We believe that the financial continuity should not be part of the ISAE 3402, as it regards a controls audit. The user organization should base its opinion on the financial continuity of its service provider on the results of a financial audit, i.e. the financial statements audit.


6.4 Further research In our research, we have looked into the ISAE 3402 framework and have dedicated special focus on the lack of the framework on providing information future operating effectiveness. No research is done on other limitations of the ISAE 3402 framework.

Based on this research we have identified and discussed conceptual additions with the stakeholders, proving its added value by applying it in the case studies.

Although we have identified the conceptual addition and proved its added value, we have not researched the additions in detail. The conceptual additions are described as a concept and broad guidelines on implementing these additions are given, but we have not researched how these additions have to be included in the existing audit approach, with an adjusted audit report as an result. We have not described, for example, where the information on the amount of automated or Meta controls should be mentioned. In addition, no detailed information can be derived from the research on what information (aspects) should be provided on the organization change management or in the directive report. This is only mentioned in a briefly manner. Further research can be done on the detailed implementation of these conceptual additions in the audit approach and audit report.


7 Bibliography AICPA. (2001). Attest Engagements. New York, New York, United States of America.

AICPA. (2010). Service Organization Controls, managing risks by obtaining a Service Auditor's Report. New York, New York, United States of America.

AICPA. (2011). Reporting on Controls at a Service Organization. New York, New York, United States of America.

AICPA. (2014). TRUST SERVICES PRINCIPLES AND CRITERIA. New York, New York, United States of America.

British Standard Institute. (2005). BS ISO/IEC 27001:2005. BSI Catalogue “International Standards Correspondence Index”.

Buitendijk, D., & van Gerner, M. (2011). Third Party Audits. Amsterdam: Vrije Universiteit.

Buuren, van, J., Koch, C., Nieuw Amerongen, van, C., & Wright, A. (2011). The use of Business Risk Audit perspectives by non-Big 4 audit firms. Nyenrode Business Universiteit (July).

Ernst & Young. (2009). Planning for the new service organization reporting standard. IT Risk and Assurance Insights Issue 4.

Ewals, R. (2009). Zekerheid bij uitbesteding (SAS70). Handboek EDP auditing Volume 37.

Heiser, J., & Caldwell, F. (2010). SAS 70 Is Not Proof of Security, Continuity or Privacy Compliance. Gartner, 8.

Holcomb, T., & Hitt, M. (2007). Toward a model of strategic outsourcing. Journal of Operations Management, 25(2), 464-481.

IAASB. (2008). ISAE (3000) Assurance Engagements Other Than Audits or Reviews of Historical Financial Information. IAASB.

IAASB. (2009). International Standard on Assurance Engagements (ISAE) 3402 Assurance reports on controls at a service organization. IAASB.

IFAC. (2009). INTERNATIONAL STANDARD ON AUDITING 520 ANALYTICAL PROCEDURES. IAASB.

Leenders RA, E. N., & Nagy RO, L. Z. (2013, September). De verwachtingskloof van ISAE 3402. Audit magazine(3), 16-19.

Leung, J. (2011). Auditor reporting on controls at service organizations .

Mertens, P., Meliefste MSc, S., & Blij CFA, D. (2013 йил 1-2). Continuïteit van Nederlandse beursfondsen: een continu punt van aandacht. Amsterdam: NBA.

NBA. (2013, February 26). Continuïteit van Nederlandse beursfondsen: een continu punt van aandacht. Retrieved March 2, 2013, from Accountant.nl: http://www.accountant.nl/Accountant/Nieuws/Jaarverslag+beursfonds+onduidelijk+over+continuite.aspx


PCI Security Standards Council. (2013). Payment Card Industry (PCI) Data Security Standard v3.0. Payment Card Industry (PCI) Data Security Standard v3.0.

PricewaterhouseCoopers. (2012). PwC ISAE3402 library.

PricewaterhouseCoopers. (2014). PwC Auditguide ISA 520. PwC.

Roozendaal. (2011). Spotlight Volume 18 - edition 1. Spotlight (openly published company literature).


Appendix Additional information is included to support the main part of this thesis. The following appendices are included:

A. Exploratory interview: Domain Expert

B. Exploratory interview: Service Provider

C. Exploratory interview: Auditor, Client of Service Provider

D. Exploratory interview: External Auditor

A Exploratory interview : Domain Expert Role and background

• Ex-KPMG partner, responsible for the SAS70/ISAE 3402 product development within KPMG globally and working for the bigger clients.

• Nowadays retired, but still involved in advisory engagements regarding ISAE 3402 and involved in the developments within NOREA and NBA regarding SOC2.

Benefits and limitations of ISAE 3402 • In the current practice many organizations request an ISAE 3402 report, mainly

for one of these three reasons:

1) It is required by clients or governments. 2) The service provider itself wants to measure its own performance and

control, in order of being in control. Results of the 3402 audit are of essence for internal decision-making.

3) The service provider wants to proof to their (potential) clients that their services are in control, and on what matters. This proof (i.e. the ISAE 3402 audit) can be requested by one of the clients, but the service provider can also initiate the ISAE 3402 audit as a report which they can use an unique selling point in comparison with its competitors.

• One of the biggest benefits the standardized audit approach that the ISAE 3402 framework entitles. The report framework provides a clear structure, which also shows in the resulting reports. The only thing being variable in an ISAE 3402 framework are the control objectives and controls in scope, the other aspects are included in the provided structure.

• One of the limitations regards the misunderstanding on the purposed use of the ISAE 3402. TheISAE 3402 has been designed as an auditor-to-auditor report, and its framework is defined based on this principle. So by nature the framework does not require that the auditor provides information on aspects such as availability, continuity, confidentiality etc. An auditor is merely interested in the controls, which have an impact on the reliability of the financial data, as stated in the ISAE 3402 framework. In practice, this understanding does not directly affect the work in the ISAE 3402 audits; it only creates an expectation gap between the auditor, user organization and service provider when the scope is not clearly defined.


• Another limitation is the lack of focus on the dynamics within organizations. Processes and controls can change over time, but in the current ISAE 3402 framework little attention is required on this matter. There is one paragraph included in the report in which the auditor is required to mention this matter, but no further guidelines on the depth of the test work and/or description of the changed processes/controls and its impact on the audit are given in the ISAE 3402 framework. These aspects are directly related with the thesis’ object of research, the future operating effectiveness.

• Limitation regards the use of the carve out and inclusive options of subservice organizations functionalities within the ISAE 3402 framework due the vagueness and lack of in-depth knowledge of the standard. A third option is possible is the monitoring approach, which is not mentioned in the standard. One can choose to cover the monitoring controls needed to verify reliability of the sub service organization(s).

Suggestions for improvements mitigating its limitations • The Domain Expert suggested, related to the limitation of the lack of focus on

the dynamics within organizations in the ISAE 3402 framework, that the ISAE 3402 report should also include the processes and controls regarding the change of processes/controls. With this insight the user of the report is able to conclude that when the processes/controls has changed since the ISAE 3402 audit, that it is changed in a controlled manner with very limited impact on the services of the service provider.

• Additionally, Meta process/controls should be included, so the user of the ISAE 3402 report gets insight in the controller manner of controls (controls over controls). These controls are considered stronger because a second line of defence is implemented as part of a companies internal control system.

• When a sub service provider brings out Service Level Reports (SLR), the user organization gets more insight over the year on the performance of the service provider. When dedicating special focus on the processes on which the SLR’s are defined and determine the reliability of the SLR, more information on the future operating effectiveness can be acquired.

• NOREA in cooperation with the public sector is working on a form of certificate on the most common online financial administration software, such as Exact online, AFAS online etcetera. This certificate will be based on the ISAE 3402 and includes possibly aspects of continuity and availability (www.zeker-online.nl).

• ISO27001 only requires description of the management controls in design and operations but this framework does also refer to some continuity aspects that might relate to future operating effectiveness. He suggested us to perform some research to this framework to get more ideas on proving future operating effectiveness of controls.

• A transmittal letter clearly states that the audit is performed on the controls data in the past. It does not provide any assurance on the operating effectiveness of the controls at this moment.


What are the boundaries of the audit standard.

• The current ISAE 3402 framework does not allow any adjustments (or removals). For our ideas of additions to the ISAE 3402 report on providing more information on future operating effectiveness, he suggested to perform the audit under ISAE 3000, based entirely on the ISAE 3402 framework. Within the ISAE 3000 we can add our additions on future operating effectiveness. This is allowed within the audit standard. Important is that with our information on future operating effectiveness we clearly state in the report that is it no assurance on future operating effectiveness, as nothing can/may be said about the future. We should clearly state that the information regards leads or clues on which the user of the report can based his/her view on.

Developments in practice regarding ISAE 3402 • The Domain Expert informs us that in his opinion there might be a shift from

operational effectiveness (Type II) to design and operation (Type I) reporting. Because of continuous assurance, one would like to know how a company performs at any given moment. Therefore, the operational effectiveness information can be system generated based on logging, supported by a Type I report, which is required to be composed by the external auditor.


B Exploratory interview : Service Provider Role and background

• Controller at a fast growing Payment Service Provider organization. • He was involved in the process of making the organization ISAE 3402 ready for

the first year’s audit.

Benefits and limitations of ISAE 3402 • In practice, few companies request the ISAE 3402 report for their internal

control function. The main reason for distribution of the report regards Request for Proposal (RfP) requirements for the bigger clients of the service provider.

• The control matrix is large, complex and controls are spread over organizational structure making it more difficult to assign ownership to controls. The report should be structured such that the main process is clearly described and other less relevant processes are marked as sub processes.

• Additionally, it is not easy to stay in control about the achievement of all control objectives during the year. You would need someone, supported by an automated system, to verify during the year that controls are performed as stated in the control framework.

• The positive part about the ISAE 3402 framework is that is gives you an objective opinion about your company, which you can show to your customers or other parties who are interested.

• What is missing in the current report is a section in which the management elaborates on developments regarding the organization, processes, people, technology and related controls. The management assertion, which is mandatory, only addresses control objectives that are not achieved.

Suggestions for improvements mitigating its limitations • The Service Provider suggested that a Directive Report (which is part of the

annual audit of financial statements) should be included in a ISAE 3402 report stating at least the following with the coverage of one year after the date the report will be issued:

1) Organization 2) Processes 3) People 4) Technology 5) Related controls

• A web based portal should be implemented which gives you a direct insight in

the status of control objectives. This overview can be used in quarterly reviews which can be reported to the clients.


Service provider specific questions

• Without an IAD and supporting system, it is not possible to know the exact status of controls in the current year. Therefore, it is impossible to make assumptions about the future. However, as a business one does know what happens in which department and processes and how it could affect the controls part of an ISAE 3402 report.

• Important conditions are for instance the amount of automated controls per process. The more automated controls, the more likely it is that the system will keep on function (relying on an effective change management process regarding infrastructure and applications).


C Exploratory interview : Client of Service provider Role and background

• Senior partner at Atos Consulting & Technology Services • As a Senior Manager at KPMG, he was assigned to attestation services, which

includes the previous framework SAS70. • Currently, he is consulted when ISAE 3402 reports are requested by user

organizations (clients of service provider) to address their needs. He thereby functions as an intermediate on behalf of the user organization.

Benefits and limitations of ISAE 3402 • From a Business perspective (COO), the ISAE 3402 framework is found to be:

1) Hard to understand / interpret 2) It is obligated from their accountant and/or local legislation 3) Expensive 4) Covers only the past 5) It is an auditor to auditor report

• In the opinion of the Client of the service provider, the COO is becoming more

important than the CFO. The COO also has a focus rather on the future than the past, in contrary with the CFO. Therefore, the ISAE 3402 framework fits better with the CFO’s need than the COO’s needs.

• Both the SAS70 and ISAE 3402 framework are being used for other purposes than originally intended and therefore, a situation is created in which the value of the report is found to be limited. The expectations (reporting on the organization’s business performance) do not meet with the report’s offerings (reporting on the organization’s controls, which affect their financial performance). Because of this reason, fewer companies are willing to pay for the report. SAS70 (and later ISAE 3402) remains an auditor-to-auditor report. The interviewee’s opinion is supported by an article from (Heiser & Caldwell, 2010) as he suggested himself.

Suggestions for improvements mitigating its limitations • As a start, the auditors should not be the ones who defines the need for

assurance, this should be done by the client of a Service Organization. This results into a suggested approach in which: 1) The auditor discusses the need for assurance with the customer(s) 2) The correct mean is selected which may vary from an ISAE 3402 report to a

simple memo, dependable on the nature of the audit object 3) The goal for usage of the work to be performed (and its result in a report)

should be clearly defined between auditor and auditee


• The report needs to be concise and per control, the following needs to be

stated:

1) Control measure 2) Detailed information on test work per measure 3) Per items a statement of approval or rejection from the auditor 4) Clear references to documents and/or data on which the conclusion is based

upon

Based on the detailed information of the test work the user of the report can combine this information with its self-gathered information and determine its own view on the future operating effectiveness of the specific control measure.

• Forward looking assurance (in business terms) is an upcoming movement, which is driven by for example James Turling from EY. As a company, one should have Key Assurance Indicators in place. These indicators can be covered by an ISAE 3402 report when the scope regards the financial statements.


D Exploratory interview : External auditor Role and background

• Partner at PwC in Amsterdam • He is involved in performing ISAE 3402 audits and can be seen as subject

matter expert within PwC the Netherlands

Benefits and limitations of ISAE 3402 • Limitation regarding the scope of controls that only controls with financially

relation are permitted within the ISAE 3402 scope has been discussed recently internally. From this discussion can be concluded that controls, which have a financial impact, are permitted in the ISAE 3402 scope; so also controls, which secure business continuity, are allowed within the ISAE 3402 scope.

• Both auditors as a small number of organizations are experiencing the lack of information on future operating effectiveness of controls. Because the final work of the audit takes place after the period of review, no assurance can be given on the period after the audit work performed.

• Important in the conceptual additions to the regular audit approach is that no assurance can be given on the future (operating effectiveness). The auditor can only facilitate in providing information on which the user of the report can form its opinion on the future operating effectiveness.

Suggestions for improvements mitigating its limitations • With a Type 2 report, assurance is given over the operating effectiveness over

the period of review. Together with (the operating effectiveness of) the controls framework and the COSO elements as identified at the Service Organization, the stakeholders can form an image of how the Service Organization has managed its control objectives and the related risks. With this information, the stakeholder of the report can form its own opinion on the future operating effectiveness of the controls in scope. In this case, it is important for the external auditor to verify the description of the COSO elements. These COSO elements can also be included in the controls framework.

• Currently, based on the interim work early warning reports and/or meetings are organized. These meetings only regard the external auditor and the auditee (Service Provider), so no information is obligated to be given to the stakeholders of the report.

• In the current report, information on the developments within the organization can be given in the paragraph regarding “Other information”. These developments can affect the future operating effectiveness of the controls in scope, and can therefore be described in this section of the report. However, this part of the report is no part of the auditor’s opinion so the reliability of the information has not been verified.


• Suggestion is made on including Meta controls in the controls framework. These

Meta controls can include monitoring controls on the key controls but also controls regarding the reliability of Service Level Reports, which the service organization sends to their user organizations. This way the user organization gets reliable insight in the performance of the Service Provider during the year, even after the audit report has been issued. Important is that the KPI’s internally within the service organization are aligned with the KPI’s mentioned towards the user organization, on which the latter relies on. With reliable periodic Service Level Reports, the user organization can better anticipate on possible failure of controls. As these controls are included in the controls framework, no special adjustment on the audit approach is required.

• When the service provider is IT driven and/or has a high amount of automated controls implemented, it is important to give the IT General Controls appropriate attention within the audit. Also within the COSO elements, as mentioned before, appropriate attention on the IT should be given as this can influence the user of the report in its opinion on the future operating effectiveness of controls. For example when a IT driven organization is a laggard (or an early adopter) on IT developments, this can give an user organization a view on the possibility of the automated controls not meeting its control objectives. Although this point of view does not distinctly point out what the chances are on failing control objectives, as there are more variables in the situation that should be taken into account.

ISAE 3402 - VUrORE · The ISAE 3402 framework is used to provide comfort to user entities and their...

Documents

Transcript of ISAE 3402 - VUrORE · The ISAE 3402 framework is used to provide comfort to user entities and their...