CONSULTATION Systems Integration Test (SIT) … · Smart Metering Programme Consultation on SIT...

Smart Metering Programme

Consultation on SIT Approach

of 75 v1.2

DCC Public

CONSULTATION

Systems Integration Test (SIT) Approach

Consultation opens: 22nd July 2014

Consultation closes: 19th

August 2014



of 75 v1.2

DCC Public

Contents

1. Executive summary ........................................................................................ 3

2. Background .................................................................................................... 5

3. Consultation Scope ......................................................................................... 6

4. Consultation questions ................................................................................... 7

5. How to respond .............................................................................................. 8



of 75 v1.2

DCC Public

1. Executive summary

The Data and Communications Company (DCC) is committed to the Smart Metering Implementation Programme (SMIP) which will help empower electricity and gas customers in Great Britain to enjoy the benefits of being in control of their energy usage. Systems Integration Testing tests the capability of the DCC and the component parts of the DCC Systems together with the Communications Hubs selected to interoperate with each other and with the RDP Systems. The Systems Integration Test (SIT) Approach sets out the DCCs approach to testing during this phase. It is therefore seeking views on the Systems Integration Test (SIT) Approach which is designed to ensure that Testing remains on schedule whilst maintaining the integrity of the whole DCC Solution.

The DCC solution is being delivered by six Service Providers:

1. Data Service Provider (DSP) - CGI IT UK Ltd

2. Communications Service Provider (North) – Arqiva Smart Metering Ltd

3. Communications Service Provider (Central and South) – Telefónica UK Ltd

4. Trusted Service Provider (Smart Metering Key Infrastructure) – British

Telecom Plc

5. DCC Enterprise Solutions (Billing, BI/MI) – Capita IT Services

6. Parse and Correlate Solution – Critical Software

Each Service Provider will test its complete solution in isolation during the Pre-Integration Test (PIT) Phase. On completion of PIT, the Data Service Provider acting in its role as the DCC’s systems integrator will integrate all six solutions, ready for the start of the Systems Integration Test (SIT) Phase in March 2015. SIT will validate that the DCC and component parts interoperate with each other and Registration Data Provider RDP Systems to form a solution that meets the agreed functional and non-functional requirements. It is therefore seeking RDP stakeholders’ views on whether the SIT Approach presents the appropriate provisions for ensuring timely and correct delivery of the SMIP’s SIT Phase.

This consultation paper briefly summarises the SIT Approach on which we are consulting, it asks questions to which we would welcome responses and informs stakeholders of how to respond and by when.

The document on which we are consulting concerns the Systems Integration Test Phase aspect of the Smart Metering Implementation Programme, as set out in Section T2 of the Smart Energy Code (SEC) Stage 3: Final Version.

Consultation on this SIT Approach closes on the 19th August.

Following this consultation, and taking respondents’ feedback into account, this Test Approach will undergo a further review cycle with the Test Advisory Group (TAG). The SEC Panel has established the TAG to support it in discharging its SEC duties. Once the TAG review cycle is complete, the DCC will seek final SEC Panel approval against the SIT Approach. When submitting the SIT Approach Document to the Panel, the DCC shall also submit copies of the consultation responses received from the Registration Data Providers. In addition, the DCC shall publish such consultation



of 75 v1.2

DCC Public

responses (to the extent not marked confidential) on the DCC Website. With SEC Panel approval in place the DCC will publish the SIT Approach on the DCC Website at least three months prior to SIT Phase commencing. The publication date will be 1st December 2014 at the latest.



of 75 v1.2

DCC Public

2. Background

The contents of this Test Approach have been derived from a series of SIT Planning workshops organised by the DSP, involving Test Managers and End to End Architects from the DSP, the CSPs, the DCC Licensee, Xoserve, and St Clements representing the Electricity Registration Data Providers (RDPs). Six separate workshops were held with them to agree the content of the document. The workshops were:

1. Workshop 1

a. Objectives, Scope & Approach

b. Entry & Exit Criteria

c. Roles & Responsibilities

d. Processes

2. Workshop 2

a. Integration sequence

b. Test scenarios

c. Test data

d. Test stubs & simulators

3. Workshop 3

e. Non-Functional

4. Workshop 4

a. Environments

b. SM equipment

c. Test tools

d. Resilience testing

5. Workshop 5

a. Electricity Registration Data specific with St Clements

6. Workshop 6

a. Gas Registration Data specific with Xoserve

Following these workshops the SIT Approach was drafted and has undergone three review cycles with the DCC, Service Providers, Xoserve and St Clements.



of 75 v1.2

DCC Public

3. Consultation Scope

For RDP consideration

The SIT Approach is open for comment by RDPs. However there are sections of the SIT Approach that are not relevant to RDPs’ testing during SIT because they form elements of the solution that are not in scope for RDP testing; they therefore do not form part of this consultation.

Sections open for consultation:

i. Section 1 – Introduction

ii. Section 2 – Scope

iii. Section 3 – Objectives

iv. Section 5 (omitting section 5.7) – Test Phase Description

v. Section 6 – Test Procedure

vi. Section 7 – Test Result Management & Reporting

vii. Section 8 – Test Incident Management

viii. Section 9 (omitting section 9.3) – Test Resources

ix. Section 10 – Roles & Responsibilities

x. Section 11 (omitting section 11.2) – Environments and Labs

xi. Section 12 – Test Scenarios

xii. Section 13 – Test Incident Severities

Not for RDP consideration

Sections not open for this consultation:

xiii. Section 4 - Deliverables

xiv. Section 5.7 – Device Strategy

xv. Section 9.3 – Test Stubs and Prototypes

xvi. Section 11.2 – Test Labs

xvii. Section 14 – Device Selection Methodology

xviii. Section 15 – Auditing of SIT Exit Criteria

xix. Section 16 – Compliance with SEC 3



of 75 v1.2

DCC Public

4. Consultation questions

Q1. Based on the consultation scope as described in section 4, do you agree with our proposed content within the SIT Approach? Please provide a rationale for your views.

Q2. If applicable, as an Electricity RDP do you agree with the approach to connection proposed in Section 5.5.2 (i.e. a direct link into the DSP per RDP) based on your current plans to connect to the DCC? Please provide a rationale for your views.

Q3. If applicable, if your answer to Q2 was no, do you still agree with the Test Approach defined in Section 5.5.2? Please provide a rationale for your views.



of 75 v1.2

DCC Public

5. How to respond

Please provide responses by 19th August 2014 to the DCC at [email protected]. If you have any questions, please contact [email protected].

If you have questions about our approach to consultations, please contact our Regulation Manager at [email protected].

When submitting the SIT Approach Document to the SEC Panel, the DCC shall also submit copies of the consultation responses received from the Registration Data Providers. In addition, the DCC shall publish such consultation responses (to the extent not marked confidential) on the DCC Website.

mailto:[email protected]




Systems Integration Test Approach

of 75 v1.2

DCC Public



v1.2



of 75 v1.2

DCC Public

Document History

Version Date Comment Author

0.1 04/04/14 Initial draft for internal DSP review DCC

0.2 25/04/14 Updated with DSP review comments DCC

0.3 23/05/14 Updated with Test Forum comments, including Xoserve and St Clements

DCC

0.4 23/06/14 Updated with comments from Test Forum and DCC Licensee. Aligned with SEC3 Legal drafting

DCC

0.5 04/07/14 Updated with DSP review comments DCC

1.0 04/07/14 Up-issued without change from v0.5 DCC

1.1 14/07/14 Added text for Section 15 DCC

1.2 17/07/14 Updated with Test Forum comments on v1.0, including Xoserve and St Clement

DCC

References

Ref Title Source Date Version

[1] Joint Test Strategy DCC Feb 2013 v2.3

[2] Smart Energy Code Stage 3 Legal Drafting

DECC Jun 2014

[3] Glossary of Testing terms ISTQB Oct 2012 v2.2

[4] Test Approach for the Pre-Integration Test Phase

Arqiva Apr 2014 001

[5] PIT Phase Test Approach Telefonica Jan 2014 v1.0

[6] PIT Phase Test Approach CGI Dec 2013 v1.1

[7] Schedule 6.2 (DSP version) CGI Sep 2013 v.1

[8] Schedule 6.2 (CSP N version) Arqiva Sep 2013 v.1

[9] Schedule 6.2 (CSP C/S version) Telefonica Sep 2013 v.1

[10] Integrated Solution Delivery Plan DCC June 2014 v3.1



of 75 v1.2

DCC Public

Contents

1 Introduction .................................................................................................... 15

1.1 General ..................................................................................................... 15

1.2 Change Forecast ....................................................................................... 16

1.3 Reviews, Approvals and Appeals .............................................................. 17

1.4 Terminology .............................................................................................. 18

2 Scope .............................................................................................................. 20

2.1 Overview ................................................................................................... 20

2.2 Out of Scope ............................................................................................. 20

3 Objectives ....................................................................................................... 22

3.1 SIT Objective ............................................................................................ 22

3.2 Test Approach Objectives ......................................................................... 22

4 Deliverables .................................................................................................... 23

4.1 General ..................................................................................................... 23

4.2 By Test Phase ........................................................................................... 23

4.3 By Test Stage ........................................................................................... 23

4.4 Maintenance ............................................................................................. 24

5 Test Phase Description .................................................................................. 25

5.1 Focus areas for Systems Integration Testing ............................................ 25

5.2 Key planning assumption .......................................................................... 25

5.3 De-risking SIT ........................................................................................... 26

5.3.1 Introduction ........................................................................................ 26

5.3.2 DSP Review of PIT ............................................................................ 27

5.3.3 Early SIT ............................................................................................ 27

5.3.4 Technical Readiness .......................................................................... 27

5.4 Service Provider and RDP sequence ........................................................ 28

5.5 Testing with the RDPs ............................................................................... 29



of 75 v1.2

DCC Public

5.5.1 Gas .................................................................................................... 29

5.5.2 Electricity ........................................................................................... 30

5.6 Systems Integration sequence .................................................................. 31

5.7 Device strategy ......................................................................................... 31

5.7.1 Certified equipment – Smart Meters ................................................... 31

5.7.2 Certified equipment – Comms Hubs ................................................... 32

5.7.3 Risk mitigation – representative Test Stubs/prototypes ...................... 33

5.7.4 Risk mitigation – certified Comms Hubs ............................................. 33

5.7.5 Test Stub requirements ...................................................................... 34

5.8 Non-Functional Testing ............................................................................. 35

5.8.1 Performance Testing .......................................................................... 35

5.8.2 Resilience Testing .............................................................................. 36

5.9 Test Method .............................................................................................. 36

5.9.1 Test Scenarios ................................................................................... 36

5.9.2 Test Data ........................................................................................... 39

5.10 Regression Testing ................................................................................... 43

5.11 V Model ..................................................................................................... 43

5.12 Requirements Traceability ......................................................................... 44

6 Test Procedure ............................................................................................... 46

6.1 Test Stages ............................................................................................... 46

6.1.1 Introduction ........................................................................................ 46

6.1.2 Solution Test ...................................................................................... 46

6.1.3 Service Provider UAT ......................................................................... 46

6.2 Test Stage overlap .................................................................................... 46

6.2.1 SP UAT Test Stage with Interface Test Stage .................................... 46

6.2.2 Solution Test Stage with Interface Test Stage .................................... 47

6.3 Division of work ......................................................................................... 47

6.4 High Level Plan ......................................................................................... 47



of 75 v1.2

DCC Public

6.5 Dependencies ........................................................................................... 49

6.6 Entry & Exit Criteria ................................................................................... 49

6.6.1 General .............................................................................................. 49

6.6.2 Entry Criteria ...................................................................................... 50

6.6.3 Exit Criteria ........................................................................................ 50

6.7 Quality Gates ............................................................................................ 52

7 Test Result Management & Reporting .......................................................... 54

7.1 Tracking .................................................................................................... 54

7.2 Reporting .................................................................................................. 54

8 Test Incident Management ............................................................................ 56

8.1 RDPs ........................................................................................................ 56

8.2 Logging and triage of Test Incidents.......................................................... 56

8.3 Resolution of Test Incidents ...................................................................... 56

8.4 Assurance and Disputes ........................................................................... 57

8.4.1 Assurance .......................................................................................... 57

8.4.2 RDP Dispute ...................................................................................... 57

8.4.3 DCC Licensee Dispute ....................................................................... 57

8.5 Reporting of Test Incidents ....................................................................... 57

8.6 Test Incident Management Process .......................................................... 57

9 Test Resources .............................................................................................. 60

9.1 Test Team ................................................................................................. 60

9.2 Test Tools ................................................................................................. 61

9.3 Test Stubs and Prototypes ........................................................................ 62

10 Roles & Responsibilities ............................................................................ 63

10.1 General ..................................................................................................... 63

10.2 DSP as Systems Integrator ....................................................................... 63

10.3 Service Providers ...................................................................................... 64

10.4 DCC Licensee ........................................................................................... 64



of 75 v1.2

DCC Public

10.5 RDPs ........................................................................................................ 65

10.6 St Clements .............................................................................................. 66

10.7 Network Operators .................................................................................... 67

11 Environments and Labs ............................................................................. 68

11.1 Environments ............................................................................................ 68

11.2 Test Labs .................................................................................................. 68

11.2.1 CSP N ................................................................................................ 68

11.2.2 CSP C/S ............................................................................................ 68

12 Test Scenarios ............................................................................................ 70

13 Test Incident Severities .............................................................................. 71

14 Device Selection Methodology .................................................................. 73

15 Auditing of SIT Exit Criteria ....................................................................... 74

15.1 Procurement of Independent Test Auditor ................................................. 74

15.2 Independent Test Auditor Role .................................................................. 74

16 Compliance with SEC 3 .............................................................................. 75



of 75 v1.2

DCC Public

1 Introduction

1.1 General

This document sets out how Systems Integration Testing (SIT) will be conducted for the Smart Metering Programme. Readers are expected to be familiar with the Joint Test Strategy (Reference [1]).

The Smart Metering eco-system is depicted in the following diagram.

Figure 1 – DCC Solution

This Test Approach is based on:

the Joint Test Strategy

the DSP and CSP Service Provider Contracts (References [7], [8], [9])

the Smart Energy Code (Reference [2]).

Evidence of this Test Approach’s compliance with the Smart Energy Code can be found in Section 16.

In the event of any discrepancy between these documents, the DCC Licensee will investigate, giving due regard to the relative precedence of the documents as set out in the Contracts. Where necessary, the DCC Licensee will raise Change Requests with the relevant organisations so that the discrepancy can be resolved.

The contents of this Test Approach have been derived from a series of SIT Planning workshops organised by the DSP, involving Test Managers and End to End Architects from the DSP, the CSPs, the DCC Licensee, Xoserve, and St Clements representing the Electricity Registration Data Providers (RDPs).



of 75 v1.2

DCC Public

The diagram below demonstrates how this Test Approach (enclosed within the red oval) fits in with the hierarchy of test documents that will be produced for the programme.

Figure 2 – Test Documentation Hierarchy

The testing described in this document will be further elaborated in the Solution Test Plan and Service Provider User Acceptance Test (UAT) Plan documents (shown underneath the SIT Approach document in Figure 2), which will be developed in collaboration with the Service Providers and RDPs.

1.2 Change Forecast

This document will be updated and re-issued when approved versions of the following documents are available:

Device Selection Methodology

Common Test Scenarios

High Level Design for Billing

High Level Design for Business Intelligence (BI) / Management Information (MI)

High Level Design for Smart Meter Key Infrastructure (SMKI)

revised High Level Design for Service Management

PIT Test Approach for Billing

PIT Test Approach for BI/MI

DCC Business Processes



of 75 v1.2

DCC Public

further Legal Draftings of the Smart Energy Code (SEC).

The DSP will keep this document up to date.

Updates to this document will follow the review, approval and appeal process outlined below.

1.3 Reviews, Approvals and Appeals

This document will be subject to internal DSP review prior to it being issued for external review and approval.

Version 1.1 of this document was conditionally approved at the DCC Licensee’s Design Assurance Board (DAB) on 15th July 2014. The DCC Licensee will:

issue version 1.2 of the document to RDPs for formal Consultation, and then

address RDP Consultation responses, and then

issue the document to the SEC Panel for review along with the responses from the RDP Consultation, after it has published the Device Selection Methodology on its website, and then

publish the RDP Consultation responses on its website.

Once the SEC Panel review comments have been addressed, the DCC Licensee will submit the document to the SEC Panel for approval.

If the SEC Panel decides not to approve the SIT Approach, it will notify the DCC Licensee and the RDPs of its reasons.

If the DCC Licensee and RDPs accept the SEC Panel’s decision, the SIT Approach will be revised to address such reasons and a second consultation with the RDPs will take place prior to the document being re-submitted to the SEC Panel for approval.

If the DCC Licensee or RDPs do not accept the SEC Panel’s decision, each will have the ability to refer the matter to the Gas and Electricity Markets Authority (GEMA), who will decide whether the document should be:

approved as is, or

approved after it has been revised in line with the SEC Panel’s reasons, or

revised in line with the SEC Panel’s reasons, submitted to the RDPs for a second consultation, and then re-submitted to the SEC Panel for approval.

Once the SIT Approach is approved the DCC Licensee will:

publish it on its website, not more than 3 months before the start of Solution Test execution

confirm the start date of Solution Test execution with the Service Providers (SPs) and RDPs.



of 75 v1.2

DCC Public

1.4 Terminology

In this document the term “Service Provider” includes all of the following:

the DSP

both CSPs

the Trusted Service Provider (TSP), supplier of the SMKI solution element

the DCC Licensee in its role as supplier of Enterprise systems such as Billing and BI/MI.

The term “User Integration Testing” (UIT) refers to the test phase that follows SIT, comprising Integration Testing and End to End Testing.

The term “Test Stub”, as defined in the SEC and used in this document, means systems and actions which simulate the behaviour of Devices and User Systems.

This document uses standard testing terminology, a glossary (Reference [3]) of which can be found on the International Software Testing Qualification Board website, www.istqb.org.

Abbreviations used in this document are listed in the following table.

Abbreviation Meaning

BI Business Intelligence

CI Configuration Item

CSP Communication Service Provider

DAB Design Assurance Board

DCC Data Communications Company

DCC KI DCC Key Infrastructure

DECC Department of Energy and Climate Change

DSP Data Service Provider

GBCS Great Britain Companion Specification

HAN Home Area Network

IHD In Home Display

iGT Independent Gas Transporter

MI Management Information

MPRS Meter Point Registration System

PIT Pre-Integration Testing

RDP Registration Data Provider

RTM Requirements Traceability Matrix

SEC Smart Energy Code

SIT Systems Integration Testing

SMKI Smart Meter Key Infrastructure

http://www.istqb.org/



of 75 v1.2

DCC Public

Abbreviation Meaning

SP Service Provider

TSP Trusted Service Provider

UAT User Acceptance Testing

UIT User Integration Testing

Table 1 - Abbreviations



of 75 v1.2

DCC Public

2 Scope

2.1 Overview

In its role as Systems Integrator, the DSP will manage SIT with support from the SPs and RDPs, and all parties will take an active part in the planning, preparation and execution activities.

SIT will verify that the SP and RDP system elements integrate together to form a working system that meets the agreed functional and non-functional requirements. The system consists of the whole Smart Metering eco-system with the exception of a Service User solution. Service User elements will be “stubbed” for SIT, and will be tested in the User Integration Test phase.

SIT will be undertaken on a CSP Region by CSP Region, RDP System by RDP System basis, with CSP Regions and RDP Systems being tested in parallel wherever it is efficient to do so.

At least two sets of certified Smart Metering Equipment will be used for SIT, chosen according to the Device Selection Methodology which has yet to be produced and agreed. Should certified Smart Metering Equipment not be available during SIT, then Smart Meter Test Stubs or prototypes will be used.

All five certified Comms Hub variants will be tested, with each variant being tested against two sets of Smart Metering Equipment.

The DCC Licensee will witness an agreed subset of tests, during the SP UAT stage of SIT.

2.2 Out of Scope

The following are outside the scope of SIT:

certification and accreditation of Smart Metering Equipment (this is not part of the DCC Licensee’s responsibilities)

certification of Comms Hubs (this is not part of the DCC Licensee’s responsibilities)

testing of the Smart Metering Equipment other than its interaction with the DCC solution (this is the responsibility of Meter manufacturers)

testing of the Home Area Network (HAN) other than its interaction with the DCC solution (this is not part of the DCC Licensee’s responsibilities)

testing of Hand Held Terminals (this is not part of the DCC Licensee’s responsibilities)



of 75 v1.2

DCC Public

testing the inter-changeability1 of Smart Metering Equipment (this is not part of the DCC Licensee’s responsibilities)

testing the inter-changeability of the Home Area Network Equipment (this is not part of the DCC Licensee’s responsibilities)

testing the interaction of each Comms Hub variant with more than two sets of Smart Metering Equipment (further sets will be covered in UIT)

testing the interaction with Service User systems (this will be covered in UIT)

testing of any pre-SMETS2 solutions or their interaction with the DCC solution (this is not part of the DCC Licensee’s responsibilities)

testing of Service User Business Processes (this will be covered in UIT)

testing the interaction with the DSP and CSP Security Operations Centres (this will be covered in Operational Acceptance Testing, to be run after SIT has completed and before the system is put live)

any testing requiring use of Production environments, e.g. Disaster Recovery Testing (this will be covered in Operational Acceptance Testing)

full end to end performance testing (this will be covered in Operational Acceptance Testing).

________________________

1 The ability to exchange one device with another without affecting the original functionality



of 75 v1.2

DCC Public

3 Objectives

3.1 SIT Objective

The objective of SIT is to demonstrate, by testing at the relevant volume scenarios (see Section 5.8.1), that the solution elements supplied by the various Service Providers and RDPs operate with one another as per the agreed requirements, and thereby show:

the DCC Licensee is capable of delivering its services as specified in Sections E (Registration Data), G (Security) and H (DCC Services) under SEC

the RDPs (through the Network Operators) are capable of meeting their SEC obligations under Section E (Registration Data) in relation to the transmission of registration data.

3.2 Test Approach Objectives

The objectives of this Test Approach document are to:

define the scope, objectives, deliverables, activities, resources and test processes required for SIT

define the roles and responsibilities of those involved in SIT.



of 75 v1.2

DCC Public

4 Deliverables

4.1 General

The following sections list the SIT deliverables, the contents of which are described in the Joint Test Strategy. For each deliverable, the DSP will provide the DCC Licensee with at least 10 working days’ notice prior to submission.

4.2 By Test Phase

The table below lists the deliverables for the SIT Phase as a whole.

Deliverable Timing

Test Approach (this document) Final version to be submitted to DCC Licensee no later than 3 months before commencement of Solution Test execution

Test Scenarios To be submitted with the Test Approach

Test Phase Completion Report Draft version to be submitted to DCC Licensee no later than 10 working days before planned end of test execution.

Final version to be submitted on completion of test execution.

Table 2 - Test Phase Deliverables

4.3 By Test Stage

The table below lists the deliverables for each of the two Test Stages that comprise the SIT Phase, namely Solution Test and Service Provider UAT.

Deliverable Timing

Test Plan Final version to be submitted to DCC Licensee no later than 30 working days before commencement of test execution

Test Schedule To be submitted with the Test Plan

Test Specifications To be submitted to DCC Licensee no later than 20 working days before commencement of test execution

Requirements Traceability Matrix – initial version of iteration 4

To be submitted before the commencement of test execution: see Section 5.12 (applies to Solution Test stage only)

Requirements Traceability Matrix – final version of iteration 4

To be submitted with the Test Stage Completion Report (applies to Solution Test stage only)

Test Readiness Reports Issued weekly, to start 20 working days before commencement of test execution

Test Execution Reports Issued weekly, for the duration of test execution

Test Results Available for inspection throughout test execution

Incident Log Available for inspection throughout test execution

Regression Test Pack To be submitted with the final Test Stage Completion Report



of 75 v1.2

DCC Public

Deliverable Timing

Test Stage Completion Report Draft version to be submitted to DCC Licensee no later than 10 working days before the planned end of test execution.

Final version to be submitted on completion of test execution.

Work-Off Plan To be submitted with the final Test Stage Completion Report

Table 3 - Test Stage Deliverables

4.4 Maintenance

Updated versions of the Test Approach, Test Plans, Test Specifications and Regression Test Pack will be supplied to the DCC Licensee on the anniversary of the completion of UIT.



of 75 v1.2

DCC Public

5 Test Phase Description

5.1 Focus areas for Systems Integration Testing

Under their own Pre-Integration Test (PIT) phase, each Service Provider (SP) and RDP will fully test their elements of the Smart Metering solution in isolation for all functional and non-functional requirements (including exception paths and negative conditions), as laid out in their PIT Test Approach and Test Stage Plans.

There is no need to repeat all such testing in SIT: instead, the primary focus of SIT will be on the dynamic interaction between solution elements that span SP/RDP boundaries.

The other focus areas for SIT are:

operation of SMKI: this security feature is fundamental to the processing of Service Requests, Responses and Alerts but is not available to the DSP or CSPs during their PIT

use of certified Comms Hubs: CSPs will use Test Stubs or prototypes in their PIT if certified Comms Hubs are not available

use of certified Smart Meters: CSPs will use Test Stubs or prototypes in their PIT if certified Smart Meters are not available.

operation of Service Management: this complex feature involves multiple interactions between the DSP and CSP systems.

5.2 Key planning assumption

The approach to SIT outlined in this document and the underlying timescales, quality levels and costs are predicated on the assumption that all solution elements will be available at the start of SIT, meeting all requirements and fully PIT tested, i.e. there will be no:

late deliveries

phased deliveries (e.g. partial functionality delivered at the start of SIT, and residual functionality delivered later)

significant outstanding tests or defects in the SP/RDP PIT Work-Off Plans.

The only exception to this is Comms Hubs:

certified versions of Comms Hubs may not be available at the start of SIT, in which case protocol-certified versions will be used

CSP C/S may not have protocol-certified versions of all four Comms Hubs variants available at the start of SIT, but at least one must exist at the start of SIT for CSP C/S.

Certified Smart Metering Equipment may not be ready at the start of SIT but this is not part of the DCC-supplied solution and so does not constitute an exception.



of 75 v1.2

DCC Public

5.3 De-risking SIT

5.3.1 Introduction

To de-risk SIT, there are three preliminary verification steps that need to be completed before SIT starts: DSP Review of PIT, Early SIT and Technical Readiness.

Figure 3 – Verification steps

DSP Review

Service Provider PIT

RDP PIT

Early SIT

With CSPs Technical Readiness

Service Providers

RDPs



of 75 v1.2

DCC Public

5.3.2 DSP Review of PIT

The DSP will review Service Provider PIT using the following methods:

review of SP PIT Approach

review of SP System Test Plan and FAT Plan

review of SP Requirements Traceability Matrix (3rd Iteration)

review of SP FAT test scenarios and data

review of SP Test Completion Reports and Work-Off Plans for System Test and FAT

review of SP Test Completion Report for PIT

attendance at SP Quality Gate Review 2.

The DSP will review RDP PIT by inspection of the relevant test planning, design and reporting documentation.

5.3.3 Early SIT

Early SIT will be conducted by the DSP and CSPs, subject to this not impacting PIT. The objective of Early SIT is to verify the “motorway traffic” between the DSP and CSP. It takes place at the same time as but is separate to PIT, and uses “early” versions of the DSP and CSPs solution elements, i.e. versions which:

have sufficient maturity to confirm that sample Service Requests, Responses and Alerts can be successfully exchanged between the DSP and CSP systems, but

have not yet completed System Testing or Factory Acceptance Testing.

The DSP will lead the planning and control of Early SIT, with input from the CSPs. Test preparation and execution activities will be shared across the DSP and CSPs.

In addition to this “dynamic” testing of the interfaces between the DSP and CSP systems, “static” testing of this and other interfaces will be undertaken by all Service Providers, Xoserve and St Clements whereby sample interface files will be manually exchanged and desk-checked.

5.3.4 Technical Readiness

All SPs and RDPs will undergo a Technical Readiness assessment before participating in SIT, which will confirm:

whether the SPs’ or RDPs’ PIT is complete

what the impact is on SIT of any outstanding PIT defects, and whether the dates in the Work-Off Plan are acceptable

have the relevant SP/RDP SIT environments been:

o built



of 75 v1.2

DCC Public

o configured

o loaded with the required solution elements (including Test Stubs, tools and devices)

o connected up (e.g. firewalls opened up and validated)2

o smoke tested

o penetration tested

has appropriate evidence of security controls been provided to the DSP Security Governance Team

has the base test data required to commence SIT been loaded to the relevant data stores

has the Test Management tool been set up and made available to the relevant personnel

have guidelines for the Test Management tool been developed and have the relevant SP/RDP personnel been made familiar with its use.

Further details of the Technical Readiness assessment will be provided in the Solution Test Plan.

5.4 Service Provider and RDP sequence

SIT will be performed on a CSP Region by CSP Region and on an RDP System by RDP System basis, so that achievement of the SIT Objective can be demonstrated for each CSP Region and each RDP System separately. The technical solution for CSP Regions Central and South is identical, and so only one of these Regions will be tested. CSP Regions North and Central/South will be tested in parallel wherever it is efficient to do so. The Gas and Electric RDP Systems will be tested in parallel wherever it is efficient to do so.

SIT will start once the DSP, in its role as System Integrator, has deemed that the following minimum set of SPs is ready:

at least one CSP

either the Gas RDP (Xoserve) or the Electricity RDPs represented by St Clements for functional testing and a designated Electricity RDP for non-functional testing, or both

the DCC Licensee (in its role as supplier of Enterprise Systems)

the TSP.

________________________ 2 For RDPs, this will include set up of the connectivity into the Hydra network and set up of the DDC Key

Infrastructure (KI)



of 75 v1.2

DCC Public

Any SP or RDP not ready at the start of SIT will join at a time agreed with the DSP.

5.5 Testing with the RDPs

5.5.1 Gas

The following diagram shows the high level Gas RDP system:

Figure 4 – Gas RDP system

Xoserve is the Gas RDP and runs its RDP system on a central platform. Independent Gas Transporters (iGTs) and Shippers send their registration data to the Gas RDP, and receive data updates from the Gas RDP.

Xoserve tests changes to this RDP system under the following test stages within their Pre-Integration Test phase:

Unit Testing

Integration Testing

Business Testing

User Trials.

Xoserve will fully test the interaction of their system with the systems of the iGTs and Shippers during the User Trials test stage, including a two-way connectivity test of the new communications links.

SIT will focus on the functional and non-functional interaction of the Gas RDP system and the DSP system, including:

a performance test of the transmission of a full refresh from the Gas RDP to the DSP



of 75 v1.2

DCC Public

a resilience test of a failed transmission between the Gas RDP and the DSP

a two-way connectivity test of the new communications links between the Gas RDP and DSP.

5.5.2 Electricity

The following diagram shows the high level Electricity RDP system:

Figure 5 – Electricity RDP system

St Clements supplies and maintains the Meter Point Registration System (MPRS) run by all Electricity RDPs on a distributed basis. St Clements will fully test the MPRS changes with Electricity RDP representatives during the following test stages within their Pre-Integration Test phase:

Unit Testing

System Testing

User Acceptance Testing

Regression Testing

Performance Testing

User Testing.

St Clements will fully test the MPRS changes with Electricity RDP representatives during the User Acceptance test stage of their Pre-Integration Testing.

St Clements will represent the Electricity RDPs in functional SIT with the DSP. The DSP’s non-functional SIT with the RDPs will comprise:

a performance test of the transmission of a full refresh from a large Electricity RDP to the DSP

a resilience test of a failed transmission between an Electricity RDP and the DSP



of 75 v1.2

DCC Public

a two-way connectivity test of the new communications links between the DSP and each of the Electricity RDPs.

5.6 Systems Integration sequence

The various elements of the solution will initially be integrated and tested as a series of “sub-assemblies”. This testing will be pipe-cleaning rather than full functional testing, and is designed to verify the basic connectivity between the various elements on an incremental basis. This initial integration testing will occur in four steps as illustrated by the diagram below. Note that this diagram is an abstraction, and does not show all solution elements or all connections between those elements.

User Message

Gateway

Request

Management

(incl. Transform

Service)

Data

Communication

Telefonica

SMWAN

Gateway

Arqiva

SMWAN

Gateway

Communications

Hub

Meter Simulators

Communications

Hub

System Integration Sequence

Service User

Simulator

Meter Simulators

Registration

Systems

Gateway

Data

Management

(incl.

Registration)

Registration

Systems

Data Warehouse

& Reporting

Coverage

Databases

Self Service

InterfaceSMKI

DSP PKI

Repository

CSP PKI

Repositories

Order

Management

Billing

Enterprise

Systems

Gateway

BICGI Remedy

Telefonica

Remedy

Arqiva Remedy

Event

Management

1 2 3 4

Integration Sequence

Figure 6 – Systems Integration sequence

Note that Step 1 requires RDP data to be loaded, but testing of the Registration Systems Gateway occurs in Step 2.

Once this initial integration testing is completed, the full Smart Metering eco-system will be tested end to end, within the boundaries of the scope of SIT.

5.7 Device strategy

5.7.1 Certified equipment – Smart Meters

The DCC Licensee will develop, agree, publish and operate the process used to select the certified Smart Metering Equipment to be used in SIT, known as the Device Selection Methodology: details can be found in Section 14.



of 75 v1.2

DCC Public

Ideally, SIT will start with all sets of certified Smart Metering Equipment resulting from the Device Selection Methodology. If this is not be possible, SIT will start with either Smart Meter Test Stubs (worst case), Smart Meter prototypes, some sets of certified Smart Metering Equipment (best case) or a combination. Should improved versions of such Smart Meter Test Stubs/prototypes or certified Smart Meters become available during SIT, they will be introduced into SIT in a controlled manner subject to an assessment of the following:

how thoroughly has the improved Test Stub/prototype or certified Smart Meter been tested by the manufacturer and/or the CSP prior to SIT

what additional confidence will be provided by testing with the improved Test Stub/prototype or certified Smart Meter in SIT

how much of the SIT testing that has already been completed needs to be repeated in order to mitigate the risk of regression and create the requisite level of confidence3

the size of the remaining SIT test window. Should the window be too short to test all Service Requests, Responses and Alerts with the improved Test Stub/prototype or certified Smart Meter, then a subset will be selected and tested based on:

o the priority of the test

o a sampling principle, whereby similar Service Requests, Responses and Alerts are grouped together and a representative example for each group is tested.

Note that this assessment will be carried out by the DSP in consultation with the CSPs.

5.7.2 Certified equipment – Comms Hubs

Ideally, SIT will start with fully certified versions of all five Comms Hubs variants. If this is not be possible, SIT will start with at least one protocol-certified4 Comms Hub prototype from each CSP. When improved versions of Comms Hubs become available during SIT, they will be introduced into SIT in a controlled manner subject to an assessment of the following:

how thoroughly has the improved Comms Hub been tested by the manufacturer and/or the CSP prior to SIT

________________________ 3 Each test will have two classifications: “Final test to be run with certified Comms Hub Y/N” and “Final

test to be run with certified Smart Meter Y/N”, based on the classifications of the test’s parent Test Scenario listed in Section 12

4 A “functional” prototype has the same functionality as the certified Comms Hub but may use different

hardware components, circuit design and form-factor; an “integration” prototype has the same functionality and only minor differences between hardware components, circuit design and form-factor; the “protocol-certified” prototype is identical to the Comms Hub that will be mass-produced for Consumer premises



of 75 v1.2

DCC Public

what additional confidence will be provided by testing with the improved Comms Hub in SIT

how much of the SIT testing that has already been completed needs to be repeated in order to mitigate the risk of regression and create the requisite level of confidence

the size of the remaining SIT test window. Should the window be too short to test all Service Requests, Responses and Alerts with the Comms Hub, then a subset will be selected and tested based on:

o the priority of the test

o a sampling principle, whereby similar Service Requests, Responses and Alerts are grouped together and a representative example for each group is tested.

Note that this assessment will be carried out by the DSP in consultation with the CSPs.

5.7.3 Risk mitigation – representative Test Stubs/prototypes

There is a significant risk that any device Test Stubs/prototypes will not be fully representative of the certified devices, and so any associated test results need to be regarded as provisional. The following strategies will be used to mitigate this risk during SIT and improve the confidence that can be placed on the test results:

setting and applying explicit selection criteria for both Smart Meter Test Stubs and prototypes, and Comms Hubs prototypes. Note that such criteria will be set and applied by the DSP in consultation with the CSPs.

making the availability of a protocol-certified Comms Hub prototype part of the Entry Criteria for SIT. Note that Comms Hub Test Stubs will be required for performance testing in SIT

making testing with fully-certified Comms Hubs part of the Exit Criteria for SIT.

5.7.4 Risk mitigation – certified Comms Hubs

Both CSPs are working closely with their Comms Hubs manufacturers to ensure that certified Comms Hubs are available for and have been tested prior to the start of SIT:

CSP N plans to have its Comms Hub protocol-certified by 30th January 2015, and security-certified by 30th April 2015

CSP C/S has a contract in place with both its manufacturers to provide certified Comms Hubs by 30th April 2015.

CSP N will be using one Comms Hub variant for SIT, provided by EDMI (the Fylingdale variant will not be available until after the system has gone live).

CSP C/S will be using four Comms Hub variants for SIT, provided by Toshiba and WNC:

Cellular only - WNC



of 75 v1.2

DCC Public

Cellular only – Toshiba

Cellular and Mesh, cellular aerial port – Toshiba

Cellular and Mesh, 2 aerial ports – Toshiba.

5.7.5 Test Stub requirements

This section describes the requirements for Test Stubs. Note that:

a Smart Meter Test Stub will be used if there are no suitable Smart Meters available

a Comms Hub Test Stub will be used for limited performance testing (see Section 5.8.1)

For simplicity, these requirements are expressed as being for a combined Comms Hub/Smart Meter Test Stub, but in practice the implementation of these requirements will result in the creation of multiple Test Stubs, e.g. a Comms Hub Test Stub and a Smart Meter Test Stub.

The summary requirements are as follows:

the Test Stub will receive a variety of Service Request messages output by the DSP Communications Handler and forwarded by the CSP Gateway, interpret those messages and produce an appropriate Service Response. The Test Stub will also need to produce Alert messages to be sent to the DSP. Whilst the scope of SIT is to deal with all Service Requests, Responses and Alerts for pragmatic purposes the Test Stub need only deal with a prioritised subset (e.g. Install and Commission, Meter Read). The remainder will be tested once certified Comms Hubs and Smart Meters are available. The Test Stub will therefore need to:

o authenticate and, where necessary, decrypt the message, involving application of the public keys of the various message senders and the private key of the Smart Meter

o interpret the message to determine what sort of response is required

o for certain messages include specific test data in a response

o build and, where necessary, encrypt the response using appropriate keys and references to enable the DSP system to link the response to the original message.

the Test Stub should also be capable of producing scheduled responses, where the response is triggered by the schedule and not by the receipt of a Service Request. However for the purposes of the test it will be acceptable for these responses to be triggered manually (rather than building the scheduling mechanism into the Test Stub).

The Test Stub should be capable of operating either as a combined Comms Hub or Smart Meter Test Stub as above, or as a Smart Meter only Test Stub, on the basis that certified Comms Hubs may be available before certified Smart Meters.

The Test Stub should be capable of acting as several different Smart Meters either simultaneously or at different times to cater for electricity and gas, and to



of 75 v1.2

DCC Public

allow testing to continue if data for any one Smart Meter becomes unusable. The Test Stub will need to determine which Smart Meter any particular message is meant for in order to respond appropriately.

Each CSP will require its own Comms Hub/Smart Meter Test Stub rather than share a common one, e.g. as a result of differences in WAN technologies.

A more formal and detailed definition of the Test Stub requirements will be elaborated as part of the Design phase of the programme.

5.8 Non-Functional Testing

5.8.1 Performance Testing

During PIT, each SP/RDP will use their performance test results to develop a Performance Model to demonstrate that, in isolation, their solution elements can meet the relevant performance requirements. In the case of the DSP and CSPs:

a set of volume scenarios (capacity levels to which the system will be tested) will be derived from the five roll-out Profiles defined in the SP Contracts

these volume scenarios will be based on the capacity of the PIT test environments

where the capacity of the test environments is not fully live-like, test results will be extrapolated to demonstrate that requirements can be met for all five Profiles.

In SIT, these separate Performance Models will be synthesised into a single End to End Performance Model to demonstrate that the SP/RDP solutions together meet the relevant performance requirements.

The development of this synthesised model may require each SP/RDP to undertake additional “stand alone” performance testing of their solution elements on their own live-like test environment should:

such environments not have been fully available during PIT, and/or

analysis of the End to End Performance Model indicate errors or omissions in the individual Performance Models.

The development of this model may also require some elementary (e.g. latency) End to End performance tests to be run across the integrated solution but it should be noted that:

there are wide variations in the capacity of each SP’s and RDP’s non-functional SIT environment and the comms links between them

such variations constitute a major constraint on:

o the ability to undertake SIT End to End performance tests, and

o the value of undertaking such tests.



of 75 v1.2

DCC Public

The specific volume scenarios to be exercised during SIT are:

mass firmware update and rollback

processing large refreshes of Registration data

a variety of functional tests will be run at volume with parts of the system:

o running at reduced capacity

o subject to “denial of service” attacks.

Further detail of these volume scenarios (and the full set of tests scenarios) can be found in Section 12.

Full End to End performance testing on production environments connected together with production comms links will take place in Operational Acceptance Testing, after SIT has completed and before the system is put live. Any constraints on such testing (e.g. the inability to exercise the access network, environments already in live use) will be taken into account in the design of the tests and the interpretation of the results.

5.8.2 Resilience Testing

The environment and comms link capacity variations noted in Section 5.8.1 mean that full resilience testing in SIT is not feasible. Instead, the resilience test results from each SP’s PIT will be analysed and SPs may be required to undertake additional “stand alone” resilience testing of their solution elements on their own live-like test environment should such environments not have been fully available during PIT.

Full resilience testing will take place on production environments in Operational Acceptance Testing.

5.9 Test Method

5.9.1 Test Scenarios

A series of high level test scenarios has been developed to support the approach outlined in Sections 5.1 to 5.8 above, and these scenarios can be found in Section 12. The scenarios cover both functional and non-functional aspects of the dynamic interaction between solution elements spanning SP/RDP boundaries and are based on:

the DSP and CSP contracted requirements

the Service Request Catalogue

Business Scenarios listed in the Registration Data Interface Specifications

the DSP and CSP PIT Test Approach documents ([6], [5], [4])



of 75 v1.2

DCC Public

the SIT Planning workshops held with the SPs and RDPs.

The scenarios have been grouped as follows:

Registration/de-registration of Service Users

At least one of each type of Service User will be created and at least one dual fuel supplier.

Interaction with Registration Data Providers

The initial “full refresh” of registration files from St Clements and Xoserve will provide the basis from which all other test data will be developed.

The data in these files will be updated initially with changes to ensure consistency with CSP data and to ensure some locations are capable of dual fuel.

Daily update files from RDP to DSP will cover the various change scenarios identified.

Daily update files from DSP to RDP will be triggered by certain Service Requests below.

Set up of Comms Hubs

At least one Comms Hub will be created for each Service User and RDP created above. It is anticipated that approximately 40 Comms Hubs will be required in all, although there will be separate provisions for non-functional tests requiring larger numbers, e.g. single devices simulating multiple Comms Hubs.

Service Requests

The bulk of testing will comprise the testing of numerous Service Requests. These will be grouped as follows:

On Demand vs Scheduled vs Future Dated, with the On Demand versions of the Service Requests (SRs) addressed first

High Priority vs Low Priority SRs. This distinguishes SRs which must be applied to every meter (e.g. commissioning) from those which will only be applied in specific circumstances, e.g. reset PIN.

This will give a small subset of SRs which will be addressed first to give an early view of the integration issues to be encountered and with SRs sequenced in a meaningful way. This subset will be repeated for each of electricity and gas, and for CSP N and CSP C/S.

Each SR scenario (over 100 have been identified) could have multiple variations (9 types of Service User, 5 Comms Hub variants, 8 Command Variants). It is not practical to test all variations, but a pragmatic spread ensuring coverage of all Service Users, CSPs and Comms Hub variants will be developed (see Risk-based testing sub-section below). The sets of Smart Meters resulting from the Device Selection Methodology (a minimum of two) will be distributed across the five Comms Hubs variants (two sets per Comms Hub variant), but not all combinations will be tested. Approximately 100 Meter Points will be required, with some Smart Meters



of 75 v1.2

DCC Public

being credit, some pre-pay and some export. The spreadsheet of test scenarios will make it clear which tests could be developed and which will be.

Examples of how the overall number of tests will be reduced include:

Not testing the On Demand and Future Dated versions of the same Service Request on the same Comms Hub, although each Comms Hub will have a mix of On Demand, Future Dated and Scheduled SRs applied to it

Not testing the Command Variant for Local and WAN delivery combined and the Command Variant which generates these individually on the same Comms Hub.

Later in testing a further subset of these tests will be repeated but with the system artificially broken to generate Service Level violations and incidents.

Billing and BI/MI

The list of SR tests above should ensure that sufficient variation exists to adequately test Billing and BI/MI. This will be reviewed and further tests added as necessary.

Self-Service Interface

The list of SR tests above should ensure that sufficient variation exists to adequately test the various options in the Self Service Interface. This will be reviewed and further tests added as necessary.

Housekeeping

Again, later in testing, a further repetition of a subset of SR tests will be repeated but the system altered to ensure that unprocessed messages exist to which various housekeeping rules can be applied.

Exceptions

Generally it is expected that exception processing will be tested in PIT, e.g. the identification of invalid SRs. However, where an error message is required to pass between SPs (e.g. RDPs generating an ERROR file to return to the DSP where the file supplied by the DSP is found to be invalid), such tests will be included.

Non-Functional testing

System performance will generally be tested during PIT. Non-functional tests for SIT include tests of resilience, where a file transmission is interrupted and restarted, and tests where parts of the system are operating in failover mode.

Risk-based testing

The Test Scenarios will be prioritised using a risk assessment of:

the business importance of the various solution elements being tested

the technical probability of defects being present in each solution element.

A draft assessment of these characteristics is included in the Section 12 spreadsheet, and will be ratified by discussions with the relevant stakeholders. The confirmed priorities will be used to set:



of 75 v1.2

DCC Public

the depth of coverage for each Test Scenario (e.g. how many test specifications aka test scripts will be generated for a given Test Scenario)

the sequence in which Test Scenarios are elaborated into test scripts

the sequence in which test scripts are executed.

Further work

These scenarios are a work in progress and require further discussion with the SP/RDPs to complete. They will be elaborated in the Solution Test Stage by the DSP with support from the other SPs and RDPs.

5.9.2 Test Data

Test data will be designed to support the SIT test scenarios, and all test data used in SIT will comply with the UK Data Protection Act.

The Department of Energy and Climate Change (DECC) has authorised the use of Xoserve’s “old live” MPRN data for SIT.

Should a requirement for data from external parties be identified, the DSP will notify the DCC Licensee to allow the DCC Licensee to liaise with the party concerned. Should a requirement for data anonymisation (aka obfuscation) be identified, appropriate security approval will be sought.

All test data will be prepared in time for loading to the SIT test environments during the Technical Readiness stage (see Section 5.3.4).

An analysis of the main data stores across the DCC eco-system has been started in order to identify the sources and volumes of data required. The work in progress results are detailed in the spreadsheet below, and the DSP data stores from the spreadsheet are shown on the following pages to illustrate the content.

Data Stores and their Population for Systems Integration Testing v0.4.xlsx



of 75 v1.2

DCC Public

Data Store Type

(Functional)

Short

Form

Data Store Description Approximate

No of Records

Needed for

Testing

Stat

ic D

ata

Dat

a C

reat

ed b

y R

egis

trat

ion

Pro

cess

Dat

a C

reat

ed b

y Se

rvic

e R

equ

ests

Dat

a C

reat

ed b

y C

SP R

esp

on

ses

Dat

a C

reat

ed b

y D

ata

W'h

ou

se P

roce

ss

Dat

a C

reat

ed b

y K

ey G

ener

atio

n.

Comments

Registration ENPT Energy Participant Name and contact details of a participant energy company. Tens

Data created for PIT may be used, may need to be augmented or

replaced.

ENRO Energy Role The type of role that an energy participant can have. Single figures


replaced.

PTRO Participant Role Links the energy participant to the energy role Tens

Derived from the data stores above. Data created for PIT may be used,

may need to be augmented or replaced.

MPRO Meter Point Roles At least one of these per meter point and linked to a participant role Tens of

thousands

Populated with an initial load of data from St Clements and Xoserve.

Requires the 3 data stores above (ENPT, ENRO, PTRO) to be already

populated.

Schedules Pre-installation Service

Request

Links a Service Request to a device pre-installation Tens

Populated by testing (I'm not sure why the logical data model has

grouped this with schedules)

Schedules These are the schedules for meter readings Hundreds Populated by testing.

Capabilities SURM SEC User Role Matrix Links a SEC User role to a Service Request thus controlling access to

Service Request creation

Tens


replaced.

SRTP Service Request Types The Service Request types listed in the DCC User Gateway Interface

Specification

87


replaced.

GBCS GBCS Use Cases Description and Energy Type of each Use Case. There will be at least one

of these for each Service Request.

Hundreds


replaced.

GUCD GBCS Use Case Device

Compatibility

Links a GBCS Use Case to the device models and firmware versions

capable of supporting that use case.

Hundreds


replaced.

DSRC Device SR Compatibility Links a Service Request to the device models and firmware versions

capable of supporting that use case.

Hundreds


replaced.



of 75 v1.2

DCC Public

Data Store Type

(Functional)

Short

Form


No of Records

Needed for

Testing

Stat

ic D

ata

Dat

a C

reat

ed b

y R

egis

trat

ion

Pro

cess

Dat

a C

reat

ed b

y Se

rvic

e R

equ

ests

Dat

a C

reat

ed b

y C

SP R

esp

on

ses

Dat

a C

reat

ed b

y D

ata

W'h

ou

se P

roce

ss

Dat

a C

reat

ed b

y K

ey G

ener

atio

n.

Comments

Devices MPDV Meter Point Devices Links a device to the MPxNs that are served by that device Hundreds Populated by testing.

DVCE Device All Smart Metering devices attached to the DCC Network or pending

commissioning

Hundreds

Populated by testing.

SMS Groups devices according to the SMWAN Communications Hub through

which they communicate.

Hundreds

Populated by testing.

CSP Address Gateway Links the device ID to the rep ID (?) ? Populated by testing.

Device Model The model name and approval status of each type of device Tens


replaced.

Firmware Versions The firmware versions linked to each device model Tens


replaced. Additional entries populated by testing.

DMFR Manufacturer The manufacturer id description and details for each device model Tens


replaced.

DMDL Device Type The device type and description, with many device models belonging to

each type

Single figures


replaced.

Meter Points PREM Premise The address and postcode where a Meter Point is located Tens of

thousands



populated.

MPRO Meter Point Holds every MPxN notified by the registration data providers regardless

of whether a smart metering system has been commissioned

Tens of

thousands



populated.

Registration Data Provider The RPD id and their details Single figures


replaced.

DCC/SEC Party SPTY SEC Party An SEC party Tens


replaced.

SRTP SEC Party Role An SEC party operating in a particular SEC role, becoming a DCC user Tens


replaced.

SRTP SEC Role Type A specific SEC role e.g. EIS, GIS, ENO, GNO etc. Single figures


replaced.

PPRM Party Participant Role Matrix Links a SEC Party Role to one or more Participant Roles in the Registration

data

Tens


replaced.

Other (CSP) CSP The CSPs identity and associated details 2


replaced.

CSP Region The regional details linked to the CSP id 3


replaced.



of 75 v1.2

DCC Public

Table 4 - DSP Data Stores and their derivation

Data Store Type

(Functional)

Short

Form


No of Records

Needed for

Testing

Stat

ic D

ata

Dat

a C

reat

ed b

y R

egis

trat

ion

Pro

cess

Dat

a C

reat

ed b

y Se

rvic

e R

equ

ests

Dat

a C

reat

ed b

y C

SP R

esp

on

ses

Dat

a C

reat

ed b

y D

ata

W'h

ou

se P

roce

ss

Dat

a C

reat

ed b

y K

ey G

ener

atio

n.

Comments

Data Store Type

(Data

Warehouse,

Reporting)


No of Records

Needed for

Testing

Source

Inventory and Reporting

Database

A subset of the device inventory data stored in the functional data stores

above.

Hundreds

Created from the datastores above and updated by certain of the test

scenarios.

Audit Database Created from the transaction logs that are created as Service requests are

processed.

Thousands

Each Service Request created during testing should create an entry on the

audit database.

Data Store Type

(PKI)


No of Records

Needed for

Testing

Source

SMKI PKI Repository Certificates containing the public keys of all Service Users and Smart

Meters.

Tens

Organisational certificates will be requested by Service Users, these are

sent by the Certificate Authorities (CAs) to the PKI repository. Service

Users will supply batch requests for Smart Meter Certificates, these are

also sent by CAs to the PKI repository.

Data Store Type

(Volt DB Caches)


No of Records

Needed for

Testing

Source

Population is automated based on population of master data stores.



of 75 v1.2

DCC Public

5.10 Regression Testing

SPs and RDPs will perform a regression test on each new release of their solution elements according to the principles laid out in their PIT Test Approach document, prior to the new release being accepted into and installed in the SIT environment.

Once installed, the new release will be subject to a SIT regression test prior to it undergoing detailed functional and non-functional testing. This SIT regression test will be a selection of tests from the full SIT regression test set, based on an assessment of the risk that the new release will cause features previously verified in SIT to stop working. This assessment will be undertaken by the SP Architect, SP Development Team Leader and the SIT Test Manager.

The full SIT regression test set will be developed by the DSP with support from the SP/RDPs and cover all solution elements to an appropriate depth, and will be maintained in line with:

changes in the scope of the system under test

experiences of earlier regression failures.

The regression test approach will be elaborated in the Solution Test Stage Plan.

The Regression Test Pack (test scripts, test data and documentation) will be submitted to the DCC Licensee at the end of the Solution Test Stage and the SP UAT Test Stage. Any agreed omissions will be rectified promptly.

5.11 V Model

The diagram below illustrates how:

the test phases map onto the early stages of the system development lifecycle, and

the primary specification documents (shown on the left-hand side) on which each test phase is based.



of 75 v1.2

DCC Public

Requirements

User

Integration

Test

System

Integration

Test

Pre-

Integration

Test

Detailed

Design

System

Architecture

Build

SEC

Service User Business Processes

Requirements

Interface Specification

Code of Connection

DCC Business Processes

Functional Specification

Detailed Design Specification

Component Design Specification

Figure 7 – V Model

5.12 Requirements Traceability

The SPs will each use their own tools to manage their requirements and demonstrate traceability to both the Solution Design and the Pre-Integration tests. The DSP and CSPs will each provide the DCC Licensee with a PIT Requirements Traceability Matrix (RTM), extracted from these separate tools.

Prior to the start of SIT, the DSP will consolidate and extend these PIT RTMs to include the SIT tests and supply the results to the DCC Licensee. This constitutes the initial version of the 4th iteration of the RTM, as shown in the diagram below.

Figure 8 – Evolution of the Requirements Traceability Matrix



of 75 v1.2

DCC Public

At the end of SIT, any new tests which have been created during SIT will be added to the RTM and the final version of the 4th iteration will be supplied to the DCC Licensee.



of 75 v1.2

DCC Public

6 Test Procedure

6.1 Test Stages

6.1.1 Introduction

There are two test stages in SIT:

Solution Test

SP User Acceptance Test.

6.1.2 Solution Test

The testing described in Sections 5.1 through 5.10 above will take place in the Solution Test stage.

6.1.3 Service Provider UAT

The purpose of the SP User Acceptance Test stage is to allow the DCC Licensee to witness an agreed subset of the tests carried out in the Solution Test stage as part of its overall Acceptance Activities, details of which can be found in the Joint Test Strategy. This subset of tests will be described in the SP UAT Test Plan.

Giving at least 10 working days’ notice, the DSP will provide the DCC Licensee with a schedule of when and where these tests will be executed and invite the DCC Licensee to witness either on-site or remotely. Ahead of this witnessing, each SP/RDP will invite the DCC Licensee to participate in a series of familiarisation visits to their premises so that the witnesses can acquaint themselves with the SP/RDP systems, processes and personnel.

Execution of the agreed set of tests will be managed by the SIT Test Manager and performed by the relevant SP/RDP test analyst, and there will be:

no deviation from the scripts (e.g. in response to “what if” questions raised by witnesses)

no hands-on execution by witnesses.

Test Incidents raised during witnessing will be entered in Quality Center and progressed through the Test Incident Management process (see Section 8).

As far as possible, any queries and issues arising during the witnessing period will be addressed at the time with the relevant subject matter experts. A wash-up session will be convened at the end of the witnessing period to discuss the outcome of the witnessing and to agree any outstanding queries and issues.

6.2 Test Stage overlap

6.2.1 SP UAT Test Stage with Interface Test Stage

As part of the programme re-plan (CR035), the decision was taken to overlap the SP UAT Test Stage of SIT with the Interface Test Stage of UIT in order to minimise the impact of Great Britain Companion Specification (GBCS) delays on the go-live date. This overlap introduces a risk that any significant defects found in SP UAT will cause



of 75 v1.2

DCC Public

re-work and delay in Interface Testing, however this risk is small given that SP UAT is a merely a repeat of a subset of the tests run in the Solution Test Stage. This risk will be mitigated by scheduling the high risk tests for the start of SP UAT.

6.2.2 Solution Test Stage with Interface Test Stage

Given that SIT will be conducted on a CSP Region by Region basis, the DCC Licensee may, after consultation with the SPs and RDPs, recommend to the Secretary of State that Interface Testing for one or more Regions is started before Solution Testing is completed.

The DCC Licensee will publish any such proposal and consultation results on its website.

Where the Secretary of State agrees with the DCC Licensee’s recommendation, Interface Testing shall commence from the time recommended for the identified Regions.

6.3 Division of work

The Test Scenarios referenced in Section 5.9.1 will be divided between the SPs and RDPs for test preparation and test execution work according to the following criteria:

in which part of the end to end system does the majority of the test processing take place

what skills and expertise are needed to design and run the test

the availability of re-usable testware from PIT.

A draft division of these Test Scenarios between SPs and RDPs is provided in column G (“Responsibility for Preparation and Execution”) of the spreadsheet in Section 12, and this division will be discussed and agreed with the SPs and RDPs.

The DSP will support the other SPs and the RDPs in their assigned test preparation and test execution activities, ensuring that they have access to the relevant information and resources.

The DCC Licensee will support the DSP in the planning, control and operation of SIT.

6.4 High Level Plan

The diagram below shows the high level plan for SIT, with the SIT Execution dates lifted from the latest Integrated Solution Delivery Plan [Reference 10].

A series of estimating and scheduling workshops will be held with the SPs and RDPs in July 2014 to take this plan down to the next level of detail and validate:

the overall duration of SIT Execution and

the relative split between Solution Test and SP UAT.



of 75 v1.2

DCC Public

Figure 9 – High level plan for SIT



of 75 v1.2

DCC Public

6.5 Dependencies

The following table describes the dependencies that need to be satisfied by parties outside the SP/RDP Test Teams involved in SIT.

No. Description Dependent Upon

Due Date

1 Device Selection Methodology published and in operation

DCC Licensee 27/10/14

2 Great Britain Companion Specification (GBCS) v0.8 issued

DECC 08/07/14

3 Certified Smart Metering Equipment available Meter Manufacturers

01/03/15

4 DCC Licensee Business Processes issued DCC Licensee 30/11/14

5 Design for all solution elements agreed (including Service Management and SMKI)

SP Design Teams 30/11/14

6 Test Stub design agreed SP Design Teams 30/11/14

7 Test Environment design agreed DSP 30/09/14

8 Shared instance of Quality Centre available DSP 13/10/14

9 Completion of SP and RDP PIT

9a DSP, CSP N, TSP, DCC Licensee, first RDP system SP and RDP Test Team

27/02/15

9b CSP C/S (excluding FAT), second RDP system CSP C/S, RDP Test Team

27/02/15

10 Set up of DCC User Gateway between DCC and RDPs RDPs 31/01/15

11 DCC Key Infrastructure (KI) service ready for use DCC Licensee 31/01/15

Table 5 - Dependencies

Note that for Dependency 9, there will be a range of dates for each SP as per the Integrated Solution Delivery Plan, the very latest of which will be 27/02/15.

6.6 Entry & Exit Criteria

6.6.1 General

The Entry and Exit Criteria below relate to the SIT Phase. Entry and Exit Criteria for the Solution Test Stage and the SP UAT Test Stage will be detailed in the relevant Test Stage Plans.

The Entry and Exit Criteria listed below apply to all SPs and RDPs, and the DSP will assess whether SPs/RDPs meet these Criteria.

With the support of its Network Operator, each RDP is expected to:

co-operate with the DCC Licensee in its assessment of these Entry Criteria

take all reasonable steps to meet these Entry Criteria by the date required



of 75 v1.2

DCC Public

notify the SEC Panel and the DCC Licensee if it considers it will not meet the Entry Criteria by the date required.

The DCC Licensee will report the DSP’s assessment of each RDP to the SEC Panel. Any disagreement between an RDP and the DCC Licensee over this assessment will be referred to the SEC Panel. The RDP and DCC Licensee can each appeal the Panel’s ruling to the Authority, whose decision is final.

6.6.2 Entry Criteria

The Entry Criteria for SIT are:

SIT Test Approach (this document) approved by the DCC Licensee

RTM (initial version of Iteration 4) prepared

Approval to Proceed certificates issued by the DCC Licensee for the DSP, at least one CSP, at least one RDP system, and the TSP

Solution Test Plan approved by the DCC Licensee

Solution Test Specifications (and supporting test data) prepared

Technical Readiness Assessments completed for each SP/RDP (see Section 5.3.4)

at least one “protocol-certified” Comms Hub prototype is available for each CSP Region

at least one set of certified Smart Metering Equipment for each fuel type is available (or a DSP/CSP-approved prototype or a DSP/CSP-approved Test Stub)

other Test Stubs required for SIT are available (see Section 9.3)

test environments prepared

SP and RDP resources are available to support the testing (see Section 10.2).

6.6.3 Exit Criteria

The Exit Criteria for SIT are:

at least one CSP Region meets the test success criteria listed below (note that SIT will continue until the second CSP has exited)

the DSP, TSP and DCC Licensee meet the test success criteria for tests run, tests passed and defects as listed below

both RDP systems meet the test success criteria for tests run, tests passed and defects as listed below

all fully-certified Comms Hub variants required for go-live tested



of 75 v1.2

DCC Public

both sets of certified Smart Metering Equipment for each fuel type have been tested (or accepted prototypes or accepted Test Stubs)

all planned tests run, or any exceptions documented and agreed with the DCC Licensee

at least 85% of tests passed, or any exceptions documented and agreed with the DCC Licensee

the level of defects for each SP/RDP is within the following thresholds, or any exceptions are documented and agreed with the DCC Licensee:

o 0 Severity 1s (see Section 13 for Severity definitions)

o 0 Severity 2s

o 15 Severity 3s

o 30 Severity 4s

o 60 Severity 5s

the Test Completion Report has been issued

the Work-Off Plan has been produced

test results have been documented and evidence captured

a complete set of test incident logs produced

RTM (final version of Iteration 4) prepared

regression testing is complete

the Regression Test Pack has been produced

Test Stage Completion Certificates for the Solution Test Stage and the SP UAT Test Stage issued by the DCC Licensee

Test Phase Completion Certificate for SIT issued by the DCC Licensee.

The DCC Licensee will procure an independent auditor to confirm that the above Exit Criteria have been met for a CSP Region or RDP system, and will publish the auditor’s report on its website (the approach for implementing and operating this auditing is defined in Section 15). The DCC Licensee will then:

notify the Secretary of State, the GEMA, the SEC Panel, the SEC Parties and the RDPs that SIT for the CSP Region or RDP system in question has ended

provide the GEMA, the SEC Panel and (on request) the Secretary of State with copies of the Test Completion Report and the auditor’s report, along with a list of those sections of such reports that it considers should be redacted



of 75 v1.2

DCC Public

on direction from the SEC Panel, provide the SEC Parties, RDPs and SPs with copies of the Test Completion Report and the auditor’s report, having first redacted any sections specified by the SEC Panel.

SIT will end, and achieve its Milestone, on attainment of its Exit Criteria. This is expected to coincide with the last day of SP UAT execution.

6.7 Quality Gates

A series of Quality Gate Reviews will be held between Test Stages within the PIT and SIT Phases, to confirm that the Exit Criteria of the preceding Test Stage and the Entry Criteria of the upcoming Test Stage have been met. The following table describes these Quality Gate Reviews:

No. Quality Gate Review Chair Approver Attendees

1a Between CSP Link and System Test CSP CSP DCC Licensee

1b Between DSP Link and System Test DSP DSP DCC Licensee

1c Between DCC Licensee Link and System Test DCC Licensee

DCC Licensee

1d Between TSP Link and System Test TSP TSP DCC Licensee

2a Between CSP System Test and FAT CSP CSP DSP,

DCC Licensee

2b Between DSP System Test and FAT DSP DSP DCC Licensee

2c Between DCC Licensee System Test and FAT DCC Licensee

DCC Licensee

DSP

2d Between TSP System Test and FAT TSP TSP DSP,

DCC Licensee

3 Between FAT and Solution Test DSP DCC Licensee

SPs, RDPs

4 Between Solution Test and Service Provider UAT DSP DSP SPs, RDPs,

DCC Licensee

5 Between Solution Test and Interface Test DSP DCC Licensee

SPs, RDPs

Table 6 - Quality Gate Reviews

Note that for Quality Gates 3, 4 and 5 St Clements will represent the RDPs.

The “Approver” of each Quality Gate Review Meeting will set the outcome as one of the following:

preceding Test Stage can close, upcoming Test Stage can start, only minor (if any) remedial actions required

preceding Test Stage cannot close until remedial actions have been completed, upcoming Test Stage can start

preceding Test Stage can close, upcoming Test Stage cannot start until remedial actions have been completed



of 75 v1.2

DCC Public

preceding Test Stage cannot close, upcoming Test Stage cannot start, until remedial actions have been completed

In its role as Systems Integrator, the DSP will attend Quality Gates 2a, 2c, 2d.

Quality Gate Review 3 also covers the Exit of the PIT Phase and the Entry of the SIT Phase. Quality Gate 5 covers the Exit of the SIT Phase and the Entry of the User Integration Phase.

Each Quality Gate Review meeting will be a short, checklist-driven event at which previously assembled and validated evidence relating to the Exit and Entry Criteria is considered and decisions made to close the current Test Stage and start the next Test Stage. It is expected that Quality Gate Review meetings will be dry-run to enable issues to be identified and resolved in a timely manner, and thereby avoid impacting the start date for the upcoming Test Stage.

The current Test Stage/Phase will complete (and achieve its Milestone) on attainment of its Exit Criteria. The next Stage/Phase will commence (and achieve its Milestone) on attainment of its Entry Criteria. The Quality Gate Review meeting will take place during the transition from the current to the next Test Stage/Phase.

To facilitate the operation of Quality Gate Reviews and the timely achievement of Test Stage Entry Criteria within SIT, the DSP will publish weekly Test Readiness Reports for the Solution Test Stage and SP UAT Test Stage, as described in the Joint Test Strategy.



of 75 v1.2

DCC Public

7 Test Result Management & Reporting

7.1 Tracking

HP’s Quality Center Test Management tool will be used to manage SIT testing and SIT test incidents. The DSP will establish, maintain and support a shared instance of this tool for local and remote use by SPs/RDPs in support of SIT preparation and execution activities, and supply up to 50 royalty-free licences in total for this purpose (further licences will be available at a charge). The proposed allocation of these licences during the SIT phase is shown in the following table.

Organisation Licence count

DSP 18

CSP N 12

CSP C/S 12

TSP 2

DCC Licensee 2

RDPs 4

Table 7 - Proposed allocation of QC licences

Note that these licences will be re-distributed at the start of the UIT phase to cater for the participation of Service Users in test preparation and execution activities.

Quality Center will be used to store:

the RTM

the test scripts

execution details of each test script (e.g. when run, by whom)

evidence of system behaviour (e.g. screen shot, log file) observed during execution

the result of execution (pass, fail)

defects raised for failed tests (which will be linked to the failed tests).

A guideline for the use of Quality Center will be developed as part of the Solution Test Plan.

7.2 Reporting

Data held in Quality Center will be used to populate the weekly Test Execution reports and the Test Completion reports sent to the DCC Licensee, the contents of which are described in the Joint Test Strategy. Significant test failures will be notified to the DCC Licensee once they have been confirmed.



of 75 v1.2

DCC Public

The DCC Licensee will provide the SEC Panel and the Secretary of State with a copy of the weekly Test Execution Reports for information, with details of any test incidents anonymised and redacted as required in accordance with Section H14.44 of SEC 3 [2].

At the end of each Test Stage, the DSP will provide the DCC Licensee with a Test Completion Report, the contents of which are described in the Joint Test Strategy.



of 75 v1.2

DCC Public

8 Test Incident Management

8.1 RDPs

RDPs are considered to be SIT “Test Participants” as defined by SEC 3 [2], and will undertake tests as directed by the DSP. RDPs are expected to take reasonable steps to diagnose and resolve test incidents before raising them with the DSP.

8.2 Logging and triage of Test Incidents

All test incidents will be logged in Quality Center by the person executing the test. Full details will be recorded in Quality Center to enable speedy resolution. New incidents will be reviewed at least daily by the relevant SP/RDP Test Manager, who will:

classify them as one of:

o a defect

o not a defect

o duplicate

o change

o need more information, and

set their Severity (see Section 13 for definitions) and Priority (the urgency with which they need to be resolved for testing purposes)

assign the incident to the relevant resolver group. Should this lie outside the SP’s or RDP’s organisation, the incident will be assigned to the DSP Incident Manager.

8.3 Resolution of Test Incidents

The DSP Incident Manager will:

regularly review all outstanding incidents to ensure that they are resolved at the requisite speed

involve the Gas RDP and St Clements as required in the triage and resolution of defects associated with Registration Data

agree with the relevant SP/RDP Test Managers the defect fixes to be included in each Release to the SIT environment

report progress directly to stakeholders.

The following table lists the proposed maximum fix times for confirmed defects, to be measured from the point at which the defect is assigned to the resolver group to the point at which a fix is released to the relevant SIT stage. It requires discussion with the SPs and RDPs before it is finalised.



of 75 v1.2

DCC Public

Priority Maximum Fix time (days)

1 2

2 4

3 5

4 10

Table 8 - Maximum defect fix times

8.4 Assurance and Disputes

8.4.1 Assurance

The Incident Review Board (IRB), comprising each SP’s Design Authority, Xoserve and St Clements’ Design Authorities and the DCC Licensee, and chaired by the DSP Incident Manager, will meet twice weekly (and on demand for urgent incidents which are delaying testing) to:

resolve cases where the ownership of a test incident is disputed

confirm, by a process of sampling, that test incidents are being given the correct Severity by the local triage process

confirm, by a process of sampling, that Priority 1 defects are being resolved at the requisite speed.

8.4.2 RDP Dispute

Where an RDP is dissatisfied with the triage (assigned severity/priority, expected resolution timetable, resolution actions assigned to itself) results of any test incidents it has raised, it may refer the matter to the DCC Licensee.

If the RDP disagrees with the DCC Licensee’s findings, it may request that the DCC Licensee refers the matter to the SEC Panel.

The SEC Panel will publish dispute decisions on its website.

8.4.3 DCC Licensee Dispute

The severity of unresolved test incidents listed in the Test Completion Reports will be agreed with the DCC Licensee. If the parties are unable to agree the severity, the Dispute Resolution Procedure defined in the relevant SP Contract will be followed.

8.5 Reporting of Test Incidents

Information on the status of test incidents will be reported by the DSP to the DCC Licensee in the weekly Test Execution reports described in the Joint Test Strategy.

8.6 Test Incident Management Process

A detailed Test Incident Management process will be developed in conjunction with the Solution Test Plan.



of 75 v1.2

DCC Public

The high level lifecycle for test incidents raised during SIT is shown in the following diagram.

New

Incident

Triage

Open

Work in

Progress

Fixed

In PIT Build

PIT

Test

Passed PIT

In SIT Build

SIT

Test

Passed SIT

Closed

Need more

information

Rejected

Failed PIT

Failed SIT

Duplicate

Change

Not a

Defect

Figure 10 – Test Incident Lifecycle



of 75 v1.2

DCC Public

Note that incidents can be passed back to “Triage” from various steps (e.g. “Failed PIT”), but these links are not shown on the diagram in order to preserve clarity.



of 75 v1.2

DCC Public

9 Test Resources

9.1 Test Team

The diagram below shows the structure of the Test Team.

Figure 11 – Structure of Test Team

The roles in this structure are as follows:

Integration Test and Acceptance Manager

o responsible for all DSP Integration Test activities, including User Integration Testing

SP and RDP Test Managers

o responsible for PIT testing of their solution elements

o single point of contact for all design, development and technical queries on their solution elements

o provides “local” management of preparation and execution for SIT tests within their agreed test boundary

Security Architect

o responsible for advising on all Security aspects of testing

Configuration Manager

o owns the master Configuration Plan which defines a) the Configuration Items (CIs) comprising the Smart Meter eco-system and b) the inter-dependencies between these CIs

o responsible for configuration management of DSP CIs

Release Manager

o owns the master Release Schedule which shows a) when the various SP/RDP Releases are deployed to the SIT environments and b) the inter-dependencies between these releases



of 75 v1.2

DCC Public

o responsible for releases of DSP solution elements

Environment Manager

o owns the master Environment Plan which shows a) the architecture of the various SP/RDP SIT environments and b) the communications links between these environments

o responsible for the DSP SIT environment

Incident Manager

o responsible for chairing the IRB, expediting the resolution of outstanding incidents, agreeing which defect fixes go into which SIT Releases, and reporting progress to stakeholders

Test Architect

o responsible for designing Integration test scenarios and test data, supporting Integration test Planning, designing and assuring Integration test procedures

SIT Test Manager

o responsible for planning and control of SIT

Test Preparation Lead

o responsible for designing and building SIT test scripts

Test Execution Lead

o responsible for execution of SIT test scripts

Test Analysts

o responsible for writing and executing SIT test scripts

o sourced from each SP/RDP, to work on tests within their agreed test boundary

o numbers and profile to be determined as part of the “Estimating and Scheduling” activity shown on the High Level Plan in Section 6.2.

9.2 Test Tools

HP’s Quality Center Test Management tool will be used to manage SIT testing and SIT test incidents.

The following table lists the other test tools that may be used in SIT.

Tool Type Service Provider

JMeter Performance DSP

Gatling Performance DSP



of 75 v1.2

DCC Public

Tool Type Service Provider

Selenium Automation DSP

TBC Performance CSP N

TBC Automation CSP N

SilkPerformer Performance CSP C/S

LoadRunner Performance CSP C/S

JMeter Performance CSP C/S

QuickTestPro Automation CSP C/S

Selenium Automation CSP C/S

TBC TSP

TeamForge Collaboration DCC Licensee

Table 9 - Test Tools

The TSP test tools will be identified by the TSP Test Manager once test planning for SMKI is further advanced.

9.3 Test Stubs and Prototypes

The following table lists the Test Stubs and prototypes that will be used in SIT, along with the organisations that are responsible for their design and for their development.

Test Stub Design responsibility

Development responsibility

Comment

Service User Test Stub – functional

DSP DSP The DSP PIT Test Stub will act as the basis, but will need to be enhanced (e.g. to deal with SMKI).

If feasible, the real Parse & Correlate software will be used

Service User Test Stub – performance

DSP DSP The DSP PIT Test Stub could act as the basis, but will need to be enhanced

Smart Meter Test Stub

CSPs CSPs The CSP PIT Test Stubs could act as the basis, but will need significant enhancement (e.g. to deal with SMKI)

Protocol-certified Comms Hub prototype

CSPs CSPs

Comms Hub Test Stub

CSP CSP Only required if CSPs are unable to supply a protocol-certified Comms Hub prototype

Comms Hub/Smart Meter Test Stub – performance

CSPs CSPs

Table 10 - Test Stubs and Prototypes



of 75 v1.2

DCC Public

10 Roles & Responsibilities

10.1 General

All parties involved in SIT are expected to:

follow the SEC guidelines for “Good Industry Practice”, i.e. the exercise of that degree of skill, diligence, prudence and foresight which would reasonably and ordinarily be expected from a skilled and experienced person engaged in a similar type of undertaking as that Party under the same or similar circumstances

take all reasonable steps to facilitate achievement of the SIT Objective.

10.2 DSP as Systems Integrator

In its role as Systems Integrator, the DSP will lead SIT and is responsible for the following areas:

producing and maintaining the SIT Approach, the Solution Test Plan and the SP UAT Test Plan

ensuring that the SIT Approach, the Solution Test Plan and the SP UAT Test Plan align with the Joint Test Strategy

ensuring that SIT activities are carried out in line with the SIT Approach, the Solution Test Plan and the SP UAT Test Plan

overall planning and control of SIT, including chairing Quality Gates 3 (between FAT and Solution Test), 4 (between Solution Test and SP UAT) and 5 (between Solution Test and Interface Test)

maintaining Risk, Assumption, Issue and Dependency Logs

leading the design and creation of test scenarios, test scripts, test data and test environments

preparing test execution and environment usage schedules

supporting the other SPs and the RDPs in their assigned test preparation and execution activities

managing test incident resolution, and supporting SPs and RDPs in the resolution process

producing the Test Stage Plans, Test Specifications, Requirements Traceability Matrices, Progress Reports and Test Completion Reports

operating the master Configuration Management Plan (see Section 9.1)

operating the master Release Schedule (see Section 9.1)

operating the Environment Plan (see Section 9.1).



of 75 v1.2

DCC Public

10.3 Service Providers

The SPs (including the DSP in its role as provider of the DSP system elements, and the DCC Licensee as provider of Enterprise Systems) will:

support the DSP as Systems Integrator in:

o planning and control of SIT

o design and creation of test scenarios, test scripts, test data and test environments

o preparing test execution and environment usage schedules

o diagnosing test incidents

o producing the Test Stage Plans, Test Specifications, Requirements Traceability Matrices, Progress Reports and Test Completion Reports

o contributing to the master Configuration Management Plan

o contributing to the master Release schedule

o contributing to the Environment Plan

establish, maintain and control their own test environments, in terms of software/hardware configuration and access control

for tests within their agreed test boundary, under the direction of the Systems Integrator:

o execute and monitor test scripts

o capture evidence

o report progress

resolve test incidents for their solution elements and undertaking PIT testing (including regression testing) of any fixes required.

The CSPs will:

establish, maintain and control their own Test Labs

procure and install Comms Hubs in their Test Labs for use in SIT

install sets of Smart Metering equipment in their Test Labs for use in SIT

procure, install, maintain and support Comms Hubs and Smart Meter Test Stubs for use in SIT.

10.4 DCC Licensee

The DCC Licensee will:

comply with its obligations under the approved SIT Approach (this document)



of 75 v1.2

DCC Public

use its reasonable endeavours to ensure that SIT is completed as soon as is reasonably practicable to do so

procure sets of Smart Metering equipment from Meter Manufacturers for use in SIT

support the DSP in the planning, control and operation of SIT

assure SIT planning, preparation and execution activities undertaken by the Systems Integrator, Service Providers and RDPs as detailed in the Joint Test Strategy

review and approve the relevant Test Documents, and issue the Approval to Proceed certificates as described in Schedule 6.2 ([7] [8] [9]). This includes the need to approve the SIT Phase and Stage Completion Reports promptly

participate in Quality Gate Reviews as described in Section 6.6

agree with the DSP, other Service Providers and RDPs a subset of the Solution Tests to be witnessed in the UAT stage

witness the execution of these tests in UAT

define and implement a process to audit the achievement of SIT Exit Criteria

produce the Device Selection Methodology in collaboration with DSP and CSPs, and issue

produce and issue the DCC Licensee Business Processes

perform the role of Service Provider for DCC Enterprise systems such as Billing and BI/MI.

10.5 RDPs

The RDPs will:

support the Systems Integrator in:











of 75 v1.2

DCC Public



o capture evidence

o report progress


10.6 St Clements

St Clements will:

act as the representative of Electricity RDPs in testing discussions with the DSP and DCC Licensee

help co-ordinate test planning, preparation and execution required of the Electricity RDPs

support the Systems Integrator in:











o capture evidence

o report progress




of 75 v1.2

DCC Public

10.7 Network Operators

The Network Operators will:

ensure that its RDP/RDPs complies with its/their obligations under the approved SIT Approach.



of 75 v1.2

DCC Public

11 Environments and Labs

11.1 Environments

The following table shows the names of the test environments that will be used for SIT.

Test activity DSP CSP N CSP C/S

DCC Licensee

TSP Gas RDP

Elec. RDPs

Early SIT Sandpit Dev 2 Ref 1 Test N/A N/A N/A

SIT – functional SIT Pre-Prod Pre-Prod Test SMKI Test

Test Test

SIT – non-functional SIT or Pre-Prod

Pre-Prod Pre-Prod or Ref 2

Test SMKI Test

Test Various

Table 11 - Test Environments

Each SP/RDP is responsible for establishing, maintaining and controlling their own Test Environments.

Note that the impact on the above environment usage of overlapping the SP UAT Test Stage of SIT with the Interface Test Stage of User Integration Testing is currently being assessed.

11.2 Test Labs

Each CSP will provide a Test Lab housing Smart Meters, Comms Hubs and Smart Metering equipment for use during SIT.

11.2.1 CSP N

CSP N will use the Internal Arqiva Test Lab that will be individually connected to the Pre-Production environment for SIT (and the Dev 2 environment for Early SIT).

The Internal Test Lab will provide an environment that smart devices such as the Comms Hub and meters may be tested with the SMWAN Infrastructure and vice versa. It will be possible to configure each Comms Hub separately so that it connects to a specific Access Network base station, and Access Networks for each Test Environment (and the Production Environment if necessary) will be within radio range of the Test Labs.

This will allow flexibility in the number of devices connected within each Test Environment.

The Internal Test Lab will provide two mechanisms for loading smart meters within the facility. A limited number of meters will be connected to adjustable loads, which will allow the power consumption of the meter to be varied to a preconfigured setting. The remaining meters will be connected to static loads.

11.2.2 CSP C/S

Telefonica’s Test Lab in Slough hosts various sets of Smart Metering equipment i.e.

electricity meters, gas meters, In-Home Displays and Communication Hubs. These

equipment sets will be configurable in order that various environments can be



of 75 v1.2

DCC Public

connected and the configuration for dual fuel, single fuel, communications hubs

variants etc. can all be amended as necessary for any combination required for

testing. Note that once set up for a given environment, these sets of equipment

cannot be switched between environments.

The features of the Slough Test Bed\Lab include:

932 square metres floor space, capable of housing 380 cabinets

one dedicated Faraday cage, and 8 cages in total

49 dedicated test benches

GSM, GPRS and 3G have full network capability monitoring

Radio capability representative of live.



of 75 v1.2

DCC Public

12 Test Scenarios

The spreadsheet below contains the list of Test Scenarios identified to date for SIT. It is a work in progress, and requires further discussion between the SPs and RDPs.

The following key information is recorded for each Test Scenario:

Description

Responsibility for preparation and execution

Type (Normal, Exception, Alternative)

Verification method

Business importance

Technical probability

Priority

Final Test with Certified Comms Hub ?

Final Test with Certified Smart Meter ?

Mode of operation

Test variations

Pre-requisites

Comments

Command Variants/Test Conditions.

DT.0034 System Integration Test Scenarios v0.5.xlsx



of 75 v1.2

DCC Public

13 Test Incident Severities

The following table taken from the Joint Test Strategy lists the standard Test Incident Severities

Test Incident Severity

Description

1 An Incident which:

prevents a DCC Service User or large group of DCC Service Users from using the DCC Service User Systems;

has a critical adverse impact on the activities of the DCC;

could cause significant financial loss and/or disruption to the DCC services or DCC Service Users; or

results in any material loss or corruption of Data.

Non-exhaustive examples:

An Incident leading to Non-availability of the DCC Data Services;

An Incident leading to Non-availability of the CSP Core solution element(s); or

All test progress is blocked.


has a major (but not critical) adverse impact on the activities of the DCC but the service is still working at a reduced capacity; or

causes limited financial loss and/or disruption to the DCC


An Incident leading to Non-availability of the Network Management Centre;

An Incident leading to loss of resilience of the SMWAN Gateway;

Large areas of functionality will not be able to be tested; or

Testing not completely blocked but has significant impact.


has a major adverse impact on the activities of the DCC but which can be reduced to a moderate adverse impact through a work around; or

has a moderate adverse impact on the activities of the DCC.


Testing can progress but the work-around will impact test progress.


has a minor adverse impact on the activities of the DCC.


Minor service interruptions in the business process or functionality of the DCC Systems and / or service.



of 75 v1.2

DCC Public

Test Incident Severity

Description


has minimal impact to the activities of the DCC.

Non-exhaustive examples

Trivial incidents with workarounds which are noted for future releases but minimal impact of running existing services; or

Tests can still pass but there are cosmetic issues.

Table 12 - Test Incident Severities



of 75 v1.2

DCC Public

14 Device Selection Methodology

This is a placeholder section, to be populated by the DCC Licensee



of 75 v1.2

DCC Public

15 Auditing of SIT Exit Criteria

15.1 Procurement of Independent Test Auditor

To meet the requirement in the Smart Metering System & Equipment Testing – Consultation Response, and the SEC regarding the exit of SIT, the DCC Licensee will appoint an independent auditor (that is sufficiently independent of the DCC Licensee, the SPs and RDPs) to assess the attainment of the agreed exit criteria. The auditor will produce a report that will be provided to the SEC Panel for information.

To appoint the independent auditor the DCC Licensee will utilise the Audit and Assurance Framework that it has created for this purpose. The framework is a competitively tendered agreement, created to allow DCC Licensee to award all types of audit and assurance service requirements including technical audit. The agreement is in line with the DCC Licensee procurement strategy and enables the DCC Licensee to award work that meets the economic and efficient requirement of the license. A requirement can be met either via direct award process (using the framework evaluation criteria) or via mini-competition.

15.2 Independent Test Auditor Role

The independent auditor selected by the DCC Licensee will monitor the matters being tested pursuant to SIT, and confirm that the appropriate and agreed exit criteria has been achieved for a CSP Region or an RDP System.

The auditor will create an independent report in accordance with this SIT Approach document confirming that the exit criteria have been met in respect of that CSP Region or RDP System.

This report along with the DCC Licensee’s SIT Test Completion Report, will be distributed to the DECC, SEC Panel, and if required the Secretary of State to notify all parties to the end of SIT.



of 75 v1.2

DCC Public

16 Compliance with SEC 3

The spreadsheet below contains each of the SEC 3 statements that relate to SIT, and a reference to the Section number(s) in this document where text can be found to demonstrate compliance.

It is accurate as at v0.5 of this document, and will not be updated until SEC 4 has been concluded.

DT.0033 System Integration Test Approach - SEC 3 Traceability Matrix v0.5.xlsx

CONSULTATION Systems Integration Test (SIT) … · Smart Metering Programme Consultation on SIT...

Documents

Transcript of CONSULTATION Systems Integration Test (SIT) … · Smart Metering Programme Consultation on SIT...