Laboratory capacity and verification testing report

Prepared for the

Vietnam Energy Efficiency Standards and Labelling (VEESL) Program

Produced by

Chris Evans, Kevin Lane

399 Silbury Boulevard, Milton Keynes

MK9 2AH, United Kingdom

Email: enquiries@S2E4.com

Telephone: +44 (0)1908 303 600

www.S2E4.com

June 2014

Table of Contents

1 INTRODUCTION ................................................................................................................................... 7

2 EXISTING ACCREDITATION AND APPROVAL OF TESTING ORGANISATIONS ............................................ 9

2.1 ACCREDITATION BY THE BOA .............................................................................................................................. 9 2.2 APPROVAL BY MOIT ......................................................................................................................................... 9

3 PROVISION OF EXPERT TRAINING ...................................................................................................... 10

3.1 ELECTRIC FANS ............................................................................................................................................... 11 3.1.1 Report back to VEESL and MOIT from the electric fans expert ........................................................... 11

3.2 RICE COOKERS ............................................................................................................................................... 13 3.2.1 Report back to VEESL and MOIT from the rice cookers expert ........................................................... 13

3.3 COMPACT FLUORESCENT LAMPS ........................................................................................................................ 14 3.3.1 Report back to VEESL and MOIT from the CFLs expert ....................................................................... 15

4 INTER-LABORATORY COMPARISON TESTS ......................................................................................... 17

4.1 RESULTS OF WITNESS TESTING ........................................................................................................................... 17 4.2 RESULTS OF TESTING ELECTRIC FANS ................................................................................................................... 18

4.2.1 Discussion of results of testing electric fans ....................................................................................... 21 4.3 RESULTS OF TESTING RICE COOKERS .................................................................................................................... 23

4.3.1 Discussion of results of testing rice cookers ....................................................................................... 26 4.4 RESULTS OF TESTING CFLS ............................................................................................................................... 28

4.4.1 Discussion of results of testing CFLs ................................................................................................... 30

5 CONCLUSIONS AND RECOMMENDATIONS ......................................................................................... 33

5.1 PRODUCT TESTING RELATED ADVICE FOR EACH OF THE QUATEST LABORATORIES ....................................................... 33 5.2 RECOMMENDATIONS APPLICABLE TO ALL APPROVED TESTING ORGANISATIONS IN VIETNAM ........................................... 34 5.3 THE ACCREDITATION OPERATED BY THE VIETNAMESE BOARD OF ACCREDITATION (BOA) .............................................. 34 5.4 SUPPORT FOR THE MOIT APPROVAL PROCESS ...................................................................................................... 34 5.5 RECASTING THRESHOLD FOR ENERGY LABEL BANDS ................................................................................................ 35

APPENDIX 1 EXAMPLE OF TEST RESULTS SHEET FOR ONE MODEL OF ELECTRIC FAN .............................. 36

APPENDIX 2 EXAMPLE OF TEST RESULTS SHEET FOR ONE MODEL OF RICE COOKER ............................... 37

APPENDIX 3 EXAMPLE OF TEST RESULTS SHEET FOR ONE MODEL OF CFL .............................................. 38

2

List of Figures Figure 1: Testing of electric fans with support from the expert .............................................. 11 Figure 2: Testing of rice cookers with support from the expert .............................................. 13 Figure 3: Testing of CFLs with support of the expert ........................................................... 15 Figure 4: Electric fan manufacturer label and the energy label .............................................. 19 Figure 5: Star rating of the 20 fans as measured by the three laboratories ............................. 20 Figure 6: Maximum-to-minimum star rating of the 20 fans as measured by the three laboratories ................................................................................................................................... 20 Figure 7: Reported minimum-to-maximum efficiency of the 20 fans, by three laboratories ........ 22 Figure 8: Rice cooker and its energy label ......................................................................... 23 Figure 9: Star rating of the 20 rice cookers as measured by the three laboratories .................. 25 Figure 10: Maximum-to-minimum star rating of the 20 rice cookers as measured by the three laboratories .................................................................................................................. 25 Figure 11: Reported average cooking efficiency (%) of the 20 models of rice cookers .............. 26 Figure 12: Example of CFL label ....................................................................................... 29 Figure 13: Average efficacy (lm/W) measured for the 20 models of CFLs ............................... 30 Figure 14: Average measured power (W) of the 20 CFL models ............................................ 31 Figure 15: Measured/reported initial luminous flux (lm) of the 20 CFL models ......................... 32 List of Tables Table 1: Summary of findings by electric fans testing expert ................................................ 12 Table 2: Summary of findings by rice cookers testing expert ................................................ 14 Table 3: Summary of findings by CFLs testing expert .......................................................... 16 Table 4: Brand/model list of electric fans tested ................................................................. 18 Table 5: Brand/model list of rice cookers tested ................................................................. 24 Table 6: Energy efficiency bands given in TCVN 8252:2009 .................................................. 27 Table 7: Brand/model/size list of CFLs tested ..................................................................... 28 Table 8: Difference between the average efficacies of the tested CFL lamps ............................ 30 Table 9: Example of test results sheet for one model of electric fan ....................................... 36 Table 10: Example of test results sheet for one model of rice cooker ..................................... 37 Table 11: Example of test results sheet for one model of CFL ............................................... 38

Disclaimer The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any agency of the Australian Government. The authors have made their best endeavours to ensure the accuracy and reliability of the data used herein, however make no warranties as to the accuracy of data herein nor accept any liability for any action taken or decision made based on the contents of this report.

3

Executive summary The Australian Government Department of Industry (DoI) is providing assistance to Vietnam through the Vietnam Energy Efficiency Standards and Labelling (VEESL) Program, funded by the Australian Agency for International Development (AusAID). This report has been compiled by consultants with extensive experience of product testing who are providing direct assistance and advice to the VESSL Program. It provides results for that part of the Program intended to help build laboratory testing capacity in three state-owned QUATEST laboratories. The work being reported was undertaken in partnership with Vietnam’s Ministry of Industry and Trade (MOIT) and comprised of two main components:

• Visits to the test laboratories, QUATEST 1, 2, 3, by experts in the energy efficiency performance testing of electric fans, rice cookers and compact fluorescent lamps (CFLs). The purpose of the visits being to provide supplementary training to the already experienced laboratory testing staff;

• The inter-laboratory testing of 20 models of electric fans, rice cookers and CFLs (which followed on from the visits by the experts).

Achieving accurate and consistent test results by these three laboratories is crucial for effective standards and labelling in Vietnam. Methods for measuring declared values for energy consumption and performance characteristics must be of sufficient accuracy to provide confidence to governments, consumers and manufacturers. Any significant variations in the reproducibility1 of results achieved by test laboratories could reduce the effectiveness of the implementation of the MEPS and labelling regulations. It must be recognised that some variation in measurement is unavoidable. This can be due to the variance of the product itself and to the measurement method applied. While the first one is the responsibility of the manufacturer, the second one is the responsibility of either or both the test laboratory(s) and the organisation publishing the test standard. The part of the Program reported here began with the supplementary training visits made to the three QUATEST laboratories. In each case, the experts provided advice focussed on making improvements to testing procedures - the precise application of testing methods, the use of calibrated measuring equipment and the routine assessment of sources of errors of measurement.

1 reproducibility conditions

conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment

[IEC 61923:1997, 3.9]

4

The results of the inter-laboratory testing that then followed were mixed. Those of multiple samples of the same models of CFLs showed, with the exception of consistently lower results from one of the QUATESTs, good reproducibility – as should be expected. The results of the testing of electric fans and rice cookers were much less reproducible with large and inconsistent variations in performance measured for the same samples of some models. The experts identified a number of reasons, which could have contributed to these variations, which were the responsibility of the test laboratories:

• Some inconsistencies in the precise application of techniques between the laboratories; • Some equipment not being as specified in the test standard;

• Some equipment not being calibrated.

Additionally, in the case of the rice cookers, the expert identified that the published test method needed amending as it could not be undertaken in ways that avoided significant errors of measurement. The challenges faced by laboratories when conducting performance tests on products should not be underestimated. Relatively small changes in procedures or in the accuracy of measuring instruments can have a significant impact on the energy efficiency ratings that are subsequently calculated from the laboratory test results. Implementation of the advice that the experts gave to each of the QUATEST laboratories was intended to have a positive impact on improving the consistency of test results. The authors of this report recommend further actions are taken. A multi-agency approach, in which the Vietnam Board of Accreditation (BoA) takes part, should implement a programme to assist the test laboratories to improve the suitability and calibration of their measuring equipment. In addition, estimating and eliminating sources of measurement errors are also important actions to be taken.

Some specific recommendations:

• Amend the measurement standard TCVN 8252:2009 for rice cookers in order to improve its repeatability2;

• Assist the BoA to implement best practice when accrediting test laboratories to ISO/IEC17025;

2 repeatability conditions conditions where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time [IEC 61923:1997, 3.6]

5

• Introduce a requirement that inter-laboratory trials are to be undertaken for each new test standard to identify and resolve any inconsistencies in application of the test methodology and so ensure that reproducible test results are consistently obtained;

• Each QUATEST to implement a procedure under which uncertainties of measurement3 are estimated for all the test programs they undertake.

• Further and specific refinements to laboratory procedures should be made at each QUATEST. The guidance provided by the testing experts and the comments presented in this report are relevant to this.

• Request the BoA to then undertake a re-accreditation program (which would normally be undertaken annually) for the QUATEST laboratories;

3 The expression “uncertainties of measurement”, which is used frequently in this report, deals with assessing the variations in results that can occur when making a measurement. Testing a product to measure its performance will always be affected by errors due to instrumentation, methodology, presence of confounding effects and so on. Consequently, experimental uncertainty estimates need to be undertaken by test laboratories to identify (and, where possible, reduce) the different sources of errors and to provide an assessment of the confidence in the results. It is a requirement of the international standard used for accrediting test laboratories, ISO/IEC 17025 General requirements for the competence of testing and calibration laboratories, that testing laboratories have and apply procedures for estimating uncertainty of measurement.

6

1 Introduction

The Government of Vietnam has introduced legislation to implement Minimum Energy Performance Standards (MEPS) for appliances and equipment alongside a program of Energy Labelling. The Australian Government is supporting the Vietnam Energy Efficiency Standards and Labelling (VEESL) Program, using funds supplied by the Australian Agency for International Development (AusAID) and the Australian Department of Industry (DoI). These funds have been used to employ consultants with extensive experience in this area who are providing direct assistance and advice to the Australian and Vietnamese Government officials managing VESSL. The aim of the VEESL Program is to improve prosperity in Vietnam while increasing energy security and reducing carbon emissions. This will be achieved through a range of activities designed to build sustainable capacity within Vietnam to set and enforce appliance and equipment energy standards, and monitor and evaluate the MEPS and Energy Labelling Program. Two of the main activities, within the scope of this report, are to:

• Support the monitoring of compliance through verification testing; • Support the development of local test capacity and technical and commercial skills in

laboratories through training, and through inter-laboratory (round robin) testing. The legislation to implement MEPS and Energy Labelling for appliances and equipment became effective during 2013. This has required new products to meet MEPS levels and, in many cases, being required to display an energy label at the point of sale. All affected products must be compliant with the legally imposed performance requirements. Though the responsibility for this lies with the product supplier, the Vietnamese market surveillance authorities also have a responsibility for checking that these products are in compliance with these requirements. Some checks, such as whether a label is being correctly displayed, can be made at the point of sale4 but the technical performance of the product can only be checked through tests conducted by a suitably skilled and equipped testing organisation. Since the results of these check tests may be used as evidence for enforcement action, it is vital that tests are conducted accurately according to the specified test methodology, and that the results are sufficiently repeatable and reproducible. Much of the current responsibility for conducting compliance tests is likely to be with the state testing organisations, Quality Assurance and Testing Centre (QUATEST) 1, 2 & 3, and a program was developed on behalf of VEESL to support these organisations. The VEESL program reported here was undertaken in partnership with the Vietnam Ministry of Industry and Trade (MOIT). It consisted of the following components:

4 Survey of Energy Efficiency Labelling of Appliances in Vietnam was published by VEESL in April 2014

7

• Providing supplementary training for laboratory staff undertaking the testing of electric fans, rice cookers and compact fluorescent lamps (CFLs);

• Witnessing of the testing of these products by the experts that had provided the training;

• Production of testing methodological guides for electric fans and rice cookers; • Inter-laboratory testing of 20 models of electric fans, rice cookers and CFLs.

Products were purchased from the Vietnamese market for testing. The results of testing were intended to enable the program to achieve the following:

• An exploration of whether the standard test procedures, their regulated performance levels and the testing being conducted were sufficient to deliver the performance standards required by the Vietnamese authorities; and, if so,

• A check on whether these products were compliant with their regulatory requirements and performed as claimed on their energy labels.

8

2 Existing accreditation and approval of testing organisations

There are two official accreditation or approval programs that are applicable to testing organisations in Vietnam. Both have been applied to the QUATEST laboratories. The first is based on the accreditation process managed by the Bureau of Accreditation Vietnam (BoA). The second, approval to undertake the tests required by the standards and labelling regulations, is undertaken by MOIT.

2.1 Accreditation by the BoA

The BoA is the official body in Vietnam responsible for accrediting certification bodies under the ISO/IEC Guide 65 and specifically for accrediting test laboratories in accordance with ISO/IEC 17025. The BoA has international recognition as the Vietnamese member of the International Accreditation Forum (IAF), Asia Pacific Laboratory Accreditation Cooperation (APLAC), International Laboratory Accreditation Cooperation (ILAC) and the Pacific Accreditation Cooperation (PAC). QUATEST laboratories 1, 2 & 3 each have a generic accreditation certificate for ISO/IEC 17025 issued by the BoA. These certificates do not state which specific TCVN standards that the laboratories have been accredited for (accreditation under ISO/IEC 17025 is specific to identified standards; it cannot be applied generically to the laboratory). Examination of the content of the BoA’s website, in which detailed schedules of accredited tests are displayed for each organisation, reveals that the detailed lists they publish are not up to date. Enquiries are on-going, but currently it has not been possible to confirm whether each QUATEST has been specifically accredited to ISO/IEC 17025 for testing the energy efficiency of electric fans, rice cookers and CFLs.

2.2 Approval by MOIT

An approval process of testing organisations is undertaken by MOIT. This is undertaken separately to the ISO/IEC based accreditation process undertaken by the BoA. The VEESL program has previously provided guidance to MOIT that was intended to assist them to conduct audits of testing organisations. This guidance was prepared to ensure the quality systems operated by approved testing organisations extended beyond the minimum requirements required by ISO/IEC standards.

9

3 Provision of expert training

The QUATEST laboratories were already able to test electric fans, rice cookers and CFLs so were seeking expert guidance to help them further refine their expertise. International experts for each product group were commissioned to support the VEESL program. The role the experts were asked to adopt was one of providing support and training rather than one of critique and supervision. To do this, each expert separately spent three days in November 2013 at each QUATEST and undertook a similar program based on the following framework: • Inspect the current test arrangements and compare with those required by the applicable

TCVN standard. Make note of any deficiencies and, where practicable, assist the staff to remedy them. Where possible, assist the staff to improve these to best practice standards.

• Check the samples to be witnessed tested by the laboratories. These should have been selected in advance by MOIT and consist of a minimum of three different models taken from the retail market.

• Assist the staff to conduct tests and record results, providing guidance and training to correct any deficient practice wherever possible.

• Formally witness the testing (i.e. post-training) of the three models. Note any deficiencies in the tests carried out and take a record of the results obtained.

• Throughout the period spent at each laboratory, note any other aspects of laboratory management that fall short of best practice and which may in some way prevent the laboratory achieving consistent results in the testing it does on the other product categories. Provide briefings to laboratory management where improvements could be made.

• At the completion of testing at the first laboratory, supervise the packing of the three models tested5 and ensure arrangements have been made to immediately ship these samples to the next laboratory to be visited.

• Provide report backs to the laboratory management using a Laboratory Report Template. This report must also be provided in a verbal briefing to senior laboratory management at the end of each training visit.

• Provide a training mission report back to VEESL and MOIT using a Training Mission Report Template.

• Use, where appropriate, materials from the laboratories’ existing testing guidance manuals, to produce a test guidance manual for the product under review. This document is intended to replace the individual guides currently being used and so ensure that all laboratory staff followed the same guidance in future. (This requirement applied to electric fans and rice cookers only.)

5 For rice cookers and electric fans; three separate batches of the same model were provided to each laboratory for CFLs

10

3.1 Electric fans

The expert engaged to provide support and training was: Johnny Chow Technical Manager Intertek-Consumer & Commercial Electrical Hong Kong

Training was focussed on the requirements of the applicable Vietnamese national standards TCVN 7826, and TCVN 7827.

Figure 1: Testing of electric fans with support from the expert

3.1.1 Report back to VEESL and MOIT from the electric fans expert

In addition to supplying an individual verbal and written report back to the managers of each QUATEST laboratory, Mr Chow submitted a written report of his key findings to VEESL. A summary of the content of each of these reports is provided in Table 1. These summaries cover whether the conditions for undertaking the test were as required by the relevant test standard, whether the measuring equipment had been calibrated (i.e. checked for accuracy), whether the staff were evaluating and estimating the applicable uncertainties of measurement and whether the test was being conducted exactly as required. All of these must be undertaken correctly in order to achieve repeatable and reproducible test measurements.

11

Table 1: Summary of findings by electric fans testing expert

QUERIES QUATEST1 QUATEST2 QUATEST3 Were all the conditions specified in the test method for undertaking the tests found to be satisfactory when expert arrived?

No. Significant defects were found

No. Significant defects were found

No. Significant defects were found

Had all measuring equipment been calibrated?

No Yes Yes

Were uncertainties of measurement being estimated?

No No No

Were significant improvements in undertaking the test procedures achieved during the visit of the expert?

Yes. Most improvements that were possible to make in the time available were to improve the alignment of the test fan with the positioning of the measuring instrument.

Yes. Most improvements that were possible to make in the time available were to improve the alignment of the test fan with the positioning of the measuring instrument.

Yes. Most improvements that were possible to make in the time available were to improve the alignment of the test fan with the positioning of the measuring instrument.

What are the main further actions recommended for the laboratories to take?

• Test rooms must be draught-free and of the correct dimensions; all substantial gaps in the wall fabric need to be eliminated.

• The accuracy of the test result is dependent on the alignment of the test fan with the positioning of the measuring instrument. The current inaccurate alignment marking on the floors of the chambers need to be removed and laser pointers used to obtain and constantly check the necessary precision of alignment.

• The wind speed meters used must meet the requirements given in the test standard.

• The laboratories must apply measurement uncertainties procedures and so identify and, where possible, reduce sources of measurement error. This is a requirement of the accreditation standard ISO/IEC17025.

12

3.2 Rice cookers

The expert engaged to provide support and training was: Michael Cheng Technical Manager Intertek-Consumer & Commercial Electrical Hong Kong

Training was focussed on the requirements of the applicable Vietnamese national standard TCVN 8252.

Figure 2: Testing of rice cookers with support from the expert

3.2.1 Report back to VEESL and MOIT from the rice cookers expert

In addition to supplying an individual verbal and written report back to the managers of each QUATEST laboratory, Mr Cheng submitted a written report of his key findings to VEESL. A summary of the content of each of these reports is provided in Table 2. These summaries cover whether the conditions for undertaking the test were as required by the relevant test standard, whether the measuring equipment had been calibrated (i.e. checked for accuracy), whether the staff were evaluating and estimating the applicable uncertainties of measurement and whether the test was being conducted exactly as required. All of these must be undertaken correctly in order to achieve repeatable and reproducible test measurements.

13

Table 2: Summary of findings by rice cookers testing expert

QUERIES QUATEST1 QUATEST2 QUATEST3 Were all the conditions specified in the test method for undertaking the tests found to be satisfactory when expert arrived?

No. Significant defects were found.

No. Significant defects were found.

Yes

Had all measuring equipment been calibrated?

No No Yes

Were uncertainties of measurement being estimated?

No No No

Were significant improvements in test procedures achieved during the visit of the expert?

Yes. Primarily to the detailed application of test method.

Yes. Primarily to the detailed application of test method.

Yes. Primarily to the detailed application of test method.

What are the main further actions recommended for the laboratories to take?

• The accuracy of the test result is dependent on temperature measurement and the way in which stirring (of the contents of the rice cooker) needs to be undertaken to achieve a uniform temperature. The expert has developed test guidance for all the laboratories to follow in order to achieve this in a more consistent way.

• The testing needs to take place in substantially draught-free areas. • The weighing equipment needs to be upgraded.

• The laboratories must apply measurement uncertainties procedures and so identify and, where possible, reduce sources of measurement error. This is a requirement of the accreditation standard ISO/IEC17025.

3.3 Compact fluorescent lamps

The expert engaged to provide support and training was: Shuming Hua Chief Executive Global Efficient Lighting Centre Beijing

14

Training was focussed on the requirements of the applicable Vietnamese national standard TCVN 7896.

Figure 3: Testing of CFLs with support of the expert

3.3.1 Report back to VEESL and MOIT from the CFLs expert

In addition to supplying an individual verbal and written report back to the managers of each QUATEST, Mr Hua submitted a written report of his key findings to VEESL. A summary of the content of his report is provided in Table 3. These summaries cover whether the conditions for undertaking the test were as required by the relevant test standard, whether the measuring equipment had been calibrated (i.e. checked for accuracy), whether the staff were evaluating and estimating the applicable uncertainties of measurement and whether the test was being conducted exactly as required. All of these must be undertaken correctly in order to achieve repeatable and reproducible test measurements.

15

Table 3: Summary of findings by CFLs testing expert

QUERIES QUATEST1 QUATEST2 QUATEST3

Were all the conditions specified in the test method for undertaking the tests found to be satisfactory when expert arrived?

No. Significant defects were found.

No. Significant defects were found.

No. Significant defects were found.

Had all measuring equipment been calibrated?

No No No

Were uncertainties of measurement being estimated?

No No No

Were significant improvements in test procedures achieved during the visit of the expert?

Yes. Primarily because the testers had not previously received sufficient training for testing lamps.

Yes. Primarily because the testers had not previously received sufficient training for testing lamps.

Yes. Primarily because the testers had not previously received sufficient training for testing lamps.

What are the main further actions recommended for the laboratories to take?

• Staff to receive more training for testing lamps and (in some laboratories) to conduct testing more frequently in order to build experience and maintain skills.

• Test equipment needs to be managed better, it needs to be calibrated and included within a test instrument quality system.

• Some of the test equipment should be updated. • The laboratories must apply measurement uncertainties procedures

and so identify and, where possible, reduce sources of measurement error. This is a requirement of the accreditation standard ISO/IEC17025.

16

4 Inter-laboratory comparison tests

The three QUATEST laboratories undertook inter-laboratory comparisons. Each tested the same samples of 20 different models of electric fans and rice cookers. Testing of CFLs was undertaken on 30 samples of 20 different models requiring a total of 600 CFLs to be obtained. For each of the 20 models, the 30 samples were subdivided into three batches of 10 samples; one batch went to each QUATEST laboratory. The main purposes of these trials were to provide:

• A measure of the extent of the variation and uncertainty of results; • An understanding of the cause of these variations in order to…

• Be able to advise MOIT and the laboratories on steps that could be taken to minimise these variations and so achieve repeatable and reproducible results.

The first, of two, phases of the inter-laboratory comparison tests was initiated when the expert trainers were present at the QUATEST laboratories. Part of their role was to witness the initial tests being undertaken. These tests were performed after the experts had provided some training/guidance and had assisted the testing staff to achieve the best possible set-up using the test equipment available at the time. The second phase of the inter-laboratory comparison testing took place after the experts had departed. The laboratory staff then tested all the products using their existing knowledge supplemented by the expert training.

4.1 Results of witness testing

Each expert was required to formally observe (“witness”) the testing of a selection of products that they had trained the laboratory staff to test. This had two purposes:

• To enable the experts to review how well the tests were being conducted and so note whether the advice and training they had been given previously had improved the ways in which the tests were performed; and

• To establish the organisation of the inter-laboratory comparison tests that would continue to take place after their departure.

Each expert submitted detailed report test sheets to VEESL of the test results he witnessed being measured. An example of one of the result sheets for each of the three product types is given in Appendices 1-3. The content of the experts’ reports, which mainly deal with sources of measurement uncertainty i.e. the reasons for variations in results obtained from reproduced tests of the same product, are discussed later in this report. The measurements obtained in the tests that were witnessed have not been reported here since they have been superseded by those that were subsequently repeated by the three QUATEST laboratories following them making further improvements to their testing procedures.

17

4.2 Results of testing electric fans

Twenty different models of electric fan were tested, after the expert training had been provided. The brand and model identifications are listed on Table 4. Table 4: Brand/model list of electric fans tested

Reference number Fan name 1 ĐIỆN CƠ TICO: B300 2 CHING HAI: BF168 3 LIOA: QB-409K 4 CHING HAI: HS802 5 SHARP: PJT1621V 6 ĐIỆN CƠ TICO: B400 7 VINAWIND: QĐ400-MS 8 LIOA: QT-409KLG 9 VINA ASIA: L16003

10 ĐIỆN CƠ HD: HĐ-450 (treo) 11 SANAKY: SK-16C 12 ĐIỆN CƠ 91: QB-450 (dung) 13 VINAWIND: QĐ450-ĐM 14 DAIICHI: DC-SF525 15 DAIICHI: DC-SF535 16 HOTOFAN: QĐC400 17 DAIICHI: DC-SF650 18 LIOA, MODEL: Q-409K 19 ĐIỆN CƠ TICO: B300 20 ĐIỆN CƠ 91: QĐ-CN450P

They were a mixture of table top and pedestal fans; there was one wall fan. Blade diameters ranged from 300-450mm; input powers were in the range 35-65W (example shown in Figure 4).

18

Figure 4: Electric fan manufacturer label and the energy label

Each sample was measured in accordance with TCVN 7827:2007, the official test method for measuring the airflow performance of these types of product. The results obtained were then used in combination with TCVN 7826:2007, which provides the levels of energy efficiency of the fan (k) and the corresponding energy label star rating. Figure 5 gives the reported level of energy efficiency or star rating (based on R6) of the 20 fans by the three laboratories.

6 TCVN 7826 and TCVN 7827 use “k” and “R” to mean the same thing (ratio of measured efficiency to minimum requirement), which some users may find confusing.

19

Figure 5: Star rating of the 20 fans as measured by the three laboratories

There is clearly a difference between the star ratings measured by the laboratories for some of the fans. This may be more clearly seen if the measured star range for each fan is only plotted as a line.

Figure 6: Maximum-to-minimum star rating of the 20 fans as measured by the three laboratories

20

4.2.1 Discussion of results of testing electric fans

The variation in results for the star rating for the label, reported in Figures 5 & 6 is very large for some of the models tested. Since the tests were conducted on the same samples, using the same test procedure and the same guidance manual was available to follow, the following may have

contributed to the variation:

1. Large measurement uncertainties being caused by the test method not being reproduced in exactly the same way by each of the three QUATESTs;

2. Large measurement uncertainties being caused by the test method because it is not possible to repeat it in a consistent way;

3. The threshold values for the energy label levels, the star values, are too compressed with the consequence that small differences in performance measurements are increased through being multiplied in the labelling calculation, which results in having the effect of exaggerating the real difference in energy efficiency.

Examining each of these in turn: Large measurement uncertainties being caused by the test method not being reproduced in exactly the same way by each of the three QUATESTs The training expert identified this as the most significant factor at the time when the very first tests were being conducted. The following causes of measurement uncertainties were reported:

• The wind speed measuring instrument was not being accurately aligned with the wind flow from the fan under test;

• None of the test chambers at the three QUATESTs were fully correct as they did not completely eliminate draughts from external sources and, in one case, the chamber was of incorrect dimensions;

• The wind speed test instruments used by two of the QUATESTs did not meet the specification given in TCVN 7827;

• Some equipment had not been calibrated;

• Some wind flow readings had to be taken from the wind speed measuring instrument manually which meant that test personnel had to be in the vicinity of the wind flow being measured, a factor that could interfere with the wind flow being measured.

All of these factors, which were possibly contributing to increasing the level of measurement uncertainties, were fully identified to the QUATEST staff prior to them conducting the performance tests that are reported in Figure 5 & 6. At the time of writing this report, it was not known to what extent remedial actions had been undertaken by any of the three QUATESTs in the period following the departure of the training expert and prior to them conducting the main set of tests.

21

Large measurement uncertainties being caused by the test method because it is not possible to repeat it in a consistent way The opinion of the training expert was that this was not a significant factor. However, the wind flow from the sample under test has to be meticulously measured under precisely maintained conditions. The laboratory testing staff must understand that this test procedure has to be undertaken using methods requiring a high level of precision. The only enhancement recommended for the basic test methodology was for a guidance note to be added to TCVN 7827 to draw the attention of testing staff to the importance of accurately aligning the wind flow measuring instrument with the fan being tested and the importance of ensuring nobody is standing in the vicinity of the air flow when it is being measured. Given the sensitivity of the measurement process, an additional precaution would be to require the test to be conducted under repeatability conditions to ensure that the results being obtained are consistent. The banding scales for levels of energy efficiency are too compressed with the consequence that small differences in performance measurements are accounting for large (exaggerated) differences in the levels of energy efficiency The average difference in the k values between the minimum and maximum values for each fan is 0.4 (Figure 7). Since the current labelling scale has increments of 0.1 i.e. a 10% increase in efficiency at each level, the variation in test results from the three laboratories is, on average, spanning four levels of efficiency. It is clear from this variation, that the results of inter-laboratory tests were not sufficiently reproducible to support the current energy label.

Figure 7: Reported minimum-to-maximum efficiency of the 20 fans, by three laboratories

22

Furthermore, most of fans comfortably exceed the best performance requirements required in the energy labelling regulations as can be seen from the test results of the samples displayed in Figure 7. An examination of the MOIT registration database for electric fans confirms that almost all the fans listed have 5 stars. It is clear that the thresholds need to be recast if they are to provide any meaningful signals to the market, and be useful to policy makers. The re-grading of the energy label should be done using other available evidence, e.g. examination of the latest registration database, examination of other countries regulations on electric fans, and finding best-practice products. Ideally, this re-grading process would be done after the laboratories have succeeded in improving the level of reproducibility and so further reduce the level of test result differences between themselves.

4.3 Results of testing rice cookers

Twenty different models of rice cookers were tested. All were table top designs with capacities ranging from 1 – 3 litres; input powers ranging from 350-1000W (an example is shown in Figure 8).

Figure 8: Rice cooker and its energy label

The brand and model identifications of the twenty models tested are listed in Table 5.

23

Table 5: Brand/model list of rice cookers tested

Reference number Rice cooker name 1 SHARP KSH-211 2 HIỆP HƯNG: HH105 3 SUPOR: CFXB50YB5B-70 4 SUPO VIET: SP-30R 5 GOLDSUN: ARC-G18CP1 6 ELECPOR: SR-P1C 7 HONEY’S: H0901-M22D 8 SUPO VIET: SP-22R 9 GOLDSUN: ARC-GDF18C2

10 DAEWOO: DFC-5018 11 COMET: CM8018 12 SHARP: KSH-219V 13 AIDI: MR-SM07J 14 SUPO VIET: RC-18SB 15 DAELUX: DXR50J-G 16 CARIBBEAN: CRC-182 17 SHARP: KSH-222 18 KANGAROO: KG376 19 SUPO VIET: SP-12R 20 MIDEA: MR-CM18SD

Each sample was measured in accordance with TCVN 8252:2009, the official test method for measuring the energy efficiency performance (K) of these types of product. Figure 9 gives the reported energy efficiency grade or star rating (based on the Energy Efficiency Index, K) of the 20 rice cookers by the three laboratories.

24

Figure 9: Star rating of the 20 rice cookers as measured by the three laboratories

There is clearly a difference between the laboratories for some of the rice cookers. This may be more clearly seen if the measured star range for each rice cooker is only plotted as a line (Figure 10).

Figure 10: Maximum-to-minimum star rating of the 20 rice cookers as measured by the three laboratories

25

The large variation in the star rating is mainly due to the large variation in the measured cooking efficiency. NOTE: the efficiency reported by QUATEST2 as exceeding 100% is assumed to be an error.

Figure 11: Reported average cooking efficiency (%) of the 20 models of rice cookers

4.3.1 Discussion of results of testing rice cookers

The variation in results for the star rating for the label, reported in Figure 9 and Figure 10, is very large for some of the models tested. Since the tests were conducted on the same samples using the same test procedure and the same guidance manual was available to follow, the causes for the variation are expected to be one or some of the following:

1. Large measurement uncertainties being caused by the test method not being reproduced in exactly the same way by each of the three QUATESTs;

2. Large measurement uncertainties being caused by the test method because it is not possible to repeat it in a consistent way;

3. The threshold values for the energy label levels, the star values, are too compressed with the consequence that small differences in performance measurements are increased through being multiplied in the labelling calculation, which results in having the effect of exaggerating the real difference in energy efficiency.

Examining each of these in turn:

26

Large measurement uncertainties being caused by the test method not being reproduced in exactly the same way by each of the three QUATESTs; The training expert identified this as one of the two most significant factors at the time when the very first tests were being conducted. The main cause of the measurement uncertainties being that the accuracy of the test result is dependent on temperature measurement. The stirring of the contents of the rice cooker in a thorough and consistent way in order to achieve a uniform temperature distribution is crucial to this and, perhaps not surprisingly, different laboratory staff undertake this manual activity in slightly different ways. Large measurement uncertainties being caused by the test method because it is not possible to repeat it in a consistent way The training expert confirmed that this was the other significant factor. For example, the measurement uncertainty of 0.5oC permitted by the existing standard already accounts for one label grade. Furthermore, the standard needs to be changed to make it less sensitive to temperature measurement. This could be achieved by using a longer heating time in order to make the temperature rise larger. This would ensure that the label grade calculation becomes less sensitive to the temperatures being measured. Alternatively, the test method could be changed to measuring the energy required to raise the water temperature in the rice cooker for a larger fixed value, say 50K. The banding scales for the energy efficiency grades are too compressed with the consequence that small differences in performance measurements are accounting for large (exaggerated) differences in the final energy efficiency grade The current grade performance thresholds given in TCVN 8252 appear very compressed, see Table 6 below. The impact of this is significant since a change in measured temperature as small as 0.5oC results in a change of label grade. Table 6: Energy efficiency bands given in TCVN 8252:2009

Grade Energy Efficiency Index

K

1 ≥ 1,00

2 ≥ 1,05

3 ≥ 1,10

4 ≥ 1,15

5 ≥ 1,20

However, examination of the MOIT registration database for rice cookers shows a wide spread of ratings from 1 through to 5 stars i.e. most rice cookers do not exceed the best performance requirements required in the energy labelling regulations.

27

Consequently, from a marketplace perspective, there is no requirement yet to recast the thresholds and this leads to the conclusion that expanding the banding scales would not provide a practicable solution for dealing with the spread of performance results reported by the three QUATEST laboratories.

4.4 Results of testing CFLs

Twenty different models of CFLs were tested, with each of the three laboratories having 10 of each model to test from the same purchased batch of 30 samples. The lamps were not transferred between three QUATEST laboratories. Brand and model identifications are listed on Table 7. Rated powers were in the range 5-23W. Some models claimed to have high efficiency performance and so were qualified to display the endorsement energy label. Table 7: Brand/model/size list of CFLs tested

Reference number CFL name 1 Rạng Đông 5W 2 Rạng Đông 11W 3 Rạng Đông 15W 4 Rạng Đông 20W 5 Rạng Đông 40W 6 PHILIPS, GENIE 5W 7 PHILIPS, GENIE 11W 8 PHILIPS, GENIE 14W 9 PHILIPS, Essential 18W

10 PHILIPS, Essential 23W 11 Điện quang 5W 12 Điện quang 8W 13 Điện quang 14W 14 Điện quang 18W 15 Điện quang 26W 16 OSRAM DULUXSTAR COMPACT 8W 17 OSRAM DULUXSTAR COMPACT 11W 18 OSRAM DULUXSTAR COMPACT 14W 19 OSRAM DULUXSTAR COMPACT 20W 20 OSRAM DULUXSTAR COMPACT 23W

An example of one of the CFLs tested with its energy label marking is shown in Figure 12.

28

Figure 12: Example of CFL label

Each sample was measured in accordance with TCVN7896:2008, the official test method for measuring the energy efficiency performance of these types of product. The initial luminous flux (lm) and the power (W) were measured, and the ratio reported (lm/W). The results for this (as the average of the 10 samples tested for each model) from each of the three QUATEST laboratories are shown in Figure 13.

29

Figure 13: Average efficacy (lm/W) measured for the 20 models of CFLs

4.4.1 Discussion of results of testing CFLs

These results from the three QUATEST laboratories are, when compared to those obtained for electric fans and rice cookers much more consistent. All models appeared to be compliant in respect of efficacy (lm/W), which was the only compliance parameter monitored in this testing program. The results reported by QUATEST3 showed consistently lower values, a few per cent lower than the other two laboratories on average – see Table 8, below. Table 8: Difference between the average efficacies of the tested CFL lamps

Declared* Q1 Q2 Q3 Average efficacy (lm/W) 56.24 56.97 56.87 54.50 Q3=100% 103% 105% 104% 100%

* average of 17 lamps The cause of this variation could be due to variations in the measurement of input power or variations in measuring the light output. A further examination of the input power recorded by each laboratory for each was made to determine whether this was the factor responsible. The average power for each model by each laboratory is given in Figure 14.

30

Figure 14: Average measured power (W) of the 20 CFL models

These data show that the measured input powers are very close to each so are not causing the variation in reported efficacy. The variation is therefore coming from the measurement of light output (Figure 15). An examination of the remarks made by the expert trainer suggests that the failure to calibrate the standard (calibration) lamps used by each of the three QUATEST laboratories to the same national (reference) standard lamp is most likely to be the cause of the consistent variation in initial luminous flux measurements. The following figures shows the reported initial luminous flux, which clearly shows that QUATEST 3 is reporting lower figures than the other two laboratories, especially for the larger CFLs.

31

Figure 15: Measured/reported initial luminous flux (lm) of the 20 CFL models

NOTE: In the figure above, the display of most of the results from QUATEST 1 are hidden by the nearly identical results from QUATEST 2.

32

5 Conclusions and recommendations

Vietnam has a number of testing laboratories that have the ability and potential to undertake the testing required to support the appliance MEPS and energy labelling testing regime. The state owned QUATEST laboratories, for which this program from VEESL provided support, were already able to test electric fans, rice cookers and CFLs so were seeking expert guidance to help them further refine their expertise. The results obtained by the inter-laboratory testing of electric fans and rice cookers will need careful examination by MOIT in order to identify whether any can be used for enforcement purposes. Many, though, appear too variable to be used for this purpose. The results for CFLs are much more consistent and could have been used for enforcement purposes if any had performed below the minimum permitted performance level. None did. The main finding of this report is that the variability in test results for two of the three product sectors (rice cookers and electric fans) is very large for some models. Examination of the test methodologies used, the procedures followed and the data recorded by the test laboratories has identified that the following are likely to be the main reasons for the variability:

• Not all measurements are being reproduced in a consistent way by each of the three laboratories;

• The specification and calibration of measuring instrumentation needs to be improved in some cases;

• The published test method for rice cookers needs to be amended to enable it to be consistently reproduced by different test laboratories;

• Thresholds for energy label bands for electric fans are too narrowly defined. These reasons are covered in the discussions below. Actions have been recommended that should enable the results of laboratory testing to become sufficiently accurate to provide confidence to the Government, consumers and manufacturers.

5.1 Product testing related advice for each of the QUATEST laboratories

The deployment of testing experts to provide training and guidance to testing staff at the QUATEST laboratories has been beneficial since they have identified where further refinements can be made. The results of the inter-laboratory testing show that, if not done already, the following needs to be done at each laboratory:

• Upgrade and calibrate testing equipment as already recommended by the experts directly to each of the QUATEST laboratories;

33

• Implement a program to apply measurement uncertainties procedures and so identify and, where possible, reduce sources of measurement error;

• Implement the use of the guidance manuals for testing electric fans and rice cookers. Note: the use of these and the routine estimation of uncertainties of measurement should support the continuing development of a performance testing culture at the QUATEST laboratories – one in which performance testing is always undertaken in a meticulous way;

5.2 Recommendations applicable to all approved testing organisations in Vietnam

Poor reproducibility of test results between testing organisations is not uncommon. It has previously been found in similar programs conducted in other countries. As a result of these, a number of processes have been developed to improve the repeatability and reproducibility of testing at different locations, and these are recommended by the authors of this report:

• Use of common testing manuals (or shared interpretation sheets) for all products impacted by the standards and labelling regulations;

• Undertaking of regular inter-laboratory trials to check that the test results being obtained are reproduced consistently and, where not, to have the test laboratories to work cooperatively together to establish and eliminate the causes of poor reproducibility;

• Feed the experiences of using the test methodology standards by the laboratories back to the standards development body (VSQI, for Vietnam) in order to support the continuing development (to improve the repeatability and, hence, the reproducibility) of those test methods.

5.3 The accreditation operated by the Vietnamese Board of Accreditation (BoA)

The reports from each of the experts suggest that the QUATEST laboratories do not fully match the minimum standards that are required of laboratories that have been accredited to the requirements of ISO/IEC 17025. For example, uncalibrated or incorrect measuring equipment was sometimes used and none of the laboratories were routinely estimating uncertainties of measurement. This is a matter that the BoA can assist the QUATEST laboratories with. The BoA can advise the QUATEST laboratories of the detailed requirements in the ISO/IEC17025 standard and can make a detailed examination of these at their next accreditation visits to the laboratories. This is particularly important as the accreditation process is there to provide the assurance to all stakeholders that testing is being carried out properly, and with sufficient accuracy.

5.4 Support for the MOIT approval process

Currently, MOIT manage a separate approval process to the accreditation process operated by the BoA. Such approval processes are not unusual as many regimes recognise that the formal accreditation process has some limitations and so wish to impose further conditions.

34

MOIT may wish to review the procedures it follows in its approval process once it is satisfied that repeat accreditation visits by the BoA have resulted in further improvements being implemented by the QUATEST laboratories.

5.5 Recasting threshold for energy label bands

The evaluation of results reported in section 4.2.1 suggests that the thresholds for the energy label bands for electric fans would benefit from being re-cast. The evaluation identified that the market is already saturated with 5 star products and most of the fans tested easily exceeded the energy efficiency requirements for 5 stars. NOTE: further work relating to this topic of recasting thresholds is covered by another part of the VEESL Program.

35

APPENDIX 1 Example of test results sheet for one model of electric fan

Table 9: Example of test results sheet for one model of electric fan

Measurement point

Radius Power consumption

(at the highest

Setting with Osc)

Wind speed (at the highest Setting)

Average of Wind speed

Check point

Cross sec. of donuts

Wind flow blowing

through the donuts

Total wind flow

blowing through

the donuts

Energy Efficiency

Coefficient

Level of Energy

Efficiency of Fans (stable, stand, wall)

Level

(I/D) ( r) (P) Left Right V = (Vl + Vr )/2

S = 2*(PI)*r*b*10-

6

Qn = V.S = (m2)*(m/min)

Q = sum (Q1

…Qn)

EER do = Q/P

R = EER do / EER min

(mm) (W) (m/min) (m/min) (m2) (m3/min) (m3/min) (m3/min.W) 1 20 47.80 162.6 202.2 182.40 Go 0.005 0.917 0.917 0.019 0.02 0 2 60 47.80 197.4 208.2 202.80 Go 0.015 3.058 3.975 0.083 0.10 0 3 100 47.80 205.2 212.4 208.80 Go 0.025 5.248 9.223 0.193 0.24 0 4 140 47.80 184.8 167.4 176.10 Go 0.035 6.196 15.419 0.323 0.40 0 5 180 47.80 156.0 153.0 154.50 Go 0.045 6.989 22.408 0.469 0.59 0 6 220 47.80 142.8 108.6 125.70 Go 0.055 6.950 29.359 0.614 0.77 0 7 260 47.80 111.0 99.6 105.30 Go 0.065 6.881 36.239 0.758 0.95 0 8 300 47.80 99.6 56.4 78.00 Go 0.075 5.881 42.120 0.881 1.10 2 9 340 47.80 80.4 26.4 53.40 Go 0.085 4.563 46.684 0.977 1.22 3

10 380 47.80 60.6 0.0 30.30 Go 0.096 2.894 49.577 1.037 1.30 3 11 420 47.80 40.2 0.0 20.10 Stop 0.106 2.122 51.699 1.082 1.35 4

APPENDIX 2 Example of test results sheet for one model of rice cooker

Table 10: Example of test results sheet for one model of rice cooker

Witness test results for Energy Efficiency test of Rice cookers TCVN8252:2009

Product description: Model No.: Test staff:

Ratings 220V 50Hz 700W Actual Power 666WCooker weight: Aluminium, Wg1 (g) 304 Staninless steel, Wg2 (g) 0

CP1 (cal/g K) 0.22 CP2 (cal/g K) 0.12

Ambient temperature 20.6 ~ 21.8°C Relative humidity 52% ~ 72% Test date: 22 Nov & 25 Nov-2013

Test voltage 219.8V Frequency 50Hz Cooker mode: Cook

Test T1 (ºC) T2 (ºC) ΔT (ºC) Water volume (g) Energy consumption, Ec (Wh)

Initial test 19.2 31.0 11.8 2648 44.9

Trail 1 19.2 30.8 11.6 2648 44.9

Trail 2 19.4 31.2 11.8 2648 44.9

Trail 3 19.4 31.2 11.8 2648 44.9

Average of trails 1,2,3 19.3 31.1 11.7 2648.0 44.9

Rd% 82.11910478

Energy Efficiency Index, K Rd/Rmin 1.03 label grade 1

{ΔT x [Q + (Wg1 x CP1) + (Wg2 x CP2)] x 100} / (0,24 x Ec x 3600)

Sample 4

Rice Cooker Model #4

Formal witness test

37

APPENDIX 3 Example of test results sheet for one model of CFL

Table 11: Example of test results sheet for one model of CFL

Witness test results for Energy Efficiency test of CFL TCVN 7896 Lamp model Information Brand A, 11W

Identifier Voltage (V) Current (A) Power (W) PF Flux (lm) Efficacy

(lm/W) Colour coordinate

CCT CRI x y

VEESL-DR11-22 220.0 0.074 9.6 0.593 529.0 55.10 0.3164 0.3469 6199 81.9 VEESL-DR11-25 220.0 0.074 9.6 0.591 531.7 55.39 0.3160 0.3443 6231 81.5 VEESL-DR11-26 220.0 0.076 9.9 0.591 547.0 55.25 0.3173 0.3476 6157 82.0 VEESL-DR11-28 220.0 0.075 9.8 0.592 552.4 56.37 0.3155 0.3450 6252 81.4 VEESL-DR11-29 220.0 0.074 9.7 0.592 532.2 54.87 0.3170 0.3473 6170 82.0 Average 220.0 0.075 9.7 0.592 538.5 55.40 0.3164 0.3462 6202 81.8 non-uniformity factor / 2.68% 3.09% 0.002 4.35% 2.71% 0.0018 0.0033 95 0.6

38