PCCG Android Test Automation

Release notes WW37 Sept 14, 2014

1

Copyright & Disclaimer

INFORMATION IN THIS DOCUMENT PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.

*Other names and brands may be claimed as the property of others

This content is for internal use ONLY.  DO NOT redistribute!

2

Contents

1 Introduction.....................................................................................................................5

1.1 Purpose..............................................................................................................................................5

1.2 Contour mapping of Automated Test Case........................................................................................5

1.3 Supported Platform and Operating System.......................................................................................5

2 Change Log.......................................................................................................................6

2.1 Domain wise automated TC List........................................................................................................6

2.1.1 Power Management...................................................................................................................6

2.1.2 Connectivity................................................................................................................................6

2.1.3 Graphics......................................................................................................................................6

2.1.4 Multimedia.................................................................................................................................6

2.1.5 Kernel.........................................................................................................................................6

2.1.6 PUPDR.........................................................................................................................................6

2.2 Known Limitations, Assumptions.......................................................................................................6

2.3 Known Issues.....................................................................................................................................7

3 Procedure to run Test-Case...............................................................................................8

3.1 Host Setup Instructions.....................................................................................................................8

3.1.1 Dev Setup....................................................................................................................................8

3.1.2 User Setup..................................................................................................................................8

3.2 Target Setup Instructions...................................................................................................................8

3.3 Generic Introduction to ACS (User Guide).........................................................................................9

3.4 Fetching Test-cases............................................................................................................................9

3.5 Steps to run ACS test cases................................................................................................................9

3.6 Result Analysis.................................................................................................................................10

4 Setup and TC Classification.............................................................................................12

4.1 Setup Classification..........................................................................................................................12

4.1.1 Standard Setup.........................................................................................................................12

4.1.2 Power/Relay Setup...................................................................................................................12

4.1.2.1 USB Relay Connection Diagram.............................................................................................14

4.1.2.2 Sample Relay Connection......................................................................................................15

3

4.1.3 PUPDR Setup.............................................................................................................................15

4.1.4 Storage Setup...........................................................................................................................15

4.2 Test-Case Classification....................................................................................................................15

5 Debug Wiki.....................................................................................................................16

5.2 Host set-up debug...........................................................................................................................16

5.3 Software package version................................................................................................................16

5.4 Support Structure............................................................................................................................17

4

1 Introduction

1.1 PurposeThis document intended for,

Deployment of ACS automated System Test cases for BYT-M P1 priority test cases. Document limitations/assumptions for the developed test cases Classify bench-setups and list required board components.

This document’s primary focus is to provide release notes and it is not intended to replace ACS framework guide, ACS installation guide etc.

The intended audience for this document is:

Validation S/W Engineers (validating the System test cases) Automation S/W Engineers (writing the ACS test cases)

Pre-assumption is that the audience is familiar with ACS and the way this framework is used to run test cases.

1.2 Contour mapping of Automated Test Case

Contour filter of ACS automated test cases, which are available in this release, named as ACS_automated, which appears under Filter tab.

All Test Case Script files are maintained in ACS git. The path for above mentioned automated test-cases is included in Contour test case description. It is suggested to use Contour to relate Test case to its Test Script.

Eg: To find test case script related to PSPV-PSPVTC-67144 Stream test case, look up this test-case in Contour, and check for script path in Pre-Work section.

For this TC, script path is:

Link Automation Script Path (ACS): <acs_source>/acs_test_suites/PCCG/pnp/TC/LIVE_PERFMEAS_STREAM.xml

1.3 Supported Platform and Operating System

Target Device Configurations for this release is:

5

2 Change Log

2.1 Domain wise automated TC List

Test case change logs as per domains is

Note: This part will be included after review completion of all tests.

2.1.1 Power ManagementI32e84f : [PSTV] Time update check for S3 and S5

2.1.2 Connectivity

2.1.3 Graphics

2.1.4 Multimedia

2.1.5 Kernel

2.1.6 PUPDR

2.2 Known Limitations, AssumptionsACS automation has below mentioned known limitation for the test cases.

For all test cases, visual artifacts, graphics anomalies validation not automated. Audio/video quality validation not automated. Partially automated tests need manual intervention to verify results. On BYT M CRB board thermal sysfs is not functional, due to this Performance test cases gives

warning while trying to access these interfaces. For Power Management test cases, Fan and Postcode validation not automated.

6

Serial No. Platform Operating System1. BYT-M CRB Android 4.4.4

For all test cases, the below mentioned assumptions made:

Device is booted, screen lock is set to none, and screen time is set to 30 minutes, stay awake enabled.

For detailed information on all assumptions please check the Automation Wiki

2.3 Known IssuesAdb connection hang observed when multiple instances of adb servers are running on host side. Issue is more visible when ACS execution is forcibly interrupted by user.

Work around for this issue is to kill de-funct adb server instances via below command.

#kill -9 $(ps aux | grep ‘[a]db’ | awk ‘{print $2}’)

7

3 Procedure to run Test-Case

3.1 Host Setup InstructionsACS Host setup is classified in 2 categories based on the targeted use.

3.1.1 Dev SetupACS developers for writing test cases use it. Dev Setup installation procedure is detailed in the below link:

Link http://acs.tl.intel.com/portal/index.php/dg-prerequisites/linux-bench-setup

If Dev setup used for deployment purpose, eclipse installation can skipped, and TC can be run using command line.

3.1.2 User SetupThis setup can use for deploying Test Cases. User Setup installation can be done by following the below link:

Link http://planetblue.ith.intel.com/gethelp/acs/documentation/p/get-started.aspx

For now, we are using Dev Setup for deployment of test cases, as the user setup not yet explored.

3.2 Target Setup InstructionsBYT M CRB Target test bench setup can do as per this presentation.

By following this presentation, device setup should be suitable for executing TCs using ACS.

Things to note:

Device and host should be able to communicate over adb, and host should be having Intel Network connectivity to fetch Test load from ACS source mirrors and from Artifactory.

For debugging ACS installation related issues, please go through Debug Wiki.

8

ACS Automation-Bench Setup.pptx

3.3 Generic Introduction to ACS (User Guide)Please go through ACS user guide to get an overview about ACS, ACS terminology, relation between TC, Campaign, Bench-Config, ACS Test Verdict types and how to run tests cases with ACS :

Link http://acs.tl.intel.com/portal/index.php/acs-documentation/user-s-guide

3.4 Fetching Test-casesPre-requisite:

ACS development set up is ready and ACS repository is present on host, which has Intel Network access.

Step1: Go to ACS source base folder on host Ubuntu system.

Step2: $repo init -u ssh://android.intel.com/manifest -b platform/android/main -m acs-main_ts

Step3: $repo sync

Step4: Test cases will be located in <acs-source>/acs_test_suites/PCCG folder.

3.5 Steps to run ACS test cases Run the below command from _Execution folder which is base folder to run test cases.

#python ACS.py -d DEVICE_MODEL -b BENCH_CONFIG -c TEST_CAMPAIGN

Where:

DEVICE_MODEL, BENCH_CONFIG and TEST_CAMPAIGN should point to actual configuration of the device, bench and the campaign which user is using.

A working example of command line parameters for running Performance test cases is:

#python ACS.py -d BYT_M_CRB-Android-KK-AIA -b PCCG/pnp/BENCHCFG/Bench_Config -c PCCG/pnp/CAMPAIGN/PerfMeas_Campaign -n 1 –dev

For making changes the below example can be used.

E.g.: Modify the ipAddress parameter in BenchConfig to point to IP address of target device.

<BenchConfig> <Phones> <Phone name="PHONE1" description="Phone to use during test campaign. Please refer to Device_Catalog.xml to add device parameters"> <Parameter name="useHardShutdown" value="true" /> <Parameter name="retrieveDeviceCrashLog" value="True" /> <Parameter name="ipAddress" value="192.168.1.6" /> <Parameter name="adbPort" value="5555" />

Points to note:

For test-cases which control relay, set the parameter a) isIoCardUsed in CampaignConfig to True.b) finalDutState in CampaignConfig to False.

9

For adding new TCs to CampaignConfig file, mention the path under tag TestCases

Snapshot:<TestCases> <!-- Testcase campaign - Id: Testcase name--/> <TestCase Id="../../SETUP/SETUP"/> <TestCase Id="../TC/FASTBOOT_MODE" /> <TestCase Id="../TC/WIPE_CACHE_RECOVERY_MODE" />Add another entry here for TC which is needed to be run.</TestCases>

3.6 Result AnalysisACS collects aplog, test run console log under _Reports folder placed at <acs_path>/src/. This is divided in 2 parts.

AWR report Detailed logs

Sample of AWR report is:

Link https://acs.tl.intel.com/AWR/?mode=dev#campaigndetails:/uuid=c66aa096-68b4-4584-9c33-6d4333eb3834

ACS test verdict classified in three categories.

PASS FAIL BLOCKED

TC will be marked as PASS, when all the test steps executed successfully and the intended result achieved as mentioned in the expected result field in Contour TC definition.

TC will be marked as FAIL,

When not all the test steps did execute and the intended result, as mentioned in the expected result field in Contour TC definition not achieved. Or,

Test case is Partially Automated, and it will always FAIL. Please look for Comments as part of AWR report. Such test cases have comments to point that it is a Partially Automated test case, and the validator needs to check for Video/Screenshots in _Result folder under the execution log folder.

Sample AWR report:

Link https://acs.tl.intel.com/AWR/?mode=dev#campaigndetails:/uuid=8b5ef0c1-49b2-437e-93f5-f5ebda8fa6ab

Snapshot: Look for comment section in AWR report.

10

TC will be marked as BLOCKED,

When ACS encountered issues while executing the Test and due to it, BLOCKED the test case.

11

4 Setup and TC Classification

This section details about the various Hardware setups required to run the ACS test cases. This classification is done based on the accessories which are required to be connected to setup while executing the test cases. Based on the type of setup is required, each test is tagged with the setup type.

4.1 Setup ClassificationBroadly, setups classified in below mentioned four categories.

4.1.1 Standard SetupThis setup is the basic setup required to run ACS test cases on the target device.

DetailsBYT M CRB Mother Board + CPU + SW + Keyboard + Mouse + Touch eDP panel

WPA2 + Antenna, BT Cable(jumper)USB to ETH connector OR Ethernet NIC Card

AC Charger + BatterySensor Card

USB Serial cable for teraterm loggingExternal SD card (8GB or higher)Internet connectivity over WiFi

Audio Headset (connected to 3.5mm jack)USB Camera

USB2.0 Pendrive

4.1.2 Power/Relay SetupThis setup includes Standard setup and in addition of relay connection with the below mentioned accessories. Below information is BYT-M CRB specific.

12

USB Relay Setup (Standard + Relay control) Port NumberUSBRelay08 + USB Cable connected to Host

Wiring on the board for Mouse (Only Left click) Port 8Wiring on the board for Lid ON/OFF (J8G2) Port 7

Wiring on the board for BT Headset ( NokiaBH214 ) Port 6Wiring on the board for Power button (SW3H1) Port 5

Wiring on the board for Key Board (Only Windows Key) Port 4Wiring on the board for AC Charger Port 3

Wiring on the board for Volume UP (SW9G1) Port 2Wiring on the board for Volume Down (SW9G2) Port 1

The below is sub-setup within the above-mentioned Setup.

USB Relay Setup (Standard + Relay control Battery)Standard + Relay Control Battery

13

4.1.2.1 USB Relay Connection Diagram For relay control, USBRelay08 (http://www.robot-electronics.co.uk/htm/usb_rly08tech.htm) is used. The below diagram shows wiring connection between relay ports and the different board components which are used in automation. For all components, connect via NC and C to the relay.

Notation:

NC is Normally Close Contact, and represented in Light Green line.

C is Common Contact, and is represented in Black line.

NO is Normally Open Contact, and is represented in Light Sky Blue line.

14

4.1.2.2 Sample Relay ConnectionAttached below is detailed reworking for BT headset Nokia BH214 using relay. Similar rework is applicable for rest all accessories listed above.

4.1.3 PUPDR SetupThis setup includes direct Ethernet connection between Host and DUT in addition of Standard Setup. This setup is required to run Device Upgrade related test-cases.

PUPDR Setup (Standard + direct)Direct Ethernet connection between Host and DUT, IP in 192.168.42.x range

4.1.4 Storage SetupThe following setup is required to test Storage domain test cases. This setup includes Standard setup and the below storage devices as add-on.

Storage Setup (Standard + Storage)USB pen drive 2.0USB pen drive 3.0

SD card2.0USB External Hard disk2.0USB External Hard disk3.0

USB hub, preferably self-powered

4.2 Test-Case ClassificationTest cases classified in four categories based on the setup dependency.

Detailed Test Case classification described in the attached document.

Before executing the test cases, it advised to look up for the setup requirements for running the test case in this document, and ensure that setup has all the needed components as mentioned in the above setup classification.

15

5 Debug Wiki

5.1 Summary:

This wiki* contains troubleshoot information for the generic errors related to setup and Test case runs.

Make sure that Ubuntu version is: 12.04.1 LTS, which is officially supported by ACS community.

Important: Android SDK, with fastboot and adb to be replaced with Intel specific adb/fastboot binaries.

5.2 Host set-up debugTo rule out local setup related issues early, it recommended running the following TCs as part of ACS BAT successfully on each System with ACS. This will ensure that local setup issues are not existing, in case these test cases fail, please troubleshoot using section-5.2

Category Test Case

ACS BAT acs/src/_ExecutionConfig/PCCG/SETUP/SETUP.xml

Graphics acs/src/_ExecutionConfig/PCCG/multimedia/TC/WALLPAPER/LIVE_IMAGE_WALLPAPER.xml

Performance

acs/src/_ExecutionConfig/PCCG/pnp/TC/LIVE_PERFMEAS_STREAM.xml

Connectivity acs/src/_ExecutionConfig/PCCG/connectivity/TC/WIFI/WIFI_DOWNLOAD.xml

Multimedia acs/src/_ExecutionConfig/PCCG/multimedia/TC/VIDEO/H264_High_profile_VIDEO_PLAYBACK_TEST.xml

5.3 Software package versionWorking setup includes the following python and related utilities. It is good to check the versions at your end and update if required.

Package VersionPython 2.7.6(Officially ACS supported)

numphy 1.8.1imaging 1.1.7pyserial 2.7

psutil 2.1.1requests 2.3.0pyYaml 3.11

lxml 3.3.5libxslt 1.1.28cython 0.20.2

py-dom-xpath 0.1pyaudio 0.2.8.1libxml2 2.7.2

16

5.4 Support StructureFor issues related to ACS setup/Test cases:

Primary Contact:Secondary Contact:

Escalation Point:

Link Official ACS Support/Discussion is ongoing @ https:// soco.intel.com/groups/forum * This page does not contains all debug/troubleshoot information and is not meant to replace official ACS Support, so it is suggested to contact ACS community, whenever required.

17