Driver Testing - USB

Driver Testing - USB Version 1.1 – Aug 2014

Confidential: Under NDA only 2 of 23

Introduction This document defines the steps for installing and running the tests associated with Ubuntu USB driver requirements. These tests are designed to help Independent Hardware Vendors (IHVs) efficiently deliver high-quality drivers for use with the Ubuntu operating system. Canonical integrates these drivers into Ubuntu to deliver a first-class operating system experience for OEMs and their customers.

Table of Contents

Introduction Table of Contents Testing

Test System Setup Installation Running the test suite Refining your test run Interacting with the test run Manual test options Monitoring your test run Run Status Results submission Restarting the application

Resuming a previous session System Tests

USB device tests Support Test Detail Glossary

Basic Device Detection Storage Device Detection HID/Keyboard USB Mount USB3 Mount USB remove/unmount USB3 remove/unmount USB File Transfer USB3 File Transfer USB Storage Check


USB3 Storage Check USB Performance (Manual) USB3 Performance (Manual) USB Audio


Testing

Test System Setup This version of the hardware testing suite is focussed on LTS releases. It is therefore recommended that you install the latest Ubuntu LTS from the following location and use it as the base OS for testing your device drivers. http://www.ubuntu.com/download/desktop For instructions on installing the Ubuntu LTS Release please see our installation documentation found here: http://www.ubuntu.com/download/help/install-desktop-long-term-support

Installation Installation of the testing suite is completed using our Get Started script that is included in your testing package. The first step is to unpack the test package. To do this, simple copy the file to the test machine, and right click on the file.

This dialog will appear with the option to “Extract here”. This operation will unpack the package in the same location, making it easier to find and work with.

The remainder of this guide assumes that the package and the extracted directory are on the default user’s Desktop. If your location differs from this, please make the appropriate modifications to the paths listed.


http://www.google.com/url?q=http%3A%2F%2Fwww.ubuntu.com%2Fdownload%2Fdesktop&sa=D&sntz=1&usg=AFQjCNFRU5utfKQMTRk6rV7aEeosnngfQA

http://www.google.com/url?q=http%3A%2F%2Fwww.ubuntu.com%2Fdownload%2Fhelp%2Finstall-desktop-long-term-support&sa=D&sntz=1&usg=AFQjCNHeGYdzT69ycC1g3WQySeCw9TyMJA

Once extracted, open a file browser and double click the Canonical Driver Testing icon. The script will then prompt you for your user password, enter it and click “Ok”

If an incorrect password is entered, the installer will exit. Please validate the password is correct and relaunch the install script by running the previous command again in the terminal.

When the correct password is entered the installation will begin. The installer will begin to install the packages.


Running the test suite Now that you have successfully installed the test suite, simply launch the application from the dash, and you should be greeted by the testing application’s welcome screen. From here follow the onscreen instructions for each test. The suite contains both automated and manual tests, so you will need to periodically check the status of the test run and perform any manual tasks required.

The first step in beginning your test run is to select the component driver you would like to test. Multiple components can be selected at a time (Audio and Graphics for example).

Once you have selected your test suite, click the Ok button to send your selection to the next screen for further refinement of your test run.


Refining your test run The Test Selection screen allows you to further investigate the types of tests you would like to include in your test run. It provides with the ability to add or remove tests from the test run, and see advanced details of each test including type, estimated execution time, and test requirements.

Selecting or deselecting tests can be performed by either the checkboxes next to each test name or the Select All, Deselect All buttons. You can also access additional details about your test run. Clicking on the Info button allows you to view additional test case details as well as statistics of your test run.


A warning dialog is triggered by the de-selection of any tests in the suite. Removing any tests from the default whitelist may compromise the effectiveness of your test run.

Interacting with the test run The Manual Interaction screen allows the user to interact with the tests being run. Some tests require the user to perform manual actions or confirm the results of an automated test. When this interaction is necessary the Manual Interaction screen is triggered.

During the test run, please read the instructions on the manual tests carefully. These instructions will ensure you execute the test accurately. The test button is connected to automated tests that will help you to determine if the test passes or not. Once you have completed the actions required by the manual test, set the tick box to the appropriate answer and click Continue.


Skipping a manual test will require you to enter a reason for bypassing this test in the comment field. A warning dialog is provided to remind the user of this requirement.

Manual test options The Manual Interaction screen provides several options when called. The options allow the user to dismiss the test with a status, exercise code embedded in the test or add contextual information. ● Pass - Test result conforms to the users expectations ● Fail - Test result shows errors or lack of functionality ● Skip - Test cases is not valid or unnecessary for this situation (Requires Comment) ● Comments - Any additional information that could be helpful during analysis ● Test - Execute the script associated with the test case


Monitoring your test run The Run Manager screen allows the user to monitor the progress of the active test run. It provides individual test status, the actual elapsed time, details of the test execution, and the ability to re-execute any test.

To see the detailed log for any test run, simply click on the details icon next to the completed test. To re-execute a completed test you must wait for the test run to complete. Then simply click the re-execute icon in the actions column then click the Re-Run button. The icon will then change and previous results will disappear.

* Although the results disappear on the Run Manager screen when re-executing a test, all results are saved and can be reviewed in the test report. * Multiple tests can be re-executed at once. If you select a test that has a requirement, the required test will be automatically re-run.

Run Status The Run Manager screen displays a series of icons. Each icon provides either the status of the test itself or related results and actions. Below is a quick reference for the icons and their meanings.

Passed Failed User Skip Machine

Skip Re-Execute

View

Details


Results submission When the test run is complete, you will be presented with the results screen. This screen gives the user the options to simply view the final test report or to save it to a location accessible from the system under test.

The most common method for results capture in a lab environment is saving the results file off to an external storage device. To do this, simply enter a path where you would like to save the report and click Submit. Other methods for submission will become available later.

Attempting to exit the program before viewing or saving the final test report will trigger a warning dialog.


Restarting the application

Restarting the application is best done from the dash. Simply press the “Super key” and type “Canonical Driver Testing. Clicking the icon will return you to the application Welcome screen. From there you can select a suite and begin testing again.

Resuming a previous session Canonical Driver Test Suite offers a session restore feature for times when the test run is interrupted. This allows the user to select from three options:

● Rerun last test: Resume your test run beginning with the last test run completely.

● Continue: Resume your test run beginning with the test, skipping the incomplete test.

● Restart: Restart your test run from the beginning (This deletes any previous test data).


System Tests

USB device tests ● Driver detection

○ The system should accurately provide information about the driver in use. ● USB2/3

○ Both USB 2.0 and USB 3.0 devices should be supported. ○ USB 3.0 ports should be backwards compatible with USB 2.0 devices.

● Device detection ○ When a device is inserted the system should detect this and perform an

appropriate action. ○ When a device is removed the system should detect this and perform an

appropriate action. ● Audio

○ USB audio devices such as headsets should function properly when plugged into the port(s).

● Keyboard/Mouse ○ USB HID (Human Interface Device) devices should function properly when

plugged into the port(s). ● Storage

○ USB storage devices like thumb drives and HDD should function properly when plugged into the port(s).

○ Filesystems stored on these devices should be fully accessible and allow transfer of data to/from the storage device without errors or corruption.

● Performance ○ Devices should be able to transfer data at the speed appropriate for the USB

specification they support (when plugged in to a port of the correct type). This is 480 Megabits/s for USB 2.0 and 5 Gigabits/s for USB 3.0.

● Recovery after suspend ○ All ports should function correctly after suspend, as per the above

requirements.

* Average time for a single test run is .5 hours


Support ● IHVs are encouraged to join Canonical’s Technical Partner Programme. The Technical

Partner Programme enables collaboration between Canonical and partners. It helps partners understand how Ubuntu is developed and enhanced, and provides access to the roadmap. Partners can share their plans for developing new hardware, ensuring full support in Ubuntu from launch.

● For further support for the test tool and related tests, please contact your Canonical

representative for further details.


http://www.google.com/url?q=http%3A%2F%2Fwww.ubuntu.com%2Fpartners%2Ffind-a-partner%2Fhardware&sa=D&sntz=1&usg=AFQjCNFYgF2s23efI7fK4_fEe5AFxAh9Mg

Test Detail Glossary

Basic

Device Detection

Name usb/detect

Category USB

Description Detects and shows USB devices attached to this system.

Type Automated

Purpose Detect and record the USB devices discovered on the system

Action Execute script which parses system for USB devices

Command lsusb | sed 's/.*$ID .*$/\1/' | head -n 4 || echo "No USB devices were detected" >&2

Expected Result All USB devices are recorded

Certification Blocker

Y

Storage Device Detection

Name usb/insert

Category USB

Description This test will check that your system detects USB storage devices.

Type Manual

Purpose To detect any USB connected storage devices

Action STEPS: 1. Plug in one or more USB keys or hard drives. 2. Click on "Test"

Command removable_storage_test -l usb


Expected Result Removable storage is detected


Y

HID/Keyboard

Name usb/HID

Category USB

Description This test will check that you can use a USB HID device

Type Manual

Purpose Confirm functionality of USB HID interface

Action STEPS: 1. Enable either a USB mouse or keyboard 2. For mice, perform actions such as moving the pointer, right and left button clicks and double clicks 3. For keyboards, click the Test button to lauch a small tool. Type some text and close the tool.

Command keyboard_test

Expected Result Keyboard is functioning as expected


Y

USB Mount

Name usb/insert

Category USB

Description This test will check that the system correctly detects the insertion of a USB storage device

Type Manual

Purpose Determine if the system properly detects and mounts removable storage


Action STEPS: 1. Click "Test" and insert a USB storage device (pen-drive/HDD). (Note: this test will time-out after 20 seconds.) 2. Do not unplug the device after the test.

Command removable_storage_watcher insert usb

Expected Result Removable storage is detected and mounted


Y

USB3 Mount

Name usb3/insert

Category USB

Description This test will check that the system correctly detects the insertion of a USB 3.0 storage device

Type Manual

Purpose Validate that USB3 devices are detected and mounted

Action STEPS: 1. Click "Test" and insert a USB 3.0 storage device (pen-drive/HDD) in a USB 3.0 port. (Note: this test will time-out after 20 seconds.) 2. Do not unplug the device after the test.

Command removable_storage_watcher -m 500000000 insert usb

Expected Result USB3 devices is detected and mounted


N

USB remove/unmount

Name usb/remove

Category USB

Description This test will check that the system correctly detects the removal


of a USB storage device

Type Manual

Purpose Validate that system properly unmounts USB storage when removed

Action STEPS: 1. Click "Test" and remove the USB device. (Note: this test will time-out after 20 seconds.)

Command removable_storage_watcher remove usb

Expected Result System unmounts the USB device


Y

USB3 remove/unmount

Name usb3/remove

Category USB

Description This test will check that the system correctly detects the removal of a USB 3.0 storage device

Type Manual

Purpose Validate that system properly unmounts USB storage when removed

Action STEPS: 1. Click "Test" and remove the USB 3.0 device. (Note: this test will time-out after 20 seconds.)

Command removable_storage_watcher -m 500000000 remove usb

Expected Result USB3 device is unmounted


N


USB File Transfer

Name usb/storage-transfer

Category USB

Description This test will check your USB connection.

Type Manual

Purpose Validate the USB connection

Action STEPS: 1. Plug a USB HDD or thumbdrive into the computer. 2. An icon should appear on the Launcher. 3. Click "Test" to begin the test.

Command removable_storage_test -s 268400000 usb

Expected Result File is transferred without error


Y

USB3 File Transfer

Name usb3/storage-transfer

Category USB

Description This test will check your USB 3.0 connection.

Type Manual

Purpose Validate USB3 connection

Action STEPS: 1. Plug a USB 3.0 HDD or thumbdrive into a USB 3.0 port in the computer. 2. An icon should appear on the Launcher. 3. Click "Test" to begin the test.

Command removable_storage_test -s 268400000 -m 500000000 -p 7 usb


Expected Result File is transferred


N

USB Storage Check

Name usb/storage-automated

Category USB

Description This test is automated and executes after the usb/insert test is run.

Type Automated

Purpose Validate that the storage is mounted properly

Action Execute automated script to check mounted storage

Command removable_storage_test -s 268400000 usb

Expected Result Drive is mounted and check passes


Y

USB3 Storage Check

Name usb3/storage-automated

Category USB

Description This test is automated and executes after the usb3/insert test is run.

Type Automated

Purpose Validate that the storage is mounted properly

Action Execute automated script to check mounted storage

Command removable_storage_test -s 268400000 -m 500000000 -p 7 usb

Expected Result Drive is mounted and check passes



N

USB Performance (Manual)

Name usb/performance

Category USB

Description Verify USB external storage performs at or above baseline performance

Type Automated

Purpose Validate that USB bus is operating at expected speeds

Action Execute performance test on all detected USB ports

Command removable_storage_test -s 268400000 -m 500000000 -p 15

Expected Result USB ports operate at or above expected speeds


N

USB3 Performance (Manual)

Name usb3/superspeed_performance

Category USB

Description Verify USB3 external storage performs at or above baseline performance

Type Automated

Purpose Validate that USB3 bus is operating at expected speeds

Action Execute performance test on all detected USB3 ports

Command removable_storage_test -s 268400000 -m 500000000 -p 60

Expected Result USB ports operate at or above expected speeds



N

USB Audio

Name audio/alsa_record_playback_usb

Category USB

Description Check that a USB audio device works correctly

Type Manual

Purpose Validate that USB audio devices can record audio and have audio played back through them.

Action Request user to record a sound and play that sound back over the USB audio device.

Command audio_settings store --file=$CHECKBOX_DATA/pulseaudio_settings; audio_settings set --device=usb --volume=50; alsa_record_playback; EXIT_CODE=$?; audio_settings restore --file=$CHECKBOX_DATA/pulseaudio_settings; exit $EXIT_CODE

Expected Result The tester hears the recorded sound played back over the USB audio device.


Y


Driver Testing - USB

Documents

Transcript of Driver Testing - USB