Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf...
Transcript of Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf...
![Page 1: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/1.jpg)
ConfidentialPA110/23/20141
Open Source QA -What will it take
to get to the next level?
Tim Bird
Fuego Test System Maintainer
Sr. Staff Software Engineer, Sony Electronics1
![Page 2: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/2.jpg)
ConfidentialPA110/23/20142
Problem
• We can do better with Linux QA, as an industry and community
![Page 3: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/3.jpg)
ConfidentialPA110/23/20143
Problem Opportunity
• We can do better with Linux QA, as an industry and community
• Not saying we’re doing something wrong• Lots of great projects and efforts
• I think that we’re missing opportunities to do better
![Page 4: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/4.jpg)
ConfidentialPA110/23/20144
Outline
Attributes of Open SourceStatus of Open Source QAObstacles to SharingSolutions and Next Steps
4
![Page 5: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/5.jpg)
ConfidentialPA110/23/20145
Outline
Attributes of Open SourceStatus of Open Source QAObstacles to SharingSolutions and Next Steps
5
![Page 6: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/6.jpg)
ConfidentialPA110/23/20146
Attributes of Open Source
• Openly available
• Easy to contribute to
• Generalized• Applies to a broad range of uses
• Has a development community
• Community effect• Build on the work of others
• As contributor pool increases, better ideas are found
• Feedback loops
• The essence of Open Source is “exchange” or sharing
![Page 7: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/7.jpg)
ConfidentialPA110/23/20147
Outline
Attributes of Open SourceStatus of Open Source QAObstacles to SharingSolutions and Next Steps
7
![Page 8: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/8.jpg)
ConfidentialPA110/23/20148
Status of Open Source QA
• Software Landscape• Tools
• Tests
• Hardware
• Industry QA efforts• In-house
• Ad-hoc solutions
• Result = lots of unshared stuff
![Page 9: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/9.jpg)
ConfidentialPA110/23/20149
What tools in QA are Open Source
• Systems:• Buildbot, Jenkins, LAVA, LKFT, Fuego, KernelCI, CKI
• Lab/board management: Labgrid, SLAV, libvirt, r4d
• Harnesses: pytest, ptest (Yocto Project), ktest
• Services: 0-day, Phoronix, CKI?
• Tests suites: LTP, kselftest
• Tests• See next page
![Page 10: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/10.jpg)
ConfidentialPA110/23/201410
Open Source tests
• kernel testing• LTP – actually multiple test suites
• sysbench, unixbench, hackbench, dhrystone, etc.
• system testing:• lsb-test, yocto ptest, debian autopkgtest
• filesystem – iozone, xfstest, bonnie, dbench
• realtime – cyclictest, pitest
• network – iperf, netperf
• security – vuls
• vertical tests – Android Compatibility Test Suite (CTS)?
![Page 11: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/11.jpg)
ConfidentialPA110/23/201411
Test hardware
• Some off-the shelf components:• PDUs (power distribution units)
• Commercial multi-boards: ACME
• Video capture: Numato Opsis, Lenking LKV373a HDMI extender
• Some open hardware:• Multiboards: ACME, MuxPi, Sony debug board
• Analyzers: Sigrok-based boards (eg. BeagleLogic)
• Different labs use different combinations of things• Often manually manage the hardware
• Companies have lots of in-house custom hardware solutions• e.g. Sony debug board has custom USB switching capability
![Page 12: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/12.jpg)
ConfidentialPA110/23/201412
Industy QA Efforts
• In-house
• Ad-hoc
• Lots of legacy manual testing
![Page 13: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/13.jpg)
ConfidentialPA110/23/201413
Unshared
• Many QA artifacts are not even tangible• Knowledge specific to the QA objectives
• What tests to run?
• Dependencies
• Expected values – How to interpret results?
• What metrics are important?
• Hardware testing is extremely silo-ed
• Exceptions:• Vendor-provided: Android compliance testing (CTS)
![Page 14: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/14.jpg)
ConfidentialPA110/23/201414 ConfidentialPA110/23/201414
An analogy:
Today’s QA software = Yesterday’s RTOS
![Page 15: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/15.jpg)
ConfidentialPA110/23/201415
Embedded OS landscape 20 years ago
• Fragmented
• In-house
• Ad-hoc
• Unshared
• Some exceptions:• Commercial offerings: VxWorks, pSOS
• Regional industry standard: µItron
• Some open source solutions: mbed, ecos, etc.
![Page 16: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/16.jpg)
ConfidentialPA110/23/201416
Open Source Software vs. Testing
• Samsung, LG, Sony all produce TV sets
• 80% of software stacks in TVs are Open Source
• What percentage of QA software is Open Source?
• Which of these companies:• Contribute to Open Source test projects?
• Share their TV functionality tests?
![Page 17: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/17.jpg)
ConfidentialPA110/23/201417
Outline
Attributes of Open SourceStatus of Open Source QAObstacles to SharingSolutions and Next Steps
17
![Page 18: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/18.jpg)
ConfidentialPA110/23/201418
Silos... Why did it have to be silos...?
![Page 19: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/19.jpg)
ConfidentialPA110/23/201419
A tale of
• Fragmentation
• Specialization
![Page 20: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/20.jpg)
ConfidentialPA110/23/201420
It’s worse than just silos;The silos are all different
![Page 21: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/21.jpg)
ConfidentialPA110/23/201421
Obstacles to sharing
• Fragmentation
• Custom hardware
• Unique lab configurations
• Dependency on test framework (different APIs)
• Unique software
• Different testing goals
• Organizational inertia
• Licensing
![Page 22: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/22.jpg)
ConfidentialPA110/23/201422
Fragmentation of Test hardware API
• No consistent API for lab hardware with similar functionality
• Example: PDUs• There are many different ways to control power to a board in a lab
• Serial-controlled power devices• APC devices, YKush, ...
• Network-controlled power switches• Digital Loggers web power switch, Devantech devices
• Custom-built network relays
• USB-controlled power devices• Sony debug board, PowerUSB, etc.
• No single API for all of them
![Page 23: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/23.jpg)
ConfidentialPA110/23/201423
Specialization: Lab configuration
• Even with off-the-shelf hardware, labs use different hardware• Everyone seems to have a different PDU (Power Distribution Unit)
• Different external power measurement devices
• Labs mix-and-match equipment
• Each lab ends up with unique combinations
• Tests written for one lab don’t work in another
![Page 24: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/24.jpg)
ConfidentialPA110/23/201424
Specialization: Test Framework
• Different test frameworks have different APIs, test models
Fuego LAVA Yocto Project
test driven from host target target
target lifecycle multiple tests per boot re-provision every test n/a
languages host: bash, python, yaml, json
target: sh
sh, awk, yaml python
APIs X Y Z
dependencies permission, kconfig, mem, storage permission, packages packages
![Page 25: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/25.jpg)
ConfidentialPA110/23/201425
Specialization: Unique software
• Vendors have different software stacks above their Open Source layers• e.g. Samsung’s TV stack doesn’t look like Sony’s TV stack
• More and more software is open source, and therefore common• Estimate is that for many products, 80% of the software is Open
Source
• Lots of legacy test software
• Target distribution may have different installed software• e.g. may or may not have ‘expect’, ‘awk’, ‘sed’, or even ‘grep’.
![Page 26: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/26.jpg)
ConfidentialPA110/23/201426
Fragmentation: Different test goals
• Developers and testers with different roles:• System software developers (Kernel developers)
• Distribution developers (e.g. AGL, Android, CIP, other stacks)
• Product developers (hardware/software integrators)
• Testing to:• Find software regressions
• Find integration problems
• Meet criteria for shipping
• Interested in different parts of the system
• Different threshold of sensitivity to bugs
• Focus on local remediation vs. upstream reporting and fixing
![Page 27: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/27.jpg)
ConfidentialPA110/23/201427
Organizational inertia
• QA department is not yet interacting with OSS community• Some companies still working to have their software teams learn
to interact with OSS communities
• There’s a learning curve going from using to contributing
• Testing has not historically been an open activity• Concern that tests reveal product info prematurely
![Page 28: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/28.jpg)
ConfidentialPA110/23/201428
Licensing
• License may not require sharing
• Requirement to publish is not invoked• QA software is not distributed by a company
• There’s no trigger for license publication requirement
![Page 29: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/29.jpg)
ConfidentialPA110/23/201429
Outline
Attributes of Open SourceStatus of Open Source QAObstacles to SharingSolutions and Next Steps
29
![Page 30: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/30.jpg)
ConfidentialPA110/23/201430
Solutions
• Create objects that can be shared• Common CI reference model
• Standardize test definitions• Common APIs
• Common languages
• Standardize APIs to lab hardware
• Create places where objects can be shared
• Test generalization
• Interchange
![Page 31: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/31.jpg)
ConfidentialPA110/23/201431
A software stack
![Page 32: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/32.jpg)
ConfidentialPA110/23/201432
Need similar APIs
![Page 33: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/33.jpg)
ConfidentialPA110/23/201433
Need similar APIs
![Page 34: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/34.jpg)
ConfidentialPA110/23/201434
Need similar APIs
![Page 35: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/35.jpg)
ConfidentialPA110/23/201435
Common CI reference model
• Like protocol stack• Define discrete entities and interfaces
• Organizes operations and responsibilities into layers or modules
• Standard interfaces allow for interchangeable implementations
• At recent Automated Testing Summit, we produced a draft reference model
![Page 36: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/36.jpg)
ConfidentialPA110/23/201436
![Page 37: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/37.jpg)
ConfidentialPA110/23/201437
Sharable objects
• Create sharable objects• Test definitions usable in multiple frameworks
• Sharable pass criteria
• Sharable (mineable) results• Common result format
![Page 38: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/38.jpg)
ConfidentialPA110/23/201438
Sharable objects
![Page 39: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/39.jpg)
ConfidentialPA110/23/201439
Test definitions
• Meta-data and instructions for a running a test
• Elements:• Pre-requisites and dependencies
• Information about a test
• Instructions for test execution
• Output parsing or conversion
• Customizations (knobs and dials)
• Visualization control
• See my presentation later today
![Page 40: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/40.jpg)
ConfidentialPA110/23/201440
Define standards for lab environment
• Board control (power, bus control, multiplexing)
• Multi-machine tests (servers, peers, simulators)
• External hardware (monitors, analyzers)
• API above and below lab controller entities
• Goal is to create ecosystems of plug-and-play modules
![Page 41: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/41.jpg)
ConfidentialPA110/23/201441
Place to share objects
• Create:• Project neutral site for collecting/disseminating objects
• or...
• Agreement to consolidate tests in one repository
• Issues:• Peer-to-peer test sharing
• Eliminate gatekeeping for collaboration in testing community
• Allow ad-hoc test customization and enhancement• For diagnosing problems
• Support diagnosis by remote developer who has hardware needed for test
![Page 42: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/42.jpg)
ConfidentialPA110/23/201442
Test generalization
• Allow tests to be generalized• by making tests more customizable
• Ways to make a test customizable:• Skiplists (control of testcases)
• Use dependencies to automate
• Test variables• Expected values/Test outcomes (pass criteria)
• Localized results interpretation
• Data files for different use cases• e.g. filesystem workload - For example, dbench supports custom “loadfiles”
which specify the set of operations to perform
• Preferably do automatic customization• e.g. Set benchmark value threshold based on previous results
![Page 43: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/43.jpg)
ConfidentialPA110/23/201443
Generalization vs. specialization
![Page 44: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/44.jpg)
ConfidentialPA110/23/201444
Generalization
• Take our pirate pieces, and turn them into LEGOs• Or make them compatible with LEGOs
![Page 45: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/45.jpg)
ConfidentialPA110/23/201445
Next steps
• Experimenting
• Creating standards
• Cross-system interactions
• More face-to-face meetings
![Page 46: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/46.jpg)
ConfidentialPA110/23/201446
Next steps – Standards
• At ATS 2018:• Agreed to use pdudaemon as standard Power controller
• Is the first lab API to be standardized• Need to actually document and standardize the API
• Working towards standardizing:• Test definition
• Results definition
• Backend API – KernelCI vs. Squad
![Page 47: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/47.jpg)
ConfidentialPA110/23/201447
Next steps – Interchange
• Utilize Test Stack Survey• See https://elinux.org/Test_Stack_Survey
• Prototype cross-use of results• Between Fuego and Linaro projects (LAVA, LKFT, Squad)
• Unified results format
• Prototype cross-use of tests• Between Fuego and Linaro projects
• Fuego running Linaro tests
• LAVA running Fuego tests
![Page 48: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/48.jpg)
ConfidentialPA110/23/201448
Next steps – More meetings
• Meetings this week
• Some meeting this fall:• Automated Testing Summit #2, or
• Plumbers Microconference
• or both
![Page 49: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/49.jpg)
ConfidentialPA110/23/201449
Advice from my Uber driver
![Page 50: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/50.jpg)
ConfidentialPA110/23/201450
Promote the sharing of automated CI components, artifacts, and results, the way code is shared now
Vision – super high level
• Allow components to specialize
• Support collaboration between projects
Do for testingwhat open source
has done for coding
![Page 51: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/51.jpg)
ConfidentialPA110/23/201451
TL; DR for the talk...
• Being Open Source is not enough
• Also have to be non-fragmented
• And we have to generalize• That is, we have to work together
Let’s try to reduce our silos
![Page 52: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/52.jpg)
ConfidentialPA110/23/201452
Image sources
• tux crash test from LTP project
• Ralls_texas silos image from Wikipedia, User Leaflet CC BY SA 3.0
• silo-coal from heiligbv.com
• white-silo from liberaldictionary.com
• Port_Giles silo from wiktionary.com
• muxpi - Pawel Wieczorek, Samsung
• Lego, pirate toys, blocks, ladder chair toys from respective toy company web sites and marketing materials
• Uber driver - http://vietnamnews.vn/tags/51757/taxi-operator.html
• china dynasty picture – by Thomas Lessman (http://www.thomaslessman.com/)
• Fire images, sony debug board - by Tim Bird
![Page 53: Open Source QA - Amazon S3 · 1110/23/2014 PA1 Confidential Test hardware • Some off-the shelf components: • PDUs (power distribution units) • Commercial multi-boards: ACME](https://reader033.fdocuments.us/reader033/viewer/2022042400/5f0ee7557e708231d44181e2/html5/thumbnails/53.jpg)
ConfidentialPA110/23/201453
Thanks
Tim Bird
Fuego Test System Maintainer
Sr. Staff Software Engineer, Sony Electronics53