Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile...

22
Parallel Automated Testing on Mobile Devices through BDD-Cucumber and Open Source Resources Presented by Jorge Asensio © All rights reserved 1

Transcript of Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile...

Page 1: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Parallel Automated Testing on Mobile Devices through BDD-Cucumber and Open Source Resources

Presented by Jorge Asensio

© All rights reserved 1

Page 2: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Summary

1. Why Mobile Testing, DevOps and BDD • Background

• The Mobile DevOps challenge

• BDD and Mobile Projects

2. A bit of history • Our needs and how we tackled them

3. The chain of command • Workflow

• Tools, languages and frameworks

• Test run

4. Demo of UAT Automation • Live demo with four different devices

• Case Study example

••

© All rights reserved

User Conference on

Advanced Automated Testing 2

• Trends

• The Mobile DevOps challenge

• BDD and Mobile Projects

2. A bit of history • What we needed

3. The chain of command • Workflow

• Tool

4. Demo of UAT Automation • Live demo with 4 different devices

5. Outlook in the future • App usage will increase

• Pitfalls

• What remains as is and prospective improvements

Page 3: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

1- Why Mobile Testing, DevOps and BDD

© All rights reserved

User Conference on

Advanced Automated Testing 3

Page 4: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Background

• Apps are leading the market.

• Many different devices with different OS type/Versions.

• Quick releases: automation solution.

• High demand for mobile testing in DevOps environments.

© All rights reserved

User Conference on

Advanced Automated Testing 4

Source: ComScore Media

Page 5: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Background

• Apps are leading the market.

• Many different devices with different OS type/Versions.

• Quick releases: automation solution.

• High demand for mobile testing in DevOps environments.

© All rights reserved

User Conference on

Advanced Automated Testing 5

Source: ComScore Media

Page 6: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

The Mobile DevOps challenge

• Market fragmentation.

• Different tooling than Desktop DevOps.

• Speeding-up DevOps activities: cloud/local/3rd parties.

• No continuous deployments.

• No rollback in releases.

© All rights reserved

User Conference on

Advanced Automated Testing 6

Page 7: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

The Mobile DevOps challenge (2)

• Scalable automation.

• Continuous Everything against real environments.

• Poor app instrumentation.

• Back-end changes: services may not serve mobile.

© All rights reserved

User Conference on

Advanced Automated Testing 7

Page 8: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

BDD and mobile projects

• Mobile testing is particularly user XP-oriented.

• Gherkin standardizes documentation.

• BDD enhances interdepartmental collaboration.

© All rights reserved

User Conference on

Advanced Automated Testing 8

Page 9: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

2- A bit of history

© All rights reserved

User Conference on

Advanced Automated Testing 9

Page 10: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Our needs and how we tackled them

© All rights reserved

User Conference on

Advanced Automated Testing 10

Requirement\Issue Solution

Real environments. Real devices, self-hosted solution.

Time, costs, customizable tools and frameworks. Free open-source tools, self-hosted CI solution.

Privacy and security of the data. Self-hosted solution.

Portability. Wheeled structure.

Improve \ modify framework. Scalable code (e.g. Page Object Model techniques).

Current commercial tools not satisfying. Customisable open-source tools.

Dead batteries. USB-powered on hub machine.

Page 11: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

© All rights reserved

User Conference on

Advanced Automated Testing 11

Page 12: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

3- The chain of command

© All rights reserved

User Conference on

Advanced Automated Testing 12

Page 13: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Workflow • Test project triggered from CI.

• Test automation run on devices connected to the slave machine.

• Unified report delivered.

© All rights reserved

User Conference on

Advanced Automated Testing 13

Page 14: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Tools, languages and frameworks

© All rights reserved

User Conference on

Advanced Automated Testing 14

Page 15: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Test run

© All rights reserved

User Conference on

Advanced Automated Testing 15

Page 16: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

4- Demo of UAT Automation

© All rights reserved

User Conference on

Advanced Automated Testing 16

Page 17: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Live demo with 4 different devices

© All rights reserved

User Conference on

Advanced Automated Testing 17

Samsung A5 under

Android 7.0

iPhone 7 under iOS 11.2.4

Sony Xperia XZ1 under Android

8.0

HTC U11 under Android 7.1.1

• Threads run in parallel.

• Unified reports.

Page 18: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

5- Outlook in the future

© All rights reserved

User Conference on

Advanced Automated Testing 18

Page 19: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

App usage will increase • App download estimation: over 79% increase in the next 4 years

• Market points towards mobile app direction

© All rights reserved

User Conference on

Advanced Automated Testing 19

Source: TechCrunch

Page 20: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Pitfalls • SDKs and tool versions not aligned with OS.

• Test environment restrictions.

• iOS device driving requirements.

• iOS parallel testing is tricky.

• App code is not test-friendly.

• Powerful hub machines.

• Appium API deprecations.

• Different Android implementations.

• Steep learning curve.

© All rights reserved

User Conference on

Advanced Automated Testing 20

Page 21: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

What remains as is and prospective improvements • Solution remains self-hosted and using Gherkin.

• Performance tests: JMeter/Locust: • Load/Stress Testing: web and webservices.

• Measure global performance in terms of time.

• Security tests: OWASP Zed Attack Proxy/ LinkedIn QARK.

• Improved test reports.

© All rights reserved

User Conference on

Advanced Automated Testing 21

Page 22: Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile DevOps challenge BDD and Mobile Projects 2. A bit of history What we needed 3. The chain

Thank you for your attention!

© All rights reserved

User Conference on

Advanced Automated Testing 22