Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to...
Transcript of Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to...
![Page 1: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/1.jpg)
Testing IoT Applications: From Embedded to Cloud
Andrey Madan – Lead Solution Architect
September 15, 2018
![Page 2: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/2.jpg)
Intro
•Physics / Computer Science, Novosibirsk State University, Russia
•BS, MS in Computer Science from Purdue University
•Medtronic – Firmware Development / Test Automation: ~10 years
•Parasoft – Solution Architect . 5+ years (current)
•Maple Grove, MN
•Three children + one wife + one dog
•Hobbies: Soccer, Martial Arts, Children and Creativity
![Page 3: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/3.jpg)
Before we start a serious talk
• https://www.youtube.com/watch?v=f0ncl5pEYxg
![Page 4: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/4.jpg)
Why IoT?
Everything is connected, but why?
![Page 5: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/5.jpg)
http://www.iteveryone.com/internet-of-things/
![Page 6: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/6.jpg)
![Page 7: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/7.jpg)
Why does IoT testing start to matter?
•Connected “things” become integral part of our LIFE
•Basic Human Needs: Safety & Security
9/14/2018 7
![Page 8: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/8.jpg)
![Page 9: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/9.jpg)
Talk outline – high level
•IoT Introduction
•IoT Edge is moving into safety critical domain
•Embedded (Edge) testing
•IoT Platform and Enterprise testing
•Unique trends in Testing IoT
•Q&A
Goal: Interactive information exchange
![Page 10: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/10.jpg)
Embedded Testing: Testing the Edge
•Future proof for functional safety
•Application Security
•CI/CD integration
What is specific to IoT domain?
![Page 11: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/11.jpg)
API Testing: IoT Platform and Enterprise layers
• Additional communication protocols
• DevOps implementation in heterogeneous test environments
• Test Data Management
• Who should test APIs?
• API’s are shared across domains: Security
• API Simulations
![Page 12: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/12.jpg)
Internet Of Things:Definition and Trends
12© 2018 Parasoft CorporationAutomating Software Testing
![Page 13: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/13.jpg)
Future: Connecting people with services
![Page 14: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/14.jpg)
Industry 4.0
![Page 15: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/15.jpg)
Autonomous Vehicles
![Page 16: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/16.jpg)
![Page 17: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/17.jpg)
Embedded Systems(barebone C)
Simplistic example of a connected automotive system.
Sensors
Controllers
cloud(s)
CAN bus
Infotainment REST over HTTP(s)
Main Controller
MQTT
V2V, V2X
CarHub: QNX / C++ / Android Java
![Page 18: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/18.jpg)
![Page 19: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/19.jpg)
Trends in IoT Testing
•Edge is moving into safety critical domain.
•End To End Testing and Debugging is expensive
•Focus on interoperability and API testing
•Embedded and IT must coexist
• Security
![Page 20: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/20.jpg)
But - Is IoT testing anything special?
• Back to the fundamentals of test automation
• In 2012 Martin Fowler wrote about the test pyramid, this was a concept developed by Mike Coen.
https://medium.com/@timothy.cochran/test-pyramid-the-key-to-good-automated-test-strategy-9f3d7e3c02d5
Quick Poll
![Page 21: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/21.jpg)
Building a solid Testing Pyramid in IoT
Minimize brittle end to end and UI driven tests, focus on highly automated API
centric testing
Focus on API testing and integrate into the CI
pipeline
Establish a solid foundation of early stage
Unit Tests
![Page 22: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/22.jpg)
IoT:Testing on the Edge
22© 2018 Parasoft CorporationAutomating Software TEsting
![Page 23: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/23.jpg)
Continuous Testing Pipeline
Requirements
Defined
Policy Management
Development
Defect Prevention
Development
Development Testing
Static Analysis
Unit/Component
Peer Review
Automated Tests
Integration Testing
API/Service Tests
Smoke Test
Security Tests
Automated Tests
System Testing
Functional Tests
Scenario Tests
Performance Tests
CI Build
Defect Remediation Tasks
Go
Release Path
No GoBusiness
Decision
Test Environment Access
Code Coverage
![Page 24: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/24.jpg)
Embedded Systems(barebone C)
Part 1: Development Testing practices
Sensors
Controllers
Infotainment
Main Controller
CarHub: QNX / C++ / Android Java
• Static Analysis
• Unit Testing
• Integration Testing
• Code Review
• Functional Safety Compliance
• Safety and Security
• …
![Page 25: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/25.jpg)
The story of Shift Left
Find detects earlier, save money!
Quick poll
![Page 26: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/26.jpg)
Functional Safety Standards
![Page 27: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/27.jpg)
Future proof your application for compliance
A real use case in an automotive market
• A vision sensor was developed and tested as a prototype
• Major issues were fixed and startup was sold to Tier 1 supplier
• Manufacturer requires ISO 26262 compliance
• The question:
How do I get to compliance when all the code is written and operational?
![Page 28: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/28.jpg)
The Answer
![Page 29: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/29.jpg)
ISO 26262: ASIL Levels and Safety
ASIL D
ASIL C
ASIC B
ASIL A
The risk of safety impacts the rigor that must be followed
RISKAutonomous VehiclesNo Driver
Advanced Driver Assistance Software ADAS
Diver assist- Driver in the Loop
Controls and InfotainmentDriver full Control
Cost of Compliance
![Page 30: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/30.jpg)
ISO 26262 – the V diagram
Requirements
Architecture
Design
Code
Unit Test
Integration Test
System Test
These parts of the model are detailed in Section 6
![Page 31: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/31.jpg)
![Page 32: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/32.jpg)
Coding Standards and Guidelines
• MISRA C 2012
• CERT C/C++, JAVA (Computer Emergency Readiness Team)
• JSF
• UL 2900
• AUTOSAR
• CWE –SANS, TOP 25
• High Integrity C++
• MISRA C++2008
• OWASP Top 10
• Barr Group Coding Standard
• Effective C++ (Scott Meyers)
• Power of Ten
Quick Poll
![Page 33: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/33.jpg)
MISRA compliance framework
• Guideline Enforcement Plan - demonstrates how each MISRA guideline is verified
• Guideline Re-categorization Plan - is used to communicate the agreed upon importance for a guidelines as part of the vendor/client relationship.
• Deviations report - documents the deviated violations of guidelines (i.e. exceptions to while the violation will not be remediated) with appropriate rationale.
• Guidelines compliance summary - is the primary record of overall project compliance.
Defined by MISRA 2016 Compliance Document
![Page 34: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/34.jpg)
Embedded Security: What can we realistically do with SA?
• Invoking unsafe macros with invalid arguments. (CERT C, Rule 01)
• Conversions on integers (CERT C, Rule 04)
• Protecting information leakage from trusted (kernel) to untrusted (application) level space. (CERT C, Rule 02)
• Using data outside of declared object’s storage durations (CERT C, Rule 02)
• Avoid accessing arrays out of bounds (CERT C, Rule 06)
• Heap and Stack usage violations (CERT C, Rule 08)
• Handle standard library errors (CERT C, Rule 12)
• Passing a non-null terminated string to a function that expects a string (CERT C, Rule 07)
• Avoid multithreading issues: deadlocks, priority inversion, race conditions (CERT C Rule 14)
• Ensure correct pointer arithmetic (CERT C, Rule 04)
• General rules to eliminate undefined behaviors that can lead to exploitable vulnerabilities.
Example - CERT C.
![Page 35: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/35.jpg)
Static Analysis Adoption
• False Positives rate
•Running slow
• “Don’t see value is fixing the violations. My code works fine. “
• “Too busy focusing on real issues”
•Commercial tools are expensive, open source tools are noisy.
• “Takes too much time to analyze the problems reported”
Quick Poll
![Page 36: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/36.jpg)
Practical approaches for Static Analysis for IoT
• Integrate Static Analysis practice into CI/CD workflow
•Create a baseline.
• Start small and solidify and issue resolution workflow.
• Focus on deltas between previous build and your changes.
• Integrate Static Analysis suppression review with Code Review workflow
• Integrate with development workflow (IDE, command line)
![Page 37: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/37.jpg)
Static Analysis Tools
https://www.softwaretestinghelp.com/tools/top-40-static-code-analysis-tools/
![Page 39: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/39.jpg)
![Page 40: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/40.jpg)
Unit Testing
INPUTS
CHECK OUTPUTS
![Page 41: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/41.jpg)
Unit testing in real life
• “Unit Testing is useless – it does not find real issues”
• “I don’t have time to create and maintain unit tests”
• “There is no internal quality policy to create unit tests. It’s a best practice that I do when I have time. But instead, I’d rather spend time being productive and add more product features that customers will benefit from.”
• “We pay our developers to develop, not to test. We are outsourcing testing to save money to be able to pay the developers top salary they demand”
• “We’d rather spend time and money creating more top level tests to better verify integrated components”
Industry adoption outside of safety critical industry is low. Why?
Quick Poll
![Page 42: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/42.jpg)
The process of creating unit test is beneficial by itself
• Developer looks at the code through difference lenses –> additional code review
• Review the interface to the functionality from external point of view
• How will my code be used? -> simplify the interface
• What if I get invalid data? -> adding error handling
• Revisit the interactions of the UAT to other components
• Simulate fault injections
• Test negative / positive responses from dependencies
• Decide on boundaries of a Unit
• Only create meaningful tests
• Easier traceability to Requirements and User Stories -> reduces # of functional test cases
![Page 43: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/43.jpg)
Unit Testing for Embedded : Host Vs Target
• Option 1: Compile with host compiler (gcc, visual studio) and execute on x86 (host)
Pros:
Fast execution
Easier integration into CI/CD pipeline
Easier setup in some cases
Cons:
Requires stubbing of hardware drivers
Different compiler and architecture. (harder to build functional safety case)
Need to maintain multiple projects
![Page 44: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/44.jpg)
Unit Testing for Embedded : Host Vs Target
• Option 2: Compile unit tests with cross compiler and execute on target
Pros:
Testing is more realistic – easier to qualify for FuSa
Can test HW drivers
Cons:
Requires a dev board
Harder or impossible to run in CI/CD pipeline
Harder to setup for some boards and communication channels
requires cross compiler license and debugger interface
![Page 45: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/45.jpg)
Unit Testing for Embedded : Run on simulator!
• Solution: Compile unit tests with cross compiler and execute on simulator
•Run / debug tests on simulator.
• Integrate with CI/CD
•Re-run the same tests on board for formal verification
Simulators:
QEMU (Open Source), Fast Models and Keil (ARM), IAR, WindRiver VxWorks sim, GreenHills,
![Page 46: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/46.jpg)
Code Coverage Story
![Page 47: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/47.jpg)
Unit Testing – stubs to help drive code coverage
![Page 48: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/48.jpg)
Unit Testing: Popular frameworks and tools
• Open Source frameworks:
• Google test
• Unity
• cppUTest
• Boost
• Unit Testing tools for functional safety:
• Parasoft C/C++test
• LDRA
• Cantata
• VectorCast
![Page 49: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/49.jpg)
Summary
• Identify potential Risks and Safety Integrity Level (SIL)
• Isolate safety critical functionality in files, modules, libraries, applications
• Establish and follow Coding Standards
• Run Static Analysis
• Establish Unit Testing baseline
• Test algorithmic functions at lower level
• Establish Code Review workflow
• Require frequent check ins of code changes.
• Integrate Static Analysis and Unit/Regression testing into your CI/CD pipeline
• Focus on isolating and testing components: e.g. RTOS Tasks, logical modules, ISR’s.
![Page 50: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/50.jpg)
IoT API testingTesting Platform and Enterprise layers
51© 2018 Parasoft CorporationAutomating Software TEsting
![Page 51: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/51.jpg)
API Testing: IoT Platform and Enterprise layers
• Additional communication protocols
• DevOps implementation in heterogeneous test environments
• Test Data Management
• Who should test APIs?
• API’s are shared across domains: Security
• API Simulations
![Page 52: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/52.jpg)
https://www.researchgate.net/publication/313494365_Internet_of_Things_A_Comprehensive_Review_of_Enabling_Technologies_Architecture_and_Challenges
IOT Communication Stack
![Page 53: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/53.jpg)
Embedded Systems(barebone C)
Focus on API Testing and Test Environments
Sensors
Controllers
cloud(s)
CAN bus
Infotainment REST over HTTP(s)
Main Controller
MQTT
V2V, V2X
CarHub: QNX / C++ / Android Java
Verify Infotainment API: MQTT, REST
Verify Sensors:API: CAN
Verify Car:API: REST, V2V, V2X
Verify IoT platform:API: REST, V2V, V2X
![Page 54: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/54.jpg)
RESTful API (synchronous)
![Page 55: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/55.jpg)
MQTT – publish/subscribe (asynchronous)
https://www.skyfilabs.com/project-ideas/iot-based-real-time-data-acquisition-using-mqtt-protocol
![Page 56: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/56.jpg)
Continuous Testing Pipeline
Requirements
Defined
Policy Management
Development
Defect Prevention
Development
Development Testing
Static Analysis
Unit/Component
Peer Review
Automated Tests
Integration Testing
API/Service Tests
Smoke Test
Security Tests
Automated Tests
System Testing
Functional Tests
Scenario Tests
Performance Tests
CI Build
Defect Remediation Tasks
Go
Release Path
No GoBusiness
Decision
Test Environment Access
Code Coverage
![Page 57: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/57.jpg)
Challenge – how to isolate Application Under Test?
![Page 58: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/58.jpg)
Virtual Services
• Simulate response payload
• Can respond with realistic data and simulate delays
• Lightweight, can be deployed in application container (e.g. Docker), reusable
• Use Case: Integration Testing
• NOT Virtual Machines with real services deployed.
![Page 59: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/59.jpg)
Application Under Test
IoT: Increasing complexity of testing requirements
Web
![Page 60: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/60.jpg)
Application Under Test
Parasoft SOATest
Omni/Multi-Channel Testing
WebWeb
![Page 61: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/61.jpg)
Application Under Test
Omni/Multi-Channel Testing
Unavailable or fee-based 3rd
party systems
Uncontrollable behavior
“Agile Roadblock”
Unable to ‘shift-left’ performance
testing
WebWeb
Parasoft SOATest
![Page 62: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/62.jpg)
Application Under Test
Parasoft SOATest
Parasoft Virtualize
Total control of the Test Environment
500 Internal Server Error
Malformed Response
Expose a security
Exception
Test the boundaries of performance
SLAs
WebWeb
Test Data
![Page 63: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/63.jpg)
Digital Transformation applied to an IoT Ecosystem
Edge Computing
![Page 64: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/64.jpg)
What am I testing (Integration Layer)
Edge Computing
![Page 65: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/65.jpg)
Test to Edge
Edge Computing
AUT
![Page 66: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/66.jpg)
Test to Edge
Edge Computing
AUT
![Page 67: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/67.jpg)
Simulate Cloud
Edge Computing
AUT
![Page 68: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/68.jpg)
Simulate Cloud
Edge Computing
AUT
![Page 69: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/69.jpg)
Simulate Edge
Edge Computing
AUT
Validate
![Page 70: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/70.jpg)
Isolation enables DevOps: Testing on Demand
![Page 71: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/71.jpg)
Testing a component of IoT system
Infotainment REST over HTTP(s)
Test
MQTT
V2V, V2X
CarHub: QNX / C++ / Android Java
Virtual Service
Virtual Service
Virtual Service
Virtual Service
![Page 72: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/72.jpg)
API Simulators
•Market: API providers and API consumers
•API providers make development kits for the API (e.g. AWS, Azure)
•API consumers have access to API simulators
•API consumers develop tests against simulators.
•API providers can validate and certify correct API usage by API consumers
![Page 73: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/73.jpg)
Car API Simulator -example
https://car-simulator.developer.mercedes-benz.com/orgs/5vzj/emulators/483D1C27D27A39132C#/
![Page 74: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/74.jpg)
IoT Platform and Enterprise testing
• HTTP, MQTT, RabbitMQ, JMS, XML, JSON, REST, SOAP, … many more
Comprehensive protocol stack
• Complex test environment require programming skills
Interoperability between
Dev and QA
• Run early stage performance tests leveraging functional tests created by the team
Security and Performance
Testing into CI/CD
• Combine API, Web, Mobile and DB interactions into one test scenario
End-To-End testing
• Deploy and Destroy Test Environment modelIntegration with DevOps
![Page 75: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/75.jpg)
Key takeaways
•Focus on API Testing
•Make API’s simple and stateless
•Create reusable virtual services to isolate components under test
•Define who owns API testing domain
• Integrate with CI/CD
•Perform security and pen testing earlier at component level
![Page 76: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/76.jpg)
IoT: Embedded and Cloud
77© 2018 Parasoft CorporationAutomating Software TEsting
![Page 77: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/77.jpg)
Similarities between Enterprise IT and Embedded/IoT
•Functionalities spanningacross many layers
•Open Architecture
•24x7 Connectivity
•Constant Security threat
![Page 78: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/78.jpg)
What can Embedded learn from IT?
• Connected embedded systems resemble architectures known in IT
• Complexity stemming from interconnections shifts the gravity in testing
• Embedded Left-focus strategy no longer optimal
![Page 79: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/79.jpg)
Meet in the middle?
Embedded development
• Focus on defect prevention and early detection
• Unit Testing, Software Coding Guidelines
• Minimize testing fully assembled integrated systems
• Avoid debugging of integrated systems
• Focusing on compliance, not speed
Enterprise IT development
• Focus on agile, rapid development
• Volume of code too large to widely apply unit testing
• Automated component/API level testing
• Testing and debugging integrated solutions
![Page 80: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/80.jpg)
Continuous Testing Pipeline
Requirements
Defined
Policy Management
Development
Defect Prevention
Development
Development Testing
Static Analysis
Unit/Component
Peer Review
Automated Tests
Integration Testing
API/Service Tests
Smoke Test
Security Tests
Automated Tests
System Testing
Functional Tests
Scenario Tests
Performance Tests
CI Build
Defect Remediation Tasks
Go
Release Path
No GoBusiness
Decision
Test Environment Access
Code Coverage
![Page 81: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/81.jpg)
Wrap up
82© 2018 Parasoft CorporationAutomating Software TEsting
![Page 82: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/82.jpg)
Key takeaways
• Prepare for compliance
• Focus on creating clean simple and testable API’s.
• API Testing is the key.
• IoT testing : Safety and Security is the focus of the future to avoid crisis
• Automated testing using simulations to enable DevOps
• Service Virtualization to help isolate components under test
• Understand the cultural differences
• Test early to avoid debugging costs at later stages.
• Integrate all testing phases into one CI/CD pipeline
• Don’t try to do all of the above all at once.
![Page 83: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/83.jpg)
Developer QA / Tester Release Mgmt
• Focused on innovation and minimizing testing burden• Easy, quick, and integrated• Minimize overhead• Get back to development
• Focused on eliminating defects and reducing risk• Limited time + complex
testing problem• Test Coverage, Security,
Performance
• Focused on assessing risk and application readiness
Static Code Analysis
API Testing & Service Virtualization
Unit Testing
Coverage & Traceability
Reporting & Analytics
Load and Performance Testing
HOW PARASOFT SUPPORTS THE SDLC
SOAtest, Virtualize
Load Test
Parasoft DTP
Jtest, C/C++test, dotTEST, Insure++
![Page 84: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/84.jpg)
Comprehensive Testing Solutions for the Entire SDLC
Static Analysis
Runtime Analysis
Unit Testing
Functional & Non-
Functional Testing
Service Virtualization
Management & Analytics
Developer Tester/QA Management Testing Challenges
✓ Security✓ Scalability✓ Reliability
✓ Quality✓ Safety✓ Risk
SDLC Integration
JenkinsBambooTeamCityVSTS
JIRA / RALLYHP / ALMGITSVN
Strategic Ecosystem Financial Services Clients
END-TO-END TESTING
![Page 85: Testing IoT Applications: From Embedded to Cloud€¦ · Testing IoT Applications: From Embedded to Cloud Andrey Madan –Lead Solution Architect September 15, 2018. Intro •Physics](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec55b179e7020370409bb44/html5/thumbnails/85.jpg)
Thank you!
Contact information:[email protected]
https://www.linkedin.com/in/andreymadan/
86© 2018 Parasoft CorporationAutomating Software TEsting