Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software...
Transcript of Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software...
![Page 1: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/1.jpg)
Analyzing and Testing Non-functional Software Properties
Sudipta Chattopadhyay Kick-off Meeting
![Page 2: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/2.jpg)
Objective• Learn about non-functional software properties
• Timing
• Energy
• Robustness
• Security (primarily side-channel leaks)
• Reliability
![Page 3: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/3.jpg)
End of the seminar• Answers to questions you should know (or have some ideas at least)
• Why should we care about non-functional properties or should we?
• Is it harder to analyse than functional properties?
• Is it harder to test than functional properties?
• What is the formal definition for non-functional bugs?
• Which applications are more critical if they are prone to non-functional bugs?
• Are non-functional bugs dependent on execution platforms (processors, memory) or are they independent?
• Are you kidding me? So how do developers fix non-functional bugs or they just ignore them?
![Page 4: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/4.jpg)
End of the seminar• Answers to questions you should know
• What happens if I blindly apply a technique developed for analysing/testing functional properties?
• What does performance/energy debugging mean to you?
• Does it differ from classic debugging? Is it harder than classic debugging?
• Isn’t it a compiler’s job? Why should I care about?
• Meh, Sudipta is bluffing….People must have looked at it in the past. Have they? If not, why not?
• Should we care about this problem for next 5 years?
• Can I work on it for my PhD thesis now?
• Should we care about this problem for next 10 years?
• Can I work on it for my PhD thesis after 5 years (I have no time to do PhD now)?
![Page 5: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/5.jpg)
Organizational matters• No unexcused absence is accepted
• You need to give a valid reason in advance (unless it is accidental)
• Submit all summaries by Wednesday each week
![Page 6: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/6.jpg)
Organizational matters• Final presentation
• Need to present a paper at the end of the semester
• It’s not just the presentation of the paper
• Study related literature
• We should get a better understanding of the overall area
![Page 7: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/7.jpg)
Reading papers• Read every detail
• possibly some related paper if needed
• Understand the main concept
• Ask questions
• Is this contribution fundamental or incremental?
• General idea vs. simple improvement over state-of-the-art
• Is the proposed claim justified?
• Is this contribution theoretical with no hope of being applied?
• Is this pure engineering?
• Is this just one-off paper or there could be a series of follow-up papers?
• Can I write a paper in this area now (or technology has overshadowed the contribution)?
![Page 8: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/8.jpg)
Writing summaries• Summary is not a short version of the paper
• Do not choose sentences from the paper
• Do not strictly follow the flow of the paper (Section 1 mentions this, …..)
• Do not write in first person to refer the work (Our work addressed….)
• Not providing citations from another paper/blog/website
• Highlight
• Problem addressed
• Main idea
• Contribution
• Weaknesses
• Evaluation
• Questions highlighting the potential ambiguity of the technique, evaluation or the paper at large
![Page 9: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/9.jpg)
Writing summaries• DART: Directed Automated Random Testing
• “In Section 5, authors have introduced their technique.”
![Page 10: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/10.jpg)
Writing summaries• “In Section 5, authors have introduced their technique.”
• This sentence does not convey anything
• Which technique?
• What problem?
• Who cares about section 5?
• The paper addresses the problem of testing software functionality bugs via dynamic symbolic execution.
![Page 11: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/11.jpg)
Writing summaries• “The paper addresses the problem of testing
software functionality bugs via dynamic symbolic e x e c u t i o n . T h e p ro p o s e d t e c h n i q u e i s extraordinary.”
![Page 12: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/12.jpg)
Writing summaries• “The paper addresses the problem of testing software functionality
bugs via dynamic symbolic execution. The proposed technique is extraordinary.”
• Do not use flashy words
• extraordinary
• unbelievable
• genius
• great
• poor, bad, pointless, I don’t care
![Page 13: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/13.jpg)
Writing summaries• “The paper addresses the problem of testing
software functionality bugs via dynamic symbolic execution. Although it was really hard for me to understand, the proposed technique seems to work well.”
![Page 14: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/14.jpg)
Writing summaries• “The paper addresses the problem of testing
software functionality bugs via dynamic symbolic execution. Although it was really hard for me to understand, the proposed technique seems to work well.”
• Do not express your emotion
• hard for me, easy for me
• I enjoyed a lot reading it
![Page 15: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/15.jpg)
The Seminar• Divided into categories
• Analysis/Verification
• Testing
• Profiling/Mining
![Page 16: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/16.jpg)
The Seminar• Performance/energy bugs
• Waste performance or energy
• Unnecessary memory access
• Unnecessary GPS access
• Worst-case performance or energy
![Page 17: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/17.jpg)
The Seminar• Reliability
• Many software systems process input derived from sensors
• Embedded software
• Sensor input
![Page 18: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/18.jpg)
The Seminar• Robustness
• Stability of control software
• Operate on non-deterministic environment
![Page 19: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/19.jpg)
The Seminar• Side channel attacks
• Break implementation by sensing timing, power etc.
• Discover private information e.g. secret keys of ATM card
![Page 20: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/20.jpg)
Choosing Papers• Send an email to [email protected]
• 6 preferences, preference by area and not by paper
• One paper (or “any” paper) from each area, ordered by your preference
• Profiling
• Performance analysis and testing
• Energy consumption analysis and testing
• Reliability analysis (only one paper)
• Robustness analysis and testing
• Side channels
• Deadline: 4th Nov, 2015, 23:59:59
![Page 21: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/21.jpg)
Assigning Papers• First-come-first-served
• Order of preference
• It is possible that all your preferences have been assigned
• I’ll contact you to resolve conflict, most likely by suggesting a paper in the area of your preference
![Page 22: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/22.jpg)
Upcoming meetings• 13th November, 2015:
• Efficient Path Profiling (MICRO 1996)
• 20th November, 2015:
• Where is the energy spent inside my app?: fine grained energy accounting on smartphones with Eprof (EUROSYS 2012)
• The paper for next meeting should be assigned now
![Page 23: Analyzing and Testing Non-functional Software …...Analyzing and Testing Non-functional Software Properties Sudipta Chattopadhyay Kick-off Meeting Objective • Learn about non-functional](https://reader034.fdocuments.us/reader034/viewer/2022042806/5f740c3515aaa3092f3ff5b5/html5/thumbnails/23.jpg)
Objective• Learn about non-functional software properties
• Timing
• Energy
• Robustness
• Security (primarily side-channel leaks)
• Reliability
The Seminar• Performance/energy bugs
• Waste performance or energy
• Unnecessary memory access
• Unnecessary GPS access
• Worst-case performance or energy
Choosing Papers• Send an email to [email protected]
• 6 preferences, preference by area and not by paper
• One paper (or “any” paper) from each area, ordered by your preference
• Profiling
• Performance analysis and testing
• Energy consumption analysis and testing
• Reliability analysis (only one paper)
• Robustness analysis and testing
• Side channels
• Deadline: 4th Nov, 2015, 23:59:59
Upcoming meetings• 13th November, 2015:
• Efficient Path Profiling (MICRO 1996)
• 20th November, 2015:
• Where is the energy spent inside my app?: fine grained energy accounting on smartphones with Eprof (EUROSYS 2012)
• The paper for next meeting should be assigned now