What does QA mean to Sakai? Megan May – Sakai Foundation Aaron Zeckoski – CARET Alan Berg –...
-
Upload
gary-miles -
Category
Documents
-
view
218 -
download
0
Transcript of What does QA mean to Sakai? Megan May – Sakai Foundation Aaron Zeckoski – CARET Alan Berg –...
What does QA mean to Sakai?
Megan May – Sakai FoundationAaron Zeckoski – CARET
Alan Berg – UVADavid Horwitz – Cape TownSeth Theriault- Columbia
Linda Place – U of M
2
Today’s Session
• What is QA? • Unit/integration testing• Static code review• Release/Branch management • QA server network• QA WG• Performance testing
3
What is quality assurance?
What is the role of the QA Director?
the set of planned and systematic actions necessary to provide appropriate confidence that a product or service will satisfy the requirements for quality.
5
Test Driven Development
• Create the Class/API -> Write the Test -> Program the Implementation -> Run the Test
• Forces you to use your own method (and hopefully check if it is intuitive to use)
• Requires an immediate check against the javadoc (API)
• Makes the developer think about how the method will work and what it does BEFORE they write any code
• http://en.wikipedia.org/wiki/Test-driven_development
6
Bug fix test cases
• When there is an issue created against your code you should create a test case to test for that specific failure before you fix it
• These steps– Create a test case which fails as indicated in the issue
from the issue tracker (JIRA)– Reference the JIRA URL in the comment for the test case– Run the test and make sure it fails– Fix the problem– Rerun the test to make sure it passes– Now you are safer from regressions
7
Tests vs. Code Review
• Static code review is valuable and helpful• Peer code review is more valuable• Neither is a substitute for unit testing and
integration testing though• Why?
– They do not provide for protection from regression– They do not supply the context that tests do
• Tests show how to use the code and provide a type of documentation
9
Static Code Review
Functional testing– Guarantees a minimum quality– Black box looks from the outside in– Does not cover all the code– Labor intensive– Does not totally scale as the code base scales
10
Static Code Review
• Static Code Reviews– Looks at the source code not a running instance– White box, looks from the inside out– Looks through all the code base once a day– Automated– Steps over commit boundaries– Information overload– Requires the right mentality from the community– Need to eat the bugs with little fuss
14
Evolutionary
14
• Using PMD, Findbugs, QJ Pro, Jlint and custom Perl code
• Tools getting better• Findbugs does not have many False positives• PMD has configurable rules• More shaping the information to avoid information
overload• Engaging in debate on how to deal with bugs• Need to change developers perceptions
15
Other
15
• Code coverage• Automatic removal of simple bug types script• Tracking of long term use of memory• Provisioning for performance testing• More to come
17
Branch Management & Enlightened Self interest
• Why would we want to be branch managers?– first to go to 2.5.x – wanted a say in ensuring a high quality branch
for production– Its not actually a big overhead outside of the
release cycle
19
QA Server Network
Amsterdam Boston UCambridge Cape TownColumbia IndianarSmart Georgia Tech
Charles Sturt U
Coming soon - server in Japan
20
Running a qa server
• What you need:– a computer – a network connection– a database– know how to compile and install Sakai
• Initial setup– Build server
21
Running a qa server
• Updating and maintenance– downloading source– build deploy– upgrade data (maybe)– usually once a week during release
23
Collab https://collab.sakaiproject.org/portal
JIRA
Confluence
http://jira.sakaiproject.org/
http://confluence.sakaiproject.org/confluence/display/QA/Home
WG Tools
Sakai Performance Testing
Current model and limitationsDesired model and benefits
Dr. Linda M Place, University of Michigan
25
Talking Points
• Current model– Black box testing– Performance failure debugging and fixing
• Desired model– White box testing– White box testing with baseline load– Minimize production performance failures
26
Black Box Testing
• Black box testing– Running system with projected use-case scenarios
for acceptable throughput and response times• How will users experience application?
– Baseline comparison• Throughput and response times compared against
previous version test results• Looking for as good as or better results from new version
– Typically occurs at end of QA process
27
Qualities Being Tested
• Push application load against a system to identify quality of performance– Transaction response time for end users– Throughput (hits per second)– Reliability (percentage of failed
transactions)– Scalability (what happens when scale
increases significantly)– Capacity (upper limits of system)
28
Results
• Are metrics being met?– Measures of success
• Current Sakai test metrics– Baseline or better response times and throughput– New transactions do not disrupt baseline metrics– New transaction stress tests
• How many occurrences before system becomes unusable?
29
Failed Performance Debugging
• Designing isolated tests to address obvious performance problems in production code– Use tests to isolate performance cause– Use tests to verify fix before adding to
production instance
• Putting out fires
30
White Box Testing
• Pushing system to identify application, database, operating system, or network problems– Tune environment to identify and address
specific problems– Tests watched by developers, DBA, system and
network administrators, and performance engineers
• Typically occurs during code development
31
Little in Sakai Today
• Developers may design performance tests as part of development– Process not codified as required standard for
development– Most developers do not have a system of
sufficient scale to push code adequately– All components of production environment not
used and monitored• Developers; database, system & network
administrators; performance engineer
32
Value to Sakai
• Adding white box testing into Sakai QA process finds and fixes numerous performance problems before they reach formal QA process
• Engages Operations team early in development cycle rather than after performance problem affects user community
• Debugging easier prior to full integration with application suite
33
Want White Box Testing?
• Stay in room for next session– Sakai Community Performance Testing: A
Working Proof of Concept
• Participate in community model to improve your code prior to subjecting it to QA
• Don’t expect someone else to identify and fix your codes performance shortcomings