1

2

3

4

Abstract: Are you getting ready to jump into the Agile world for your product development/testing? Have you recently started with Agile and would like to learn from someone else's mistakes instead of your own? Do you want to learn some of the best practices involved in testing products following Agile methodology? If you have answered Yes to any one of the above, then you should attend this session. In this talk, Ravi Shanker will walk you through the "Microsoft Visual Studio Test & Lab Management" (VSTLM) team's journey, the team that builds testing products for Visual Studio, as they moved from waterfall/iteration based development/testing cycles to more of Agile development/testing approach. This journey, which started in early 2008 during the development of Visual Studio 2010, continues on today as part of development of Visual Studio 11.

Session Objectives:

Share our experience of adopting “agile software development”

Key Takeaways:

5

1. Clearly understand the reasons to jump onto “agile software development”

2. Learn from VSTLM team’s experience – teething troubles & how to

overcome them, best practices etc as you adopt “agile software

development” First gotcha being that this is not a recipe to be followed “as-is” – it definitely requires customization to suit your team/org needs

5

Before I start on agile, let us look at the software industry at large : Just look around – how many of you are using the same mobile phone that you used 2 or 3 years ago? With always ON, high-speed internet, web 2.0, mobile 4G, and a multitude of devices with a life span of a couple of years, the market is changing very rapidly. That in turn means that the product requirements are changing rapidly as well. All in all, there is tremendous pressure to accelerate software delivery. There are certain products, including Visual Studio where a new release is still shipped every couple of years. Even in those cases, it is extremely important to get early feedback on the software because requirements don’t always match the end user intent. On top of that, SW development is still an art. It involves solving new problems every time. Therefore, even effort estimation is really hard for reasonably complex projects.

6

The problems raised could be attributed to issues with existing software methodologies – Waterfall or Iterative

7

The software industry is responding to these dynamics by adopting Agile practices.

8

Lets take a look at what Agile means? Before that, some questions: How many of you are currently using agile methodologies? How many of you know the Agile Manifesto? How many of you are aware of the principles behind the manifesto? The agile manifesto calls out 4 things: - Individuals and interactions - Working software - Customer collaboration - Responding to change

9

If you look at the principles behind the Agile manifesto, the core things that stand out are: - Satisfying the customer needs - Adapting to changing requirements - Focus on Continuous delivery - Working software to show progress - Collaboration within the team and with the customers - Emphasis on continuous feedback

These are the prime reasons why Agile is catching up fast in software development.

10

This report from Forrester clearly shows that Agile methodologies have overtaken the Waterfall

ones (even though this is about 2 year old data).

When I asked for how many people know about agile methodologies, almost <percentage>

went up. This was clearly not the case couple of years back.

Another important aspect is that “Scrum” is by far the most adopted agile methodology.

May 5, 2010

The Forrester Wave™: Agile Development Management Tools,

Q2 2010

by Dave West and Jeffrey S. Hammond

Scrum process diagramatically

12

Now that we have looked at why Agile is catching up, lets now look at our team and our experience of adopting scrum Every team generally follows different project methodologies – for sake of operational efficiency, we all align to same sprint boundaries at ALM level. The key thing is that we belong to a large division (DevDiv) and have to follow all the divisional constraints associated with release activities. Even in such complex and diverse set of teams, we are able to adopt agile methodologies, primarily Scrum, for our team.

13

We entered into the Agile world because it was mandated by our Management – How many of you can associate with this? As a result, bunch of teething problems cropped up, such as: - Entire team not fully on-board with Agile - Team in complaining mode - No formal training provided - What should Devs do, what should testers do? When should testers start testing?

What should PMs do?

14

Let us look at how we overcame the teething problems using these 3 drivers and transitioned into a well oiled machine Since the focus of Agile is to having continuous delivery of value to customers, there are 3 important drivers for this: - People - Process - Tools

15

When I use the word team, it is all inclusive – developers, testers, designers, business analysts, architects, project managers, etc. First thing was that everyone in the team went through a formal training on Agile and Scrum. This ensured that everyone was speaking the same lingo Managers and leads were a must in addition to the entire team We also identified a list of Agile experts within MS but in different teams – created a DL with these folks and used it to clarify any doubts we ran into as we adopted scrum As managers and leads, be prepared to accept lower productivity during the first few sprints as the team adjusts to the new operating principles. Some people will be confused and need coaching. Some will adapt quickly and others will take time. Some may never get comfortable with the scrum model and may quit! Remember, the change is always hard. Some short term pain has to be absorbed to get long term gains. Management went about gaining the trust of the team – instead of using the Top-Down method, they opened it up for the team saying that we all would give Scrum a try for couple of sprints. If at the end of it, folks were not convinced on the advantages, we would go back to old ways. With this approach, everyone was open to give this a fair shot

16

Next thing we did was to clearly identify the roles and responsibilities within the team Scrum Master – QA/Dev lead, this was a rotating job, whoever was passionate about Scrum would become the Scrum Master Product Owner – primarily the Program Manager, who would define what needs to be done and owned the product backlog Pig – the entire team members consisting of Dev, QA, UX etc who would actually do the work Chicken – our Management that approved what we were building Since everyone was done with training, they clearly understood what they had to do. There was still some confusion around what testers do and when. We will tackle that in next few slides.

17

Now that we tackled PEOPLE, lets look at how we approached the Process itself The process can be broken into - Sprint Planning - Sprint Execution - Sprint Closure

18

Before we started on any sprint, we had out “Done List” completely nailed. We had 2 “Done lists” – one at the User Story level and the other at Experience/Epic level User Story Done list would come into picture for every feature sprint. This included - All feature bugs fixed - Exploratory testing completed - All automation debt completed - Accessibility testing completed - High contrast testing completed - FxCop clean - Leads walkthru completed - Manual Perf measurements

If a user story did not meet this done criteria, it was considered in-progress and it carried over to the next sprint.

19

We had a dedicated integration/stabilization period to pay-off the rest of ‘release’ debt at each major milestone. We planned the Epics in such a way that they would align along the “Release Sprints”. The Epic Done list included all the Quality Gates/Tenets that are required for Release: - Acquisition tenet (Setup/Upgrade testing) - Compatibility (with prior versions) - Compliance (LCA/Privacy) - Ecosystem (Extensibility) - Performance - Reliability (Stress/Memory) - Security - User Experience - World Ready (Localization/Globalization)

It requires a lot of discipline to not create feature bug debt. This is something we failed to do in Dev10 but have been quite successful in Dev11 so far.

20

A prioritized product backlog is must for success of the project - the product owner is tasked with maintaining this. In the Storypoint estimation, Product owner is ready with a detailed spec – most cases it is a storyboard. The product owner continues to work on this one sprint in advance. Just before the current sprint is ending, Product owner meets with the QA/Dev leads and reviews the storyboard and come up with storypoints – this is done so that we know how many stories to take to the team before they estimate it. When everyone participates in Sprint planning, it brings everyone on the same page on what the sprint goals are. Program managers/Business analysts help clarify user stories. It also forces the team to consider non-functional investments as well. For example, QA team building perf infrastructure. It increases visibility, helps with better resource allocation and higher confidence plans. We generally do this on the first day of the sprint (after lunch) Story Point Estimation – Best practice – anything over 5 points will not get done within the sprint, so we try to break the stories down into smaller chunks The focus is on collaboration, including collaboration with the customer and making sure that all team members understand the user stories. It is very common in our team for the QA to file spec bugs because the code implemented as per the spec may not meet user or usability requirements.

21

Smaller user stories really help in two important ways 1. Divide and conquer - they reduce the complexity of task at hand 2. Less spill over from sprint to sprint

21

Design & Task breakdown Once we have the scope clear and what needs to be done, we discuss the potential design and then get into creation of tasks. Creation of additional tasks is something that is helping us very nicely – these are tasks that are added to any User story. It helps guide the Done list for the user story AND at the same time clearly identifies who has to do what and at what times. Quality is a shared responsibility and it is not an afterthought. Testing activities are accounted for in the sprint plans and they start from day one of the sprint. Developers and testers have a vested interest to understand functional specs. Similarly, test teams typically review developer design specs. Since the devs create the unit tests, this forces them to review the plan and suggest modifications/additions. In Microsoft where most of our software testers write code, it is easier to load balance developer-tester workload. However, in other companies, it is hard for developer/tester personas to interchange. If you don’t have SDET profile testers in your team, the testers can still add value to the team by defining the Test Plan and doing exploratory testing – these are core skills for a tester. In MS, our QA folks also write scenario automation tasks and some unit tests. These are 2 areas where Dev/QA work interchangeably – depending on who is free and what the next task is.

22

Remember, one of the Agile goals is to deliver customer value frequently. You can do that only if you have shorter sprints. How short is short? It is relative. We started with 6 week sprints when we build Visual Studio 2010. Now we are practicing 3 week sprints. Some teams practice even one or two week sprints. Optimal sprint duration is a function of the process overheads, such as, security, code coverage, upgrade, compatibility, globalization, localization, etc. Longer sprint mean that your reaction time to any external change is proportionately longer.

22

Show how using the new VS11 Agile dashboard to: - manage Product backlog

- move items within the backlog using drag & drop - create new stories, - assign certain stories for the upcoming sprint - Managing OOF schedule

- Manage sprint backlog - Add storypoints - Create tasks – add time estimates - Calculate capacity for the team - View past Velocity of the team - Commit stories to a sprint

23

Limit the daily stand up meeting to no more than 15 minutes. Everyone in the team, including testers should participate in this meeting. Typically, each person answers the following questions: 1) What did I accomplish yesterday 2) What do I plan to accomplish today 3) What are impediments to progress, if any

It is extremely important to time cap scrum of scrum. It is hard to do so in the beginning. Our stand up meetings used to run up to 30-45 mins. Now, they are down to 15-25 mins. To make the process light weight, we use post-it notes on walls to quickly update the task status. Visibility and transparency can be scary! For example, it took for team members to come to realization that daily update of tasks was not meant to micro-manage their work. Instead it was required to accurately measure team velocity and remove impediments. Similar issues cropped up when testers had to share test infrastructure development backlog. It requires trust among all parties and that takes time to develop.

24

Show how VS11 task board can be used to run daily standup meeting, such as: - Moving tasks from Proposed to Active to Completed - Updating hours - Adding new tasks - Pivot by User story or by Team member - View Burn down charts instantaneously as you update the tasks

25

During Sprint Execution – each team member is focused on the tasks they pick up from the list. As the Devs start the implementation, the QA folks come up with the test plan and the tests that they would cover for that user story. Devs & PMs review this Test Plan since they Devs will be creating unit tests for P0/P1 items, and PMs will be doing a walkthru before the sprint closes. We make it a point not to have more than 2 user stories “in progress” at the same time. Being hard-core about this helped us out during the end of the sprint – where appropriate, only 1 or 2 stories would overflow into next sprint, instead of multiple being overflowing. We also have regular Weekly bug bashes scheduled – primarily driven by the QA team

One form of creativity is Exploratory Testing (XT). Testing can never cover all combinations of inputs, environments, transient conditions, etc. To uncover errors of omissions, our team uses XT. Modern XT tools, such as, Microsoft MTM allow testers to file actionable bugs even when exploring an app. It is quite effective! Too often it happens that developers keep coding new user stories till the last day of the sprint. As a result, testing teams do not get enough time to validate then. This causes the testing activities to spill over to the next sprint. It also causes confusion in

26

the team whether those user stories should be marked done or not. In our team, unless a user story has been thoroughly tested, it is not marked done. There are multiple ways to avoid this situation, as suggested above. We also focus on having an infrastructure in place – like Rolling Builds, Nightly Automation Runs (NAR), Build Verification Tests (BVT) that can catch regressions early.

26

Sample Rolling Build report – for every DLL, we have the tests that failed as part the last changeset, list of recurring failures. If a changeset has broken some of the rolling build tests, then that person has to stop working on the story and make sure the rolling build tests pass It took a little while for us to get the Rolling build infrastructure in place – before this, we were running into multiple regressions every 1-2 sprints. This has clearly saved us lot of time/trouble as we start to approach the end-game

27

Show how using VS11, tester can: - Start exploratory testing for a user story OR without any story - File rich bugs during the exploratory session - Were appropriate, also file a corresponding test case that gets added to the User

story automatically - Analyze the effectiveness of the exploratory sessions across different testers

28

During Sprint Demo – we talk about all the stories that team took up, how many got “done done” meeting the defined done list. We only demo the ones which were completely done. Generally the demos are done by the Dev who implemented it or the QA who tested it – gives an oppurtunity for individual team members to present their work before the management In Post Mortems – we list out on the white board, - what went well – so that we can continue to do those things in upcoming sprints - What to improve – list all the items down, identify what are actionable and then

assign owners to come up with solutions. - List how folks felt the sprint went – was there a rush to get the stories in, was it

going as per plan or people were sitting idle. This is something we started recently to address the work-life balance issues. As we approach the end-game in release, general expectations are that sprint is going to be hot

- The entire team nominates who went above and beyond to help others – collaboration aspect and give them “star awards” – this makes the team members empowered and get peer recognition.

29

You can track all tasks, whether dev or test in one place.

30

Testers need to have visibility into various metrics, such as, code coverage and code churn and test impact to really gauge the quality of the build and take go or no-go decisions.

31

With that we come to the end of this presentation – hopefully I have covered

that Key takeaways for you as listed:

Key Takeaways:

1. Clearly understand the reasons to jump onto “agile software development”

2. Learn from VSTLM team’s experience – teething troubles & how to

overcome them, best practices etc as you adopt “agile software

development”

First gotcha being that this is not a recipe to be followed “as-is” – it definitely requires customization to suit your team/org needs

32