Software Assessment Booklet

8/9/2019 Software Assessment Booklet

1/19

Software

Assessment

Booklet

Tudor Grbawww.tudorgirba.com
http://www.tudorgirba.com/http://www.tudorgirba.com/


2/19


3/19


4/19

Assessment is defined by the Webster dictionary asthe evaluation or estimation of the nature, quality,or ability of someone or something. In otherwords, assessment is what we do when we want toknow about the state of facts, and it is what weshould to do before taking a decision.

Assessment is an act of reasoning. If we want toknow, we have to get to know. If we want to know,we have to gather the facts, understand theirstructure and relationships, and interpret themfrom the point of view of the problem at hand.

An important challenge is posed by large amountsof data. When we cannot grasp all details in a shortamount of time, we tend to perceive the problemto be primarily a tool issue: all will be solved if webuy the perfect reporting tool. This is a rather falseperception.

We can indeed externalize parts of the numbercrunching and learn indirectly from the summaryof the analysis. If we know what the problem athand is, and how to solve it, we can have tools thatautomate the tedious parts. However, we cannotjust treat a tool as a magic black box that analyzesthe data and tells us whether to go left or right. We

have to go beyond the surface and interpret thesituation ourselves.

Any indirect learning involves risks, because asummary presents by definition a reduced set ofdetails that are typically obtained through aninterpretation. The risks can come from thepossibility of the set of details being incomplete orfrom the possibility of the interpretation being

incorrect.

What is assessment?


5/19

To properly grasp the output of a tool, we need toknow what details it looks at and whatinterpretations are involved in the analysis process.For example, lets suppose we can have a tool tomeasure some characteristics of our system. Wehave to know what parts of the system weremeasured and how they are measured. Only

afterwards can we interpret the result.It is not always feasible to control and knoweverything. Due to practical reasons, such as timeconstraints, we sometimes do have to rely onsummaries provided either by tools or by otherpeople. We just have to make sure that we do notpretend we know when we do not.

For example, suppose we need to find the bestresource for assessment. One solution is to spawn

a search engine and search for assessment. We canbe happy with the first result we get (probably thewikipedia article that talks about educationalassessment). However, we cannot claim that this isthe best article because we have not much of anidea about what were the decisions taken by thesearch algorithm and we can certainly have not

much of an idea of the quality the input data. Wecan say that we trust the result, but we cannot claimthat we know.

The difference might look small, but it is critical.Knowledge implies control, trust impliesacceptance of risks. Assessment is all about makingthe situation as explicit as possible, so that thedecision can be as informed as possible.
http://www.google.com/search?rls=en&q=assessment&ie=UTF-8&oe=UTF-8http://www.google.com/search?rls=en&q=assessment&ie=UTF-8&oe=UTF-8


6/19

Software assessment is a critical activity that shouldbe addressed explicitly in the development process.

Why? Because software systems are so large andcomplex that teams spend up to 50% of theiroverall effort to understand the system.

Given that 50% is considerably large, you would

expect a significant amount of tools to be gearedtowards this side. In practice, developers mostlyrely on code reading. However, software systemsare so large that relying on reading them just doesnot scale when we want to understand them as awhole. To put it in perspective, a person that readsone line in two-seconds would requireapproximately one month of work to read aquarter of a million lines of code. And this is justfor reading the code.

So, do developers actually read the entire code?Not really. They typically limit the reading to a part

of the system, while the global view is left for thedrawing board from the architects office. Thus,most decisions tend to be local, while those that areglobal are mostly based on inaccurate information.

To rectify the situation we need tools that help uscrunch the vastness of data. However, not any tooldoes the job. Many tools exist out there to dealwith various aspects of data and software analysis,but most of them take the oracle way: they offersome predefined analyses that provide answers tostandard questions.

Receiving a standard answer is great when you havea standard question. However, it turns out thatmost of the time our questions are not quitestandard. In these situations, regardless how smartthe analysis is, it is of limited use.

We need customized tools that provide contextualfeedback.

Rethinking software assessment


7/19

One reason regression testing is so useful is that itprovides feedback that is continuous and highlycontextual, thus it has great potential to leading tocorrective actions.

But, building regression tests was not always sowelcome. It took more than a decade of agility tomake tests gain their rightful place. And its noteven a close chapter as we can still find skepticspointing their finger to the costs of writing tests,and how this takes away from overall effort thatcould otherwise be spent on active programming.

While testing is important, it concerns but oneaspect of a software system, namely the expectedfunctionality. Ideally, we would like to createassessment tools just like we now create tests. Theonly trouble is that the cost of building such toolsis too high. Or at least it is perceived to be,especially if you have to write them from scratch.A middle ground can be found by having a

platform upon which these tools can be built.Drawing from the testing parallel, we would needthe correspondent of an xUnit-like platform.

With such a platform, the process of assessmentwould be turned around. Instead of conforming togeneric tools, that are often prohibitively expensive,development teams would start crafting their owntools, tools that make sense for their project, fortheir process, for their practices. Instead of relyingprimarily on manual reviews, quality assuranceteams would build automatic reporting tools withdedicated analyses. Instead of specifyingarchitecture rules on paper, architects wouldencode them in automatic checkers.

It would be a new way of approaching softwaredevelopment. A way in which the possibility ofmaking unintentional mistakes is embraced andactively guarded against.


8/19

Moose is an extensive platform for software anddata analysis that makes software assessmentpractical.

It is a free and open-source project started in 1996.Since then it spread to several research groups andhas been applied in multiple industrial contexts.

The design of Moose is rooted in the core idea ofplacing the analyst at the center and ofempowering him to build and to control theanalysis every step of the way.

Data from various sources and in various formatsis imported and stored in a model. For example,Moose can handle out of the box variouslanguages: Java, JEE, Smalltalk, C, C++, XML.

On top of the created model, the analyst performsanalyses and combines and compares results tofind answers to specific questions.

On the one hand, Moose comes with multiplepredefined services such as: parsing, modeling,metrics computation, visualization, data mining,duplication detection, or querying. These basicservices are aggregated into more complex analyseslike computation of dependency cycles, detectionof high level design problems, identification ofexceptional entities and so on.

On the other hand, Moose offers an infrastructurethrough which new analyses can be quickly builtand can be embodied in new interactive tools.Using this platform the analyst can easily definenew analyses, create new visualizations, or buildcomplete browsers and reporting tools altogether.

More information about Moose can be found at:

http://moosetechnology.orghttp://themoosebook.org

Practical software assessment with Moose
http://themoosebook.org/http://moosetechnology.org/http://themoosebook.org/http://themoosebook.org/http://moosetechnology.org/http://moosetechnology.org/


9/19

Importers Model

Repository

Meta-models

Tool building

infrastructure

Generic

analysis

services

Dedicated

interactive

tools

Aggregated

analysis

services}

{

The Moose workflow


10/19

System Complexity

Dependency

structure matrix

Class Blueprint

Side-by-side

duplications


11/19

Generic Moose browser

A code browser revealing annotations


12/19

Our client was responsible for the developmentand maintenance of a large portfolio of softwareprojects. The software development wasperformed both by in-house teams and outsourcedto external teams. As the number of projectsincreased, the need for a continuous assessment ofthe software became apparent so as to ensure thequality and compliance of the work.

We were mandated to introduce softwareassessment techniques and tools to improve thesoftware development productivity.

We identified two areas of improvement. On theone hand, teams needed an overview of theirsystems. On the other hand, the architects neededto check conformance with the organization-wideguidelines.

To address these issues in two ways: we introducedcustom reporting tools, and we coached the

developers to make explicit their technicaldecisions.

These tools were custom in that they containedconcerns that were identified as being important bythe development team and by the architects. Theywere integrated in the continuous integrationprocess and produced dedicated reports with

highly contextual feedback every night.

To get the stakeholders to define explicit concerns,we held several interactive workshops in which weidentified problems and answered them on the fly.

The improvements was positively received. Afteran analysis session held together with the team, oneof the developers said: I have never seen my code

in this way. It is eye-opening.

Success story: Continuous assessment


13/19

A report example showing a map of classes

A report example showing guidelines violations


14/19

Our clients IT-Architecture department publisheda set of architectural guidelines and codingconventions for the internal and external softwaresystems. They needed a way to verify complianceof one of their software system.

Our role was to review the application and verifyits conformance to the guidelines.

After obtaining an overview, we then focussed onthe architectural layer and interface boundaries. Wevalidated the coarse grained architectural rules. Weused visualization techniques and applied variousdetection strategies to highlight points of interestand irregularities in the code. On closer codeinspection, we identified several violations of theguidelines.

Once the non-conforming parts were identified, weproposed concrete recommendations of how to

make the application conform. For example, one ofthe detected shortcomings of the application wasthe poor exception handling that violated thearchitectural constraints. We proposed concretemeasures of how to provide a number ofrefactorings for these parts of the system.

Though the main focus of the review was in the

guidelines verification, we also provided a summaryof the overall code quality. Among others wepinpointed how the logic is distributed over thesystem and how the code duplication should berefactored.

The architectural violations were scheduled forrefactoring before the application was to go intoproduction.

Success story: Architecture conformance


15/19

A visualization highlighting violations


16/19

The client had a large software system composedof several hundred subsystems, each being writtenin Java. However, the overall system was based on acustom engine that was put together via amultitude of custom configurations written inXML and in another custom language. Given theparticularities of the project, a dedicated tool wasneeded to analyze it.

We received the task of crafting a dedicated toolthat would be able to provide an overview of theseconfigurations and expose their variousrelationships.

The prerequisite for any data analysis is thespecification of the meta-model for the data. Thus,the first step was the analysis of the variousconfiguration files with the aim of capturing theclass diagram. The diagram on the next page showsthe anonymized result of the configurationanalysis.

Once the meta-model specified, we encoded it inthe tool.

The next step was to create the importer to take allconfiguration files as input and then to create acoherent model. The last step was to create severalanalyses and visualizations on top of this model.For building these, we used the tool building

capabilities of Moose. An screenshot of the toolcan be seen on the next page.

These views were then integrated into aninteractive tool. When used by developers, the toolrevealed several unexpected dependencies andincomplete specifications in the system.

All in all, the effort totaled a mere 10 person-days.

Put in the overall context of the multiple hundredperson-years of development effort spent in theproject, it was an insignificant investment with highreturn on investment.

Success story: Crafting a custom analysis tool


17/19

A part of the custom tool for dependency overviewA schematic view of the complex meta-model

B

L

C

A

I

O

E

M

*

*

*

F

G

H

D

*

J *

*

K

*

P

*

Q

*


18/19

Assessment is all about feedback, and the bestfeedback is continuous and contextual.

Assessment requires a driving set of questions andaccess to data.

Specific questions and custom data require specificanswers provided by custom tools.

Figuring out what data is important and how toexpose requires dedicated training.

There always is a way to find an answer out ofdata.

Building an assessment tool is similar to developingany software project and thus appropriate attentionand investment should be allocated to it.

Using a proper infrastructure can significantlydecrease the overall cost of building a tool.

While technology is important, we have to

remember that assessment is a human activity.

A good tool does not guarantee ideal solutions, butit can provide accurate input for better decisions.

Software assessment wisdom


19/19

2010 Tudor Grba

www.tudorgirba.com
http://www.tudorgirba.com/http://www.tudorgirba.com/

Software Assessment Booklet

Documents

Transcript of Software Assessment Booklet