Survey of Effort to Fix Violations of Good Architectural...
19
Survey of Effort to Fix Violations of Good Architectural and Coding Practice Purpose This survey is designed to collect estimates of the time required to fix each of the weaknesses in the four CISQ Quality Characteristic measures. These estimates are needed to develop a measure of the Technical Debt in an application. Average estimates based on the data collected in this form will be used for estimating the effort to fix critical weaknesses in an application, and thus provide an estimate of its Technical Debt in hours of effort which can be transformed into costs in a local currency. Your accurate estimates are critical to this effort. Instructions for Estimating the Time to Fix for Each Weakness The tables below present the 86 weaknesses that comprise the 4 CISQ Quality Characteristic measures. Each violation is listed with its identification number, a description of the weakness, and fields for estimating the time required to fix the weakness and whether it involves fixes to multiple components. We use the term ‘component’ to represent a single code unit such as a class, method, module, subroutine, or similar program element. Please observe the following guidelines when making your estimates. • Estimate the effort that you personally would need to fix each of the violations of good architectural and coding practice listed in the next four sections. To do so, select one of the following values o < 20 minutes o 20 - 60 minutes o 1 - 4 hours o 4 - 8 hours o > 8 hours • If you have not fixed this kind of weakness before, or you do not have a good idea of how long it would take you to fix it, please leave the question blank. • Use the 'best case scenario' as the basis for your estimate. That is, consider that the component(s) to update are not overly complex, not overly exposed to or coupled with the rest of the system, and are well designed for their purpose. Also, consider that the physical environment where you are implementing the change is free of distractions such as noise, interruptions, or a lack of resources such as processing power or memory. • Make your estimates from the perspective of a single language for which you have considerable experience, such as Java, .NET, C++, Python, ABAP, or COBOL. Please indicate if correcting the weakness frequently involves changes to multiple components. • Consider only the time required to analyze the weakness, design and implement the change, and unit test the change. Please include all tasks performed prior to submitting the changed component(s) to a build. Since the CISQ measure is a predictive estimate of technical debt, it does not have to include all activities, especially since integration and testing processes are merged for all the weaknesses corrected during a release.
Transcript of Survey of Effort to Fix Violations of Good Architectural...
Survey of Effort to Fix Violations of Good Architectural and Coding Practice
Purpose This survey is designed to collect estimates of the time required to fix each of the weaknesses in the four CISQ Quality Characteristic measures. These estimates are needed to develop a measure of the Technical Debt in an application. Average estimates based on the data collected in this form will be used for estimating the effort to fix critical weaknesses in an application, and thus provide an estimate of its Technical Debt in hours of effort which can be transformed into costs in a local currency. Your accurate estimates are critical to this effort.
Instructions for Estimating the Time to Fix for Each WeaknessThe tables below present the 86 weaknesses that comprise the 4 CISQ Quality Characteristic measures. Each violation is listed with its identification number, a description of the weakness, and fields for estimating the time required to fix the weakness and whether it involves fixes to multiple components. We use the term ‘component’ to represent a single code unit such as a class, method, module, subroutine, or similar program element. Please observe the following guidelines when making your estimates.
• Estimate the effort that you personally would need to fix each of the violations of good architectural and coding practice listed in the next four sections. To do so, select one of the following values
o < 20 minuteso 20 - 60 minuteso 1 - 4 hourso 4 - 8 hourso > 8 hours
• If you have not fixed this kind of weakness before, or you do not have a good idea of how long it would take you to fix it, please leave the question blank.
• Use the 'best case scenario' as the basis for your estimate. That is, consider that the component(s) to update are not overly complex, not overly exposed to or coupled with the rest of the system, and are well designed for their purpose. Also, consider that the physical environment where you are implementing the change is free of distractions such as noise, interruptions, or a lack of resources such as processing power or memory.
• Make your estimates from the perspective of a single language for which you have considerable experience, such as Java, .NET, C++, Python, ABAP, or COBOL. Please indicate if correcting the weakness frequently involves changes to multiple components.
• Consider only the time required to analyze the weakness, design and implement the change, and unit test the change. Please include all tasks performed prior to submitting the changed component(s) to a build. Since the CISQ measure is a predictive estimate of technical debt, it does not have to include all activities, especially since integration and testing processes are merged for all the weaknesses corrected during a release.
Initial Questions
1) How many years of professional programming experience do you have?
2) What was the highest level of education you completed?
Did not complete high school
High school
2 yr. Associates degree
4 yr. Bachelor’s degree
Masters
Doctorate
3) Do you have a college degree in computer science, information systems, engineering,mathematics, or a related field?
Yes
No
4) Please list the programming languages with which you have at least 2 years of professionaldevelopment experience.
5) What types of programs have you worked on?
6) Please list the primary programming language that you will be considering when making your estimates of time to fix on the following pages. List one programming language.
Estimates to Fix
Reliability Weaknesses
Reliability—the extent to which software contains weaknesses that cause outages, unexpected behavior, instability, data corruption, long recovery times, or other related problems.
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
1 Remove an instance where the content of the first buffer is moved into the content of the second buffer while their allocated sizes are incompatible
Yes
No
2 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. executes a CRUD SQL statement, yet the return code value of the action is not checked anywhere
Yes
No
3 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. reads, writes, or manages an external resource, yet the return code value of the action is not checked anywhere
Yes
No
4 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. contains a catch of an exception whose type is part of a list of overly broad exception types
Yes
No
5 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. throws an exception whose type is part of a list of overly broad exception types
Yes
No
6 Remove an instance where a function is allowed to call itself
Yes
No 7 Remove an instance where a variable,
field, member, etc. is declared, then is evaluated without ever being initialized prior to the evaluation
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
8 Remove an instance where a variable, field, member, etc. is declared with a data type, and then is updated with a value with a second data type, which is incompatible with the first data type
Yes
No
9 Remove an instance where a platform resource (messaging, lock, file, stream, directory, etc.) is allocated and assigned a unique resource handler, and its unique resource handler is used throughout the application, but never released
Yes
No
10 Remove an instance where a value is used as an index in a ‘Read’ or ‘Write’ access to a buffer; yet none of the operations performed prior the buffer access check the value with regards to buffer maximum size
Yes
No
11 Remove an instance where an exception handling block (such as catch and finally blocks) of the function, method, procedure, stored procedure, sub-routine, etc. does not contain any instruction
Yes
No
12 Remove an instance where the serializable field, member, etc. has no serialization operation. Notes: * in the case of technologies with classesand interfaces, this means situationswhere the serializable field, member isfrom a class that implements a serializableinterface but does not implement aserialization method as part of its listcomposed of methods* the serializable nature of an element istechnology dependent, for example,serializable capabilities come from sourcessuch as a serializable attribute in .NET andinheritance from the java.io.Serializableinterface in Java
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
13 Remove an instance where a serializable field, member, etc. is composed of a non-serializable data element. Notes: * in case of technologies with classes andinterfaces, this means situations where theserializable field, member, etc. is from aclass that is serializable but owns the non-serializable field, member* the serializable nature of an element istechnology dependent, for example,serializable capabilities come from sourcessuch as a serializable attribute in .NET andinheritance from the java.io.Serializableinterface in Java
Yes
No
14 Remove an instance where the persistent variable, field, member, etc. has no dedicated operation handling comparison operations. Note: in case of technologies with classes, this means situations where a persistent field, member is from a class that is made persistent while it does not implement methods from the list of required comparison operations (an example in JAVA, is the list composed of {'hashCode()','equals()'} methods).
Yes
No
15 Remove an instance where the application running on an application server, yet uses low-level resource management API (I/O, sockets, class loaders, etc.) and not the resource management API offered by the application server
Yes
No
16 Remove an instance where a variable, field, member, etc. contains a pointer but no dedicated copy operation or copy constructor
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
17 Remove an instance where a class can self-destruct (an example of a self-destruction in C++ is 'delete this')
Yes
No
18 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. has a variable number of parameters, thanks to the variadic parameter in its signature
Yes
No
19 Remove an instance where the float values of variable, field, member, etc. are compared for equality using regular comparison operators (an example in JAVA, is the use of ‘==’ or ‘!=’)
Yes
No
20 Remove an instance where function, method, procedure, stored procedure, sub-routine, etc. executes a data access outside of dedicated data access component thus circumventing the intended design for data access. Notes: * the dedicated data access component can be either client-side or server-side, which means that data access components can be developed using non-SQL languages. * the assumption is made that there already exist dedicated data access components.
Yes
No
21 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. owns a non-final static variable, field, member, etc. while it operates in a multi-threaded environment
Yes
No
22 Remove an instance where a singleton class is instantiated without any prior locking mechanism activation
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
23 Remove an instance where a module has references that cycle back to itself (an example ins JAVA is the existence of cycles between packages)
Yes
No
24 Remove an instance where a parent class has a reference to one of its child classes, directly or indirectly via its methods or its fields, members
Yes
No
25 Remove an instance where a class contains a virtual method yet does not declare any virtual destructor
Yes
No
26 Remove an instance where, for languages in which custom destructors can be written, the parent has no virtual destructor
Yes
No
27 Remove an instance where, for languages in which custom destructors can be written, the child class does not have its own virtual destructor, while its parent class has a virtual destructor one
Yes
No
28 Remove an instance where a variable, field, member, etc. is initialized with a hard-coded network resource identification information
Yes
No
29 Remove an instance where a synchronous call is initiated but the time-out argument is not set or is set to infinite time
Yes
No
Security Weaknesses Security—the extent to which software contains weaknesses that can be exploited to gain unauthorized access to a system to steal data, cause damage, or other malicious acts.
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
1 Remove an instance where the content of the first buffer is moved into the content of the second buffer while their allocated sizes are incompatible
Yes
No
2 Remove an instance where a user input is ultimately used in a ‘Read’ or ‘Write’ access to an array, without any range check between the user input and the array access.
Yes
No
3 Remove an instance where a user input is ultimately used in a formatting statement, without any sanitisation (based on a list of vetted sanitisation functions, methods, procedures, stored procedures, sub-routines, etc.) of the user input value between the user input and the statement. Note: the assumption is made that there already exists vetted sanitisation components.
Yes
No
4 Remove an instance where a user input is ultimately used in a file path creation statement, without any sanitisation (based on a list of vetted sanitisation functions, methods, procedures, stored procedures, sub-routines, etc.) of the user input value between the user input and the statement. Note: the assumption is made that there already exists vetted sanitisation components.
Yes
No
5 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. reads, writes, or manages an external resource, yet the value of the return code is not checked anywhere
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
6 Remove an instance where the application uses a cryptographic list which is not part of the list of vetted cryptographic libraries.
Yes
No
7 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. contains a catch which declares to catch an exception whose type is part of a list of overly broad exception types
Yes
No
8 Remove an instance where the function, method, procedure, stored procedure, sub-routine, etc. throws an exception whose type is part of a list of overly broad exception types
Yes
No
9 Remove an instance where a user input is ultimately used in a file upload statement, without any sanitisation (based on a list of vetted sanitisation functions, methods, procedures, stored procedures, sub-routines, etc.) of the user input value between the user input and the statement. Note: the assumption is made that there already exists vetted sanitisation components.
Yes
No
10 Remove an instance where a variable, field, member, etc. is declared, then is evaluated without ever being initialized prior to the evaluation.
Yes
No
11 Remove an instance where a user input is ultimately used in the loop condition statement, without any range check between the user input and the loop statement.
Yes
No
12 Remove an instance where the shared variable, field, member, etc., is accessed outside a critical section of the application.
Yes
No
Time to Fix Multiple Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
13 Remove an instance where the platform resource (messaging, lock, file, stream, directory, etc.) is deallocated using its unique resource handler which is used later within the application to try and access the resource.
Yes
No
14 Remove an instance where a variable, field, member, etc. is declared with a numerical data type in the ‘Create’ action, and then is updated with a value which is cast via a type cast action into a second numerical data type, which is incompatible with the first data type
Yes
No
15 Remove an instance where a user input is ultimately used to access a resource by name, without any sanitisation (based on a list of vetted sanitisation functions, methods, procedures, stored procedures, sub-routines, etc.) of the user input value between the user input and the statement. Note: the assumption is made that there already exist vetted sanitisation components.
Yes
No
16 Remove an instance where a platform resource (messaging, lock, file, stream, directory, etc.) is allocated and assigned a unique resource handler, and its unique resource handler is used throughout the application along a sequence of operations, but none of which is a release statement.
Yes
No
17 Remove an instance where a user input is ultimately used to execute an OS command, without any sanitisation (based on a list of vetted sanitisation functions, methods, procedures, stored procedures, sub-routines, etc.) of the user input value between the user input and the statement. Note: the assumption is made that there already exists vetted sanitisation components.
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
18 Remove an instance where a user input is ultimately used in a ‘Read’ or ‘Write’ access to a buffer, without any range check between the user input and the buffer access.
Yes
No
19 Remove an instance where a user input is ultimately displayed to the user, without any sanitisation (based on a list of vetted sanitisation functions, methods, procedures, stored procedures, sub-routines, etc.) of the user input value between the user input and the statement. Note: the assumption is made that there already exists vetted sanitisation components.
Yes
No
20 Remove an instance where a variable, field, member, etc., is initialized with a hard-coded literal value, and ultimately used to access a remote resource.
Yes
No
21 Remove an instance where a recursive function, method, procedure, stored procedure, sub-routine, etc., has no execution path to exit the recursion
Yes
No
22 Remove an instance where a user input is ultimately used in a SQL statement, without any sanitisation (based on a list of vetted sanitisation functions, methods, procedures, stored procedures, sub-routines, etc.) of the user input value between the user input and the statement. Note: the assumption is made that there already exists vetted sanitisation components.
Yes
No
Performance Efficiency Weaknesses Performance Efficiency—the extent to which software contains weaknesses that can degrade a system’s performance or cause excessive use of processor, memory, or other resources.
Time to Fix Multiple Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
1 Remove an instance where a variable, field, member, etc. is initialized in a static block of code
Yes
No
2 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc., creates immutable text data via a string concatenation (which could be avoided by using text buffer instead)
Yes
No
3 Remove an instance where a static field or member is declared as static but its parent class is not a singleton class; it does not take into account final static fields or members
Yes
No
4 Remove an instance where a very large table, that is, whose number of rows exceeds a threshold value (default is 1,000,000 rows), is accessed by a SQL statement with too many joins (default threshold value for the maximum number of joins is 5), and too many sub-queries (default threshold value for the maximum number of sub-queries is 3).
Yes
No
5 Remove an instance where the syntax of the SQL SELECT statement and the index configuration of the SQL table or SQL view causes the DBMS to run sequential searches
Yes
No
6 Remove an instance where a very large table, that is, whose number of rows exceeds a threshold value (default is 1,000,000 rows), has too many indices (default threshold value for the maximum number of indices is 3)
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
7 Remove an instance where a very large table, that is, whose number of rows exceeds a threshold value (default is 1,000,000 rows), has an index whose size is too large (default threshold value for the index range is 10)
Yes
No
8 Remove an instance where an operation causing consumption of platform resource (messaging, lock, file, stream, directory, etc.) is directly or indirectly called within a loop body or within a loop condition
Yes
No
9 Remove an instance where a server-side non-stored procedure contains too many data queries (default value for the maximum number of data queries is 5)
Yes
No
10 Remove an instance where a client-side function, method, sub-routine, etc., contains too many data queries (default value for the maximum number of data queries is 2).
Yes
No
11 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. outside a dedicated data manager component (as identified in the vetted data access component list) accesses data.
Yes
No
12 Remove an instance where a variable, field, member, etc., is an aggregate of too many (non-primitive) data types (default value for the maximum number of aggregated non-primitive types is 5)
Yes
No
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
13 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. executes a data resource management action without using a connection pooling capability (the usage of a connection pooling capability is technology dependent; for example, connection pooling is disabled with the addition of 'Pooling=false' to the connection string with ADO.NET or the value of a 'com.sun.jndi.ldap.connect.pool' environment parameter in Java)
Yes
No
14 Remove an instance where a memory resource is explicitly allocated to a variable, field, member, etc. which is used throughout the application, but is never released.
Yes
No
15 Remove an instance where a method references an object, without ever de-referencing it.
Yes
No
Maintainability Weaknesses
Maintainability—the extent to which software contains weaknesses that make software hard to understand or change, resulting in excessive maintenance time and cost as well as higher defect injection rates.
Time to Fix Multiple
Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
1 Remove an instance where the control flow is transferred outside a switch statement (for instance, depending on the technology, by using 'go to', 'continue', or 'break' statements)
Yes
No
2 Remove an instance where a class inherits from too many concrete classes (default threshold value for the maximum number of concrete class Inheritances is 1).
Yes
No
3 Remove an instance where a literal value is used to initialize a variable, field, member, etc. (exceptions are simple integers and a static constant variable, field, member, etc.)
Yes
No
4 Yes
No
6 Remove an instance where a value of a local variable, field, member, etc. used in the condition of a loop is updated within the loop body
Yes
No
7 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. contains too much commented-out code (default threshold value for the maximum percentage of commented out instructions is 2%)
Yes
No
5
Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. has a Fan-Out value that is too large, that is, with too many references to other objects within the application. (default threshold value for the maximum number of references to other objects within the application is 5)
Remove an instance where a control element contains too many commented-out code items compared to a threshold based on the percentage of instructions in the control element that are in comments. (default threshold value for the percentage of commented out instructions is 2%)
Yes
No
Time to Fix Multiple Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
8 Remove an instance where a module has references that cycle back to itself (for example, in JAVA this pattern means cycles between packages)
Yes
No
9 Remove an instance where a file has too many lines of code (default threshold value for the maximum number of lines of code is 1000)
Yes
No
10 Remove an instance where a model of the architectural layers of an application contains too may or too few horizontal layers (excluding the vertical utility layers) based on comparison to a threshold value. The default threshold value for the minimal number of horizontal layers is 4, and the default value for maximal number of horizontal layers is 8.
Yes
No
11 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. is part of two architectural layers.
Yes
No
12 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. has a Cyclomatic Complexity that is too large (default threshold value for maximum Cyclomatic Complexity is 20).
Yes
No
13 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. from a higher horizontal layer directly calls a function, method, procedure, stored procedure, sub-routine, etc. in a lower horizontal layer that is not adjacent to the upper layer making the call, as defined in a model of the application’s architectural layers (this excludes the vertical utility layers that can be referenced from any horizontal layer).
Yes
No
Time to Fix Multiple Components
< 20 minutes
20 – 60 minutes
1 – 4 hours
4 – 8 hours
> 8hours Yes or No
14 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. has too many parameters in its signature (default threshold value for the maximum number of parameters is 7).
Yes
No
15 Remove an instance where a function, method, procedure, stored procedure, sub-routine, etc. has too many SQL or file operations (default threshold value for the maximum number of SQL or file operations is 7).
Yes
No
16 Remove an instance where a variable, field, member, etc. is declared as public.
Yes
No 17 Remove an instance where a method
from a class accesses a field, or member from another class.
Yes
No
18 Remove an instance where a class inheritance level is too large (default threshold value for maximum Inheritance levels is 7).
Yes
No
19 Remove an instance where the a class number of children of a class is too large (default threshold value for maximum number of children of a class is 10).
Yes
No
20 Remove an instance of copy-paste between functions, methods, procedures, stored procedures, sub-routines, etc.
Yes
No
Final Questions
1) In your experience, how much impact does the complexity of the component have on the effort to fixa defect fully contained within its boundary? That is, how much impact would a combination of factorssuch as complicated and unstructured control flow, complex data access, excessive inward coupling,poor cohesion among the functions within the component, and a lack of explanatory comments have onthe time to fix a single defect?
No effect
Less than 50% extra effort
50% to 100% extra effort
More than 100% extra effort
2) In your experience, how does an accumulation of defects to fix within a single component affect theeffort to fix them? Compared to the effort to fix a single defect in a component, would the effort to fixfour or more defects in the component at the same time:
reduce the effort to fix each defect more than 50%
reduce the total effort to fix each defect by up to 50%
negligible, the effort to fix each defect would remain about the same
increase the effort to fix each defect by up to 50%
almost double (2X) the effort required to fix each defect, or increase the
effort to fix each defect by more than twice as much
3) In your experience, how much does the indirect coupling of a component to other components in thesystem affect the effort to fix a defect? Indirect coupling is the propagation of a component’s effectsbeyond the components to which it is immediately coupled, resulting from their connections to othercomponents. So, how much impact would a highly-coupled component have on the effort required toperform corrective actions such as analyzing the defect and the potential side effects of its fix, that youmust perform before submitting the changed component to integration and testing?
No effect
Less than 50% extra effort
50% to 100% extra effort
More than 100% extra effort
4) Would you like to be listed as a contributor on the CISQ website?
Yes
No
5) Would you be willing to speak with us (CISQ) about your responses?
Yes
No
6) If you are willing to be contacted by CISQ, please provide your name and contact information. Your name will not be made public unless you indicate interest in being listed as a contributor on the CISQ website. Your contact information will remain confidential. If you only wish to be included in the drawing for the gift certificates, just list your name and email address.Name: __________________________________________________
Title: ___________________________________________________
Organization: ____________________________________________
Email Address: ___________________________________________
Phone Number: __________________________________________
![Mining Fix Patterns for FindBugs Violations · PMD [5] while C projects use Splint [6], cppcheck [7], or Clang Static Analyzer [9], while Linux driver code are systematically assessed](https://static.fdocuments.us/doc/165x107/5f0b89917e708231d431016a/mining-fix-patterns-for-findbugs-violations-pmd-5-while-c-projects-use-splint.jpg)
