Does Your Configuration Code Smell?

Tushar Sharma, Marios Fragkoulis and Diomidis SpinellisDept of Management Science and TechnologyAthens University of Economics and Business

Athens, Greece{tushar,mfg,dds}@aueb.gr

ABSTRACTInfrastructure as Code (IaC) is the practice of specifyingcomputing system configurations through code, and manag-ing them through traditional software engineering methods.The wide adoption of configuration management and in-creasing size and complexity of the associated code, promptfor assessing, maintaining, and improving the configurationcode’s quality. In this context, traditional software engi-neering knowledge and best practices associated with codequality management can be leveraged to assess and manageconfiguration code quality. We propose a catalog of 13 im-plementation and 11 design configuration smells, where eachsmell violates recommended best practices for configurationcode. We analyzed 4,621 Puppet repositories containing 8.9million lines of code and detected the cataloged implemen-tation and design configuration smells. Our analysis revealsthat the design configuration smells show 9% higher aver-age co-occurrence among themselves than the implementa-tion configuration smells. We also observed that configura-tion smells belonging to a smell category tend to co-occurwith configuration smells belonging to another smell cate-gory when correlation is computed by volume of identifiedsmells. Finally, design configuration smell density showsnegative correlation whereas implementation configurationsmell density exhibits no correlation with the size of a con-figuration management system.

CCS Concepts•Software and its engineering → Specification lan-guages; Software maintenance tools; Software librariesand repositories; Software design engineering;

KeywordsInfrastructure as Code, Code quality, Configuration smells,Technical debt, Maintainability

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.

MSR’16, May 14-15, 2016, Austin, TX, USAc© 2016 ACM. ISBN 978-1-4503-4186-8/16/05. . . $15.00

DOI: http://dx.doi.org/10.1145/2901739.2901761

1. INTRODUCTIONInfrastructure as Code (IaC) [13] is the practice of spec-

ifying computing system configurations through code, au-tomating system deployment, and managing the system con-figurations through traditional software engineering meth-ods. For example, a server farm that contains numerousnodes with different hardware configurations and differentsoftware package requirements can be specified using con-figuration management languages such as Puppet [39], Chef[37], CFEngine [4], or Ansible [1] and deployed automaticallywithout human intervention. Such automated yet customconfigured deployment is not only faster than the manualprocess but is also reliable and repeatable.

Apart from automating an infrastructure deployment, theIaC paradigm brings the infrastructure, the code and thetools and services used to manage the infrastructure, in thepurview of a software system. Therefore, IaC practices treatconfiguration code similar to the production code and applytraditional software engineering practices such as reviewing,testing, and versioning on configuration code as well.

A lot of work has been done to write maintainable code[8, 21] and achieve high design quality [36] in traditionalsoftware engineering. Similar to production code, configu-ration code may also become unmaintainable if the changesto configuration code are made without diligence and care.In a recent study, Jiang et at. [14] argued that configura-tion code must be treated as production code due to thecharacteristics and maintenance needs of the configurationcode. Therefore, traditional code and design quality prac-tices must be adopted to write and maintain high qualityconfiguration code.

In this context, we planned a preliminary quality analysisof configuration code where we focused on the maintainabil-ity aspect of the configuration code quality. We pose thefollowing questions to achieve the above stated goal:

1. What is the distribution of maintainability smells inconfiguration code? Which smells are commonly foundand which ones are rarely found?

2. What is the relationship between the occurrence of de-sign configuration smells and implementation configu-ration smells?

3. Is the principle of coexistence applicable to smells inconfiguration projects?

4. Does smell density depend on the size of a configura-tion project?

To answer these research questions, we compiled a catalogof 24 configuration smells. We used the existing Puppet-Lint[27] tool and extended its rule-base to identify implementa-tion configuration smells. We developed a tool namely Pup-peteer to identify cataloged design configuration smells. Weextracted 4,621 repositories containing Puppet code fromGitHub and applied the tools on all the repositories. Usingthe collected data, we provide the empirical answers to eachof the research questions listed above.

Some key observations from the study are the following.

• Configuration smells belonging to a smell category tendto co-occur with configuration smells belonging to an-other smell category when correlation is computed byvolume of identified smells.

• Design configuration smells show 9% higher averageco-occurrence among themselves than the implemen-tation configuration smells.

• Design configuration smell density shows negative cor-relation whereas implementation configuration smelldensity exhibits no correlation with the size of a con-figuration management system.

Studying code quality of configuration management sys-tems can help us understand the characteristics of configura-tion code. This study can benefit the researchers by expos-ing them to a method for investigating system configurationpractices by mining repositories. At the same time, prac-titioners can identify configuration smells using the toolsemployed in this study and adopt best practices to writemaintainable configuration code.

The rest of the paper is structured as follows: Section 2provides an overview of the study with a goal and corre-sponding research questions. Section 3 describes a catalogcontaining implementation and design configuration smells.Section 4 details the developed tool, mining Puppet repos-itories from GitHub and applying the tool on the reposi-tories. Section 5 provides detailed answers to the researchquestions. We discuss related work in Section 6.

2. OVERVIEW OF THE STUDYWe planned a preliminary study to analyze the existing

configuration code and evaluate the associated code qualityto examine the existing practices towards keeping configura-tion code maintainable. Although, there are many attributesand sub-attributes of code quality (such as reliability andportability), the focus of this study is on the maintainabil-ity attribute of configuration code quality.

We formulated the following research questions towardsthe quality analysis goal of configuration code.

RQ1. What is the distribution of maintainability smells inconfiguration code? We investigate the distribution ofconfiguration smells to find out whether there exists aset of implementation and design configuration smellsthat occur more frequently with respect to another setof configuration smells.

RQ2. What is the relationship between the occurrence ofdesign configuration smells and implementation con-figuration smells? We study the instances of designconfiguration smells and implementation configurationsmells to discover the degree of co-occurrence betweenthe two categories of configuration smells.

RQ3. Is the principle of coexistence applicable to smellsin configuration projects? In traditional software en-gineering, it is said that patterns (and smells) co-existas “No pattern is an island” [3] i.e. if we find one, it isvery likely that we will find many more around it [3,36]. We investigate the intra-category co-occurrenceof a smell with other smells to find out whether thefolklore is true in the context of configuration smells.Furthermore, whether all the smells in each of the cat-egories follow the principle with a same degree.

RQ4. Does smell density depend on the size of the config-uration project? Smell density is a normalized metricthat represents the average number of smells identi-fied per thousand lines of code. We investigate the re-lationship between the size of a configuration projectand associated smell density for both the smell cate-gories to find out how the smell density changes as thesize of the configuration project increases.

We present a theoretical model of configuration smellsto study the research questions listed above. The modelcontains two categories of configuration smells namely im-plementation configuration smells and design configurationsmells with a brief description of each smell.

To detect the majority of implementation configurationsmells, we used Puppet-Lint [27]. Due to the lack of an ex-isting tool that can detect design configuration smells, wedeveloped a tool namely Puppeteer for detecting the cata-loged design configuration smells.

We identified repositories containing Puppet code and down-loaded them from GitHub [10]. We downloaded 4,621 repos-itories containing 142,662 Puppet files and 8.9 million linesof code and analyzed them with the help of Puppet-Lint andPuppeteer. We grouped the information collected based onthe data required to answer the research questions and de-duce our observations.

We assume that the reader of this paper has a fair ideaabout Puppet and syntax of its programming language. Ifit is not the case, we encourage the reader to familiar him-self/herself with configuration management [33] through Pup-pet [26].

3. THEORETICAL MODEL: CONFIGURA-TION SMELLS

We define configuration smells as follows:Configuration smells are the characteristics of a configu-

ration program or script that violate the recommended bestpractices and potentially affect the program’s quality in anegative way.

In traditional software engineering practices, bad smellsare classified as implementation (or code) smells [8], designsmells [36], and architectural smells [9] based on the granu-larity of abstraction where the smell arises and affects. Sim-ilarly, configuration smells can also be classified as imple-mentation configuration smells, design configuration smells,documentation configuration smells, and so on. In this pa-per, our focus is on two major categories of configurationsmells namely implementation configuration smells and de-sign configuration smells.

We assigned a three letter acronym starting from I foreach implementation configuration smell and from D foreach design configuration smell. We use these acronyms inthe later sections of the paper.

3.1 Implementation Configuration SmellsImplementation configuration smells are quality issues such

as naming convention, style, formatting, and indentation inconfiguration code. After studying available resources, suchas the Puppet style guide [35] and rules implemented byPuppet-Lint, we prepared a list of recommended best prac-tices. We grouped the best practices based on their similar-ity and arrived at a corresponding implementation configu-ration smell when a best practice is violated. Table 1 liststhe implementation configuration smells and correspondingset of best practices.

Here, we present a list of implementation configurationsmells with a brief description. Figure 1 shows an annotatedPuppet example with all the cataloged implementation con-figuration smells.

Missing Default Case (imd) A default case is missing in acase or selector statement.

Inconsistent Naming Convention (inc) The used namingconvention deviates from the recommended namingconvention.

Complex Expression (ice) A program contains a difficultto understand complex expression.

Duplicate Entity (ide) Duplicate hash keys or duplicateparameters present in the configuration code.

Misplaced Attribute (ima) Attribute placement within a re-source or a class has not followed a recommended order(for example, mandatory attributes should be specifiedbefore the optional attributes).

Improper Alignment (iia) The code is not properly aligned(such as all the arrows in a resource declaration) ortabulation characters are used.

Invalid Property Value (ipv) An invalid value of a propertyor attribute is used (such as a file mode specified using3-digit octal value rather than 4-digit).

Incomplete Tasks (iit) The code has “fixme” and “todo”tags indicating incomplete tasks.

Deprecated Statement Usage (ids) The configuration codeuses one of the deprecated statements (such as “im-port”).

Improper Quote Usage (iqu) Single and double quotes arenot used properly. For example, boolean values shouldnot be quoted and variable names should not be usedin single quoted strings.

Long Statement (ils) The code contains long statements(that typically do not fit in a screen).

Incomplete Conditional (iic) An “if..elsif” construct usedwithout a terminating “else” clause.

Unguarded Variable (iuv) A variable is not enclosed inbraces when being interpolated in a string.

3.2 Design Configuration SmellsDesign configuration smells reveal quality issues in the

module design or structure of a configuration project. Var-ious available sources, such as the Puppet style guide [35],blog entries [16, 17], and videos of technical talks [18] high-light the best practices to be followed for configuration code.We obtained a list of commonly occurring design configu-ration smells from the violation of these best practices atdesign-level. We assigned relevant names (many a timesinspired by the traditional names of smells) to the smellsand documented their forms representing variations of thesmells. Here, we present design configuration smells with abrief description.

Multifaceted Abstraction (dmf) Each abstraction (e.g. a re-source, class, ‘define’, or module) should be designed tospecify the properties of a single piece of software. Inother words, each abstraction should follow single re-sponsibility principle [20]. An abstraction suffers frommultifaceted abstraction when the elements of the ab-straction are not cohesive.

The smell may occur in the following two forms:

• a resource (file, package, or service) declarationspecifies attributes of more than one physical re-sources, or

• all the language elements declared in a class, ‘de-fine’, or a module are not cohesive.

Unnecessary Abstraction (dua) A class, ‘define’, or mod-ule must contain declarations or statements specifyingthe properties of a desired system. An empty class,‘define’, or module shows the presence of unnecessaryabstraction smell and thus must be removed.

Imperative Abstraction (dia) Puppet is declarative in na-ture. The presence of imperative statements (such as“exec”) defies the purpose of the language. An ab-straction containing numerous imperative statementssuffers from imperative abstraction smell.

Missing Abstraction (dma) Resource declarations and state-ments are easy to use and reuse when they are encap-sulated in an abstraction such as a class or ‘define’.A module suffers from the missing abstraction smellwhen resources and language elements are declaredand used without encapsulating them in an abstrac-tion.

Insufficient Modularization (dim) An abstraction suffersfrom this smell when it is large or complex and thuscan be modularized further. This smell arises in fol-lowing forms:

• if a file contains a declaration of more than oneclass or ‘define’, or

• if the size of a class declaration is large crossing acertain threshold, or

• the complexity of a class or ‘define’ is high.

Duplicate Block (ddb) A duplicate block of statements morethan a threshold indicates that probably a suitable ab-straction definition is missing. Thus a module contain-ing such a duplicate block suffers from duplicate blocksmell.

Table 1: Mapping Between Implementation Configuration Smells and Corresponding Best PracticesSmells Best practicesMissing default case Case and Selector statements should have a default caseInconsistent naming convention The names of variables, classes and defines should not contain a dashComplex expression Expressions should not be too complexDuplicate entity Duplicated hash keys and parameters should be removedMisplaced attribute • “ensure” attribute should be the first attribute specified

• The required parameters for a class or ‘define’ should be listed before optionalparameters• Right-to-left chaining arrows should not be used

Improper alignment • Properly align arrows (arrows are not all placed one space ahead of the longestattribute)• Tabulation characters should not be used

Invalid property value • “ensure” property of file resource should be valid• File mode should be represented by a valid 4-digit octal value (rather than 3)

or symbolically• The path of “puppet:///” url should start with “modules/”

Incomplete tasks “fixme” and “todo” tags should be handledDeprecated statement usage Deprecated node inheritance and “import” statement should not be usedImproper quote usage • Booleans should not be quoted

• Variables should not be used in single quoted strings• Unquoted node names should not be used• Resource titles should be quoted• Literal boolean values should not be used in comparison expressions

Long statement Lines should not be too longIncomplete conditional “if ... elsif” constructs shall be terminated with an “else” clauseUnguarded variable Variables should be enclosed in braces when being interpolated in a string

Figure 1: An annotated Puppet example with all the cataloged implementation configuration smells

Broken Hierarchy (dbh) The use of inheritance must belimited to the same module. The smell occurs when,the inheritance is used across namespaces where inheri-tance is not natural (“is-a”relationship is not followed).

Unstructured Module (dum) Each module in a configura-tion repository must have a well-defined and consis-tent module structure. A recommended structure fora module is the following.

Module name

mani f e s t sf i l e stemplatesl i bf a c t s . dexamplesspec

An ad-hoc structure of a repository suffers from un-structured module smell that impacts understandabil-ity and predictability of the repository.

Dense Structure (dds) This smell arises when a configu-ration code repository has excessive and dense depen-dencies without any particular structure.

Deficient Encapsulation (dde) This smell arises when anode definition or ENC (External Node Classifier) de-clares a set of global variables to be picked up by theincluded classes in the definition.

Weakened Modularity (dwm) Each module must strive forhigh cohesion and low coupling. This smell arises whena module exhibits high coupling and low cohesion.

4. MINING GITHUB REPOSITORIESWe followed the procedure given below to select and down-

load the repositories.

1. We employed GHTorrent [11, 12] to select GitHubrepositories to download.

2. There are various options to choose from to selectthe subject systems such as number of commits andcommitters, stars, and number of relevant files in therepository. Each of the options (or their combinations)present different tradeoffs. For instance, there are only838 Puppet repositories that have five or more stars.We wanted to analyze larger number of repositories toincrease the generalizability of the observations. Re-ducing the number of stars as a selection criterionwould have resulted in more number of repositories;however, the significance of the criterion would havereduced. A high number of commits in a repositoryshows continuous evolution and thus we chose num-ber of commits as the selection criterion. We choseto download all the repositories where the number ofcommits was more than or equal to 40. The above cri-terion provided us a list of 5,387 Puppet repositoriesto download.

3. We were able to download 4,621 repositories exceptsome private repositories.

Table 2: Characteristics of the Downloaded Repos-itories

Attributes ValuesRepositories 4,621Puppet files 142,662Class declarations 132,323Define declarations 39,263File resources 117,286Package resources 49,841Service resources 18,737Exec declarations 43,468Lines of code (Puppet only) 8,948,611

Table 2 summarizes the characteristics of the downloadedrepositories. We observed, by random sampling, that thedownloaded repositories were either standalone Puppet-onlyrepositories or system repositories (where production codeas well as configuration code has been put together into arepository).

4.1 Analyzing Puppet RepositoriesWe analyzed the downloaded repositories to detect imple-

mentation and design configuration smells. We used Puppet-Lint tool to detect majority of implementation configurationsmells. We executed the tool on all the repositories andstored the generated output. In addition to use Puppet-Lint, we wrote our custom rules to detect the implementa-tion configuration smells that the tool was not detecting (forinstance, complex expression and incomplete conditional).We then aggregated the number of individual implementa-tion smells that occurred in each repository using the gen-erated output and our mapping of best practices to the im-plementation smells (see Table 1).

We developed a tool, Puppeteer,1 to detect design con-figuration smells listed in Section 3.2. We discuss detectionstrategies of all the smells detected by Puppeteer in the nextsubsection.

The data generated by the tools mentioned above for boththe smell categories for all the analyzed repositories can befound here.2

4.2 Design Configuration Smells - DetectionStrategies

This section discusses detection strategies that Puppeteeruses to identify design configuration smells.

Multifaceted Abstraction The detection strategy for the twoforms of the smell is as follows.

1. We compute a metric, “Physical resources definedper resource declaration”, for each declared re-source. We report the smell when the metric valueis more than one.

2. We compute lack of cohesion for the configura-tion abstractions to detect the second form of thesmell. Traditional software engineering uses thelcom (Lack of Cohesion Of Methods) [5] metricto compute lack of cohesion for an abstraction.The same metric cannot be used for configuration

1https://github.com/tushartushar/Puppeteer2https://github.com/tushartushar/configSmellData

code due to its different structure and character-istics. We use the following algorithm to computelcom in a configuration code abstraction.

(a) Consider each declared element (such as file,package, service resources and exec statements)as a node in a graph. Initially, the graphcontains the disconnected components (dc)equal to the number of elements.

(b) Identify the parameters of the abstraction,used variables, and literals (such as file name).Call them as data members collectively.

(c) For each data member, repeat the following:identify the components that uses the datamember. Merge the identified components ina single component.

(d) Compute lcom:

LCOM =

1

|DC| if |DC| > 0

1 otherwise

(1)

Note that we compute lcom for each class, ‘define’,and file. Therefore, it is quite possible that the toolreports more than one instance of this smell in a singlePuppet file.

Unnecessary Abstraction We compute a metric namely“Sizeof the abstraction body”. A zero value of the metricshows that the abstraction doesn’t contain any decla-rations and thus suffers from unnecessary abstractionsmell.

Imperative Abstraction We compute a metric namely “To-tal ‘exec’ declarations” in a given abstraction. Thetool reports the imperative abstraction smell when theabstraction has more than two ‘exec’ declarations andratio of the ‘exec’ declarations against all the elementsin the abstraction is more than 20%.

Missing Abstraction We identify total number of configu-ration elements except classes or defines that are notencapsulated in a class or ‘define’. A module suffersfrom the smell if there are more than two such elementsin the module.

Insufficient Modularization The detection strategy for thethree forms of the smell is as follows.

1. We count the number of classes and defines de-clared in a Puppet file. We report the smell if afile defines more than one class and ‘define’.

2. We count the number of lines in an abstraction.If a class or ‘define’ contains more than 40 linesof code, it suffers from the smell.

3. We compute maximum nesting depth for an ab-straction. An abstraction with maximum nestingdepth more than three suffers from this smell.

Duplicate Block We use the pmd-cpd [6] tool to identifycode clones. A module suffers from this smell when acode clone of larger than 150 tokens gets identified inthe module.

Broken Hierarchy For all the class definitions, we identifythe inherited class (if any). If the inherited class is de-fined in any other module, the class suffers from “bro-ken hierarchy” smell.

Unstructured Module The detection strategy for the threeforms of the smell is as follows.

1. We search for a folder named “manifests” in theroot folder of the repository. If the total numberof Puppet files in the folder is more than five whilethere is no folder containing the string “modules”,the smell gets detected.

2. We find a folder containing the string “modules”and treat all the sub-folders as separate modules.Each module must have a folder named “mani-fests”. Absence of the folder shows the presenceof the smell.

3. In each module, we count the unexpected files andfolders. Expected files and folders are: “man-ifests”, “files”, “templates”, “lib”, “tests”, “spec”,“readme”, “license”, and “metadata”. A modulewith more than three such unexpected files orfolders suffers from the smell.

Dense Structure We prepare a graph for each repository todetect the smell. Each module is treated as a node andany reference from the module to another module istreated as an edge. We, then compute average degreeof the graph.

AvgDegree(G) =2× |E||V | (2)

where |E| and |V | are number of edges and nodes re-spectively. A graph more than 0.5 average degree suf-fers from Dense structure smell.

Deficient Encapsulation We count the number of globalvariables declared for each node declaration, followedby at least one include statement. If a node declara-tion has one or more such global variables, the modulesuffers from deficient encapsulation smell.

Weakened Modularity We compute modularity ratio [2] foreach module as follows:

ModularityRatio(A) =Cohesion(A)

Coupling(A)(3)

where, Cohesion(A) refers to the number of intra-module references and Coupling(A) refers to the num-ber of inter-module references from module A. We re-port the smell if the ratio is less than one.

5. RESULTS AND DISCUSSIONThis section presents the results gathered from the anal-

ysis and our observations w.r.t. each research question ad-dressed in this paper.

We use the term “total detected smells (by volume)” torefer to all the smell instances detected in a project. Weuse the term “total detected smells (by existence)” to referto the number of different types of smell instances detectedin a project. Additionally, we refer to each cataloged con-figuration smell as a three letter acronym as defined in theSection 3.

Table 3: Distribution of Detected Implementation(ICS) and Design (DCS) Configuration Smells

ICS #I(V) #I(E) DCS #I(V) #I(E)IMD 4,604 706 DMF 64,266 4,339INC 4,804 440 DUA 4,319 1,427ICE 3,994 963 DIA 4,354 1,575IDE 65 29 DMA 1,913 813IMA 22,976 1,383 DIM 96,033 4,422IIA 780,265 3,064 DUM 4,653 3,337IPV 14,360 729 DDB 17,601 1,016IIT 11,071 1,467 DBH 83 37IDS 6,466 674 DDS 1760 1760IQU 428,951 2,463 DDE 1,075 424ILS 527,637 4,115 DWM 13,944 2,890IIC 4,797 1,217IUV 71339 1,405

RQ1. What is the distribution of maintainability smells inconfiguration code?

Approach: We compute the total number of detectedsmell instances (by volume and by existence) for all thesmells belonging to both implementation and design con-figuration smells categories.

Results: The left pan of Table 3 shows the number ofsmell instances detected for implementation configurationsmells (ics) both by volume (i(v)) and by existence (i(e)).The three most frequently occurring smells by volume andby existence are iia (improper alignment), iqu (improperquote usage), and ils (long statement). Similarly, ide (du-plicate entry), imd (missing default case), inc (inconsistentnaming convention) are some of the least frequently occur-ring smells.

The right pan of Table 3 shows the similar distributionfor detected design configuration smells (dcs). The most fre-quently occurring design configuration smells are dim (insuf-ficient modularization) and dmf (multifaceted abstraction).Similarly, the least occurring smells are dbh (broken hierar-chy) and dde (deficient encapsulation).

A few observations from the above table are the following.

• There is a relatively large number of smell instances re-ported for ddb (duplicate block) by volume; however,the smell only occurs in less than one forth of the ana-lyzed repositories. This indicates that either thedevelopers of Puppet repositories do not dupli-cate the code at all or they do it massively.

• Although, investigating and establishing the potentialreasons of identified smell instances is not in the scopeof this study, the nascent maturity phase of currentconfiguration systems could be a cause for a few smells.Specifically, the support for system configuration codein terms of better tools and IDEs is still maturingwhich could potentially avoid smells such as iia (im-proper alignment).

The reported frequently occurring smells may also mo-tivate efforts in the future to identify their causes andsteps to avoid them. Such efforts may focus on im-proving existing documentation, enhancing languagesupport, and developing new tools.

• It is interesting to note that dds (dense structure) fallsin the least occurring smell category by volume butmost frequently occurring smell category by existence.It is due to the fact that there could be only one in-stance at the most for this smell in a project.

dum (unstructured module) also exhibits similar char-acteristics. Since the tool analyzes the structure of amodule as a whole, dum gets identified at the mostonce for a module. Since each project deals with onlya limited number of modules, the detected smell in-stances are relatively low whereas the smell occurredin relatively large number of analyzed projects.

RQ2. What is the relationship between the occurrence ofdesign configuration smells and implementation con-figuration smells?

Approach: We count the total number of implementa-tion and design configuration smells for each Puppet repos-itory both by volume and by existence. Next, we computeSpearman’s correlation coefficient between the counted im-plementation and design configuration smells for each repos-itory by volume and by existence.

Results: Figure 2 presents a scatter graph (with alphaset to 0.3) showing the co-occurrence between implementa-tion and design configuration smells by volume. The figureshows a dense accumulation towards the left-bottom indi-cating that implementation and design configuration smellsco-occur together for relatively small number of identifiedsmell instances.

Figure 2: Co-occurrence between implementationand design configuration smells (by volume)

Figure 3 shows a density graph showing the co-occurrencebetween implementation and design configuration smells byexistence. The figure reveals a dense correlation between im-plementation and design configuration smells (by existence)in the left bottom quadrant of the figure where the numberof identified smell types is half or less.

We compute Spearman’s correlation coefficient for boththe cases. Table 4 shows the correlation coefficients andassociated p-value. Both the analyses show positive cor-relation between implementation and design configuration

Figure 3: Co-occurrence between implementationand design configuration smells (by existence)

Table 4: Results of Correlation AnalysisCorrelation(ρ) p-value

Analysis by volume 0.66410 <2.2e−16Analysis by existence 0.44526 <2.2e−16

smells with high statistical significance; however, correlationanalysis by volume exhibits stronger correlation than corre-lation analysis by existence. It shows that high volumeof design (or implementation) configuration smellsis a strong indication of the presence of high volumeof implementation (or design) configuration smellsin a project. Whereas, a project that shows presence oflarge number of design (or implementation) configurationsmell types moderately indicates presence of large numberof implementation (or design) configuration smell types.

RQ3. Is the principle of coexistence applicable to smells inconfiguration projects?

Approach: To compute intra-category co-occurrence fora smell, we count the number of occurrences of other smellsin the same category (by existence), only when the smelloccurred. We evaluate the average co-occurrence for eachsmell across all the repositories considering only those val-ues where the smell has occurred. We compute the averageco-occurrence for all the implementation and design config-uration smells and compared their normalized values.

Results: Figure 4 presents the average co-occurrencecomputed for each smell for both the implementation anddesign configuration smells. ide (duplicate entity) with av-erage 0.75 and iqu (improper quote usage) with average 0.29are the implementation configuration smells that show thehighest and lowest co-occurrences in the category. In thedesign configuration smells category, dbh (broken hierar-chy) with average 0.73 and dim (insufficient modularization)as well as dmf (multifaceted abstraction) with average 0.36show the highest and lowest co-occurrences respectively.

It means that whenever duplicate entity or brokenhierarchy smells are found, it is very likely to findother smells from the same category in the project.

Figure 4: Average co-occurrence (intra-category)for implementation and design configuration smells

Whereas, the smells improper quote usage and insuf-ficient modularization occur more independently.

Average normalized correlation for implementation anddesign configuration smells is 0.43 and 0.47 respectively.This leads to another interesting observation: design con-figuration smells show 9.3% higher co-occurrenceamong themselves than the implementation configu-ration smells. Since a design decision impacts the softwarein many ways, it is believed that one wrong or non-optimaldesign decision introduces many quality issues. The statisticreported above affirms the belief.

RQ4. Does smell density depend on the size of the config-uration project?

Approach: We compute normalized smell density forboth the smell categories for all the repositories and plotscatter graphs between lines of code in the repository andthe smell density. We then perform correlation analysis onboth the sets and document our observations based on thereceived results.

Results: Figure 5 and Figure 6 present the distributionof normalized smell density for implementation and designconfiguration smells against lines of code (with alpha = 0.3).

The visual inspection of the above graphs reveal the fol-lowing observations.

• The distribution shown in Figure 5 is very scatteredand random in comparison with the distribution inFigure 6. Although, both the figures show a weaklinear relationship between smell density and lines ofcode, Figure 6 shows a relatively stronger linear rela-tionship than Figure 5.

• Large projects show maturity and tend to demonstratelower smell density compared to smaller projects inboth the cases.

We, then, computed Spearman’s correlation coefficient forboth the data sets. The results of the correlation analysisare summarized in Table 5.

The results show no correlation between implementationsmell density and size of the project. However, a weak nega-tive correlation is perceived with high statistical significance

Figure 5: Smell density for implementation configu-ration smells against lines of code

Figure 6: Smell density for design configurationsmells against lines of code

Table 5: Results of Correlation AnalysisCorrelation(ρ) p-value

Implementation smells 0.03833 0.00980Design smells -0.32851 <2.2e−16

between design smell density and size of the project. Itshows that as the project size increases, design con-figuration smell density tends to decrease. This resultis interesting since it is believed traditionally that the com-plexity (and therefore, smell density) of a piece of softwareincreases as size of the software grows.

We observe another interesting aspect related to config-uration smells. Our empirical study reveal a large numberof class declarations where the corresponding definitions arenot found in the same repository. We find 59% of the repos-itories that we analyze have at least one such instance. Ma-jority of such missing definitions relate to third-party mod-ules. A possible explanation is that software developmentteams exclude third-party modules from their Puppet code

under version control. This practice provides the advantageof not having to maintain the used third-party modules asthey change. However, it breaks the fundamental princi-ple of IaC, i.e. production and configuration code shouldcoevolve. Such instances hurt the configuration process’sautomation and are bound to lead to trouble in the form ofmissing dependencies. More interestingly, the Puppet lan-guage does not offer any solution to this problem since mod-ule installation, as opposed to package installation, cannotbe part of a configuration code specification.

Yet another observation concerns language design (in thiscontext Puppet). Diligent use of language features and ad-herence to best practices can drastically reduce smells dis-cussed in this paper. However, careful language design canalso significantly avoid many configuration smells. For ex-ample, dua (unnecessary abstraction), dbh (broken hierar-chy), and dde (deficient encapsulation) can be controlledand avoided by suitable changes in the Puppet language.Similarly, many implementation configuration smells such asiuv (unguarded variable), ima (misplaced attribute), and ide(duplicate entity) can also be checked at Puppet languagelevel itself and can be avoided without any compromise infunctionality and convenience.

6. RELATED WORKOur work is related to studies of code quality practices

in traditional software engineering and system configurationmanagement.

6.1 Code Quality Practices in Traditional Soft-ware Engineering

Fowler [8] characterized code smells as poor design andimplementation decisions and identified numerous code anddesign smells. Girish et al. [36] provided a comprehensivecatalog of structural design smells classified based on theprinciple that they violate. Similarly, Garcia et al. [9] pro-vided a catalog of smells that may arise at architectural level.Many of our cataloged smells have names similar to ones inthe literature [8, 36]; however, their meaning and context isaligned to the configuration domain. Additionally, we haveadded a few new ones; for example unstructured module anddense structure.

Smells lead to technical debt and affect maintainabilitynegatively [15]. A popular approach for detecting them isstatic code analysis, which we also employ in this work.Metrics-based rules [19] identify code smells, such as Godclass or blob, by comparing computed metrics to specifiedthresholds. Decor [22] provides a domain specific languagefor formulating rules that detect smells such as blob, func-tional decomposition, and swiss army knife. Designite [30]detects numerous implementation and design smells. Alongthe similar lines, various tools to identify refactoring can-didates have been proposed; for example, extract methodrefactoring [38, 29] and extract class refactoring [31, 7].

In addition to the above approaches that base their analy-sis on a given code snapshot, hist [23] and Clio [40] considerchange history information. hist [23] analyses changes be-tween software components to detect blob, divergent change,shotgun surgery, parallel inheritance, and feature envy smells.Clio [40] detects software modularity violations with thestudy of co-evolving software components throughout projecthistory. Our tool Puppeteer also relies on static code analy-sis and reveals design configuration smells.

6.2 Code Quality Practices in System Config-uration Management

In the landscape of system configuration management,empirical studies on configuration code written in languagessuch as Puppet [39] and Chef [37] are scarce. Jiang et al. [14]study the co-evolution of Puppet and Chef configuration fileswith source, test, and build code. They analyze the softwarerepositories of 256 OpenStack projects and distinguish filesas infrastructure, which contain configuration code in Pup-pet or Chef language, production, build, and test. Theyfind that configuration code comes in large files, changesmore frequently, and presents tight coupling with test files.In our study, we perform static analysis on Puppet config-uration code using three different tools (Puppet-Lint, pmd-cpd, and Puppeteer) to detect configuration smells in a largenumber of repositories.

Puppet Forge [25] — the repository of Puppet modules,provides an evaluation of configuration code quality througha quality score based on three aspects: code quality scoreprovided by Puppet-Lint [27], compatibility with Puppet,and metadata quality. Metadata quality is subject to a setof guidelines that metadata files should adhere to.

Sonar-Puppet [24] is a SonarQube [32] plug-in that hasnumerous rules to detect quality violations in Puppet code;most of the rules applied by Sonar-Puppet are common withPuppet-Lint. We have included Puppet-Lint and the addi-tional rules checked by Sonar-Puppet in our analysis andmapped the rules to implementation configuration smells.We have also implemented Puppeteer that identifies all thecataloged design configuration smells.

Various authors have published their ideas describing bestpractices for configuration code in the form of blog-posts,articles, and technical talks [16, 17, 35, 18]. However, to thebest of our knowledge, our study is the first attempt towardsformalizing and detecting smells in configuration systems.

7. THREATS TO VALIDITYConstruct validity concerns the appropriateness of obser-

vations made on the basis of measurements. False positivesand false negatives are always associated with static codeanalysis and so are applicable to the tools that we used ordeveloped for this experiment. However, the effect of falsepositives and negatives is reduced when two or more streamsof results are compared as in this experiment. A compre-hensive set of test cases can rule out obvious deficiencies.Therefore, we employed a comprehensive set of test-casesfor Puppeteer.

Similarly, chosen threshold values in any smell detectiontool play an important role since their values decide thevolume of detected smell instances. Many authors such asLinda et al. [28] have suggested the chosen threshold valuesafter careful analysis. We carefully chose threshold values byadapting commonly used thresholds in traditional softwareengineering within IaC paradigm in such a way that theyare neither too lenient nor very stringent.

Further, a source-code analysis may adopt one of the nu-merous techniques to collect source-code information suchas ast parsing, reflection, and string matching [34]. Due tothe lack of available parser library for Puppet, our tool usesregular expression based string matching extensively whichis not as efficient as ast parsing. In addition to test thetool with unit-tests that check the correctness of the used

regular expressions, we carried out manual testing to ensurethe behaviour of the used expressions.

External validity concerns generalizability and repeatabil-ity of the produced results. Our experiment analyzes onlyPuppet repositories whereas there are many other configu-ration management systems. Although, the employed toolsare specific to one configuration language, the proposed the-oretical model is general and language agnostic. We believethat it will open doors for similar studies for other configu-ration management systems.

8. CONCLUSIONS AND FUTURE WORKThe paper presents a preliminary quality analysis for con-

figuration code. We propose a catalog of 13 implementa-tion and 11 design configuration smells based on commonlyknown best practices. We analyzed 4,621 Puppet reposi-tories containing 142,662 Puppet files and more than 8.9million lines of code. We investigated four research ques-tions using smell instances detected by our analysis. Thekey findings from our analysis are the following.

• The developers of Puppet repositories either do notintroduce code-clones at all or they do it in a mas-sive scale. Code-clones and other frequently occurringsmells may motivate studies in the future to improvesupport for avoiding and refactoring such smells.

• Configuration smells belonging to a smell category tendto co-occur with configuration smells belonging to an-other smell category when correlation is computed byvolume of identified smells.

• Design configuration smells show 9% higher averageco-occurrence among themselves than the implementa-tion configuration smells. This observation affirms thebelief that one wrong or non-optimal design decisionintroduces many quality issues and therefore suggeststhe developers to take design decisions critically anddiligently.

• Design configuration smell density shows negative cor-relation whereas implementation configuration smelldensity exhibits no correlation with size of a project.

The study brings benefits for both the researcher andpractitioner community. It offers a method for investigat-ing system configuration smells by mining repositories whichaims to benefit the research community. At the same time,practitioners can identify configuration smells using the toolsemployed in this study and adopt best practices to writemaintainable configuration code.

Our future research may take many possible directionsincluding investigating various aspects relevant to qualitywhen configuration repositories evolve, applying automatedrefactoring to configuration code, and measuring the impactof smells on various system parameters.

AcknowledgementsThis work is partially funded by the seneca project, whichis part of the Marie Sk lodowska-Curie Innovative TrainingNetworks (itn-eid). Grant agreement number 642954.

We would like to thank Faidon Liambotis (Principal Op-erations Engineer at Wikimedia Foundation) for providinghelpful inputs on our early configuration smells catalog.

9. REFERENCES

[1] Ansible. http://www.ansible.com/, 2016. [Online;accessed 22-Jan-2016].

[2] P. Bhattacharya, M. Iliofotou, I. Neamtiu, andM. Faloutsos. Graph-based analysis and prediction forsoftware evolution. In 34th International Conferenceon Software Engineering, pages 419–429, June 2012.

[3] F. Buschmann, R. Meunier, H. Rohnert,P. Sommerlad, and M. Stal. Pattern-Oriented SoftwareArchitecture Volume 1: A System of Patterns. Wiley,1 edition, 1996.

[4] CFEngine. https://cfengine.com/, 2016. [Online;accessed 22-Jan-2016].

[5] S. Chidamber and C. Kemerer. A metrics suite forobject oriented design. Software Engineering, IEEETransactions on, 20(6):476–493, Jun 1994.

[6] PMD—CPD: Copy Paste Detector.https://pmd.github.io/, 2016. [Online; accessed22-Jan-2016].

[7] M. Fokaefs, N. Tsantalis, E. Stroulia, andA. Chatzigeorgiou. JDeodorant: Identification andApplication of Extract Class Refactorings. InProceedings of the 33rd International Conference onSoftware Engineering, ICSE ’11, pages 1037–1039,New York, NY, USA, 2011. ACM.

[8] M. Fowler. Refactoring: Improving the Design ofExisting Programs. Addison-Wesley Professional, 1edition, 1999.

[9] J. Garcia, D. Popescu, G. Edwards, andN. Medvidovic. Toward a Catalogue of ArchitecturalBad Smells. In Proceedings of the 5th InternationalConference on the Quality of Software Architectures:Architectures for Adaptive Software Systems, QoSA’09, pages 146–162, Berlin, Heidelberg, 2009.Springer-Verlag.

[10] GitHub. https://github.com/, 2016. [Online; accessed22-Jan-2016].

[11] G. Gousios. The GHTorrent dataset and tool suite. InProceedings of the 10th Working Conference onMining Software Repositories, MSR ’13, pages233–236, Piscataway, NJ, USA, 2013. IEEE Press.

[12] G. Gousios and D. Spinellis. GHTorrent: Github’sdata from a firehose. In 9th IEEE Working Conferenceon Mining Software Repositories, pages 12–21, June2012.

[13] J. Humble and D. Farley. Continuous Delivery:Reliable Software Releases Through Build, Test, andDeployment Automation. Addison-WesleyProfessional, 1 edition, 2010.

[14] Y. Jiang and B. Adams. Co-evolution of Infrastructureand Source Code: An Empirical Study. In Proceedingsof the 12th Working Conference on Mining SoftwareRepositories, MSR ’15, pages 45–55, Piscataway, NJ,USA, 2015. IEEE Press.

[15] P. Kruchten, R. L. Nord, and I. Ozkaya. TechnicalDebt: From Metaphor to Theory and Practice. IEEESoftware, 29(6):18–21, 2012.

[16] G. Larizza. Building a Functional Puppet WorkflowPart 1: Module Structure.http://www.webcitation.org/6g23RY7yS, 2016.[Online; accessed 15-Mar-2016].

[17] G. Larizza. Building a Functional Puppet WorkflowPart 2: Module Structure.http://www.webcitation.org/6g23YeuFl, 2016.[Online; accessed 15-Mar-2016].

[18] G. Larizza. Doing the Refactor Dance — Making YourPuppet Modules More Modular.http://www.webcitation.org/6g23dnNKo, 2016.[Online; accessed 15-Mar-2016].

[19] R. Marinescu. Detection Strategies: Metrics-BasedRules for Detecting Design Flaws. In Proceedings ofthe 20th IEEE International Conference on SoftwareMaintenance, ICSM ’04, pages 350–359, Washington,DC, USA, 2004. IEEE Computer Society.

[20] R. C. Martin. Agile Software Development, Principles,Patterns, and Practices. Pearson, 1 edition, 2002.

[21] R. C. Martin. Clean Code: A Handbook of AgileSoftware Craftsmanship. Prentice Hall PTR, 1 edition,2008.

[22] N. Moha, Y. Gueheneuc, L. Duchien, and A. L. Meur.DECOR: A method for the specification and detectionof code and design smells. IEEE Transactions ofSoftware Eng., 36(1):20–36, 2010.

[23] F. Palomba, G. Bavota, M. Di Penta, R. Oliveto,A. De Lucia, and D. Poshyvanyk. Detecting badsmells in source code using change historyinformation. In IEEE/ACM 28th InternationalConference on Automated Software Engineering, pages268–278, Nov 2013.

[24] S. Puppet. SonarQube Puppet Plugin, Last accessedon: 22nd Jan 2016. Available at:https://github.com/iwarapter/sonar-puppet.

[25] Puppet Forge: a repository of Puppet modules, Lastaccessed on: 22nd Jan 2016. Available at:https://forge.puppetlabs.com.

[26] Puppet Labs. https://puppetlabs.com/, 2016. [Online;accessed 22-Jan-2016].

[27] Puppet-lint: Puppet code style checker, Last accessedon: 22nd Jan 2016. Available at:http://puppet-lint.com.

[28] L. H. Rosenberg, R. Stapko, and A. Gallo. Risk-basedobject oriented testing. In Twenty-Fourth AnnualSoftware Engineering Workshop, Greenbelt, MD, Dec.1999. NASA, Software Engineering Laboratory.

[29] T. Sharma. Identifying Extract-method RefactoringCandidates Automatically. In Proceedings of the FifthWorkshop on Refactoring Tools, WRT ’12, pages50–53, New York, NY, USA, 2012. ACM.

[30] T. Sharma, P. Mishra, and R. Tiwari. Designite — ASoftware Design Quality Assessment Tool. InProceedings of the First International Workshop onBringing Architecture Design Thinking intoDevelopers’ Daily Activities, BRIDGE ’16, New York,NY, USA, To appear. ACM.

[31] T. Sharma and P. Murthy. ESA: TheExclusive-similarity Algorithm for IdentifyingExtract-class Refactoring Candidates Automatically.In Proceedings of the 7th India Software EngineeringConference, ISEC ’14, pages 15:1–15:6, New York,NY, USA, 2014. ACM.

[32] SonarQube. http://www.sonarqube.org/, 2016.[Online; accessed 22-Jan-2016].

[33] D. Spinellis. Version control systems. IEEE Software,22(5):108–109, September/October 2005.

[34] D. Spinellis. Tools and Techniques for AnalyzingProduct and Process Data. In T. Menzies, C. Bird,and T. Zimmermann, editors, The Art and Science ofAnalyzing Software Data, pages 161–212.Morgan-Kaufmann, 2015.

[35] Puppet language style guide.https://docs.puppetlabs.com/guides/style guide.html,2016. [Online; accessed 22-Jan-2016].

[36] G. Suryanarayana, G. Samarthyam, and T. Sharma.Refactoring for Software Design Smells: ManagingTechnical Debt . Morgan Kaufmann, 1 edition, 2014.

[37] M. Taylor and S. Vargo. Learning Chef—A Guide ToConfiguration Management And Automation. O’ReillyMedia, 1 edition, 2013.

[38] N. Tsantalis and A. Chatzigeorgiou. Identification ofExtract Method Refactoring Opportunities for theDecomposition of Methods. Journal of Systems andSoftware, 84(10):1757–1782, Oct. 2011.

[39] J. Turnbull and J. McCune. Pro Puppet. Apress, 1edition, 2011.

[40] S. Wong, Y. Cai, M. Kim, and M. Dalton. DetectingSoftware Modularity Violations. In Proceedings of the33rd International Conference on SoftwareEngineering, ICSE ’11, pages 411–420, New York, NY,USA, 2011. ACM.