A novel approach to automatic extraction of configuration parameters from industrial code

A novel approach to automatic extraction of

configuration parameters from industrial code

Ian Davis, Ric Holt †, Ron Mraz‡

†‡

OWL Computing Technologies Inc.

( www.owlcti.com ) Provides solutions that: Facilitate ultra-secure unidirectional data transfer

across the internet (very secure one-way firewall) Potential Customers:

Governments Military and Intelligence Power plants Water management, etc.

PLE Workshop 9 Dec 2013

Fact Extraction from Bash in Support of Script Migration 2

http://www.owlcti.com/

The Idea Two Unix machines in a single box Write only network card upstream Read only network card downstream Internal cable connects two network cards

10Mb/sec and 155Mb/sec (fiber optics) supported External connections for:

Incoming and outgoing network cable Local access to each Unix machine USB Sockets for each Unix machine



The Product Line



For VehiclesFor Personnel

An Application



The Unix Software Supports:

Transparent unidirectional TCP/UDP firewall SMTP email systems and other protocols Automatic file mirroring to/from secure machine All major historian/tracking/logging systems

Bash scripts: Facilitate software installation Initiate client/server and other executables Monitor real time behaviour of system Stop/restart services, etc



The Engineering Problem

Customers requirements vary They must change configuration variable values This results in Bash scripts being cloned

Why is this problematic Customers have to understand Bash Customers risk breaking their scripts Repairing broken scripts very costly Owl must support many variants of scripts Hard to switch from Bash to better languages



The Issues with Bash Problems

Antiquated (limited support) Cryptic language (people dislike it) Slow (Python can be 6 to 21 times faster) Memory leaks and other bugs (software failure) To configure software scripts must be edited

Solution Refactor all scripts to use external XML files Port Bash to Python/Ruby etc.



Benefits of XML configuration file Supports script migration

Massive consolidation of scripts Releasing updates to scripts much easier End users do not need to understand scripts

Intrinsic benefits: Users know exactly what they can change They don’t risk breaking the scripts Configurations documented in the XML files Can easily compare configurations XML files can be shared across scripts



Our Challenge Mission impossible:

Change ~1,000 scripts to use XML configuration The largest script ~2,000 lines long Scripts may not be copied off site You are only on site for a week

We have no clue what the scripts do: Don’t understand the contents of the scripts Don’t know what parameterizes the scripts Don’t know what parameters users may change No obvious way of reading an XML file in Bash



The First Hour Refactoring task

Manually examine first script Record how variables are used Guess which variables customer’s change Somehow change the script to read XML

Unworkable solution Very tedious work Requires constant attention to detail Task enormous Deliverable will be far from optimal



Prowl(Parameterizing Owl)



VariablePropertiesBASH

SCRIPTXMLFILE

The Solution

Hack the Bash 4.2 interpreter Open source C code is easily changed Continue to source scripts etc Change interpreter to “action” everything Remove ability to invoke external programs Augment variables with usage flags Output XML file when interpreter finishes



Dynamic Fact Extraction

x=2;

if (…test…)

then

…statements…

else

…statements…

fi

function($x,$y)



y=2; if (…test…)then returnficase $x in1) …;break;;2) …code…esac

Example XML output

y=10; x=1; y=2; z=$x+$y; exit $(($z))

<prowl>

<script1 version="default">

<default>

<x>1</x>

<y changed>2</y>

<z derived returned>1+2</z>

</default>

</script1>

</prowl>



XML Configuration File(Manually edited)

<prowl>

<script1 version="default">

<default>

<x>10</x>

<y>20</y>

</default>

</script1>

</prowl>



Look at:

Naming conventions

Documentation

Examine variable usage

Compare cloned scripts

What customers change

Prowler



VariableAssignments BASH

SCRIPTXMLFILE

Reading Configuration File

Small C program parses the XML file Optional script name Optional version name Optional external assignments (x=21 y=3)

Prowler outputs: Bash variable assignment statements

x="10";y="20"; This string then eval’d within the script



Just One Extra Line/Script

But where to place this one line ?

To early: Configuration values later get changed

To late: Configuration values are used before set

Just right How to tell ?



Validate Placement

Run prowl on the refactored Bash script Execute prowler (and only prowler) Complain about any variable named by

prowler already flagged as USED. Prowler invoked too late

Complain if any variable named by prowler is later assigned a value. Prowler invoked too early



Contributions

Automated analysis of all Bash scripts Automated construction of XML files One line solution for parsing XML files Ability to validate changes applied Removed a significant road block for OWL Existing customers were asking for this Meets new US purchasing requirements



What Was Learned

Think outside the box Necessity is the mother of invention Source code is a valuable resource

It can be exploited in novel ways Tools that make a good worker

Don’t refactor if a tool can do the work for you Getting it right saves a lot of time Getting it wrong isn’t an option



Conclusions

Software implemented in a week Only minor changes to Bash C source files Available as open source software http://www.swag.uwaterloo.ca/prowl http://cs.uwaterloo.ca/~ijdavis (CSMR paper)

Easy to port to other Bash environments [Almost] as robust as Bash

OWL thrilled Customers happy



http://www.swag.uwaterloo.ca/prowl

http://cs.uwaterloo.ca/~ijdavis



Thank You

That is our novel approach to automatic extraction of configuration parameters

from industrial code

A novel approach to automatic extraction of configuration parameters from industrial code

Documents

Transcript of A novel approach to automatic extraction of configuration parameters from industrial code