Functional programming languages for verification tools: a comparisonof Standard ML and Haskell

Martin Leucker1,1, Thomas Noll2, Perdita Stevens3,2, Michael Weber2

1 Department of Computer Systems, Uppsala University, Box 337, 75105 Uppsala, Swedene-mail: leucker@docs.uu.se2 Lehrstuhl fur Informatik II, RWTH Aachen, Ahornstr. 55, Aachen, Germanye-mail: {noll,weber}@i2.informatik.rwth-aachen.de3 School of Informatics, University of Edinburgh, JCMB, King’s Buildings, Mayfield Road, Edinburgh EH9 3JZ, UKe-mail: Perdita.Stevens@ed.ac.uk

Keywords:

Abstract

We compare Haskell with Standard ML asprogramming languages for verification tools basedon our experience developing the verification plat-form Truth in Haskell and the Edinburgh Con-currency Workbench (CWB) in Standard ML.We review not only technical language featuresbut also the “worlds” of the languages, for ex-ample, the availability of compilers, tools, andlibraries. We also discuss the merits and diffi-culties of comparing programming languages inthis wide sense and support our view that Truthand the CWB are similar enough to justify theconclusions drawn in this paper.

1 Introduction

Concurrent software and hardware systems play an in-creasing role in today’s applications. Due to the largenumber of states and to the high degree of non-determin-ism arising from the dynamic behaviour of such systems,testing is generally not sufficient to ensure the correct-ness of their implementation. Formal specification andverification methods are therefore becoming more andmore popular, aiming to give rigorous support for thesystem design and for establishing its correctness prop-erties, respectively (cf. [1] for an overview).

In view of the inherent complexity of formal meth-ods, it is desirable to provide the user with tool sup-port. It is even indispensable for the design of safety-critical concurrent systems where an ad hoc or conven-tional software engineering approach is not justifiable.For example, model checking is a particularly successfulautomated approach to verification in which one employsdecision procedures to prove that (a model of) a systemhas certain properties specified in a suitable logic.

One major concern in the development of model-checking tools and other verification tools is correctness.

1 Supported by European Research Training Network “Games”2 Supported by EPSRC GR/A01756/01

1 STTT0184 – January 11, 2005 9:29

Since a verification tool is used for verifying hardware,protocols, and software, it would be useless if it were nottrustworthy. Thus, a programming language employedfor developing a verification tool should support the taskof verifying the code, at least in the informal sense ofsatisfying oneself that the code is correct. In general,functional languages are often considered to provide thisfeature, especially if the language disallows side effects.

Considering the broad range of functional languagesthat have been designed, it is surprising, and in our viewunfortunate, that there is little literature comparing dif-ferent functional languages. The Pseudoknot benchmarkpaper [3] studies several implementations of functionalprogramming languages (Haskell and Standard MLamong them) with respect to their runtime and mem-ory performance, and [7] compares the module systemsof Haskell and Standard ML. However, the developertrying to choose between the languages needs to be con-cerned about a much wider class of issues, including bothtechnical language features and “environmental” aspectssuch as the availability of libraries, documentation, sup-port, and multiplatform compilers. We did not find goodsources of help for a developer trying to choose betweenthe languages based on a larger collection of relevant as-pects like this. By contrast, many comparisons of Javawith C++ are readily available.

It is unsurprising, therefore, that developers (thosewho choose a functional language at all) often choose thelanguage most used in their institution, without seriouslyconsidering alternatives. The difficulty of getting infor-mation to guide an informed choice may also contributeto developers whose workplaces do not have a history offunctional programming language use deciding againstexperimenting with one.

Why is there so little material to help developersmake an informed choice? Part of the reason must bethat it is very hard to do convincing comparisons of lan-guages without being vulnerable to the criticism that oneis not comparing like with like. We think that a fair com-parison needs to be based on real experience of peopleusing the languages to build real systems in the samedomain; otherwise it is almost impossible to be sure thatdifferences are not due to differences in the domains ofapplication. The systems themselves need to be broadlycomparable in size and complexity, need to be more thantoys, and should preferably have been developed andmaintained over years (since a language that makes de-velopment easy might nevertheless encourage the devel-opment of code that is unmaintainable). Moreover, thedomain should be one where either of the languages isa reasonable choice, and the comparison should be doneby people with a reasonably typical level of experience inthe languages. A comparison is probably most generallyuseful to developers when it is done neither by novicesin the languages compared nor by people intimately fa-miliar with the compiler internals.

2 STTT0184 – January 11, 2005 9:29

This paper recounts our experiences in using SMLand Haskell for two broadly comparable applicationsin the domain of verification tools on which the authorshave worked for some years: the Edinburgh ConcurrencyWorkbench (CWB), in SML, and the verification plat-form Truth, in Haskell. The domain of verificationtools is eminently suited to the use of a statically typedfunctional language such as SML and Haskell, andboth languages are popular choices in this domain. Allof the authors have accumulated considerable experiencewith the languages we use, but we are not functional pro-gramming researchers.

Thus our primary motivation for writing this paper isthat we believe we are in an unusually good position toproduce a comparison of Haskell with SML that maybe useful to developers choosing between the languages.A secondary motivation is to be helpful to language de-signers and developers who work to support languagesby providing a record of our experiences, good and bad,with SML and Haskell.

The rest of this paper is structured as follows. Sec-tion 2 discusses the domain of verification tools and in-troduces the two systems. Section 3 discusses the classof languages from which SML and Haskell are drawnand briefly introduces the two languages for readers whomay not be familiar with them. Sections 4 and 5 arethe main body of the paper; Sect. 4 compares Haskelland SML on the basis of their technical language designfeatures, whereas Sect. 5 considers the equally important“environment” aspects. Finally, Sect. 6 concludes.

2 Verification tools

The domain on which we compare SML and Haskellis that of verification tools. The term “verification tool”covers any tool whose task it is to assist in checking thecorrectness of some artefact. Usually the artefact con-cerned is (an abstraction of) something produced in thesoftware or hardware development process.

We introduce Truth and the CWB and briefly sum-marise their histories before discussing the characteristicfeatures of verification tools in general.

2.1 The Edinburgh Concurrency Work-bench, in SML

Work on the CWB3 began in 1986. The CWB’s keystrength is its breadth: a variety of different verificationmethods are supported for several different process alge-bras. In particular, it allows users to:

• Define behaviours given either in an extended ver-sion of Milner’s CCS (Calculus of CommunicatingSystems [5]) or in its synchronous version, SCCS,and to perform various analyses on these behaviours,

3 http://www.dcs.ed.ac.uk/home/cwb/

3 STTT0184 – January 11, 2005 9:29

such as exploring the state space of a given processor checking various semantic equivalences and pre-orders;

• Define propositions in a powerful modal logic andcheck whether a given process satisfies a propertyformulated in this logic;

• Play Stirling-style model-checking games to under-stand why a process does or does not satisfy a for-mula;

• Derive automatically logical formulae which distin-guish nonequivalent processes;

• Interactively simulate the behaviour of an agent,thus guiding it through its state space in a con-trolled fashion.

One major focus of the CWB was always research;it was a platform which researchers (especially those atEdinburgh) could use to experiment with new relationsbetween processes and new algorithms. In the earlyyears all of these changes were retained in the main tool,even those which had been added experimentally withoutmuch consideration for the integrity of the CWB over-all. This contributed to the architectural degradation ofthe CWB and its increasing fragility: an important taskfaced by Stevens on taking over the maintenance of theCWB in 1994 was to reverse this process. The currentversion of the CWB consists of around 25 kloc in SML,plus several thousand in other languages for various sup-porting utilities.

The CWB was developed in Standard ML, but vari-ations were long maintained for several major ML com-pilers because different compilers provided different ex-tensions to the SML90 standard, and especially becausethey had different system build facilities. We settled onStandard ML of New Jersey (SML/NJ) because mostusers of the CWB used that compiler and the effort inmaintaining build scripts (the major point of difference)for several compilers did not seem well spent. Perhapswe should once again target Poly/ML,4 for example, infuture. We will discuss the history of the family of ML-like languages in Sect. 3. For now it suffices to say thatthis paper inevitably considers SML/NJ more than anyother SML compiler, and that since our experience iswith SML, we do not consider in depth other languagesin the ML family, specifically Caml and O’Caml. Thecontribution made by the SML language to both thearchitectural degradation problem and its solution arediscussed later.4 http://www.polyml.org/

4 STTT0184 – January 11, 2005 9:29

2.2 The verification platform Truth, inHaskell

In terms of features, Truth5 is similar to the CWB.In its current version, it supports tableau-based modelchecking for the full µ-calculus and game-based modelchecking for the alternation-free subcalculus. Both op-erate on finite transition systems, given in terms of CCSprocesses. The latter can be visualised and simulatedin an interactive fashion, to help the user understandTruth’s answers. Current development activities con-centrate on the parallel implementation of model check-ing on a cluster of workstations and on a specificationlanguage compiler generator which, given the definitionof a language, automatically generates a correspondingparser and a semantic evaluator.

Truth’s initial version dates back to 1997, and its de-velopment could benefit a lot from the progress made inthe design of verification tools over the years. As a con-sequence, its architecture is quite modular and easy tounderstand, and a deep change of the module structurehas not been necessary so far. It now consists of approx-imately 18 kloc in Haskell. Although there are severalHaskell compilers, Truth is written for the GlasgowHaskell Compiler6 (GHC), and since it uses some non-standard Haskell extensions and libraries only presentin the GHC, we have not tried to port it.

MoreoverTruthemploys theparsergeneratorHappy7

and many of the available Haskell libraries, and it inte-grates several stand-alone systems such as the daVinci8

graph visualisation tool and the GraphViz package.9 Fur-thermore, it uses existing C and Java libraries to providefunctionality such as textual and graphical user inter-faces and network communication, comprising approxi-mately 13 kloc. It is one of the bigger real-world applica-tions that is registered in the official Haskell pages.10It is worth mentioning that Truth is one of the fewtools listed there which was developed using but not forfunctional programming.

2.3 Characteristics of verification toolsin general

The peculiarities of the verification tool domain from thepoint of view of software engineering were considered byStevens in [9]. Here we briefly summarise and then focuson the implications for language choice.

Verification tools answer precisely defined questionsabout precisely defined systems. Thus it is compara-tively easy to understand what it means for the tool’sbehaviour to be correct. The downside is that certain5 http://www-i2.informatik.rwth-aachen.de/Research/Truth/6 http://www.haskell.org/ghc/7 http://haskell.cs.yale.edu/happy/8 http://www.informatik.uni-bremen.de/daVinci/9 http://www.graphviz.org/10 http://www.haskell.org/practice.html

5 STTT0184 – January 11, 2005 9:29

classes of bugs are unacceptable in a verification tool;semantic correctness is vital. Thus any language fea-tures supporting the development of correct programsare highly desirable.

A further characteristic is that verification tools tendto be developed in research environments, where it ismore easily recognised for novel theoretical contributions,or new applications of theory, than for the application of“best practice” in software engineering, which is likelyto be discounted because it is not new. Anything thatspeeds up development is an advantage, as it enables thedevelopers to spend a higher proportion of their time onthe work which is most valued. In such environments,it is also difficult to justify spending large amounts ofeffort on academically uninteresting aspects of the tool,such as a GUI, or on “invisible” areas such as testing(!),documentation, and ensuring portability. Nevertheless,the usability and, ultimately, success of the tool dependheavily on such aspects. Therefore, those planning todevelop verification tools will do well to choose a lan-guage in which professional results can be achieved witha minimum of effort.

It is perhaps instructive to note that in some cases,the same considerations may apply to those developinglanguages and their associated tools.

3 The space of programming lan-guages

Clearly Haskell and SML, the languages of Truthand the CWB, have a great deal in common: both arebasically functional languages and both have static typesystems which are strong in the sense that a well-typedprogram will be free of certain classes of runtime errors.Moreover, both are minority languages, with their originsin academia. What is the significance of these featuresfor verification tools?

The functional paradigm. Essentially, a functionalprogramming language is one in which the natural pro-gramming style includes treating functions as first-classconcepts. For example, one expects to write higher-orderfunctions; that is, functions which take other functionsas arguments. There is, however, no universally agreeddefinition of what it is to be a functional programminglanguage, although no reasonable definition would ex-clude either SML or Haskell. The difficulty stems fromthe impure nature of most languages, which stems in turnfrom the need to permit the use of whichever paradigm ismost appropriate for a particular problem. It is possible,for example, to write higher-order functions in C; the rea-son why C is not included in definitions of a functionalprogramming language is that this is not the natural,normal way to solve problems in C.

The main reason, in our view, for using a functional

6 STTT0184 – January 11, 2005 9:29

language for a verification tool is that the paradigm isa good match for the domain, as the most importantconcepts in the domain tend to be algorithms. It isoften claimed that programs written in functional lan-guages are easier to reason about, and hence are morelikely to be correct, than those in one of the imperativeparadigms (procedural or object oriented). The theo-retical concept on which the claim rests is referentialtransparency, essentially the fact that identifiers are usedfor values, rather than for references whose values maychange. Where this property holds, it can indeed facili-tate reasoning, at least in small pieces of code. However,we have found that in practice, building a verificationtool in a way which provides reasonable modularity andefficiency necessitates the use of “impure” features of thelanguages, so that referential transparency is lost.

Today the most obvious alternative to the functionalparadigm for a verification tool writer is the object-orientedparadigm. The main argument in favour of the func-tional paradigm is that the most important concepts inthe domain tend to be algorithms, not objects. In thisrespect the verification tool domain differs from mostbusiness domains, and the use of a less popular languagemay be justified. However, as we shall see, being out ofthe mainstream carries disadvantages sufficient to giveone pause.

Static typing. In a statically typed language, the com-piler carries out certain checks to ensure that the pro-gram is free of certain types of errors which might other-wise cause incorrect behaviour at runtime. This does not,of course, ensure that the program is free of errors, but itcan enable errors to be caught early and easily corrected,thus speeding up the development process. Static typ-ing is often criticised for being inflexible; but when suchcriticisms are investigated, they turn out to be criticismsof the inflexibility of a particular type system. We willgive examples of such inflexibilities in Sect. 4. We arguethat a coherent understanding of a solution to a prob-lem includes an understanding of the types of the entitiesinvolved; if these fit the type system of the language con-cerned, it is hard to see how having errors caught by thecompiler can fail to be a benefit, although one could stillargue about the size of the benefit.

It is clear from the successes achieved by certain groupsworking with dynamically typed languages such as Er-lang (in the functional world) and Smalltalk (in the object-oriented world) that it is possible to write complex, cor-rect software without static typing. However, none ofthe authors would willingly give up the benefits of statictyping. We will discuss particular features of the typesystems of SML and Haskell below.

7 STTT0184 – January 11, 2005 9:29

3.1 Standard ML

Standard ML ([6]) is an essentially functional languagein the sense discussed above. By “essentially” we meanthat it is not a pure functional language: for example,references are permitted. ML originated at Edinburghin the late 1970s. Research and experimentation con-tinued over the succeeding decades in several centres,spawning a family of ML-like languages. In what is com-monly regarded as the mainstream, a formal languagedefinition was produced; this defined “Standard ML”, orSML90. From early on, there were several reasonablyfaithful implementations of this standard. Later a majorrewrite of the original language definition resulted in thenew definition of Standard ML, sometimes referred toas SML97. Other notable ML-like languages are Camland O’Caml. Although they have enough similarities toSML that many of the same considerations will apply,there are also some significant differences which mightaffect a user’s choice. In particular O’Caml’s supportfor software architecture is radically different, incorpo-rating aspects of object orientation. We do not considerthese languages in this paper, since our experience is withSML.

Technically, the revision to SML97 has been a sub-stantial improvement; but it has led, temporarily at least,to difficulties of tools and libraries not all being updatedat once; old SML programs cannot be compiled by newcompilers and vice versa.

A variety of compilers is still available for StandardML; by far the most widely used is Standard ML ofNew Jersey (SML/NJ), and this is the only compilersupported by the CWB.

The definition of Standard ML includes the StandardBasis Library, providing such things as string manipula-tion, operating system interfaces, and basic data struc-tures. SML/NJ comes with a more extensive library.

3.2 Haskell

Haskell is a purely functional programming language [8].The current standard is Haskell98, which fixes the syn-tax and semantics as well as a large set of standard li-braries.

Until recently the embedding of input and outputoperations, which have to be considered as side effects,in purely functional programming languages was gener-ally poor. Monadic I/O is a very elegant approach toovercoming this problem [11]. Haskell supports thisconcept and supplies versatile I/O libraries offering ex-ception handling and file manipulation operations, whichwere of great help in building a user-friendly and reliabletool.

8 STTT0184 – January 11, 2005 9:29

4 Comparison of language designfeatures

We begin by considering and comparing the more tech-nical aspects of SML and Haskell, before going on toconsider non-technical questions in the next section.

4.1 Typing

As semantic correctness is crucial to any verification tool,it is natural to believe that a strong static type system,enabling a large class of errors to be caught at compiletime, is a good thing in a language for verification tools.Our experience supports this; although verification toolshave been written in Lisp, for example, we would notlike to give up the static typing provided by both SMLand Haskell. The type systems of SML and Haskellare actually rather similar. In this subsection we beginour discussion by considering two related features whichHaskell and SML have in common: parametric poly-morphism and type inference. In the following subsec-tions, we shall discuss the major differences between thelanguages’ type systems separately.

Extensive type inference is convenient especially infunctional programming where identifiers often have com-plex higher-order types. However, it has serious draw-backs for maintainability of code. The human reader ofcode needs to understand the types involved, and it isfrustrating to know that the compiler has worked out in-formation which is not available to the reader. The nat-ural response is that good programming practice is thento include type annotations; but we have found this hardto put into practice. An annoyance is that the syntax ofHaskell sometimes makes this impossible. For example,in Haskell function types are implicitly all-quantifiedand thus it was not possible to give type annotations forcertain local functions. This has been remedied with so-called scoped type variables, which have been introducedin GHC around version 4.03 (too late for Truth), butare not legal Haskell98.

A more serious point which applies even to SMLwhich does permit type annotations is that if type an-notations are included which are descriptive enough tobe helpful, they are too specific to allow reuse throughparametric polymorphism. On the other hand the mostgeneral type is – except for utility functions – often mean-ingless to the programmer and fails to document the trueintention of the function. For example, perhaps the pro-grammer writes a function whose first argument has mostgeneral type α list × β. Maybe there is initially onlyone application of this function, to an argument of typeaction list× string. It may be that the fact that the firstargument has type action list is essential to the natureof the function; for example, perhaps this is reflected inthe name of the function, and using the function on anyother kind of list would be confusing. However, perhaps

9 STTT0184 – January 11, 2005 9:29

the type of the second argument is less important, andif the code works on another type, the programmer maybe quite happy to see it used on that type. “Morally”,the function’s argument has type action list × α. If theprogrammer thinks this through, it is possible to anno-tate the function accordingly; but it is not natural todo so, since it involves thinking about all possible futurereuse of the code at a time when it is more appropriate toconcentrate on the initial intended use. One could arguethat this is what comments are for, but the advantage oftype annotations is that the compiler can automaticallycheck that they are consistent with actual code.

There is a tension between trying to enable code reuseon the one hand and on the other hand trying to makecode understandable and trying to maintain appropri-ate encapsulation barriers. We find that these last two,though different, often go together: one encapsulates thedefinition of an important type together with appropri-ate functions for manipulating it, and then uses the newtype name in type annotations to elucidate the code.However, in doing so one loses the power of parametricpolymorphism for code reuse in clients of this new typebecause clients cannot see the structure of the type.

For example, processes in the process calculi we workwith can have restrictions applied to them. A restrictionis conveniently implemented as a list of actions, but cer-tain invariants need to be maintained. If we allow clientsto see that a restriction is a list of actions, then whenthey manipulate processes they can use the standard listfunctions on the restriction, but we cannot easily enforcethe invariants. On the other hand, if we use encapsula-tion to make available only a type restriction so that wecan enforce the invariants, we have to provide all nec-essary functions for manipulating this type. This is notunreasonable: it is the same work, for example, that wewould have to do if we worked in an object-oriented lan-guage and created a class Restriction. However, when wehave a variety of slightly different kinds of restriction, wehave to implement the manipulating functions afresh ev-ery time; to gain encapsulation we have lost parametricpolymorphism as a reuse mechanism, and we do not haveinheritance available to us as an alternative mechanism.This kind of situation arises very frequently in both theCWB and Truth because we write code to deal withvariants of process algebras and logics and with variouslyprocessed versions of them.

An additional issue in SML is that it is sometimesdifficult to decide whether a conceptual type should beimplemented at the module level or only at the core level;in the CWB we generally resolve this by using both butnot revealing that decision outside the module where itis made, so that, for example, the signature for processesexports only a type restriction, whereas an implementa-tion of that signature typically builds a structure Restric-tion, exporting a type from the content of that structure.(Note that because of the divide between module and

10 STTT0184 – January 11, 2005 9:29

core level, we do not have the option of working only atthe module level; in order to write functions that workwith restrictions – which is essential – at some point wehave to decide what type a restriction has. So functorconstruction and multiple applications of functors do notsolve this problem, though they may contribute to a so-lution.)

Further, we find that the powerful type systems ofSML and Haskell are a mixed blessing, often leadingto complex type errors which are understandable onlyto people who are familiar with the subtleties of the re-spective type system. This is to some extent inevitablein a language whose standard idioms involve complexhigher-order types, but refinements such as SML’s equal-ity types and weak types add to the problem, since theneed for these is not easy for the non-type-theorist pro-grammer to understand. Recent work on more informa-tive error messages, such as [4], is to be welcomed, buthas yet to make a difference to the compilers. Further-more, an interactive type analyser would be desirable,a tool which, requested by the user, would visualise thetypes of certain subexpressions. In the meantime, ouradvice is that there is little to choose between SML andHaskell in this respect.

4.2 Strictness vs. laziness

The most obvious difference between SML and Haskellis that SML is strict whilst Haskell is lazy. For dis-cussion of the concepts in general see, for example, [10].Basically, laziness means that values are only computedon demand, allowing the implementation of infinite datastructures. In contrast, strictness refers to the fact thate.g. the arguments of a function call have to be evalu-ated before executing the call, no matter whether theyare required or not.

In the context of verification tools, laziness seemsto be an appealing feature because one might hope toget “for free” certain “on-the-fly” verification techniquesthat normally have to be worked out in each special case.For example, consider a class of verification questionsconcerning a system, such as the model-checking prob-lem “does this system satisfy this property”. To answersome questions in the class, it will be necessary to calcu-late the entire state space of the system. For others, onlya small part of the state space, perhaps that reachablewithin three transitions from the starting state, will berelevant. A global algorithm is one which always calcu-lates the whole state space; a local one does not. Localalgorithms are generally harder to design and verify thanglobal ones and often have poorer worst-case complex-ity, though in practice they may perform much better.One might hope to be able to get a local algorithm froma global one “for free” using laziness because the code forcalculating certain parts of the state space would simplynever be evaluated if its results were not called for. Inpractice, however, the Truth team found that one has

11 STTT0184 – January 11, 2005 9:29

to implement the algorithm generating the state spacecarefully in order to guarantee the desired behaviour.For example, the use of monads (cf. Sect. 4.3) or of ac-cumulator techniques can easily destroy the on-the-flyproperty. Since there are no visual clues (program anno-tations) in the source code, it is often not entirely obviouswhy a function is not as lazy as one would have hopedwhen writing the code. Also, code which involves excep-tions or destructive updates of data structures needs tobe crafted quite carefully in a lazy context. Eager evalu-ation, on the other hand, is easier to write, comprehend,and debug because things happen deterministically, inthe order dictated by the source code.

Altogether the effort required to preserve the localityof a lazily evaluated, global algorithm often correspondsto the design of an algorithm which is local by nature.

Summarising, lazy evaluation is an attractive feature,but the Truth team would have liked a flexible mecha-nism with which to specify parts of the program, whichshould be evaluated eagerly or lazily.

4.3 Imperative features

Both the Truth and the CWB team have found imper-ative features to be essential. Sometimes the concern isefficiency, but more often it is understandability: manyof the algorithms we wish to implement are conceivedimperatively, and in such cases implementing them func-tionally makes the implementation more difficult to readand hence more likely to contain errors. Prominent ex-amples of algorithms with an imperative character aregraph algorithms, which play an important role in toolssuch as ours which deal with transition systems. Thedata structures we deal with grow too big to keep severalcopies in memory, and the usual way to extract informa-tion from them is to walk them in a given order, collect-ing information and destructively updating the structureon the way, which can be straightforwardly described andefficiently implemented in imperative ways.

Here SML scores by providing imperative featuresin the core language in a reasonable and powerful way,although they can be syntactically awkward. I/O is sup-ported by the Standard Basis Library. Haskell usesmonads for destructive updates and I/O; they add a re-stricted form of stateful computation to a pure language,retaining referential transparency ([11]). The disadvan-tage is that programs become more complicated. Also,if imperative elements of a given application were nottaken into account during its design but turn out tobe necessary later on, often major parts have to be re-designed or (at least) reimplemented, especially becausetypes change significantly. A simple but recurring exam-ple is to add printing of status information to an oth-erwise purely functional algorithm. In the worst casethis could result in having to rewrite the algorithm ina monadic style, but also to rewrite its callers (and tran-sitively their callers as well), plus adjusting all type an-

12 STTT0184 – January 11, 2005 9:29

notations on the way. Even when using opaque accessorsto data structures, the required changes cannot necessar-ily be limited to a single module, but affect large partsof the system. This is clearly undesirable from a soft-ware engineering or economical point of view. Indeedfor certain parts of the Truth system a redesign turnedout to be necessary in the past, mostly in order to imple-ment more efficient versions of algorithms by introducingimperative constructs like destructive updates.

The Truth team considers this point as one of thebiggest drawbacks of the purely functional paradigm asfollowed by Haskell.

4.4 Architecture support

The architecture of a system makes a vital contributionto its correctness. We hope to study module systems inthis context, building on [7], in future; in this paper wecan only indicate the main issues.

A Haskell module defines a collection of values,datatypes, type synonyms, classes, etc., as well as theirimport/export relationship with other modules. Over-loaded functions are provided in a structured way in theform of type classes, which can be thought of as familiesof types (or more generally as families of tuples of types)whose elements are called instances of the class. In theinstantiation the definitions of the overloaded operationsare given.

In Standard ML, structures provide the main name-space management mechanism; they may contain sub-structures, functions, values, types, etc. A structuremay be coded directly or produced by the applicationof a functor, which may be thought of as a generic orparameterised structure. The programmer may definesignatures which act as the types for structures; for ex-ample, a functor may be defined to take, as argument,any structure matching a given signature. The mod-ule system is separate from the core language; one can-not, for example, apply a functor conditionally. Whilstthis keeps the language definition clean, in the CWB ithas often caused problems leading to code duplication.The changes made to SML in SML97 are welcome; theelimination of structure sharing and the introduction of“where” clauses have solved several long-standing prob-lems for the CWB.

A basic facility which is desirable in a module sys-tem is that it should be possible to define an interface toa module separately from the module itself. This helpsdevelopers to understand the system, as they can readinterfaces to modules without being distracted by im-plementation information. We also want to be able toapply the same interface to several modules and to pro-vide several interfaces to the same module. In both theCWB and Truth this need arises, for example, becausewe often work with several variants of a process alge-bra, logic, or algorithm which share an interface. Wewant the compiler to do the work of making sure that

13 STTT0184 – January 11, 2005 9:29

the modules stay consistent, and we want to avoid du-plicating code. SML’s signatures support this way ofworking reasonably well, although not without problems.The Truth team has found that Haskell does not sup-port this situation so well: inside the module export list,entities cannot be annotated with types, so a commonpractice is to add them in comments. However, this iserror prone, since there is no way for the compiler toenforce their correctness or check their consistency withthe implementation in the module body.

We feel that SML’s architectural features are bettersuited than Haskell’s to our purposes; neither is ideal,however, and this seems an interesting area for futurestudy, especially as we do not think that the class andpackage systems of C++ or Java would be ideal either.

4.5 Exceptions

The CWB used to make heavy use of exceptions as a con-trol flow mechanism. This led to correctness problemsbecause the compiler could not check whether or not ex-ceptions were always handled. A common class of bugsoccurred when a programmer added a new piece of func-tionality to the CWB by adding a new module declaringan exception; the exception could arise outside the newmodule; but the programmer did not, for whatever rea-son, modify the CWB’s top level module to handle theexception sensibly. To make matters worse, in SML90,although one could write a handler that would catch allexceptions (using a wildcard), so that at least the userwould not see the CWB “crash” in the case of such a bug,one could not tell dynamically which exception was ac-tually being handled. This would result in a message tothe user of the CWB along the lines of “Sorry, an ML ex-ception has been raised. This is a bug: please report it.”The exception mechanism has been improved in SML97compared with SML90: it is now possible to interrogatean exception for its identity, which at least enables theCWB to give a fuller error message, which is useful fordebugging.

Still, in a language with type safety as a strength, it isa pity to use a programming style in which the program-mer cannot be certain that all exceptions are handled.From the point of view of the user of a verification tool,it is not very much better for the application to terminatebecause of an unhandled exception than it would havebeen for it to terminate because of a runtime type error.Therefore, the CWB now uses exceptions in a more dis-ciplined way which seems to work well. A small numberof specified exceptions (corresponding to such things as“error in user input”, “assertion violated”, etc.) are al-lowed to rise to the top level and are individually handledthere. All other exceptions are kept within small piecesof code (e.g. within one SML module) and in each casethe programmer verifies by eye that the exception can-not escape. Because the latter is hard work, exceptionsare only used where the alternative is really painful.

14 STTT0184 – January 11, 2005 9:29

We are not claiming, of course, that SML compil-ers could and should check whether exceptions are al-ways handled; this has been a research topic for sometime. Java’s requirement that functions document (cer-tain kinds of) exceptions that may be raised is an at-tempt to address the problem, but it is clumsy and in-teracts badly with a functional programming style. Ourpoint is that the style of Standard ML programming of-ten seen and encouraged, relying heavily on exceptions,has serious disadvantages which the software engineerneeds to guard against.

The Truth team found that exceptions interactedbadly with laziness: as a rather disturbing effect, partialevaluation enables exceptions to escape from an enclos-ing exception handler. To get the exceptions actuallyraised inside the handler, initially the Truth team hadto resort to code like if x==x then x else x to en-force the evaluation of x at the right time. Recently,better ways to trigger full evaluation have been pro-vided (deepSeq $! x), but they are non-standard andof course destroy laziness. We think it would be muchmore natural to avoid situations of this kind by adoptingstrict rather than lazy evaluation as the standard strat-egy in the language. Laziness, which is a very costlyfeature, could then be provided upon request, using an-notations of the function and constructor symbols.

We have often seen programming languages comparedon the basis of how many lines of code it takes to imple-ment some piece of functionality. We consider this a poormetric. The length of a piece of code is not well corre-lated either with the time it takes to write it or with thetime it takes to understand it; a short piece of code maywell be harder to write and to maintain than a longerone. This is why we have not tried to compare SML andHaskell on this point.

5 Comparison of non-language de-sign features

5.1 The available compilers and their char-acteristics

There are now three main freely available SML97 com-pilers, SML/NJ, Poly/ML, and Moscow ML. (HarlequinMLWorks ceased to be available when Harlequin wasbought: there was hope that it might become open source,but this now appears unlikely.) SML/NJ can now pro-duce native code for many platforms, which is importantfor a widely distributed verification tool. However, itneeds a third-party utility to produce stand-alone appli-cations, and even then there is a problem with runningthe application from outside its directory. This is hard toexplain to users of the CWB and causes embarrassment.

For Haskell, too, three compilers are available, allof them freely: NHC98, HBC, and GHC. For Truth,

15 STTT0184 – January 11, 2005 9:29

only GHC was considered feature-complete enough; italso provides some extensions to the Haskell98 lan-guage which have proven helpful, for example multipa-rameter classes and existential types.

5.2 Libraries and associated tools

Good libraries and tools can help to ensure correctness(e.g. because well-used libraries have been debugged byothers) and can cut down development time. We con-sider and compare what is available for Haskell andfor SML.

General-purpose libraries. Both Haskell and SMLcome along with standard libraries specified alongsidethe language. SML97 defines the Standard Basis Library11

(ML97SBL); Haskell98’s libraries are described in theLibrary Report12 (H98LR). Broadly similar, these pro-vide basic data structures, interface to the operating sys-tem, etc. Both GHC and SML/NJ ship with some extra,non-standard libraries.

GUI libraries. There is an X Window System toolkit,eXene,13 written in Concurrent ML, though for a longtime this was apparently not usable with SML97 (be-cause of a signal-handling bug, fixed more recently thanthe last major CWB changes). Research projects haveprovided portable GUI library facilities for use with Stan-dard ML, such as sml tk.14 Thus one can implementa GUI in SML; but really good high-level toolkits arestill lacking. The CWB has made no serious attempt todo this. For Haskell, too, some bindings for commonGUI toolkits are available, but at the time GUI sup-port was added to Truth none of them was regarded asstable or feature-complete enough to be usable for whatwas planned. In the end, the process simulation GUIfor Truth was written in Java and was interfaced tothe Haskell part via Unix pipes. (The CWB followeda similar path in a student project, as yet unreleased.)

Associated tools. A debugger is invaluable in pro-gram development, especially when experimenting withverification algorithms which may contain bugs. Un-fortunately, writing debuggers for functional languagesturned out to be harder than for imperative languageslike C. This is even more true for a lazily evaluatedlanguage like Haskell, where the inspection of a valuewould sometimes change the evaluation order. Never-theless some attempts have been made in this direction,mostly resulting in so-called tracers (like Freya, Hood, orHat), which can record program runs for later analysis.

11 http://www.smlnj.org/doc/basis/12 http://www.haskell.org/definition/13 http://people.cs.uchicago.edu/∼jhr/eXene/14 http://www.informatik.uni-bremen.de/∼cxl/sml tk/

16 STTT0184 – January 11, 2005 9:29

None of them was used during Truth development be-cause they were either not available at that time or didnot support some of the GHC features used in Truth.There is no debugger available for SML/NJ. There is,however, a debugger for Poly/ML, which is a welcomedevelopment. We have not yet used it.

There is a lexer (ML-Lex) and a parser generator(ML-Yacc) for SML. These were long unavailable in SML97versions but do now seem to work (see below re docu-mentation). The CWB uses ML-Lex but does not useML-Yacc. At a very early stage a hand-built parser wasproduced, and by the time the major reengineering workwas done on the CWB its syntax (perhaps unfortunately,but understandably) included features which were notsupported by ML-Yacc, so that to move to ML-Yaccat that point would have involved a user-visible syntaxchange. This is an example of the problems which canarise when a suitable third-party component is not avail-able at the right moment; users may not have the op-tion of adopting it later. As mentioned, Truth uses theHappy parser generator.

Overall there is little to choose between Haskell andSML in this category, but both suffer from being minor-ity languages. There are few providers of libraries andtools, and key developers are often more concerned withcompilers. This is understandable, but to us librariesand tools are just as important.

5.3 Documentation and other sources ofhelp

Famously, Standard ML has a formal specification [6],but this is impenetrable to most programmers. Fortu-nately there are also several accessible books and tu-torials available. The official specification of Haskellis given by the Haskell98 Language Report,15 whichdefines the syntax of Haskell programs and gives aninformal abstract semantics. For such a technical doc-ument it contains much plain text, and the general im-pression of local Haskell developers is that it is quitereadable. On the other hand, as was noted elsewhere:16“The informal specification in the Haskell report leavestoo much room for confusion and misinterpretation. Thisleads to genuine discrepancies between implementations,as many subscribers to the Haskell mailing list willhave seen.”

Regarding the compiler and associated tool docu-mentation, the overall impression of the authors is thatGHC’s documentation is slightly better than that of SML/NJ.(The ML-Lex documentation has not been updated forSML97, for example.) This has not always been thecase, but the GHC developers have improved the docu-mentation quite a lot in the recent past.

15 http://www.haskell.org/onlinereport/16 http://www.cse.ogi.edu/∼mpj/thih/

17 STTT0184 – January 11, 2005 9:29

In both cases documentation for libraries is patchy,especially in the case of compiler-specific libraries, whereit sometimes happens that the programmer must consultthe source code to get more information than the signa-ture of a function. H98LR and ML97SBL are better doc-umented. From 1997 to 2001 there was no complete andup-to-date documentation of the latter, which was a se-rious problem. Now, however, an updated web page isavailable;17 the documentation appeared in book form [2]in 2002, which is a welcome development. The CWB andTruth teams each had the impression initially that theother’s language’s libraries were better documented: thismay reflect that one notices faults only on close acquain-tance. A plus for Haskell is that the GHC library doc-umentation has a consistent history of being frequentlyupdated and improved.

Moving from documents to people as sources of help,we have found the newsgroups comp.lang.ml and comp.lang.functionaland the GHC mailing lists to be useful. Naturally it iseasier to get help with problems which can be describedbriefly. When we have needed help with, for example,making architectural decisions, local language expertshave been invaluable; this is something that developersshould bear in mind.

Last but not least the home pages of Standard MLof New Jersey18 and of Haskell19 provide useful collec-tions of links and references to other resources.

5.4 Foreign-function interfaces

We have made no serious attempt to bind C and SMLcode within the CWB, because the Standard ML foreign-function interfaces were perceived (and experienced ina student project) as hard to use and inefficient. MatthiasBlume’s new “NLFFI” foreign-function interface may wellchange the situation.

The foreign-function interface in Haskell has under-gone a major redesign and is now quite usable. Truthhas been extended by a parallel model-checking algo-rithm, which uses the FFI layer to call C functions fromthe MPICH resp. LAM libraries, both well-known imple-mentations of the Message Passing Interface standard.For this application the marshalling required to convertbetween Haskell and C data formats turned out tobe very inefficient, however. Another problem was theinstability of the FFI interface at the time the Truthteam were using it: it changed rapidly between releases ofGHC. The Truth team made extraordinary use of pre-processor directives and autoconf magic in an attemptto allow Truth to support many compiler versions, but

17 Unfortunately, it documents the version of the library corre-sponding to the very latest “working”, i.e. experimental, versionof the compiler. There still seems to be no freely available docu-mentation for the version of the library in the latest release-qualityversion of the compiler!18 http://www.smlnj.org/19 http://www.haskell.org/

18 STTT0184 – January 11, 2005 9:29

they were eventually forced to give up.

5.5 Stability of languages and their im-plementations

The current stable version of Haskell is Haskell98,dating from 1997. This is the fifth major version of thelanguage definition (the next will be Haskell 2 !). Com-pilers, of course, provide the effective definition of thelanguage. There have been many changes in what GHCsupports (e.g. multiparameter classes, implicit parame-ters). Not all changes to extensions have been backwardscompatible, which is inconvenient for programmers whoneed those extensions.

Regarding the stability of Haskell implementations,only GHC has been thoroughly examined, since the otherimplementations have been ruled out by other issues, asstated earlier. GHC is under steady development, andthe quality of released versions differs greatly. Some arequite stable, but for others patch-level releases have tobe made quickly to fix the worst bugs. Unsurprisingly,bugs often accompany new features. In fairness, bugsare fixed promptly by the GHC developers once they arereported to the relevant mailing list. However, faced witha show stopper, an application programmer must choosebetween waiting for an official release of GHC includinga fix, or becoming expert in building the compiler itself,which is non-trivial, time consuming, and will not furtherhis/her aims.

Standard ML was subject to a major revision, fromSML90 to SML97. The SML/NJ compiler has under-gone many releases, but now seems fairly stable. Asmentioned, tools and libraries tend to lag. The ML2000project intended to develop a future version of ML. Lit-tle has been heard of the project recently, and many ofits early ideas have been incorporated in O’Caml. Webelieve that SML97 will increase, rather than decrease,in stability over the next few years.

5.6 Performance

This is a controversial topic, but it is an important onefor developers of verification tools. Both speed and spaceusage are important, with space usage often being moreimportant, as the amount of memory used by a verifica-tion tool is normally the limiting factor. Our experiencewith both SML and Haskell suggests that performanceis particularly poor with respect to memory usage. Also,as noted, the key feature of Haskell, lazy evaluation,comes at a high cost.

6 Conclusion

We proceed by summarising the features we found (not)helpful in using functional languages in general, and spe-cifically in the use of SML and Haskell.

19 STTT0184 – January 11, 2005 9:29

6.1 Helpful features

Both teams found the functional paradigm a good fit forthe domain of verification tools. The automatic mem-ory management of SML and Haskell lets us focus onthe important parts of the algorithms instead of fiddlingwith implementation details. Data structures that arereadily available like lists and their natural use were ap-preciated as well. Also, static types proved to be helpfulin catching certain classes of errors early.

However, some of these features turned out to bedouble-edged swords, and other features which were ini-tially considered useful turned out not to be, or to beless helpful than we had hoped.

6.2 Mythical silver bullets

The price of using convenient constructs like higher-orderfunctions and functional languages in general is oftenpaid in uncompetitive runtime and memory performance.While in imperative languages it is possible, after gettingprograms right, to get them fast, we find this harder infunctional languages.

It is commonly argued that referential transparencymakes programs easier to understand, easier to reasonabout, and generally more robust. However, we did notfind this feature to be worth its cost. Constraining (inHaskell) the use of e.g. destructive updates when theyare needed turned out to waste a lot of precious devel-oper time. Reasoning about any sufficiently complex al-gorithm on the source code level is intractable as well,even when avoiding impure features.

Static types were helpful, as mentioned above, butalso got in our way. Understanding complex type errorsand their origins requires an intimate understanding ofthe type system. Easier means of exploring them andderiving complex type annotations are lacking.

6.3 SML vs. Haskell

The most outstanding difference between Haskell andSML is the evaluation order. While Haskell’s lazy ap-proach might have its merits in some cases, it was morea hindrance than a help, causing longer development,having negative impact on the performance, and makingthe source code harder to understand.

While the use of monads in Haskell is a nice theo-retical concept to capture side effects and stateful com-putation in a purely functional setting, their disadvan-tages lead us to vote for the impure SML here, especiallygiven that many of our domain’s algorithms are naturallydescribed imperatively.

We found the module systems in both SML and Haskelllacking features from a software engineering point of view.For example, in SML the “include” mechanism for sig-natures is not flexible enough to prevent one from havingto duplicate information across several SML signatures

20 STTT0184 – January 11, 2005 9:29

when different views onto a structure are required (com-parable to “private” and “public” interfaces in languagessuch as Java). The lack of conditional application offunctors, similarly, can lead to code duplication, whichis in turn a maintenance problem. Since we have notsurveyed other programming languages in this respect,it is not clear whether they would meet our demands,though.

Both teams would have wished for better support forexceptions in order to enhance runtime error messages(SML) and to better cope with astonishing evaluationorder interactions (Haskell).

Confronted with the poor availability of ready-to-use libraries for SML and Haskell compared to main-stream languages, one is often referred to their foreign-function interfaces. In Haskell, the use comes witha performance penalty. We do not have first-hand expe-rience for SML.

Aside from the language issues we have investigated,we found the tool support in both languages lacking incomparison to mainstream languages, resulting in thereinvention of wheels other languages can readily use.We regard this an important factor when planning thedevelopment schedule of SML or Haskell programs.

We found documentation for Haskell to be better,while stability of the SML implementation was found tobe superior when compared to its Haskell counterpart.

6.4 Considerations for future projects

There have been positive and negative aspects to bothour sets of experiences with Haskell and SML, as therewould doubtless have been with whatever language wehad chosen. Overall, we consider Standard ML to bea slightly better choice for our kind of application thanHaskell, more because of a more stable environmentof supporting tools than because of language features.Of course, there are many alternatives including otherfunctional languages with which we have less experience;O’Caml might be a strong candidate.

However, it turned out in our discussions that none ofus were enthusiastic about the idea of using a functionallanguage for a future verification tool because of theirimpoverished environments compared with mainstreamprogramming languages. Our impression was that SMLand Haskell can play out their advantages mainly inthe prototyping stages of a project, an arena where bothwould have to compete with dynamic languages like Lispor Smalltalk, or scripting languages like Python (whichhave faster turn-around cycles due to absence of a com-pilation phase).

Our conclusion is that, if/when we develop new ver-ification tools, we would like to conduct a study on theuses of imperative languages for verification tools. Dur-ing our investigations we got the impression that thoseseem to be better equipped for features we need in ourdomain.

21 STTT0184 – January 11, 2005 9:29

We hope that reporting our experience using majorfunctional languages will help the community to improvesuch languages and their worlds in future.

Acknowledgement. We thank the anonymous refereesfor helpful comments and discussions.

References

[1] Clarke EM, Wing JM (1996) Formal methods: stateof the art and future directions. ACM Comput Surv28(4):626–643

[2] Gansner ER, Reppy JH (2002) The Standard MLBasis Library. Cambridge University Press, Cam-bridge, UK

[3] Hartel PH et al. (1996) Benchmarking implemen-tations of functional languages with ‘Pseudoknot’,a float-intensive benchmark. J Function Programm6(4):621–655

[4] McAdam B (2002) Repairing type errors in func-tional programs. PhD thesis, Division of Informat-ics, University of Edinburgh

[5] Milner R (1989) Communication and concurrency.International Series in Computer Science. Prentice-Hall, Upper Saddle River, NJ

[6] Milner R, Tofte M, Harper R, MacQueen D (1997)The definition of Standard ML (revised). MIT Press,Cambridge, MA

[7] Nicklisch J, Peyton Jones SL (1996) An explorationof modular programs. In: Glasgow workshop onfunctional programming, July 1996

[8] Peterson J, Hammond K, et al (1996) Report onthe programming language Haskell, a non-strictpurely-functional programming language, version1.3. Technical report, Yale University, New Yaven,CT, May 1996

[9] Stevens P (1999) A verification tool developer’s vademecum. Int J Softw Tools Technol Transfer 2(2):89–94

[10] Wadler P (1996) Lazy versus strict. ACM ComputSurv 28(2):318–320

[11] Wadler P (1997) How to declare an imperative.ACM Comput Surv 29(3):240–263

22 STTT0184 – January 11, 2005 9:29