The WARM-UP Algorithm:A Lagrangean Constructionof Length Restricted Hu�man CodesRuy Luiz Milidi�umilidiu@inf.puc-rio.brEduardo Sany Laberlaber@inf.puc-rio.brPUC-RioInf.MCC15/96 September, 1996Abstract: Given an alphabet fa1; : : : ; ang with corresponding set of weights fw1; : : : ; wng, anda number L � dlogne, we introduce an O(n logn+n logw) algorithm for constructing a suboptimalpre�x code with restricted maximal length L, where w is the highest presented weight. The numberof additional bits per symbol generated by our code is not greater than 1= L�dlog(n+dlogne�L)e�2when L > dlogne + 1, where is the golden ratio 1:618.An important feature of the proposed algorithm is its implementation simplicity. The algorithmis basically a selected sequence of Hu�man trees construction for modi�ed weights.Keywords: Pre�x codes, Hu�man Trees, Lagragean DualityResumo: Dado um alfabeto fa1; : : : ; ang com pesos correspondentes fw1; : : : ; wng e um n�umeroL � dlogne, n�os apresentamoso um algoritmo de de complexidade O(n logn + n logw)para con-struit c�odigos de pre�xo sub�otimos com restri�c~ao de comprimento L, onde w �e o maior peso dodado conjunto. O n�umero de bits adicionais por s��mbolo gerados por nosso c�odigo n~ao ultrapassa1= L�dlog(n+dlogne�L)e�2 para L > dlogne + 1, onde �e a raz~ao a�urea 1:618.Um importante aspecto do algoritmo proposto �e a simplicidade de sua implement�c~ao. O algo-ritmo �constitui basicamente de uma sucessiva constru�c~ao e �arvores de Hu�man para um conjuntomodi�cado de pesos.Palavras-Chave: C�odigos de Pre�xo, 'Arvores de Hu�man, Relaxa�c~ao Lagrangeana.1

1 IntroductionAn important problem in the �eld of Coding and Information Theory is the VariableLength Code Problem [Huf52]. A particular case of relevant interest is the BinaryPre�x Code Problem. Given an alphabet � = fa1; : : : ; ang and a corresponding setof weights fw1; : : : ; wng, the problem is to �nd a pre�x code for � that minimizesthe weighted length of a code string, de�ned to be Pni=1wili, where li is the length ofthe code assigned to ai. This problem is equivalent to the following problem: givena set of weights fw1; : : : ; wng, �nd a tree T that minimizes the weighted path lengthPni=1wili, where li is height of the ith leaf of T . T must be chosen among all fullbinary trees1 with n leaves. The equivalence is due to the fact that every binarypre�x code can be represented by a full binary tree [Eve79]. If the set of weights issorted, this problem can be solved in O(n) by one of the e�cient implementationsof Hu�man's Algorithm [Huf52, Lee76, Ktj95]. Any tree constructed by Hu�man'sAlgorithm is called a Hu�man tree.In this paper we consider the binary pre�x code problem with restricted maximallength, that is, for a �xed L � dlog ne, we must minimize Pni=1wili constrained toli � L for i = 1; : : : ; n. Gilbert [Gil71] recommends to formulate this problem whenthe weights wi are inaccurately known. Choueka, Klein and Perl [Cho85] suggest theuse of length restricted codes to reduce the external path lengthPni=1 li. The objectiveis to allow space e�cient decoding of optimal pre�x codes without bit-manipulation.Zobel and Mo�at [Zob95] describe the use of word-based Hu�man codes for compres-sion of large textual databases. The application allows the maximum of 32 bits foreach codeword. For the cases that exceed this limitation, it is recommended to usecodes with length restriction.Some methods can be found in the literature to solve the Binary Pre�x Code Prob-lem with Restricted Maximal Length [HuT72, Voo74, Fra93]. The �rst polynomialalgorithm is due to Garey [Gar74]. The algorithm is based on dynamic programmingand it has an O(n2L) complexity for time and space. Larmore and Hirschberg [Lar90]presented the Package-Merge algorithm. This is an O(nL) algorithm that requireslinear space. The authors reduce the original problem to a Coin's Collector Problem,using a nodeset representation of a binary tree. Turpin and Mo�at [Tur95] discusssome aspects about the implementation of the Package-Merge algorithm. Aggarwalet al: [Agg94] have used Megiddo's parametric search paradigm [Meg83] to obtain1In a full binary tree each internal node has exactly two sons.1

an O(npL log n+n log n) time algorithm. Currently, the fastest strongly polynomialtime algorithm for the problem is due to Schieber [Scb95]. This algorithm also utilizesa parametric search. However, it runs in O(n2o(plogL log logn)) time and requires O(n)space. These last two algorithms are based on a reduction of the Binary Pre�x CodeProblem to the Concave Least Weight Subsequence Problem [Lar95].In this paper we present an approximative algorithm that constructs length re-stricted pre�x codes. The algorithm is called WARM-UP and is based on lagrangeanrelaxation [Ahu93], a powerful technique used to solve some important problems inCombinatorial Optimization. The worst case complexity of the algorithm is given byO(n log n + n logw) , where n is the number of weights and w is the highest weight.The algorithm requiresO(n) space and can be implemented in-place [Mil97b] by usingthe Hu�man procedure presented in [Ktj95].The �rst parametric search proposed in [Agg94] is similar to the WARM-UP ap-proach introduced here. Nevertheless, from a lagrangean point of view, the formerapproach uses only one lagrange multiplier, whereas the later uses a varying number ofnonzero lagrange multipliers throughout the WARM-UP process. Furthermore, Ag-garwal's algorithm solves a sequence of Concave Least Weight Subsequence Problems,while our algorithm constructs a sequence of Hu�man trees.Due to the simplicity of Hu�man's procedure, the WARM-UP ALGORITHM hasa small constant factor in its O(n log n + n logw) time complexity. Furthermore,the Warm-up can take advantage of improvements in Hu�man's procedure, as therecently one proposed by Mo�at and Turpin [Mof97].The paper is organized as follows. In section 2, we give an integer programmingformulation to the problem, and develop the theoretical background needed to justifyour algorithm. In section 3, we address the special case of constructing a Hu�man treewith height exactly equal to L, given that the Hu�man tree with maximumheight hasheight greater than L, and the Hu�man tree with minimum height has height smallerthan L. In section 4, we introduce the WARM-UP ALGORITHM. A detailed analysisof the approximation and the complexity of the algorithm is presented. Finally, insection 5 we present some conclusions and comments on the implementation aspectsof the algorithm and other applications of this approach.2

2 The Lagrangean ApproachOur approach is based on an integer programming formulation to the problem [Fra93].Given a sequence of n integer weights w1 � : : : � wn and a length L � dlog ne theproblem is to �nd a sequence of n integers l1; : : : ; ln that minimizesnXi=1wi:liconstrained to 8>>><>>>: li � LPni=1 2�li = 1 i = 1; : : : ; nli 2 N (1)We call this problem PL. The restriction Pni=1 2�li = 1 is satis�ed if and only ifl1; : : : ; ln are the heights of the leaves of a full binary tree with n leaves [Eve79]. Inmathematical programming terms we say that PL is a primal problem [Min93].Since w1 � : : : � wn, then there is a solution where l1 � : : : � ln. Throughoutthis work we use this fact. For convenience, we use Sn to denote the points withinteger coordinates that satisfy Pni=1 2�li = 1 and we use h(T ) to denote the heightof a tree T .2.1 The Lagrangean RelaxationLet � = (�1; : : : ; �n), with �i � 0 for i = 1; : : : ; n. The Lagrangean function associ-ated to problem PL is given byL(l;�) = nXi=1wi:li + nXi=1 �i:(li � L) (2)The values �i are called Lagrangean multipliers. A point (l;�) is a saddle pointof the Lagrangean function L(l;�) if and only if(a) L(l;�) � L(l;�) for all l 2 Sn(b) li � L for i = 1; : : : ; n(c) �i:(li � L) = 0 for i = 1; : : : ; nIt is a well known result [Min93] that if (l;�) is a saddle point of the Lagrangeanfunction L(l;�), then l is a global optimum to the primal problem, in this case toproblem PL.Now, we can establish a su�cient condition for the sequence of integers li; : : : ; lnto be a solution to problem PL, for the given set of weights w1; : : : ; wn.3

Theorem 1 Let � = (�1; : : : ; �n), with �i � 0 for i = 1; : : : ; n. If the vector l =(l1; : : : ; ln) corresponds to the heights of the leaves of a Hu�man tree for the modi�edset of weights given by w1 + �1; : : : ; wn + �n, with8<: li � L andli = L if �i 6= 0;for i = 1; : : : ; n, then l solves problem PL.We show that (l;�) is a saddle point of the Lagrangean function L(l;�) by provingconditions (a)� (c).(a): Since l corresponds to the heights of the leaves of a Hu�man tree for theweights w1 + �i; : : : ; wn + �n, it follows thatnXi=1(wi + �i):li � nXi=1(wi + �i):li for all l 2 SnSubtracting the expression Pni=1 �i:L from both sides of the inequality, we obtainthat nXi=1wi:li + nXi=1 �i(li � L) � nXi=1wi:li + nXi=1 �i(li � L);and therefore L(l;�) � L(l;�) for all l 2 Sn:Conditions (b) and (c) follow immediately from the theorem hypothesis. Thisresult shows that l is a global optimum to problem PL.2.2 Finding the MultipliersTheorem 1 suggests a simple way to solve problem PL. Find multipliers �1; : : : ; �nsuch that the Hu�man tree for the modi�ed weights w1+�1; : : : ; wn+�n have heightequal to L, and all the leaves associated to the e�ectively modi�ed weights wi + �i,with �i > 0, are arranged at height L. This Hu�man tree is an optimal pre�x codetree with height restriction L.As an example, let us consider the set of weights W = f1; 1; 2; 3; 5; 8; 13g. Figure1.(a) shows a Hu�man tree for this set. Suppose we add 0.5 to the two lowest weightsof the given set. The tree in �gure 1.(b) is a Hu�man tree for this modi�ed set. Sinceall the leaves with modi�ed weights are arranged at height 4, then the tree in �gure1.(b) is an optimal code tree with restricted maximal height 4 for the original set ofweights W . 4

33

13 20

8 12

5 7

3 4

2 2

1 1

13

8

21

13 8

3 5

5

34

3

1.5 1.5

2

(a) (b) Figure 1: (a) A Hu�man tree for the set of weights f1; 1; 2; 3; 5; 8; 13g. (b) An optimaltree with maximal restricted height 4 for the same set.A natural question appears: how to choose these multipliers ?.Our approach to choose these multipliers is to select a value x with w1 < x < wnand set �i = maxf0; x� wig:Then, we use a test to verify if the chosen set of multipliers, de�ned by x, isadequate. Let wx be the new set of weights obtained through this perturbation.Observe that wx = (maxfw1; xg; : : : ;maxfwn; xg). Let Tx be a Hu�man tree for theset wx. If h(Tx) = L, then we are close to the conditions of theorem 1. That is truebecause x is the lowest weight of set wx, and leaves with equal weights are alwaysarranged in consecutive levels in a Hu�man tree [Vit87], in this case levels L andL� 1.One question arises: how to select x ?The following theorem shows that if we increase the value of the parameter x,then the height of tree Tx cannot increase. This monotone property suggests that wecan determine an adequate value for x through a convenient binary search.Let us de�ne T�x as a Hu�man tree with minimum height for the set wx, and T+xas a Hu�man tree with maximum height for the same set. Schwartz [Sch64] showshow to construct both of these trees.Theorem 2 WARM-UP Theorem 5

Let s; t be two real numbers with 0 < s < t. Then, we have that h(T+t ) � h(T�s ).Let h be the heights of the leaves of T�s , and let h be the heights of the leaves ofT+t . The optimality of h and h imply thatnXi=1maxfwi; sg:(hi � hi) � 0 (3)nXi=1maxfwi; tg:(hi � hi) � 0 (4)Let k be the greatest integer value such that Maxfs;wkg = s and let k be thegreatest value such that Maxft; wkg = t. Without loss of generality, we can chooseh and h such that hi � hi+1 and hi � hi+1 for i = 1; 2; � � � ; n� 1 .Adding inequalities (3) and (4) we obtain that(t� s): kXi=1(hi � hi) + kXi=k+1(t� wi)(hi � hi) � 0 (5)Since we have k equal weights in the tree T+t , it follows [Vit87] that h1 � hi �h1 � 1 for i = 1; � � � ; k.The highest height in an optimal tree is associated to the lowest weight. Hence, itfollows that h1 = h(T�s ) and h1 = h(T+t ). Now, let us suppose for absurd that h1 > h1.In this case we have that hi � hi for i = 2; � � � ; k, because hi � h1 � 1 � h1 � hi.Under this condition, inequality (5) is not satis�ed, what is an absurd. Therefore, wemust have h(T+t ) � h(T�s ).2.3 Well Numbered TreesThe WARM-UP theorem allows us to search for an appropriate value for x by per-forming a binary search on the interval (w1; wn). If for a given x, h(T�x ) > L, then wemust increase x. On the other hand, if h(T+x ) < L we must decrease x. The specialcase where h(T�x ) < L and h(T+x ) > L, is addressed in the next section.Until now, we don't have any assurance that there exists x such that h(Tx) = L.Before we attempt to answer this question, let us de�ne the concept of well numberedtrees.De�nition 1 A weighted full binary tree is well numbered if and only if(a) To each node is assigned a di�erent integer number i, with 1 � i � 2n� 1;6

(b) A parent node weight is equal to the sum of the weights of its two childrennodes.(c) If i > j, then the weight of the node with number i is not smaller than theweight of the node with number j;(d) Let i > j. If the node with number i is son of a node with number i0, and thenode with number j is son of a node with number j 0, then i0 � j 0.13

8

21

13 8

3 5

5

34

3

1.5 1.5

2

1 2

4 5 6

10 8 9

11 12

13

3

7 Figure 2: A well numbered treeThe tree in �gure 2 is well numbered. Each number is drawn at the right side ofthe node. Now, we state three key properties concerning the well numbered trees.Proposition 1 Let y1; : : : ; y2n�1 be the nodes of a well numbered tree T . Further-more, let yi1 and yi2, with i2 > i1 be the two children of node yi, and let yi3 and yi4with i4 > i3 be the two children of node yi+1. Under these assumptions, we have thati2 = i1 + 1; i3 = i2 + 1 and i4 = i3 + 1.First, we prove that i2 = i1 + 1. Assume that i2 > i1 + 1. Let z be the node withnumber i1+1. It follows that the parent of z has number not smaller than the parentof yi1 and number not greater than the parent of yi2 . Hence, the parent of z hasnumber i, what contradicts the fact that each internal node has only two children.The proof that i4 = i3 + 1 is analogous and the proof that i3 = i2 + 1 follows by thesame arguments already used. 7

Proposition 2 Let y1; : : : ; y2n�1 be the nodes of a well numbered tree T with corre-sponding heights l1; : : : ; l2n�1. Then, 0 � li � li+1 � 1 for i = 1; : : : ; 2n� 2.First, let us prove that li+1 � li for i = 1; � � � ; 2n � 2. It follows from properties (b)and (c) that y2n�1 is the heaviest node and hence the root of T . Hence, the resultholds for i = 2n � 2 . Let us assume by induction that the result is valid for i > pand then we prove it for i = p. Observe that the parent of yp has number smaller orequal to the number of the parent of yp+1. It follows by induction that the parentof yp has a height not lower than the height of the parent of yp+1. Since a parent'sheight is exactly one unity lower than the height of its children, the result holds fori = p.Now, suppose that li > li+1 + 1 for some i. In this case, the parent yi0 of yi hasheight li0 = li � 1. Therefore, i0 � i + 1 and li0 = li � 1 > li+1, what contradictsthe inequality li+1 � li for i = 1; � � � ; 2n � 2, that we have already proved. Hence,li � li+1 + 1.Proposition 3 If T is well numbered, then T is a Hu�man tree.It follows from proposition 1 that the tree T has the sibling property, de�ned byGallager [Gal78]. Therefore, T is a Hu�man tree.2.4 The Stopping ConditionNow, we prove that there is a value x, such that h(Tx) = L. This proof gives anintuitive idea about the e�ects in the tree generated by changing the parameter x.Theorem 3 If h(T�w1) > L � dlog ne then there is a rational number x=p/q, withq � n and w1 < x < wn, such that h(Tx) = L.Observe that T�w1 is the Hu�man tree for the original set of weights, sincemaxfw1; wig =wi for i = 1; : : : ; n. Let us assume that T�w1 is well numbered. In the section 3 weshow how to construct such a tree. We use yi to denote the node with number i, �ito denote the weight of node yi and ni to denote the number of descendent leaves ofnode yi with weight x. We de�ne ni = 1 when yi is a leaf with weight equal to x. Wealso de�ne �i by �i = (�i+1 � �i)=(ni � ni+1).The well numbered tree in �gure 2 is a Tx tree, with x = 1:5. For this tree, wehave n1 = n2 = 1, n4 = n7 = n11 = n13 = 2, and the others nodes have ni = 0. We8

also have that �1 = 0, �2 = 0:5, �4 = 0 and so on. Now, we prove that one of thetrees obtained by the following algorithm is a Tx tree with h(Tx) = L.x w1T T�xRepeatLet I = fiji = arg min f�ijni > ni+1ggDefine i� to be the smallest integer in IFor each leaf yj of T with �j = x do�j x+ �i�Update all internal node weights to assure that a parent weightis equal to the sum of its children weights.In T interchange2 the subtrees with roots yi� and yi�+1Assign number i� to yi�+1 and number i� + 1 to yi�x x+ �i�Until I = ;We divide the algorithm correctness proof into six parts.(Part a) All trees constructed by the algorithm are well numbered.items (a) and (b) of de�nition 1 are clearly respected by all trees T . Hence, weneed to prove that items (c) and (d) are also respected.For the initial tree, the result follows from the hypothesis under T�w1 . When thealgorithm adds �i� to the leaves with weight x and after updating the internal nodeweights, we still have �j � �j+1 for j = 1 : : : ; 2n � 2, since �i� is the minimumvalue necessary to produce a tie between the weights of two nodes with consecutivenumbers. Therefore, items (c) and (d) of de�nition 1 hold. On the other hand, byinterchanging the two chosen subtrees and also its corresponding root node numberswill also ensure items (c) and (d).(Part b) In each cycle, the tree T is a Tx tree for the current value of x.It follows from (Part a) and proposition 1 that each tree T is a Hu�man tree.Thus, we must prove that the weights of the leaves of T are given by maxfw1; xg; : : : ;maxfwn; xg for the current value of x. The leaves of the initial tree have these weights.Let Tm and xm be respectively, the tree T at the beginning of the mth cycle and thevalue of x at the beginning of this same cycle. We assume that the weights of the2When two subtrees are interchanged, the second subtree becomes a child of the root parent ofthe �rst subtree and vice-versa. 9

leaves of Tm are given by maxfw1; xmg; : : : ; maxfwn; xmg and then we prove thatthe weights of the leaves of Tm+1 are given by maxfw1; xm+1g; : : : ;maxfwn; xm+1g.We can suppose that wi � xm < wi+1. If xm+1 � wi+1, then the result clearlyholds. On the other hand, if xm+1 > wi+1 the result doesn't hold, since Tm+1 wouldhave a leaf with weight wi+1 < xm+1. However, if xm+1 > wi+1, then the integeri� chosen during the mth cycle is such that �i� = (xm+1 � xm) > (wi+1 � xm). Letyj be the node of tree Tm such that �j+1 = wi+1. Since �j � xm, we have that�j � wi+1 � xm. It follows that i� would not belong to I and as a consequence thecase xm+1 > wi+1 is not possible.(Part c) Node yi�+1 is always a leaf.First, let us assume that yi� is a leaf and that yi�+1 is an internal node. In thiscase ni� = 1, otherwise i� would not have been selected because ni� = 0 � ni�+1. As aconsequence we get that �i� = x. The two children of yi�+1 have numbers no greaterthan i�. Therefore, these children must have weights x, otherwise we contradict eitheritem (c) of de�nition 1 or the result of (Part b). Hence, ni� = 1 < 2 = ni�+1 and i�would not belong to set I.Now, let us assume that yi� and yi�+1 are both internal nodes. Let yi0; yi0+1 bethe sons of yi� and yi0+2; yi0+3 be the sons of yi�+1. It follows from the de�nitionof �i that �i0 + �i0+1 + (ni0 + ni0+1):�i� = �i� + �i�:ni� = �i�+1 + �i�:ni�+1 = �i0+2 +�i0+3 + (ni0+2 + ni0+3):�i�. We also have that �i0 + �i�:ni0 � �i0+1 + �i�:ni0+1 � �i0+2 +�i�:ni0+2 � �i0+3 + �i�:ni0+3, otherwise there would be a node yj with nj > nj+1 suchthat �j < �i�. To satisfy the two previous expressions we must have that �i0+�i�:ni0 =�i0+1+�i�:ni0+1 = �i0+2+�i�:ni0+2 = �i0+3+�i�:ni0+3. Since ni� > ni�+1, ni� = ni0+ni0+1and ni�+1 = ni0+2 + ni0+3, then either ni0 > ni0+1 or ni0+1 > ni0+2 or ni0+2 > ni0+3.Hence, there is an index j with i0 � j < i0 + 3 � i, such that nj > nj+1 and �i� = �j.This contradiction establishes the result.(Part d) Let M be the sum of the numbers assigned to the leaves of T . The �vestatements below are true:(i) (n2 � n)=2 �M � (3n2 � n)=2;(ii) If the algorithm halts, then M = (n2 � n)=2;(iii) After each cycle the value of M either remains the same or decreases byexactly one unity;(iv) In no more than n2 + n cycles the value of M satis�es M = (n2 � n)=2;(v) If M = (n2 � n)=2, then h(T ) = dlog ne.10

Demonstrations:(i) There are n leaves in T , each leaf assigned with a di�erent number. Thus,M � Pnj=1 j = (n2 � n)=2 and M �P2n�1j=n j = (3n2 � n)=2.(ii) In order to prove this item, we just need to show that I 6= ; wheneverM > (n2�n)=2. If the sum of the numbers assigned to the leaves of T is greater thann(n � 1)=2, then there is a leaf with number k, such that k > n. Since each numberin the set f1; : : : ; 2n � 1g is a assigned to a node, then there is a node with numbersmaller than n.It follows from item (d) of de�nition 1 that the parent of y1 is the internal nodewith minimumnumber. Letm be the number of y1's parent. We have that m < n < kand nm � 1. Since �k � �m > x and yk is a leaf, then nk = 0. As nm > nk, thereexist an index i with m � i < k, such that ni > ni+1. Hence, the set I is not empty.(iii) It follow from Part (c) that yi�+1 is a leaf. If yi� and yi�+1 are leaves, thenthe value of M remains the same. On the other hand, if yi� is an internal node, thenthe value of M decreases by exactly one unity, since we interchange the numbers.(iv) First, we show that in no more than n cycles yi� is a leaf. If yi� is a leaf then�i� = x, otherwise ni� = 0 � ni�+1. Furthermore, �i�+1 > x, otherwise ni�+1 = 1 =ni�. In this case the value of x is updated to x + �i�+1 � �i� = �i�+1. Hence, thenumber of leaves with weight x is increased by one unity. Therefore, this case canhappen in no more than n cycles, otherwise we would obtain a tree with more thann leaves.When yi� is an internal node, the value of M decreases by one unity. Since(n2 � n)=2 � M � (3n2 � n)=2, then this second case can happen no more than(3n2 � n)=2 � (n2 � n)=2 = n2. Therefore, in no more than n2 + n the value of Msatis�es M = (n2 � n)=2.(v) If M = (n2 � n)=2, then the numbers of all the leaves are not greater thann. Let lj be the height of a generic leaf yj. We have that the parent of y1 has heightl1 � 1 and number greater than n. It follows from proposition 2 that the height ofany leaf yj satis�es l1 � 1 � lj � l1. Therefore, all the leaves of T are arranged inconsecutive heights. Hence, h(T ) = dlog ne.(Part e) On each cycle the height of T either remains the same or decreases byone unity.If yi� has the same height as yi�+1, then the interchange does not modify h(T ).On the other hand, if the height of yi�+1 is smaller than that of yi� by one unity, then11

the interchange either decreases h(T ) by one unity or does not modify h(T ), becauseyi�+1 is a leaf.(Part f) The variable x assumes only rational values of the form p=q with p; q 2N and 1 � q � n during the algorithm execution.Let xm be the value of x at the beginning of the mth cycle. The �rst value of x isw1, so x1 is an integer. We observe that in the next assignment x assumes the valuexm + ((�i�+1 � �i�)=ni�) = xm + (�i�+1 � A � xm:ni�)=ni� = (�i�+1 � A)=ni�, whereA is the sum of the weights of the leaves that descend from yi� with correspondingweights values di�erent from xm. Since �i�+1 � A and ni� are positive integers and1 � ni� � n, we get that x assumes only values in the form p=q with p; q 2N and q �n. Algorithm CorrectnessIt follows from (Part d), that the algorithm can only stop if M = (n2 � n)=2.On the other hand if M = (n2 � n)=2, then h(T ) = dlog ne. Since the �rst tree hasheight greater than L and after each cycle the height of T does not decrease by morethan one unity, then the algorithm must obtain a tree Tx with height exactly L inno more than n2 + n. Furthermore, (Part f) assures that x is a rational number p=qwith q � n.This result allows us to stop searching for a value x when the length of the searchinterval is smaller than 1=n2, since there is only one rational with denominator lessthan n in such interval, and it can be easily determined.3 The Procedure TiesBefore describing our algorithm we must consider �rst a special case, the one whereh(T�x ) < L and h(T+x ) > L:For the same set of weights it is possible to exist more than one Hu�man tree. Thishappens because when constructing the tree, Hu�man's algorithm faces alternativechoices on how to choose a node when there is more than one node with minimalweight.Searching for an adequate value for x, we can face a situation where h(T�x ) < Land h(T+x ) > L. If we increase x, then the new tree gets a height smaller than L.On the other hand, if we decrease the value of x, then we obtain a tree with height12

greater than L. Theorem 2 guarantees this fact. Since we have already shown thatthere is always a value x such that h(Tx) = L, then the current value of x is the onewe are looking for. In this case, we can construct a Hu�man tree with height exactlyL for the current set of weights.The trees in �gure 3 are T3 trees for the weights 1; 2; 4; 6; 10; 16; 26. The left treeis a T+3 tree and its height is 6, while the right tree is a T�3 tree and its height is 4.If we have been trying to �nd a Tx tree with height 5, we would face the special caseadressed in this section.68

42

16 26

10 16

6 10

4 6

3 3

26 42

16

26

10

4 6

68

6

3 3

10

26 16

Figure 3: A T+3 tree and a T�3 tree for the weights 1; 2; 4; 6; 10; 16; 26.Let us consider the well known [Lee76] implementation of Hu�man's algorithmthat uses a stack S to store the leaves of the tree, and a queue Q to store the internalnodes. At start, Q is empty and S contains the leaves of the tree sorted by weights,such that the leaf with lower weight is located at the top of S.During the main step of this implementation, the weight of the leaf at the top ofS is compared to the weight of the internal node at the head of Q. The node withsmaller weight is selected. After the selection of two nodes, a new internal node iscreated. This new node is the parent of the selected nodes, and it is inserted at thetail of Q. Observe that whenever a node is selected, it is removed either from thetop of S or from the head of Q. The main step is executed until S is empty and Qcontains only one node.We say that a tie occurs whenever a leaf at the top of S has the same weight as an13

internal node at the head of Q. If we choose a leaf whenever a tie occurs, we obtaina Hu�man tree with minimum height. On the other hand, if we always choose aninternal node, we get a Hu�man tree with maximum height [Sch64].3.1 Procedure Ties DescriptionLet T� be a Hu�man tree with minimum height and let T+ be a Hu�man tree withmaximum height. Both of these trees having the same set of weights. The procedureTies constructs a Hu�man tree with height L when h(T�) < L and h(T+) > L.The procedure performs a binary search on the integers 1; : : : ; 2n � 2. For eachselected integer k, it constructs a Hu�man tree T k using the implementation describedabove with an additional re�nement: choose a leaf on each one of the �rst k tiesand then choose only internal nodes on all subsequent ties. If it obtains the tree withheight L, then the procedure ends. If the obtained tree has height lower than L, thenthe binary search proceeds with a decreased value of k. Otherwise, the binary searchproceeds with an increased value of k.3.2 Ties CorrectnessLet us suppose that procedure Ties produces also a numbering of the nodes as follows.Number 1 is assigned to the �rst node removed either from the top of S or from thehead of Q. Number 2 is assigned to the second removed node, and so on. It is easyto see that the constructed tree is well numbered.Now, let yi denote the node with number i in T k, with corresponding weight andheight given by �i and li respectively. Let us also assume that p is the number ofthe internal node selected when the (k + 1)-th tie occurs, and that q is the numberof the �rst leaf selected after node yp. Observe that �p = �p+1 = : : : = �q since the(k + 1)-th tie was between nodes yp and yq and T k is well numbered. Furthermore,it follows from property 2 that lp � lp+1 : : : � lq � lp � 1. Under these assumptions,the following steps are performed to obtain T k+1 from T k.For i = q down to p + 1Interchange the subtree rooted at yi with the one rooted at yi�1Assign number i to node yi�1, and number i� 1 to node yiNextTwo cases can occur when the procedure interchanges the subtrees. If the leaf yi14

has the same height as the internal node yi�1, then the tree height is not a�ected.Otherwise, yi has a height smaller than that of yi�1 by exactly one unity. In thissecond case, the interchange either reduces the tree height by one unity or keeps itthe same. Since lp � : : : � lq � lp � 1, then the second case can only happen once.Therefore, we have that h(T k+1) � h(T k) � h(T k+1) + 1.When h(T 2n�1) < L and h(T 0) > L, one can �nd a value k such that h(T k) = L,by performing a binary search on 1; : : : ; 2n. Hence, the procedure Ties can be usedto construct a Tx tree with h(Tx) = L, since h(T�x ) < L and h(T+x ) > L.3.3 Ties ComplexityThe binary search tries no more than dlog(2n � 2)e values for k. As we spend O(n)to construct an optimal tree if the frequencies are already sorted, we have that theprocedure spends an O(n log n) time in the worst case.4 The WARM-UP ALGORITHMFinally, we describe the algorithm. A pseudocode for it is given below.WARM-UP AlgorithmPhase 0 Trying a Huffman TreeIf h(T�w1) � L then Return(T�w1)Phase 1 Binary searchinf w1; sup wnWhile sup-inf > 1=n2 dox (inf+sup)/2If h(T�x ) = L then Return(T�x )If h(T+x ) = L then Return(T+x )If h(T�x ) < L and h(T+x ) > L then Return(Ties(x))If h(T�x ) > L then inf xelse sup xEnd WhilePhase 2 Search for p=qq 1; p dq.infe 15

While (p = q.inf or p � q.sup) doq q + 1p dq.infeEnd WhileReturn(Ties(p=q))Observe that the algorithm �rst constructs a Hu�man tree for the given set ofweights. If this tree has height not greater than L, then the algorithm halts sincethis tree solves our problem. Otherwise, the algorithm performs a binary search onthe interval (w1; wn). For each selected value x, it constructs the Hu�man trees T�xand T+x . If either h(T�x ) = L or h(T+x ) = L then the algorithm halts. If that doesn'thappen, then one of the following three actions is necessary:(i) if h(T�x ) > L then increase the value of x and continue with the binary search;(ii) if h(T+x ) < L then decrease the value of x and continue with the binary search;(iii) if h(T+x ) > L and h(T�x ) < L then call the procedure Ties for the current setof weights.The binary search also stops when the length of the searching interval is smallerthan 1=n2. Whenever this happens, the algorithm �nds the unique rational numberp=q with q � n in the last searching interval. Using this value of x, the algorithmbuilds a Tx tree. This follows immediately from theorem 3.4.1 Approximation ErrorIn the previous section we proposed a method to construct a pre�x code tree Tx withheight L. This tree is not necessarily an optimal pre�x code tree with restrictedmaximal height L. Now, we give an upper bound for the error of the code inducedby this tree. Let us de�ne this error � by� = ( nXi=1 pili � nXi=1 pili);where li is the height of ith leaf of an optimal pre�x code tree with restricted max-imal height L, li is the height of ith leaf of the tree Tx constructed by WARM-UPALGORITHM and pi is the probability of symbol ai, that is given by wi=(Pnj=1wj).The value Pni=1 pili is the average code length, and it represents the average numberof bits used per symbol. In order to bound �, we state an useful lemma.16

Lemma 1 Consider the weights w1 � : : : � wn. For these weights, let l1; : : : ; ln bethe lengths of the code induced by a Tx tree with height L. Let also, lL�11 ; : : : ; lL�1n belengths of the optimum pre�x code with restricted maximal length L � 1. Thus, wehave that Pni=1 pili � Pni=1 pilL�1i .Let p be the largest integer, such that lp = L. Consider problemPL with the additionalconstrain that li � L � 1 for i = p + 1; : : : ; n. Similarly to theorem 1, one can showthat the leaf heights of a Tx tree with height L solve the modi�ed problem. Hence, wemust have Pni=1 wili � Pni=1 wilL�1i . Otherwise, l1; : : : ; ln would not be the optimalsolution for this modi�ed problem. Since pi = wi=Pni=1wi, then we obtain thatPni=1 pili � Pni=1 pilL�1iNow, we are ready to state the theorem that gives an upper bound for �.Theorem 4 The maximum average code length di�erence � between the code inducedby the WARM-UP ALGORITHM and the optimal pre�x code with restricted maxi-mum length L is such that� < 1= L�dlog(n+dlogne�L)e�2 for L > dlog ne+ 1;where is the golden ratio 1.618.Milidiu and Laber [Mil97] show that if L � dlog ne + 1, then the maximum averagecode length di�erence between the the optimal pre�x code with restricted maximallength L and the unrestricted optimal pre�x code is smaller than 1= L�dlog(n+dlogne�L)e�1.Hence, it follows from this result and from lemma 1 that � < 1= L�dlog(n+dlogne�L)e�2for L > dlog ne+ 1.This result shows that the average code length di�erence is quite small for valuesof L not too close to dlog ne. For example, if L � dlog ne � 15, then the di�erence� is less than 0:01. For L � dlog ne � 20, we have that � < 0:0003. Nevertheless,undesirable results like the di�erence being greater than 1 could occur. However, thatwould only happen if L � dlog ne+ 1.4.2 ComplexityIn Phase 0, the algorithm spends an O(n) time to construct the Hu�man tree sincethe weights are already sorted.In Phase 1, the algorithm can spend an O(n log(n2:wn)) time in the worst case.That is true because in every cycle the length of the search interval is divided by17

2. Hence, in no more than dlog n2:(wn � w1)e steps we have sup� inf � 1=n2.Moreover, during each cycle we spend an O(n) time to determine both a Hu�mantree with minimum height and a Hu�man tree with maximum height, for the currentset of weights. In phase 1, the algorithm can also execute one call to procedure Ties.However, that does not a�ect the time complexity, since it represents an additionale�ort bounded by O(n log n).If the algorithm reaches Phase 2, it spends an O(n) time to �nd the adequaterational number p=q. Therefore, we conclude that the WARM-UP ALGORITHMhas an O(n log n+ n logwn) worst case time complexity.By using Mo�at's approach [Mof97], the WARM-UP time complexity can be easilyreduced to O(n log n+ r log(n=r) logw), where r is the number of distinct weights inthe given set of weights.4.3 LimitationsTheorem 1 shows that if all the leaves with weight x in a Tx tree have height L,then Tx is an optimal code tree with restricted maximal height L. Unfortunately, forsome sets of weights there is no such x. For instance, consider L = 3 and the setof weights given by f1; 1; 1; 13; 15g. Figure 4.(a) shows the tree topology for all Txwith 1 � x � 5, and �gure 4.(b) shows the tree topology for all Tx with 5 � x � 15.Observe that, in the tree topology of picture (b), not all the leaves with weight lessthan 5 have height 3. Moreover, the average code length for this tree is 2.064. Onthe other hand, this measure is equal to 2.032 for the optimal code tree exhibited in�gure 4.(c).One could ask what happens, if we choose the multipliers in a di�erent way. Forthe set of weights presented in �gure 4, it has been proved [Lab97] that the optimalcode tree can not be obtained, even though we try all the sets of multipliers.5 ConclusionsThe algorithm introduced in this paper is very easy to implement, since it consistsof successive constructions of Hu�man trees. Furthermore, the experiments reportedin [Mil97b] suggest that the WARM-UP algorithm is a very good choice when con-structing length restricted pre�x codes for large alphabets, if we consider time andspace requirements. 18

31

15 16

13 3

1 2

1

31

28

13

3

1 14

1

2

1 1 13

31

15 16

2

1 1

15

1 Figure 4: (a) Tree Tx for the set of weights W = f1; 1; 2; 3; 5; 8; 13g and 1 � x � 5.(b) Tree Tx for the set of weights W and 5 � x � 15. (c) An optimal tree withrestricted maximal height 3 for the set W .The value logwn that appears in the worst case complexity is lower than 40 (wn � 1012 ) for practical purposes and the constants involved in the complexity aresmalls, since there is a lot of very optimized implementations of Hu�man trees. Inorder to improve its time e�ciency, the algorithm can be executed until the averagecode length di�erence � is smaller than a given acceptable error �. An upper boundfor � can be obtained after the construction of each tree Tx by using the value of thedual function.Furthermore, we believe that the techniques developed here can be useful to ad-dress the problem of constructing optimal pre�x codes satisfying other kinds of con-strains.References[Agg94] Aggarwal, A., Schieber B., Tokuyama T., Finding a minimum weight K-linkpath in graphs with Monge property and Applications, Journal of Discrete andComputational Geometry 12(1994).[Ahu93] Ahuja, R. K., Magnanti, T. L., Orlin J. B., Network Flows:theory, algorithmand applications, Prentice Hall, 1993.[Cho85] Choueka, Y., Klein S. T. and Perl, Y., E�cient variants of Hu�man Codesin high level languages, Proc 8th ACM-SIGIR Conf, Montreal, 122-130 (1985).19

[Eve79] Even, S., Graph Algorithms. Computer Science Press, 1979.[Fra93] Fraenkel, A.S., and Klein, S. T., Bounding the depth of search trees, TheComputer Journal, 36, (7), 668-678 (1993).[Gal78] Gallager, R. G., Variations on a theme by Hu�man, IEEE Transactions onInformation Theory., IT-24, 668-674, (1978).[Gar74] Garey M. R., Optimal binary search trees with restricted maximal depth,SIAM J. Computing 23(1974), 101-110.[Gil71] Gilbert, E.N., Codes based on innacurate source probabilities, IEEE Transac-tions on Information Theory., IT-17, 304-314, (1971).[HuT72] Hu, T. C., and Tan, K. C., Path length of binary search trees., SIAM J.Appl Math 22 (1972), 225-234.[Huf52] Hu�man, D.A., A Method for the Construction of the Minimum RedundancyCodes, Proc. Inst. Radio Engineers, Vol 40,No 10, 1098-1101, September 1952.[Kat76] Katona, G. O. H., and Nemetz O. H. T., Hu�man codes and self information,IEEE Transactions on Information Theory., IT-22, 337-340, (1976).[Ktj95] Katajainen, J., Mo�at, A., Inplace calculation of minimum-redundancy codes,In Proc. Workshop an Algorithms and Data Structures, Kingston University,Canada, August 1995. Springer-Verlag.[Lab97] Laber, E. S., Um Algoritmo E�ciente para Constru�c~ao de C�odigos de Hu�-man com Restri�c~ao de Comprimento, M.Sc. Dissertation, Departamento de In-form�atica, PUC-RJ, Brasil, 1997.[Lar90] Larmore, L.L. and Hirschberg, D.S., A Fast Algorithm for Optimal Lenght-Limited Hu�man Codes, Journal of the ACM, July (1990), 464-473.[Lar95] Larmore, L.L. and Przytycka, T.M., Constructing Hu�man Trees in Parallel,Siam Journal on Computing, Novemeber (1995), 1163-1169.[Lee76] Leeuwen, J. van (1976), On the construction of Hu�man trees, In proc. 3rdInt. Coll. On Automata, Languages, and Programming, pp 382-410.20

[Meg83] Megiddo, N., Applying Parallel Computation Algorithms in the Design ofSerial Algorithms Journal of ACM 30 (1983).[Mil97] Milidiu, R.L. and Laber, E.S., Improved Bounds on the Ine�ciency of Length-Restricted Pre�x Codes, Monogra�as em Ciencia da Computa�c~ao 33/97, Depar-tamento de Inform�atica, PUC-RJ, Brasil, 1997.[Mil97b] Milidi�u, R.L., Pessoa A. A. and Laber, E.S., Practical use of the WARM-UP Algorithm on Length-Restricted Codes, To appear in the Proceedings of theFourth Latin American Workshop on String Processing, Chile, 1997.[Min93] Minoux, M., Programmation mathematique, theorie et algorithmes tome 1,Collection Technique et Scienti�que des Telecomunications, Dunod (1993).[Mof97] Mo�at A. and Turpin A. E�cient Construction of Minimum-RedundancyCodes For Large Alphabets, To appear in IEEE Transactions on InformationTheory.[Sch64] Schwartz, E. S., An Optimum Encoding with Minimum Longest Code andTotal Number of Digits, Information and Control 7, 37-44 (1964).[Scb95] Schieber B. Computing a minimum-weight k-link path in graphs with the con-cave monge property., In Proc. 6th ACM IEEE Symposium on Discrete Algo-rithms, pages 405-411,San Francisco, January 1995.[Tur95] Turpin A. and Mo�at A., Practical Length-limited Coding for Large Alphabets,The Computer Journal, 38, (5), 339-347 (1995).[Vit87] Vitter J. S., Design and Analysis of Dynamic Hu�man codes, Journal of ACM34, 4, Oct, 825-845.[Voo74] Van Voorish D. C., Constructing codes with bounded codeword lengths, IEEETransactions on Information Theory., IT-20, 288-290, (1974).[Zob95] Zobel, J. and Mo�at, A. Adding Compression to full-text retriweval system.Software Practice and Experience, 25(8): 891-903, 1995.21