IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. XXX, NO. XXX, MMM-YYYY 1

Grouping tasks and data display items via thenon-negative matrix factorization

Dorrit Billman, Hera Y. He, and Art B. Owen

Abstract—Analyzing work functions and the IO variables theyneed is an important component of designing and evaluatingcomplex systems. We develop a biclustering method for jointlygrouping work functions and IO variables. Given a binary matrixindicating which IO variables are required for which workfunctions, we develop a computational method for finding densegroups of related tasks and information. We relax the binarygrouping problem to a continuous one, and then use the non-negative matrix factorization, followed by dichotomization, tosearch for useful groupings. The result is a more automatedapproach to finding groupings for interaction design. Our mo-tivating data set comes from NASA, where the tasks are thosethat a pilot must perform and the sources are displays requiredfor those tasks.

Index Terms—biclustering, human factors, interface design

I. INTRODUCTION

COMPLEX work such as flying an airplane requiresintegration of pilot and automation by providing the pilotwith relevant information at the relevant time and by enablingthe pilot to take appropriate actions. Cockpit design dependsboth on providing the large set of needed information andaction variables, and on organizing the information so that itis presented as needed for particular work functions.

The safety, effectiveness, and usability of complex, partiallyautomated systems depends on an analysis of the work thesystem is intended to support. There are key advantages ofrepresenting work at a relatively abstract level of analysis,above the specifics of “button pushing” interaction with aparticular device. A representation level above the details ofinteracting with a specific device means this work analysiscan guide design of a novel system where no specific tasksand interactions yet exist. Further, it allows comparing theeffectiveness of alternative designs for the same work. Thisapproach is thus relevant early and throughout the developmentprocess. The value of work representations that abstract awayfrom specifics of task execution has been widely noted ([1],[2], [3], [4], [5] and [6]) and our representation and analysisfalls within this family of proposals. We use the term workfunctions to indicate a level of analysis more abstract thanspecific tasks. We refer to the information that work functionsneed as input or affect as output with the term IO variables.

We represent work as three components: a set of workfunctions, a set of IO variables, and the association betweenspecific work functions and variables needed. A work functiontakes information variables as input and affects controllable

D. Billman is with the San Jose State University Psychology DepartmentH. Y. He and A. B. Owen are with the Department of Statistics, Stanford

UniversityManuscript received xxx 2005; revised xxx.

variables as the output of that work function. For example, forthe cockpit work of flying a highly automated commercial jet,a work function might be accepting an altitude clearance fromAir Traffic Control (ATC); its input variables would includethe altitude and location information in the clearance andthe current trajectory of the aircraft; and its output variableswould include communications back to ATC and input tothe automation controlling altitude. We represent these threecomponents of work in a binary matrix, with a row for eachwork function, a column for each IO variable, and cell entriesset to 1 if a given work function takes a given IO variable andto 0 otherwise.

For simple domains, it might be feasible to treat every workfunction independently, providing the ideal organization of IOvariables for each. For complex domains, the work functionsand IO variables are interrelated, and there maybe tradeoffsin layout or design because what is best for one functionmight not be best for another. In this situation it is valuable tounderstand not only the IO variables related to an individualwork function and work functions related to a specific IOvariable, but also to understand the overall organization orstructure.

We present a method for assessing what groups of taskscluster together based on the IO variables they use and whatgroups of IO variables cluster together based on the workfunctions they support. These groups carry important infor-mation about the work which could be used to guide designand evaluation of the functionality and usability of a complexautomation. Our long term goal is to provide practitioners withthis information in a useful form; it should allow practitionersto inspect the work structure easily and to inform decisionssuch as design tradeoffs.

To develop the method, we used a preliminary matrixrepresentation for cockpit aviation work. The matrix wasproduced by a consulting research pilot and one of the authors,drawing on prior analyses and experience. To provide a clearstarting point, the scope of the work domain was highlyconstrained; we aimed to capture activities on a completelynominal flight, by a large, highly automated, commercialpassenger liner operating in US airspace. The content ofthe matrix is preliminary with additional work ongoing toassess its completeness and reliability. For present purposes,it provided a good case study of matrix representation of awork domain operating complex partially automated systems.

The work analysis produced a matrix of 119 work functionsand 210 variables, the data matrix. Overall, this matrix issparse, with about 7% of cells set to 1. Individual workfunctions differ greatly in how many variables they take andIO variables differ in how many work functions they support.

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. XXX, NO. XXX, MMM-YYYY 2

Highly used input variables include speed and deviation fromspeed; highly used output variables include stating which pilothas control and setting a speed target; in contrast, checkingavailable navigation support (input) and adding a leg to theflight plan (output) are used by few functions. Work functionsalso differ in how many variables they need: “Establish aircraftof extended runway centerline” uses many, while “deployingaerodynamic drag” uses few.

Figure 1 shows a schematic view of the data matrix, withentries of 1 shown in blue. Figure 1 shows that there is partialbut not perfect “block” structure, with some variables andfunctions tending to cluster together, forming a partial butimperfect block structure. There are also conspicuous blocksthat are nearly empty. We wanted a method to identify thishigh-level, organization structure. Conceptually, we want toknow how work functions and IO variables mutually formgroupings. Visually, the problem data resemble those seenin biclustering problems. In biclustering, one seeks sets ofrows that form dense clusters with respect to some though notnecessary all columns. Conversely, the columns cluster withrespect to some but not necessarily all rows. For a survey ofbiclustering methods see [7].

0 50 100 150

IO variables0

20

40

60

80

100

work

funct

ions

Fig. 1. Heatmap of the NASA data matrix. A dark pixel represents an IOvariable needed to perform a pilot’s work function.

Several aspects of work structure are not explicit in our ma-trix representation (or bipartite graph), such as sequence, sub-goals, and set inclusion. We chose to use a matrix (specifically,bipartite graph) representation because it provides a compact,familiar, and computationally tractable way of representingwork. Further, by relating work to IO variables, the structurefound for work may be compared to the structure provided bya device.

A. Relation to prior biclustering methods

Clustering methods group instances based on shared at-tributes, e.g. grouping rows by columns. Biclustering meth-ods simultaneously group rows by columns and columns byrows, which is valuable if the joint relations are important.Further, biclustering allows overlap among the groupings, as

when a subgrouping belongs to two larger groups. Traditionalbiclustering finds the exceptionless groupings in a data set.Visually, this would be all completely blue rectangles withina data matrix such as that illustrated in Figure 2.

0 2 4 6 8 100

2

4

6

8

10

Fig. 2. Example of simple biclusters. The dark region is formed as the unionof three rectangular biclusters.

Biclustering algorithms are available online, and we usedthe BicAT-Plus software [8] an extension of BicAT [9], andparticularly its implementation of the bimax algorithm [10].This algorithm finds all exceptionless biclusters, allowingoverlap across biclusters. However, we found that this was notvery effective for our problem. We got large numbers of verysimilar biclusters. Broadly, the underlying clusters seemed tohave exceptions, but the allgorithm finds exceptionless group-ings. As a result, it returns many highly overlapping, verysimilar biclusters, each avoiding what would be an exceptionin a larger group.

The root of the problem is that our goals do not perfectlymatch those in biclustering, in several respects. First, we wanta biclustering that covers the matrix with a small number ofcategories. While overlap is important to allow, we would likethe most compact set of groupings that capture most of thestructure in a matrix. Second, our problem is asymmetric:it is worse to miss an IO variable to work function pairthan to cover it twice. Third, we also would like a groupof IO variables that are jointly useful for a large number ofwork functions. Such a subset of IO variables could be animportant design component, presented as an integrated groupand frequently available. The converse, i.e., a group of workfunctions that require a large number of IO variables to coverthem, is less useful information to capture.

The very hardest problem we face is neither mathematicalnor computational. There is no computable criterion thatperfectly measures the quality of a biclustering for interfacedesign. There are competing subjective desiderata, not leastbecause multiple users face different issues. Our approachis to construct the best proxy criterion we can, search forgood biclusterings and use those as a design aid. Others couldreasonably choose a different proxy measure.

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. XXX, NO. XXX, MMM-YYYY 3

B. Outline

An outline of this paper is as follows. Section II describesour method, giving our proxy loss function, and showing howwe use non-negative matrix factorizations to search for goodbiclusterings. Section III shows some of the results for thecockpit design problem. Section IV gives our conclusions.

II. SEARCH STRATEGY

A. The data set and representation

The problem input data take the form Xij ∈ {0, 1} wherei = 1, . . . , I indexes the work functions (tasks for short), j =1, . . . , J indexes the IO variables (items for short) and Xij = 1if and only if task i requires item j. Our specific problem datawere curated by experts at NASA. One expert produced thelist of display items and the other, a trained pilot, producedthe list of tasks. The pilot also identified the items needed byeach task.

We will represent our problem data using L biclusters, orlayers. The reason for calling them layers will be apparentbelow. Layer ℓ = 1, . . . , L includes a subset of tasks anditems. The matrix T ∈ {0, 1}I×L has Tiℓ = 1 if and onlyif task i is in layer ℓ. The matrix D ∈ {0, 1}J×L has Djℓ = 1if and only if display item j is in layer ℓ.

An ideal biclustering would satisfy Xij =∑L

ℓ=1 TiℓDjℓor, more compactly, X = TDT, for a small number L. Inthis additive interpretation we are summing L layers, withlayer ℓ comprised of column ℓ from both T and D. Then,for every task-item pair with Xij = 1, we would find i inprecisely one bicluster and j in and only in that same bicluster.The representation X = TDT is trivially attainable by takingL =

∑i

∑j Xij but that solution is not useful.

For small L, the matrix TDT is a low-rank approximationto X . Non-negative matrix factorization algorithms often seekTDT close to X according to criteria such as mean squareerror or Kullback-Leibler distance [11].

B. The proxy criterion

If we choose binary matrices T and D then we can viewthem as an approximation X̂ = TDT to X . Then X̂ij is thenumber of layers including task i and item j. In our applicationwe want each task-item combination to be covered, ideallyonce. Therefore we adopt a loss function

Loss(X̂,X) =I∑

i=1

J∑j=1

loss(X̂ij , Xij) (1)

where

loss(X̂ij , Xij) =

0, X̂ij = Xij

1, 0 = Xij < X̂ij

τ, 1 = Xij , X̂ij = 0.

(2)

There is a no penalty for a task-item pair that does not appearin either the true or estimated matrix. There is also no penaltyfor a task-item pair that appears exactly once in both matrices.We charge a penalty of 1 if a nonexistent pair is in one or morelayers and τ if the ensemble of estimated layers fails to includeone of the true pairs.

Our loss function is only a proxy for a true loss between Xand its layer representation. We chose to work with τ = 10because missing a task-item pair is more serious than includingit twice. We briefly considered making the loss function moresevere for larger values X̂ij −Xij , that is for greater amountsof overcoverage. However that introduces one more parameterto choose and we judged that the degree of overcoverage isnot an important enough issue.

C. Search space

Clustering is already an NP-hard problem [12] and biclus-tering is even worse, and so we do not expect that our proxyloss can be minimized on large problems.

Our strategy is to relax the binary search problem toone with nonnegative matrices T ∈ [0,∞)I×L and D ∈[0,∞)J×L. Then we search for TDT .= X , which is calleda non-negative matrix factorization. We search for such fac-torizations and then coerce elements of T and D separatelyto {0, 1}. Larger elements become 1 and smaller ones 0. Thethreshold between 0 and 1 is determined numerically by ourloss criterion (1).

The initial search space for this problem consists of1) 10 different NNMF algorithms2) L = 103) 50 different random seeds for NNMF starts4) 8 different thresholds for dichotomizing T5) 8 different thresholds for dichotomizing D

After some search we discovered that two of the NNMFalgorithms (Alternating Nonnegative Least Squares MatrixFactorization Using Projected Gradient (bound constrainedoptimization) method for each subproblem (LSNMF) [13] andSparse Nonnegative Matrix Factorization (SNMF) [14]) almostalways yielded the best loss. As a result, we focussed oursearch on just those two. Similarly after exploring thresholdsin the range from 0 to 1 we found that the best thresholdswere usually between 0.8 and 1 and so we focused our searchon just that range.

The refined search space consists of1) 2 different NNMF algorithms2) L = 10, 15, 203) 100 different random seeds for NNMF starts4) 20 different thresholds for dichotomizing T5) 20 different thresholds for dichotomizing D

We found that 58% of the time LSNMF outperformedSNMF.

III. RESULTS

The results from a biclustering can be presented as L dif-ferent binary matrices. The matrix for layer ℓ is the submatrixX(ℓ) of Xij for which Tiℓ = 1 and Djℓ=1. The density ofX(ℓ) is its average value, ranging between 0 and 1.

Two layers overlap if they share some columns and alsoshare some rows. Let Rℓ and Cℓ be the number of rows andcolumns (respectively) in the matrix X(ℓ). For ℓ ̸= ℓ′, let Rℓ,ℓ′be the number of rows that they have in common, that is, thenumber of indices i for which Tiℓ = Tiℓ′ = 1. Similarly, letCℓ,ℓ′ be the number of columns that they have in common.

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. XXX, NO. XXX, MMM-YYYY 4

Fig. 3. Comparison of different NNMF algorithms performance. The y axisrepresents the chance of each algorithm outperforming all the rest based oncriterion (1). The names are consistent with the the nimfa python package [15].

Then the overlap between the matrices of layers ℓ and ℓ′ isRℓ,ℓ′ × Cℓ,ℓ′/min(Rℓ × Cℓ, Rℓ′ × Cℓ′). It gives the ratio ofcommon entries to the number of entries in the smaller layer.

We have found it useful to sort the matrices by density indecreasing order. The first ones then show some very sharpstructure. We found that somewhere before L = 20 the layersare not very dense. Using a smaller L that yielded only verydense layers would leave the user wondering whether theremight be more dense layers to find. By contrast, when we seesome dense and some non-dense layers it is more likely thatthe algorithm has done nearly the best it can.

After searching through the parameter space, each parametercombination yields a design, i.e., a group of biclusters. Wegroup designs based on their average aspect ratio (AAR). Theaspect ratio for a matrix (bicluster) X(ℓ) is defined to beRℓ/Cℓ. For each design, we calculated its AAR by averagingthe aspect ratio over all of its L = 20 biclusters. Designs withsimilar AAR are collected to form a design group with anassociated AAR. For each design group, we take the designwith minimum loss to be its representative. The representativesare designs that approximately achieves the optimal loss undereach AAR contraint.

The optimal loss of each design group versus its associatedAAR is shown in Figure 4. Forcing the aspect ratio to be large,say above 2, raises the value of the loss appreciably.

We also grouped biclustering results based on the maximumnumber of IO variables in the biclusters. The optimal lossversus different maximum number of IO variables is shown inFigure 5. Groupings where all biclusters have fewer than 20or 25 IO variables perform poorly, but there is comparativelylittle to gain by allowing even greater numbers of IO variablesin the groups.

A. Example

Here we inspect one of the biclusters that our algorithmproduced. This one attained the best value of our proxy

Fig. 4. Optimized loss versus aspect ratio.

Fig. 5. Optimized loss versus maximum number of IO variables.

function. It has L = 20 layers. Figures 6 and 7 show thetwo densest layers in the model. The bicluster shown inFigures 6 groups together the variables concerned with lateralcourse, both track and heading. The cluster includes the actualcurrent value, the target value, and constraints. The workfunctions include those directly concerned with navigationacross several phases of flight and also communication andplanning functions. The bicluster shown in Figures 7 groupstogether work functions concerned with monitoring, which aredistinctive in that they occur across the phases of flight, anddirectly affect pilot’s state rather than the aircraft. Figure 8shows all 20 biclusters ordered from densest to sparsest.

The bicluster densities drop nearly linearly from 0.79 to0.37 and fifteen of them are above 0.5. There is very littleoverlap among our 20 layers. The average of 20×19/2 = 190overlaps was 0.013. The largest was 0.31 but only 4 of theoverlaps were larger than 0.1 and 137 of them were exactly 0.

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. XXX, NO. XXX, MMM-YYYY 5

Fig. 6. First bicluster from the example. The rows correspond to 14 selectedwork functions and the columns denote 10 selected IO variables. This biclusterhas a density of 110/140 .= 0.79.

Fig. 7. Second bicluster from the example. The rows correspond to 6 selectedwork functions and the columns denote 11 selected IO variables. This biclusterhas a density of 49/66 .= 0.74.

Fig. 8. This figure shows a zoom out view of all 20 bicluster layers, orderedfrom the densest to sparsest.

IV. CONCLUSIONS AND FURTHER WORK

Generating and evaluating the interaction of user and systemis difficult. Specifically, it is difficult to ensure that the systemsupports the work the user needs to accomplish. Current prac-tice relies primarily on the wisdom of the designers, with fewsupporting tools to assess how well the functionality offeredmatches the functionality needed for the work. Systematicmethods to support this process would be valuable [16].

Our research addresses the need for methods at two levels.First, we recommend a census of the work functions in thedomain and of the information needed as input or affected asoutput. The relations between work functions and the infor-mation implicated for each should be represented explicitly aswell. Here we represent this as a binary matrix scoring eachwork function for each information variables it implicates. Theresulting matrix representation alone can be used to designforto assess coverage, for example, to assess whether a deviceprovides the information needed for a particular work function.Second, we recommend the matrix be used to identify theorganization, identifying what groups of variables tend to beused together and what groups of work functions operatewith similar variables. This can support design decisions suchas what variables should be displayed together, or wherecross-task consistency of interaction may be most valuable.Looking at the organization of information and work functionsas a whole supports design decisions concerning the overallcoherence and integration of the system.

We have developed an approach to organize a large bi-nary matrix into meaningful subsets. Our biclustering methodprovides groupings of mutually relevant work functions andvariables. We needed a proxy measure to guide the algorithmicsearch for meaningful biclusters. The measure we used is aproxy, because we do not know an objective function defininga “ground truth” for biclusters in this domain. Rather, differentdesirable properties may be weighted differently, perhapsreflecting variation across work-domain or design needs. Inshort, other proxy measures can be substituted while still usingthe analysis framework developed here.

Building the input matrix requires domain knowledge. Thereis a subjective element in choosing whether Xij = 0 or 1.Multiple raters can reasonably differ, for example, in settinga scoring threshold for whether an input variable shouldbe scored for a work function if that variable is normallyneeded or needed only in extraordinary conditions. We areinterested in identifying which of our structures are stable withrespect to uncertainty in X . A straightforward approach is toperturb a small fraction of Xij elements at random and rerunthe algorithm. We may also sum or average matrices frommultiple experts and use those as input. Such analysis wouldidentify how robust the structure is, across coding variations.In turn, this could characterize how robust the analysis isacross encoding variation.

Whatever the benefits of our approach, it is most likely to beadopted by practicioners if the results are presented in a clear,flexible, interactive manner. We are researching vizualizationmethods for the output that would allow a user to reviewresults both overall and by selected biclusters. In addition, we

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. XXX, NO. XXX, MMM-YYYY 6

are exploring how to support creation of the input matrix. Webelieve that design and evaluation of system interaction canbenefit from systematic methods for identifying the structureof work, such as the biclustering method presented here.

ACKNOWLEDGMENT

This work was supported in part by grant DMS-0906056 ofthe U.S. National Science Foundation, and in part by NASAHuman Research Program, Space Human Factors Engineering,Task ID#670. We would like to thank Michael Steward, a pilotand domain expert, for work developing the input data matrix.

REFERENCES[1] D. Billman, M. Feary, D. Schreckengost, and L. Sherry, “Needs analysis:

the case of flexible constraints and mutable boundaries,” in CHI’10Extended Abstracts on Human Factors in Computing Systems. ACM,2010, pp. 4597–4612.

[2] K. A. Butler, J. Zhang, C. Esposito, A. Bahrami, R. Hebron, andD. Kieras, “Work-centered design: a case study of a mixed-initiativescheduler,” in Proceedings of the SIGCHI conference on Human factorsin computing systems. ACM, 2007, pp. 747–756.

[3] D. E. Kieras, “Task analysis and the design of functionality.” Thecomputer science and engineering handbook, vol. 23, pp. 1401–1423,1997.

[4] K. J. Vicente, Cognitive work analysis: Toward safe, productive, andhealthy computer-based work. CRC Press, 1999.

[5] E. M. Roth, “Uncovering the requirements of cognitive work,” HumanFactors: The Journal of the Human Factors and Ergonomics Society,vol. 50, no. 3, pp. 475–480, 2008.

[6] R. G. Eggleston, “Work-centered design: A cognitiive engineeringapproach to system design,” in Proceedings of the Human Factors andErgonomics Society Annual Meeting, vol. 47, no. 3. SAGE Publications,2003, pp. 263–267.

[7] S. C. Madeira and A. L. Oliveira, “Biclustering algorithms for biologicaldata analysis: a survey,” Computational Biology and Bioinformatics,IEEE/ACM Transactions on, vol. 1, no. 1, pp. 24–45, 2004.

[8] “BicAT-Plus manual,” http://home.k-space.org/Bicat-plus-manual.pdf.[9] S. Barkow, S. Bleuler, A. Prelić, P. Zimmermann, and E. Zitzler, “BicAT:

a biclustering analysis toolbox,” Bioinformatics, vol. 22, no. 10, pp.1282–1283, 2006.

[10] A. Prelić, S. Bleuler, P. Zimmermann, A. Wille, P. Bühlmann, W. Gruis-sem, L. Hennig, L. Thiele, and E. Zitzler, “A systematic comparisonand evaluation of biclustering methods for gene expression data,”Bioinformatics, vol. 22, no. 9, pp. 1122–1129, 2006.

[11] D. Seung and L. Lee, “Algorithms for non-negative matrix factorization,”Advances in neural information processing systems, vol. 13, pp. 556–562, 2001.

[12] D. Aloise, A. Deshpande, P. Hansen, and P. Popat, “NP-hardness ofEuclidean sum-of-squares clustering,” Machine Learning, vol. 75, no. 2,pp. 245–248, 2009.

[13] C.-J. Lin, “Projected gradient methods for nonnegative matrix factoriza-tion,” Neural computation, vol. 19, no. 10, pp. 2756–2779, 2007.

[14] H. Kim and H. Park, “Sparse non-negative matrix factorizations viaalternating non-negativity-constrained least squares for microarray dataanalysis,” Bioinformatics, vol. 23, no. 12, pp. 1495–1502, 2007.

[15] M. Žitnik and B. Zupan, “Nimfa: A python library for nonnegative ma-trix factorization,” The Journal of Machine Learning Research, vol. 13,pp. 849–853, 2012.

[16] R. W. Pew, A. S. Mavor et al., Human-system integration in the systemdevelopment process: A new look. National Academies Press, 2007.

Dorrit Billman is with the San Jose State University Psychology Department.

Hera Y. He is a PhD candidate in the Department of Statistics at StanfordUniversity.

Art B. Owen is a professor in the Department of Statistics at StanfordUniversity.