Diagrammatic Representation and Reasoning

Springer-Verlag London Ltd.

Michael Anderson, Bernd Meyer and Patrick Olivier (Eds)

Diagrammatic Representation and Reasoning

, Springer

Michael Anderson, MSc, PhD Department of Computer and Information Sciences, Fordham University, 441 East Fordham Road, Bronx, NY 10458, USA

Bernd Meyer, Dr rer nat School of Computer Science and Software Engineering, Monash University, PO Box 26, Clayton Campus, Victoria 3800, Australia

Patrick Olivier, MA, MSc, PhD Department of Computer Science, University of York, Hes1ington, York, YOl05DD, UK Cover images: Reproduced from Chapter 4 and used by kind permission ofProfessor Malcolm Longair (eMalcolm Longair. AII rights reserved.)

British Library Cataloguing in Publication Data Diagrammatic representation and reasoning

l.Artificial intelligence 2.Problem solving 3.Charts, diagrams, etc. I.Anderson, Michael II.Meyer, Bernd III.Olivier, Patrick 006.3

Library of Congress Cataloging-in-Publication Data Diagrammatic representation and reasoning / Michael Anderson, Bemd Meyer, and Patrick Olivier (eds.).

p.cm. Includes bibliographical references and index. ISBN 978-1-85233-242-6 (aIk. paper) 1. Automatic theorem proving. 2. Artificial intelligence. 3. Image processing. 4.

Problem solving. 1. Anderson, Michael, 1965- II. Meyer, Bemd, 1951- III. Olivier, Patrick, 1966-QA76.9.A96 D54 2000 006.3--dc21 00-063765

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of Iicences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. ISBN 978-1-85233-242-6 ISBN 978-1-4471-0109-3 (eBook) DOI 10.1007/978-1-4471-0109-3 http://www.springer.co.uk

© Springer-Verlag London 2002 OriginalIy published by Springer-Verlag London Berlin Heidelberg in 2002

The use of registered names, trademarks etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and reguIations and therefore free for general use.

The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made.

Whilst we have made considerable efforts to contact alI holders of copyright material contained within this book, we have failed to locate some of them. Should holders wish to contact the Publisher, we will be happy to come to some arrangement.

Typesetting: Camera ready by editors 34/3830-543210 Printed on acid-free paper SPIN 10748985

Preface

Diagrams are essential in most fields of human activity. There is substan­tial interest in diagrams and their use in many academic disciplines for the potential benefits they may confer on a wide range of tasks. Are we now in a position to claim that we have a science of diagrams-that is, a science which takes the nature of diagrams and their use as the central phenom­ena of interest? If we have a science of diagrams it is certainly constituted from multiple disciplines, including cognitive science, psychology, artificial intelligence, logic, mathematics, and others.

If there is a science of diagrams, then like other sciences there is an appli­cations, or engineering, discipline that exists alongside the science. Applica­tions and engineering provide tests of the theories and principles discovered by the science and extend the scope of the phenomena to be studied by gen­erating new uses of diagrams, new media for presenting diagrams, or novel classes of diagram. This applications and engineering side of the science of di­agrams also comprises multiple disciplines, including education, architecture, computer science, mathematics, human-computer interaction, knowledge ac­quisition, graphic design, engineering, history of science, statistics, medicine, biology, and others.

The chapters of this book reflect this diversity of interests in the nature and uses of diagrams and the synthesis of results presented at three recent events on diagrammatic representation and reasoning: the American Associ­ation for Artificial Intelligence Fall Symposium on Diagrammatic Reasoning held at MIT in November 1997; the Thinking with Diagrams Workshop held at the University of Wales, Aberystywth, in August 1998, and the American Association for Artificial Intelligence Fall Symposium on Formalizing Rea­soning with Visual and Diagrammatic Representations, held in Orlando in October 1998. Without these events, this book would not have come about, and therefore we would like to thank the American Association for Aritifi­cial Intelligence and the Engineering and Physical Sciences Research Council (UK) for generous funding in supporting these meetings.

Michael Anderson Bernd Meyer Patrick Olivier

April 2001

Contents

Part I. Views of Diagrams

1. Diagrams in the Mind? . . ... . .. ...... .... . . .. .. . . ... . . . . . . 7

2. Knowing About Diagrams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 29

3. A Meta-Taxonomy for Diagram Research ................. 47

4. Sketches as Mental Reifications of Theoretical Scientific Treatment ...................................... 65

5. The Fundamental Design Variables of Diagramming ...... 85

Part II. Cognitive Aspects of Diagrams

6. Psychological Perspectives on Diagrams and their Users .. 109

7. Combining Semantic and Cognitive Accounts of Diagrams ................................................. 125

8. Tactile Maps and a Test of the Conjoint Retention Hypothesis ............................................... 141

9. Spatial Abilities in Problem Solving in Kinematics ....... 155

10. Graph Comprehension: The Role of Format, Content and Individual Differences ................................ 173

11. Graphs in Print .......................................... 187

12. The Role of Representation and Working Memory in Diagrammatic Reasoning and Decision Making ........... 207

13. Mechanical Reasoning about Gear-and-belt Diagrams: Do Eye-movements Predict Performance? ................ 223

VIII Contents

14. How do Designers Shift their Focus of Attention in their Own Sketches? ...................................... 241

Part III. Formal Aspects of Diagrammatic Reasoning

15. Diagrammatic Evaluation of Visual Mathematical Notations ................................................. 261

16. A Topological Framework for Modelling Diagrammatic Reasoning Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

17. Multiple Readings of Peirce's Alpha Graphs .............. 297

18. On Automating Diagrammatic Proofs of Arithmetic Arguments ............................................... 315

19. On the Practical Semantics of Mathematical Diagrams ... 339

20. EnE Sentences and Local Extent in Diagrams ............. 357

21. Implementing Euler/Venn Reasoning Systems ............ 371

22. Visual Spatial Query Languages: A Semantics Using Description Logic ......................................... 387

Part IV. Applications of Diagrammatic Reasoning

23. Intelligent Diagrammatic Interfaces: State of the Art ..... 411

24. Towards Diagram Processing: A Diagrammatic Information System ....................................... 431

25. Using Diagrams to Understand Diagrams: A Case-Based Approach to Diagrammatic Reasoning .... 447

26. Interpreting the Engineer's Sketch: A Picture is Worth a Thousand Constraints ........................... 467

27. Diagramming Research Designs ........................... 485

28. How to Build a (Quite General) Linguistic Diagram Editor .................................................... 505

29. AVOW Diagrams: A Novel Representational System for Understanding Electricity ................................ 521

Contents IX

30. AsbruView: Capturing Complex, Time-Oriented Plans - Beyond Flow Charts .............................. 535

31. Acting with Diagrams: How to Plan Strategies in Two Case Studies ............................................. 551

32. Specifying Diagram Languages by Means of Hypergraph Grammars ................................... 565

Contributors

Herman J. Ader hj.ader.biostat@med.vu.nl Vrije Universiteit, Clinical Epidemiology and Biostatistics, Van der Boe­chorststraat 7, 1081BT Amsterdam, The Netherlands.

Gerard Allwein gtall@cs.indiana.edu Indiana University, Department of Computer Science, Lindley Hall 215, Bloomington, IN 47405, USA.

Michael Anderson anderson@cis.fordham.edu Fordham University, Department of Computer and Information Sciences, 441 East Fordham Road, Bronx, NY 10458, USA.

Daniela M. Bailer-Jones daniela. bailer-jones@uni-bonn.de University of Bonn, Department of Philosophy, LFB I, Am Hof 1,53113 Bonn, Germany.

Sidney C. Bailin sbailin@waves.kevol.com Knowledge Evolution Inc., 1050 17th Street NW, Suite 520, Washington DC, 20036, USA.

Dave Barker-Plummer dbp@csli.stanford.edu Stanford University, Center for the Study of Language and Information, Ven­tura Hall, Stanford, California, 94305-4115, USA.

Alan F. Blackwell Alan.Blackwell@cl.cam.ac.uk University of Cambridge, Computer Laboratory, Cambridge, UK.

Mark Blades m.blades@sheffield.ac.uk University of Sheffield, Psychology Department, Western Bank, Sheffield S10 2TP, UK.

Alan Bundy A.Bundy@ed.ac.uk University of Edinburgh, IRR, Division of Informatics, 80 South Bridge, Edinburgh, EHI IHN, UK.

XII Contributors

Jo Calder J.Calder@ed.ac.uk University of Edinburgh, Division of Informatics, ICCS and LTG, Uni­versity of Edinburgh, Division of Informatics, 2 Buccleuch Place, Edin­burgh EH9 9LW, UK.

Peter C-H. Cheng peter.cheng@nottingham.ac.uk University of Nottingham, ESRC Centre for Research in Development, In­struction and Thaining, School of Psychology, University Park, Nottingham NG7 2RD, UK.

Nathalie Cousin-Rittemard rittemar@maths.univ-rennesl.fr Rennes 1 University, IRMAR, Equipe de Mecanique, Campus BEAULIEU, Universite de Rennes 1, CS 74205 35042 Rennes Cedex, France.

Yuri Engelhardt yuriengelhardt@yahoo.com University of Amsterdam, Department of Computational Linguistics, Palm­gracht 35,1015 HK Amsterdam, The Netherlands.

Dale E. Fish fish@engr.uconn.edu University of Connecticut, Department of Computer Science and Engineer­ing, School of Engineering, 261 Glenbrook Road, Storrs, CT 06269, USA.

Norman Foo norman@cse.unsw.edu.au University of New South Wales, Department of Computer Science and Engi­neering, Sydney NSW 2052, Australia.

Jean-Louis Giavitto giavitto@lamLuniv-evry.fr CNRS, LaMI umr 8042, Universite d'Evry Val d'Essonne, Boulevard F. Mit­terrand, 91025 Evry Cedex, France.

Ian Green LGreen@ed.ac.uk University of Edinburgh, IRR, Division of Informatics, 80 South Bridge, Edinburgh, EHI IHN, UK.

Corin Gurr C.Gurr@ed.ac.uk University of Edinburgh, Division of Informatics, 2 BuccIeuch Place, Edin­burgh EH8 9LW, UK.

Volker Haarslev haarslev@informatik.uni-hamburg.de University of Hamburg, Computer Science Department, Vogt-Kolln-Str. 30, 22527 Hamburg, Germany.

Mary Hegarty hegarty@psych.ucsb.edu University of California, Santa Barbara, Psychology Department, Santa Bar­bara, CA 93106-9660, USA.

Contributors XIII

Mateja Jamnik M.Jamnik@cs.bham.ac.uk University of Birmingham, School of Computer Science, Birmingham B 15 2TT, UK.

Peter Johnson pete@mimir.demon.co.uk University of Newcastle, The Sowerby Center for Primary Health Care Infor­matics, Newcastle upon Tyne, NE4 6BE, UK.

Robert Kosara rkosara@ifs.tuwien.ac.at Vienna University of Technology, Institute of Software Technology, Fa­voritenstraBe 9-11/E 188, 1040 Vienna, Austria.

Maria Kozhevnikov mkozhevn@fas.harvard.edu Harvard University, Graduate School of Education, Longfellow Hall, Appian Way, Cambridge MA 02138, USA.

Ellen Levy levy@psych.stanford.edu Stanford University, Department of Psychology, Jordan Hall, Stanford Uni­versity, Stanford, CA 94305-2130, USA.

Michael Lewis ml@sis.pitt.edu University of Pittsburgh, Department of Information Science and Telecom­munications, 135 North Bellefield Avenue, Pittsburgh, PA 15260, USA.

Robert K. Lindsay lindsay@umich.edu University of Michigan, Mental Health Research Institute, 205 Zina Pitcher Place, Ann Arbor, Michigan 48109, USA.

Nadine Lucas Nadine.Lucas@info.unicaen.fr Caen University, Groupe de Recherche en Informatique, Image, Instrumen­tation de Caen (GREYC), Campus II Universit de Caen BP 5186 F-14032 CAEN CEDEX, France.

Kim Marriott marriott@cs.monash.edu.au Monash University, School of Computer Science and Software Engineering, PO Box 26, Clayton Campus, Victoria 3800, Australia.

Richard Mayer mayer@psych.ucsb.edu University of California, Santa Barbara, Psychology Department, Santa Bar­bara, CA 93106-9660, USA.

Robert McCartney robert@cse.uconn.edu University of Connecticut, Department of Computer Science and Engineer­ing, School of Engineering, 261 Glenbrook Road, Storrs, CT 06269, USA.

Bernd Meyer bernd.meyer@acm.org Monash University, School of Computer Science and Software Engineering, PO Box 26, Clayton Campus, Victoria 3800, Australia.

XIV Contributors

Silvia Miksch silvia@ifs.tuwien.ac.at Vienna University of Technology, Institute of Software Technology, Favoriten­straJ3e 9-11/E 188, 1040 Vienna, Austria.

Mark Minas Mark.Minas@informatik.uni-erlangen.de Universitat Erlangen-Niirnberg, Lehrstuhl fUr Programmiersprachen, Martens­str. 3,91058 Erlangen, Germany.

Ralf Moller moeller@informatik.uni-hamburg.de University of Hamburg, Computer Science Department, Vogt-Kolln-Str. 30, 22527 Hamburg, Germany.

Clive Richards C.Richards@coventry.ac.uk Coventry University, Coventry School of Art and Design, Priory Street, Coventry CVl 5FB, UK.

Leonid G. Rozenblit leonid.rozenblit@yale.edu Yale University, Department of Psychology, 2 Hillhouse Ave., New Haven, CT 06520-8205, USA.

Diane Schiano At the time of writing the author was affiliated with Interval Research Cor­poration, Palo Alto, CA, USA.

Priti Shah priti@umich.edu University of Michigan, Department of Psychology, 525 East University, Ann Arbor, MI 48109-1109, USA.

Yuval Shahar shahar@smLstanford.edu Stanford University, Stanford Medical Informatics, MSOB X-215, 251 Cam­pus Drive, Stanford, CA 94 305 - 5479, USA.

Sun-J 00 Shin shin.3@nd.edu University of Notre Dame, Department of Philosophy, 336 O'Shaughnessy, Notre Dame, IN 46556, USA.

Aaron Sloman A.Sloman@cs.bham.ac.uk The University of Birmingham, School of Computer Science, B 15 2TT, Birm­ingham, UK.

Christopher Spencer c.p.spencer@sheffield.ac.uk University of Sheffield, Psychology Department, Western Bank, Sheffield SlO 2TP, UK.

Michael Spivey spivey@cornell.edu Cornell University, Department of Psychology, Ithaca, NY 14853, USA.

Contributors XV

Thomas F. Stahovich stahov@andrew.cmu.edu Carnegie Mellon University, Mechanical Engineering Department, 415 Scaife Hall,Pittsburgh, PA 15213, USA.

Masaki Suwa suwa@sccs.chukyo-u.ac.jp Chukyo University, School of Computer and Cognitive Sciences, 101 Toko­date, Kaizu-chou, Toyota, Aichi 470-0393, Japan.

Nik Swoboda swoboda@cs.indiana.edu Indiana University, Computer Science Department, Lindley Hall, 150 S. Wood­lawn Ave., Bloomington IN 47405-7104, USA.

Jozsef A. Toth jtoth@ida.org Institute for Defense Analyses, 1801 North Beauregard Street, Alexandria, VA 22311-1772, USA.

Barbara Tversky bt@psych.stanford.edu Stanford University, Department of Psychology, Building 420, Stanford, CA 94305, USA.

Simon Ungar ungar@lgu.ac.uk London Guildhall University, Psychology Department, Calcutta House, Old Castle Street, London El 7NT, UK.

Erika Valencia erika@limsiJr CNRS, LIMSI, Universite Paris-XI, BBP 133, 91403 Orsay Cedex, France.

Michael Wessel mwessel@informatik.uni-hamburg.de University of Hamburg, Computer Science Department, Vogt-K611n-Str. 30, 22527 Hamburg, Germany.

Julie Wojslawowicz jwojslaw@wam.umd.edu University of Maryland, Department of Human Development, 3304 Benjamin Building, College Park, MD 20740, USA.

Jeffrey M. Zacks jzacks@artsci.wustl.edu Washington University, Psychology Department, 1 Brookings Drive, Saint Louis, MO 63130, USA.