1. Concurrent Objects, Agents, and HPCAki YonezawaAdvanced Institute of Computational Science (AICS) RIKEN, Kobe

2. My Background,Current and in the past Currently: in charge of the research and operation divisionsof the national supercomputing center, AICS. personally have a keen interest in large scaleagent-based modeling and simulation. In the past: have been engaged in research onconcurrent objects and actors,reflection, mobile objects, 3. Synopsis AICS Concurrent Objects Modeling with mobile/concurrent objects Modeling with Agents, my understanding (Reflective concurrent objects) HPC with concurrent objects (ultra) Large scale simulation with Agents Evacuation simulation Traffic simulation 4. Advanced Institute for Computational Science (AICS)Has been established as the internationalCOE for computational sciences and computer science.Since 2010. (currently ~150 staffs, including 65 researchers) Missions are: Operating the K computer efficiently for users of wide researchareas as well as of industry Carrying out the leading edge research of computational science, promoting strong collaborations between computer scientists and computational scientists Plan and develop Japans strategy for computational science, including defining the path to Exascale computing 5. Supercomputer K, a 10 peta-flops machine5 6. A Glimpse of K Computer Number of CPUs (nodes) > 80 KLocal file systemNumber of cores > 640K> 11PB Total memory capacity > 1.7PB1TB/s350GB/s Global file systems 30PBSPARC64 VIIIfx CPU designed and made by Fujitsu. 7. Location of AICS & K Computer Kobe K Computer MaeStation Tokyo423km (263miles) west from TokyoResearch Computer Building buildingComputer room 50m*60m=3,000m2Chillers Electric power up to 30MWSubstation SupplyGas-turbine cogeneration 5MW*2Water Cooling System 8. HPC ? In Japan, HPC should be read asHighly Political Computing! What about in the States or Europe? Some episode in Japan 9. Televised Congress Hearingon the plan of K supercomputer (Nov. 2009) Chairwoman asked: Any reason to be No.1? whynot the second place is enough?. She represented the intention of Ministry of Finance. Hearing verdict : 100% budget reduction A big campaign by scientists including 7 Nobel laureatesWhole budget was recovered in March 2010!!! 10. 8 Notable Large-scale Applications on K Multi-scale Multi-physics Simulation for Heart Disease/Disorder (simulation of whole human heart) Tsunami Simulation and Disaster Mitigation Development of Virus Molecular Science Based on 10M All-Atom Simulation Drug Design Applications Next-generation Fluid Design System Based on Direct Computation of Turbulence Reduction of Weather Prediction Period using Global Cloud Resolving Model (NICAM) Simulation of Supernova Explosions based on Neutrino Heating Prediction of electro-functions of nano-structures 11. Prof. ImamuraProf. Sugita(Univ. Tokyo) 12. UT Heart SimulatorHisada & Sugiura Finite Element Model 1000M degrees of freedon !! takes two days for one beat UT Heart is a multi-scale and multi-physics simulator, starting with physiological models describing the dynamics of various ion currents and sarcomeric proteins. UT Heart leads to beating of the heart and ejection of blood, which produces the blood pressure and electrocardiogram (ECG) 13. Tsunami simulations using the tsunami-coupled equation of motion in 3DMaeda and Furumura (2011) Pure and Applied Geophysics - under reviewNavier-Stokes equu uup 2u 2u 2u +u+v +w= + 2 + 2 + 2 + g xtx yzx x y z vvvvp 2v 2v 2v + u + v + w = + 2 + 2 + 2 + g ytxyzy x y z wwwwp 2w 2w 2w +u+v+w= + 2 + 2 + 2 + g ztxyzz xyz [Present] Resolution: 1kmCPU Time: 2 hour(ES 64 node)[Expected] Resolution:0.25 kmCPU Time: < 10 min(K Computer) Its possible to issue Tsunami Warning 14. Introducing my early career 1974, joined Carl Hewitts group, MIT 1977, finished PhD 1985, ABCL (concurrent OO language),at pre-1st ECOOP 1986, 1st OOPSLA, ABCL/1 paperObject-Oriented Concurrent Programming (MIT Press) 1988, 3rd OOPSLA, ABCL/R paper 1900, 5th OOPSLA/ECOOP PC chairABCL: an Concurrent Object-Oriented System (MIT Press) 15. My idea of Concurrent Objects (1975)For modeling and programming, Concurrent Object = Encapsulated(Stateful Object + thread) ObjectConcurrentthread/cpu >Object Asynchronous message passing among concurrent objects Actor Approach-C.Hewitt and H.Baker: Laws for communicating Parallel Processes, IFIP1977-G.Agha: A Model for Concurrent Computation in Distributed Systems, MIT Press 1987 16. Concurrent objectsEach CO has a globally unique identity/name, may have a protected, yet updatable local memory, has a set of procedures that manipulates the memory, receives a message that activates one of the procedures, has a single FIFO queue for arrived messages, and has autonomous thread(s) of control. - In each CO, memory-updating procedures are activated one at a time with the message arrival order.- The contents of the memory of a CO, which is the local state of the CO. 17. my Modeling of Real Worldwith Concurrent ObjectsModelingdomain Representing concurrent 18. Real World Interacting Agents Concurrent Objects & Message PassingWEBEntities,people,machines & theirinteractions 19. Example:Modeling a small Post Office (1977)Mail Box incounter-sectionPostOffice doorclients 20. Modeling Post Office in COs Post Office Building the door door CO Counter with clerks counter CO Mail Box mailbox CO Customers customer COs not messages! 21. Modeling Movement of CustomersTwo ways:1. a customer object is transmitted in a message2. a customer object moves by itself (autonomously)Object (or its code) migrates autonomously!! 22. Mobility increases COs power Increases COs modeling power intellectual power Entering a new environment > increase accessibility ofpotentially available information and knowledge If COs are equipped with appropriate means for access to surrounding environments, their power increases. 23. JavaGO: our Mobile Concurrent Objects Language (1998)andits application systems 24. Realization of Self-migration/mobile Concurrent Objects JavaGo LanguageAdding (Goto: L) statement to the languageT.Sekiguchi & L1 H.Masuhara (go-to L1)L2 Language and its implementation that enables programmers to write concurrent objects moving around network nodes (1999) 25. JavaGoX: transformation for transparent thread migration Javas support for mobile objects dynamic class loading serialization of object states JavaGoX enables efficient migration of running objects by inserting code for saving/restoring execution stack into heap implemented as a bytecode transformation system Cf. Sakamoto, Sekiguchi, Yonezawa: Bytecode Transformation for PortableThread Migration in Java, in ASA/MA00, 2000 for details. 26. An Application of Mobile Concurrent Objects Automatic Software Install/Update System 27. Installing/updating software in PCs on LAN is painful There are many PCson a single LAN. Users of the PCs do notinstall/update software They are busy or non-geek 28. Installing/updating software in PCson LAN is painful There are many PCson a single LAN. Users of the PCs do notinstall/update software They are busy or non-geek 29. Features of Mobile Automatic Installation System Mobile software (called mobile installers)migrates from administrative PCs to client PCsand installs/updates software automatically Administrative costs can be largely reduced Number of client PCs does not matter Totally unattended installation/update is possible Encryption and digital signature are supported Risks of illegally-copied software and virus programsare addressed 30. Overview of Our System Administrative PC Mobile Installer Target Software Encryption Decryption Key Client PCs Digital Signature 31. Overview of Our System Administrative PC Mobile Installer Target Software Encryption Decryption Key Client PCs Digital Signature 32. Overview of Our System Administrative PC Mobile Installer Target Software Encryption Decryption Key Client PCs Digital Signature 33. Overview of Our System Administrative PC Mobile Installer Target Software Encryption Decryption Key Client PCs Digital Signature 34. Reflection in COs and its Application[ABCL/R language in OOPSLA88]Reflection makes COs more intelligent, and closer to Agents? 35. Computational Reflection Computation about Oneself: Introspection & Self Modification MCPs: meta- M[S]interpreters, meta-objectsSA reflective system S can reason base-levelabout or act upon itself via thecausally-connected selfrepresentation M[S]. Representation in a Reflective Tower (Smith, 1982)Pioneers:S is reified as R[S] within the meta-circular processor3-Lisp (B. C. Smith, 1982) MCP1. MCP1 is also reified in MCP2, and so forth.3-KRS (P. Maes, 1986)Reflective behaviors are realized as normal operations in the meta-levels (MCPs). 36. Why Reflection in COs? Customizable meta-objects can introduce new language features Modification of semantics of objects. Application/Domain-Specific Language Features Highly Dynamic Behaviors On-the-Fly Code Acquisition Dynamically Adaptable Behaviors Object Migration Time control of distributed simulation A New Dimension of Modularization Descriptions of Non-Functional Requirements/constraints using Meta-Levels (Meta-Objects) This idea later evolved into AOP 37. ABCL/ROne Concurrent Object > One Meta-Object Each object has its own meta- the meta-object of Omethods execution object that solely governs itsengine computation. The meta-(evaluator) object reifies the structure of the object. reified message The meta-object also has itsinternal meta-object. So the reflectivestate tower exists for each object. message queue The base-level object can send messages to its meta- reflective object: this realize reflective Omessage behaviors. incoming messagean object (base-level)Watanabe & Yonezawa, OOPSLA 88 38. Now, a Bit of Agents 39. What is an Agent (Macal&North06) An agent is identifiable, a discrete individual with a set of characteristics and rules governing its behaviors and decision-making capability An agent is autonomous and self-directed, An agent can function independently in its environment. An agent is situated (and mobile), living in an environment with which it interacts along with other agents. has protocols for interaction with other agents, such as for communication, and the capability to respond to the environment An agent may be goal-directed, Has goals to achieve (not necessarily objectives to maximize) with respect to its behaviors. 40. An Agent at a Glance (Macal&North06) 41. Agents have more abilities than COs ( + goals to achieve) + Automatic update of location information andother information + can modify its self (~a la reflection in CO) + sensor facilities for environment information sight/vision field force + actuator facilities 42. Applications of COs in large scales- Lindens Second Life / Online Virtual World- In the context of HPC, Charm++ and NAMD 43. Back to Original Motivation of COs Real WorldInteracting Agents Concurrent Objects & Message PassingWEB Entities,people, machines & their interactionsLindens Second Life is a natural outcome from the motivation of COs: 44. Why COs for Second Life The idea of concurrent objects has beenadopted in Second Life (Linden Lab.) because: COs can directly simulating virtual world objects, which enableseasy modeling andeasy/safe concurrent programming! 45. COs in Second Life Jim Purbrick, Mark Lentczner,Second Life: The Worlds Biggest ProgrammingEnvironment, Invited Talk at OOPSLA2007, said: Objects and avatars cooperate and coordinateeach other by exchanging messages.COness! each object or avatar is programmed to Have its own state, Have its own method to respond to an incoming message, Have different responses to different states, and Have its own thread. About 2 millions of objects are programmed in SecondLife and they are in action. 46. Application of JavaGoXstransformation method to Second Life our JavaGoX [ASA/MA00] a bytecode transformation system that enables migration of running objects on JVM Second Life employs similar transformationfor their new Mono-based script Sims//COsexecution engine for migrating objects between regions a region is managedby one server regionImage from EVOLVING NEMO in New World Notes at Second Life Blog(http://secondlife.blogs.com/nwn/2005/06/evolving_nemo.html) 47. Concurrent Object Languages for HPCABCL/f (U. Tokyo) Charm++ (U. Illinois) 48. ABCL/f implementationon Fujitsu AP1000 (PPoPP 93) One of the earliest attempts to implementconcurrent objects as a high performance parallelprogramming language on distributed memorymassively parallel 512 processors (sparc) Kenjiro Taura, Uni. Tokyo Information Processing Society of Japan 49. Charm++ CO-based language system forparallel/supercomputers developed by Sanjay Kalesgroup (U. Illinois) since 1993. Used for development of NAMD (MolecularDynamics), OpenAtomquantum chemistry,ChaNCAGalaxy generation Operational for various supercomputers to date Full Jaguar PF Cray XT5 (224,076 cores) K computer (700,000 cores), Blue Gene/Q (65,536 cores) Has a framework for dynamic load balancing bymoving Cos from node to node 50. What is Charm++? Platform independent libraryfor writing parallel programs C++, C, and Fortran are supported Based on the notion of concurrent objects Charm++ programs consist of: Parallel running objects (named chare), and Communication among themvia asynchronous method invocation 51. Overview of Programming Model:Logical View Chare (Concurrent Objects) Asynchronous Method Invocation 52. Overview of Programming Model: Physical View Chares migrate btwn processors ProcessorProcessorProcessorterconnectInterconnect Interconnect Intercon 53. Overview of Charm++ ImplementationProcessor Processor Chare AChare B Charm++ Charm++Scheduler Runtime RuntimeProxy of Interconnectchare B Message Queue 54. Overview of Charm++ ImplementationProcessor Processor Invoke a Chare A method ofChare B chare B Charm++ Charm++Scheduler Runtime RuntimeProxy of Interconnectchare B Message Queue 55. Overview of Charm++ ImplementationProcessor Processor Invoke a Chare A method ofChare B chare B Invoke the local proxy of chare B Charm++ Charm++Scheduler Runtime RuntimeProxy of Interconnectchare B Message Queue 56. Overview of Charm++ ImplementationProcessorProcessor Chare A Chare B The proxy creates a message for Charm++ invoking chare B Charm++Scheduler RuntimeRuntime MessageProxy ofInterconnectchare BMessage Queue 57. Overview of Charm++ ImplementationProcessorProcessor Chare A Chare B Runtime transmits the message via interconnects Charm++ Charm++ Scheduler Runtime Runtime MessageProxy ofInterconnectchare BMessage Queue 58. Overview of Charm++ ImplementationProcessor Processor Chare AChare BScheduler dispatchesthe messageto invoke chare B Charm++ Charm++Scheduler Runtime Runtime MessageProxy of Interconnectchare B Message Queue 59. Applications of Charm++ Molecular dynamics: NAMD http://charm.cs.uiuc.edu/research/moldyn Quantum molecular dynamics: OpenAtom http://charm.cs.uiuc.edu/research/qmmm/ Rocket simluation: RocStar http://www.csar.uiuc.edu/rocstar/index.html Cosmology simulation: ChaNGa http://www- hpcc.astro.washington.edu/tools/changa.html etc. 60. NAMD(Nano-scale Molecular Dynamics Simulator) 61. NAMD Nano scale molecular dynamics simulator forsupercomputers Major application of Charm++ Developed jointly by two groups in Illinois Univ.Prof. Sanjay Kale (Computer Science)Prof. Schulten (Theoretical Biology) Received Gordon Bell Prize in 2006 Running on Juager PF (224,076 cores)for 100M atoms proteins 62. This still from a Quicktime movie represents a view of the drugburied in the binding pocket of the A/H1N1 neuraminidase protein. Molecular Structure 63. Analysis of Molecular Structure of Swine Flu by NAMDShowed: Structural changes of A/H1N1 protein induceanti-tamiful effects. 64. Experimental Results: NAMD on K computer Kamada(2012) 20ApoA1 (92,224 atoms) 18ATPase (327,506 atoms) 16STMV (1,066,628 atoms) 14CPU time per step [ms] 12 10 8 6 4 2 08 cores /node 0 512 1024 1536 2048 Number of nodes Benchmarks were obtained from http://www.ks.uiuc.edu/Research/namd/utilities/ 7 workers per node 65. Now, Back to Agents 66. What is an Agent? (Macal&North06) An agent is identifiable, a discrete individual with a set of characteristics and rules governing its behaviors and decision-making capability An agent is autonomous and self-directed, An agent can function independently in its environment. An agent is situated, living in an environment with which it interacts along with other agents. has protocols for interaction with other agents, such as for communication, and the capability to respond to the environment An agent may be goal-directed, Has goals to achieve (not necessarily objectives to maximize) with respect to its behaviors. 67. An Agent at a Glance (Macal&North06) 68. Agents have more abilities than COs ( + goals to achieve) + Automatic update of location information andother information + can modify its self (~a la reflection in CO) + sensor facilities for environment information sight/vision field force + actuator facilities 69. Ultra Large Scale Agent-BasedSimulation in Japan Evacuation from Tsunami Attack Whole city traffic simulation (Kyoto) 70. Mitigation of Tsuname Damages - A TOTAL Simulation APPROACH -1. An earthquake breaks out2. Propagation of strong vibration3. Simulation of the Tsunami Broadcast early warning4. Simulation of flooding cities5. Simulation of building damages6. Simulation of human evaluation Evacuation guidance 71. Earthquake Vibration Simulation Using available geological stratum dataProf. FurumuraUniv. Tokyo 71 72. Simulation of an Expected Tsunami in Japan 73. Flooding in a City 74. Buildings in part of Tokyo GIS tile ID: 09ld171 (near Shinjuku) No. of buildings : 14,000 SRA model : Fiber element model output size: 12 GB Strong ground motion data from 1995 Kobe earthquake 75. Evacuation Drill: before building damages assumedShelter Building 76. Human Evacuation after EarthquakeShelter Building 77. Agent-BasedHuman Evacuation Simulation Goal: Total evacuation time to be reduced Find best evacuation plans Parameters: # of people, evacuation routes, Modeling/predicting human behavior By monitoring human behavior, using physical driversimulator By measuring human actions 78. Measuring peoples movementin an evacuation drill 79. Large scale evacuation simulations Understanding the dynamics of mass evacuation isimportant to find strategies to save as many as possible, in future urban developments, etc. Based on Multi-Agent Simulations (MAS) Suitable for simulating heterogeneous and complex phenomena Mass evacuation is contagious in nature and have cascade effect Whole affected area has to be simulated as a single domain HPC enhancements are necessary to simulate millions in acity like Tokyo Monte-Carlo simulations are necessary Poor scalability has been reported in literature 80. Multi Agent Simulation Agents in evacuation simulationAgentThought agent mimic people interacting with Abilityeach other and the environmentSpeedAbilityDirectionVisionThoughtSpeed Agents See the environment Passing Path Uses its Think to make a decision Move according to the decision See() numerous kinds of agent to modelThink()Move()heterogeneous crowds Different levels of abilitiessuch as see, think and act Different information about the surrounding residents, visitors, officials, etc. Different levels of responsibilities police, fire fighters, community leaders,etc. 81. Simulation: 2,000,000 agentsin a part of Kochi City Using K omputer 82. Environment of evacuation simulation Space in which human or human organization acts Automatically generated from GIS data Space/Environment is modeled as a grid Grid cells are classified as occupied and un-occupied GIS data of the real environment Raster model for MAS in vector format 83. Agents visual perception and moves Sophisticated vision: Scan the environment in high resolution (at 0.5o intervals) Identify paths, obstacles, neighbors andslow moving groups within 30m Moves: avoiding the obstacles such as slow moving groups of agents choose the path closest to the direction of destination avoid collision with individuals60m Direction of destination Walking direction 84. Visual Sight an official searching people astray - 85. Scalability of evacuation simulation module12 K Ideal gradient11 lilac (6 core Xeon E5620)10 Log(runtime,2)9 Number of agents =8 500,0007 Dotted lines show the Area 19.2 km2 in Kochi city ideal gradient, not the ideal run time6 34 56 7 89 10Log(CPU cores,2)C++ in K seems to be slow May be STL is not well optimized 86. strategies for reducing pre-evacuation time Tsunami evacuation is highly time critical Anticipating Tokai-Tonankai-Nankai tsunami may arrive in 10 minutes Evacuation time = pre-evacuation time + travel time Pre-evacuation time: waking, change clothes, putting stuffs together, Pre-evacuation time may be as large as 30 minutes Appoint officials: deliver the warning personally People tends to neglect the warnings from announcements, TV, radio, etc. To personally deliver warning and give path to nearest evacuation center Need to study the effectiveness with differentinfluential power 87. Official agents personally advisingresidents to evacuate early 88. Findings Preliminary study on pre-evacuation timereduction Delivering the warning personally could reduce evacuation time significantly Employing officials + community leaders could save more lives High parallel scalability is attained Can handle several millions of agents on thousands of CPU cores 89. Traffic Flow Simulation with Agents S. Kato, G. Yamamoto,.. (IBM Res. Report RTO 759, 2008) 90. Simulator Architecture MASMITS Agent Space Simulation Space Driver Agent Message Driver ModelDriver Agent MessageDriver Model 91. Agent SpaceSimulation Space Collection of Agents Agent:messages current states of traffic Model of a car and the alignment of roads to driver behaviordriver agents current speed and positionsof vehicles,Messages fromS-space to A-space distance sbetween cars, NextSpeedMessage the curvature and gradient EnterSpeedMessage of roads on which individualvehicles are running. NextRoadMessage EnterRoadMessage Keeps states of roads and agents ExitRoadMessage StartMessage 92. Simulation Results (single processor)Kyoto City Road Network simulated the traffic of the Kyoto city. the road network of Kyoto, 32,654 links and 22,782 nodes. 894,802 pairs of origin and destination are assigned. 894,802 vehicle agents start their origins towarddestination at the same time. time step size is 0.1. used the IBM xSeries 335 (2.8GHz processor, 4GBmemory). 93. Traffic Simulation for a Kyoto festival day 94. Nation-wide Traffic Simulationon a Supercomputer Nation-wide Road network: 993,731 intersections, 2,552,160 roads The agent simulator written in X10. X10 language (IBM) for HPC X10: PGAS OO language with places and async. activities No MPI, or No OpenMP!! Tsubame2: 1.19peta flops @ Titech 95. Thanks for your attention!