Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and...
Transcript of Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and...
![Page 1: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/1.jpg)
Perception och Maskininärning iInteraktiva Autonoma System
Michael Felsberg
Institutionen för systemteknikLinköpings universitet
![Page 2: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/2.jpg)
Vad är WASP?
• Wallenberg Autonomous Systems Program
• Sveriges största individuella forskningsprogram genom tiderna
• http://wasp-sweden.se/
• Initialt 6 olika projekt
• Fokus: perceptionsprojektet
• Inom perception: datorseende
![Page 3: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/3.jpg)
Vad är datorseende?
• delområde av datalogin
• "seende” datorer• automatiskt bearbetar digitala bilder
• extraherar specifika typer av information ur bilder beroende på den uppgift som ska lösas, e.g.• konstruktion av 3D objektmodeller från 2D bilddata
• igenkänning av 3D eller 2D objekt i bilder
• styrning av robotar och fordon med hjälp av kameror
• starkt tvärvetenskapligt• biologiskt seende, neurovetenskap & visuell perception
• matematik, numerisk analys & datorgrafik
• datorteknik, mjukvaruteknik & AI
![Page 4: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/4.jpg)
Vilka skolämnen är relevanta för datorseende?
• matematik• geometri
• algebra
• sannolikhetsteori och statistik
• analys
• kombinatorik
• fysik
• teknik
• programmering
• dator- och kommunikationsteknik
![Page 5: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/5.jpg)
Var är svårt med bilder?
![Page 6: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/6.jpg)
Vad ser du?
![Page 7: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/7.jpg)
Facit
![Page 8: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/8.jpg)
Tillbaka till WASP … Autonomasystem är mer än självkörandebilar ...
http://autodato.com/wp-content/gallery/benz-patent-motorwagen/Benz-Patent-Motorwagen-DRP-37435-02.jpg
![Page 9: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/9.jpg)
Case Study: 125 Anniversary Bertha Mannheim - Pforzheim
http://autodato.com/wp-content/gallery/benz-patent-motorwagen/Benz-Patent-Motorwagen-DRP-37435-02.jpg
![Page 10: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/10.jpg)
Autonomous Driving is Easy?
• According to some experts: Google and Apple have failed with their autonomous car projects [New York Times 9/9, Bloomberg 9/12]
• Tesla’s fatal accidentcould have been avoided by Lidar instead of computervision-based perception (Mobileye)? [New York Times 7/12, ArsTechnica 9/16]https://static01.nyt.com/newsgraphics/2016/07/01/tesla-accident/10c347b26e2d2fb936647182b6b92923cb914729/crash-720.png
![Page 11: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/11.jpg)
Integrating Perception, Learning and Verification in Interactive Autonomous Systems
The project will study perception methods based on fusion of multi-modal sensory information in combination with learning, and formal verification of autonomous systems.
Researchers:
• Danica Kragic (project coordinator), KTH
• Michael Felsberg, Linköping University
• Laura Kovacs/Bengt Lennartsson, Chalmers
• Alexandre Proutiere, KTH
• Kalle Åström, Lund University
![Page 12: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/12.jpg)
Some students in the project
DanielKTH
ShahbazKTH/ABB
FredrikChalmers
MiaKTH
GustavLIU
SamuelChalmers/Autoliv
DavidLTH
BertilLIU/Saab Dynamics
JohanKTH/ABB
![Page 13: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/13.jpg)
![Page 14: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/14.jpg)
Modes of Perception
• multitude of sensor modalities• selection problem – what sensor to focus on?• fusion problem – how to combine them?
• human-compatible sensing• visualization – acquisition of relevant (visual) data• interaction – shared percept space• adaptation to environment – most systems will be
placed in environments shaped for humans
• semantic gap / symbol grounding• fused sensor readings need to be mapped to
semantic models• top-down feedback modulates perception
![Page 15: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/15.jpg)
Ex: Sensors and Semantics
![Page 16: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/16.jpg)
Ex: Fusion and Feedback
![Page 17: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/17.jpg)
Ex: Interactionand Visualization
![Page 18: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/18.jpg)
Methods
• machine learning• deep learning – lots of data
• online learning – from demonstration
• optimization• offline – hard problems
• on-the-fly – adaptation
• latent probabilistic models• enforce consistency of parts
• avoid local minima
![Page 19: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/19.jpg)
AlexNet [Krizhevsky et al. 2012]
![Page 20: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/20.jpg)
Deep Learning Revolution
• ImageNet Large Scale Visual Recognition Challenge [Deng et al. 2009]
• Today: • more than 14 million images
• more than 10 million images annotated
• more than 1 million images with bounding box
• Classification error rate 2011: 25%
• Using CNNs in 2012: 16% (!)
![Page 21: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/21.jpg)
14 million images
Visual object
• detection
• tracking
• recognition
Danelljan et al. ICCV 2015, CVPR 2016
Visual features learned from ImageNet
![Page 22: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/22.jpg)
Learning of association to actions
Öfjäll et al., IV 2016
![Page 23: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/23.jpg)
Probabilistic geometric models
Danelljan et al., CVPR 2016
![Page 24: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/24.jpg)
LARA
![Page 25: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/25.jpg)
Relevanta Program vid LiU
• Y – teknisk fysik
• D – datateknik
• U – mjukvaruteknik
• M – medieteknik
• MT – medicinsk teknik
• TB – teknisk biologi
• IT – informationsteknologi
• I – industriell ekonomi
![Page 26: Perception och Maskininärning i Interaktiva Autonoma ... · Integrating Perception, Learning and Verification in Interactive Autonomous Systems The project will study perception](https://reader030.fdocuments.us/reader030/viewer/2022040721/5e2d80e7a4a91957dc55f1f4/html5/thumbnails/26.jpg)
Frågor?
• http://users.isy.liu.se/cvl/mfe/
• http://www.cvl.isy.liu.se/
• https://beta.liu.se/employees/micfe03