ECE 599/692 –Deep Learning Lecture 19 –Beyond...
-
Upload
truongkien -
Category
Documents
-
view
217 -
download
0
Transcript of ECE 599/692 –Deep Learning Lecture 19 –Beyond...
ECE 599/692 – Deep Learning
Lecture 19 – Beyond BP and CNNHairong Qi, Gonzalez Family ProfessorElectrical Engineering and Computer ScienceUniversity of Tennessee, Knoxvillehttp://www.eecs.utk.edu/faculty/qiEmail: [email protected]
1
Schedule
• Nov. 9: Beyond BP and CNN• Nov. 14: Depth vs. Breadth
– What’s beyond ReLU?– What’s beyond cross entropy?
• Nov. 16: Projects discussion– What’s beyond leaderboard?
• Nov. 21: How to design a new network structure?• Nov. 28, 30, Dec. 5
– Final project presentation (26 presentations)
2
From Big Data to Artificial Intelligence• What is big data?• The more the better?• Can’t live without data?• AlphaGo Zero vs. AlphaGo
– No human input vs. Human knowledge• Hinton’s two new papers on capsule networks (2017)
– “I think the way we’re doing computer vision is just wrong… It works better than anything else at present but that doesn’t mean it’s right.”
– Capsule (small groups of crude virtual neurons) networks for tracking different parts of an object
– Abandon BP– “…humans should encode as little knowledge as possible into AI
software, and instead make them figure things out for themselves from scratch…”
– “The future of AI is determined by those graduate students who seriously doubt all what I have said.”
3https://deepmind.com/blog/alphago-zero-learning-scratch/
A bit history• 1943 (McCulloch and Pitts):• 1957 - 1962 (Rosenblatt):
– From Mark I Perceptron to the Tobermory Perceptron to Perceptron Computer Simulations– Multilayer perceptron with fixed threshold
• 1969 (Minsky and Papert):• The dark age: 70’s ~25 years• 1986 (Rumelhart, Hinton, McClelland): BP• 1989 (LeCun et al.): CNN (LeNet)• Another ~20 years• 2006 (Hinton et al.): DL• 2012 (Krizhevsky, Sutskever, Hinton): AlexNet• 2014 (Goodfellow, Benjo, et al.): GAN
4
• W.S. McCulloch, W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” The Bulletin of Mathematical Biophysics, 5(4):115-133, December 1943.
• F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, Spartan Books, 1962.• Minsky, S. Papert, Perceptrons: An Introduction to Computational Geometry, 1969.• D.E. Rumelhart, G.E. Hinton, R.J. Williams, “Learning representations by back-propagating errors,” Nature, 323(9):533-536, October 1986.
(BP)• Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, “Backpropagation applied to handwritten zip code
recognition,” Neural Computation, 1(4):541-551, 1989. (LeNet).• G.E. Hinton, S. Osindero, Y. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, 18:1527-1554, 2006. (DL)• G.E. Hinton, R.R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, 313(5786):504-507, 2006 (DL)• A. Krizhevsky, I. Sutskever, G.E. Hinton, “ImageNet classification with deep convolutional neural networks,” Advances in Neural Information
Processing Systems, pages 1097-1105, 2012. (AlexNet)• I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, “Generative adversarial networks,” NIPS,
2014.
A bit of history - revisited• 1956-1976
– 1956, The Dartmouth Summer Research Project on Artificial Intelligence, organized by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon
– The rise of symbolic methods, systems focused on limited domains, deductive vs. inductive systems
– 1973, the Lighthill report by James Lighthill, “Artificial Intelligence: A General Survey” -automata, robotics, neural network
– 1976, the AI Winter• 1976-2006
– 1986, BP algorithm– ~1995, The Fifth Generation Computer
• 2006-???– 2006, Hinton (U. of Toronto), Bingio (U. of Montreal, LeCun (NYU)– 2012, ImageNet by Fei-Fei Li (2010-2017) and AlexNet
5
We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College ... The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.
https://en.wikipedia.org/wiki/Dartmouth_workshophttps://en.wikipedia.org/wiki/Lighthill_report
Turing awardees in AI
– 1969, Marvin Minsky– 1971, John McCarthy– 1975, Allen Newell and Herbert A. Simon– 1994, Edward Feigenbaum and Raj Reddy– 2010, Leslie G. Valiant– 2011, Judea Pearl
7
https://en.wikipedia.org/wiki/Turing_Award
Artificial Intelligence• The three branches
– Logic-based (Symbolic method)– Network-based– Behavior-based (Self adaptive and evolution)
• Artificial intelligence vs. Human intelligence– Logical– Linguistic– Spatial– Musical– Kinesthetic– Intra-personal– Inter-personal– Naturalist– Graphics
8
Government involvement
• China’s new generation artificial intelligence– Big data intelligence– Swarm intelligence– Multimedia (Cross-domain) intelligence (speech, image, text,
natural language)– Human-machine coordination– Autonomous vehicle
• Applications– Computer vision– Speech recognition– Natural language processing– Human-machine interface– Robotics
9
Andrew Parker’s Light Switch Theory• Andrew Parker, In The Blink of An Eye: How
Vision Sparked The Big Bang Of Evolution, Basic Books, 2004.
• The Cambrian explosion: the explosion of life forms (550 million years ago)
• The theory:– It was the development of vision in primitive animals
that caused the explosion.
10
https://www.amazon.com/Blink-Eye-Vision-Sparked-Evolution/dp/0465054382
Visual intelligence: Beyond ImageNet• Fei-Fei Li’s talk at CNCC 2017• Object recognition (ImageNet – Single Object
Recognition)– 0.28 (2010) à 0.26 (2011) à 0.16 (2012) à 0.12 (2013) à 0.07
(2014) à 0.036 (2015) à 0.03 (2016) à 0.023 (2017)• Beyond object recognition à Rich scene gist
– The Visual Genome Dataset – Visual relationship à Semantic scene retrieval à Scene graph
generation• Beyond scene gist à Vision + Language & Reasoning
– The CLEVR Dataset– Image captioning à Dense Captioning à Paragraph generation
• https://www.leiphone.com/news/201710/CvdrhzTO0cndEArJ.html
11
What’s after ImageNet?
• MS COCO (Microsoft Common Objects in Context)– 4 Tasks: Detection challenge, Instance segmentation, Human
keypoint challenge, Stuff segmentation– Winners (Detection): MSRA (2015), Google (2016), Face++
(2017)– http://cocodataset.org/#home
• Places (MIT and CMU)– 3 Tasks: Scene parsing, Instance segmentation, and Boundary
detection– Winners (2017):
– Scene parsing: CAS– Instance segmentation: Face++
– http://placeschallenge.csail.mit.edu/results_challenge.html
12
Reference
• Fei-Fei Li, Visual Intelligence Beyond ImageNet, CNCC 2017,https://www.leiphone.com/news/201710/CvdrhzTO0cndEArJ.html
• Xiangyang Shen, Microsoft, CNCC 2017• Wen Gao, From Big Data to Artificial Intelligence,
JDDiscovery, 2017,https://www.toutiao.com/a6485167794265522701/?tt_from=weixin&utm_campaign=client_share&app=news_article&utm_source=weixin&iid=9368489672&utm_medium=toutiao_android&wxshare_count=1
14