PPT slides - MACHINE PERCEPTION LABORATORY

Computer Science:A Whirlwind Tour

Jacob Whitehill ([email protected])Machine Perception Laboratory,

http://mplab.ucsd.eduUCSD

Goals of This Talk

1.Introduce you to the science of computers & computation.

2.Give a super-fast tour of various sub-areas of computer science.

3.Tell you a little about our lab’s research.

4.Show some demos.

Finding the science in Computer

Science

Computer Science: Misconceptions

•Computer science is related to, but distinct from, the following fields:

•Information Technology (IT).

•Computer & network management

•Computer programming


•Information Technology (IT):

•The study, development, and management of information systems - computer & software systems that manage large amounts of information.

•Example: design an efficient database for Verizon to handle billing information for 100 million customers.


•Computer & network management - set up and maintain computers & computer networks

•Example: figure out why the Internet is not accessible from a particular school classroom, and fix the problem.


•Computer programming - create software for a computer to execute to perform a useful task.

•Example: write software to automatically download music by your favorite artist on iTunes.

My definition of Computer Science• My definition: the development of new

techniques using computers & computation to solve problems that were fundamentally not solvable before.

• Contrast with IT: there are no fundamental challenges in setting up a billing system for Verizon that have not yet been solved - it’s clear that it’s achievable.

• Contrast with programming: though programming is by no means trivial, it is usually clear at the onset that the program can be written.

My definition of Computer Science

• Computer programming represents the means by which (empirical) computer science research is conducted.

• Analogous to conducting an assay, using a microscope, staining DNA, etc.

Computer Science•Computer science is (for the most

part) an engineering science:

•We have a task we’d like to accomplish.

•We don’t know if it’s solvable, let alone how to solve it.

•We (try to ) find computational methods to accomplish our task.

Computer Science

•Contrast with the natural sciences (e.g., biology, chemistry, physics, mathematics):

•We are interested in discovering how the world works, without necessarily accomplishing a concrete task.

Sample of Various Computer Science Research Problems

Computer Vision

• Google Earth - How do you “stitch together” satellite photographs taken above the Earth into a navigable 3-D “world” (Google Earth)?

Graphics

• Pixar - How do you animate a graphical video character (e.g., Shrek) so that it looks alive?

Cryptography and Security

• How do you encrypt a message so that only the “intended persons” can read it and eavesdroppers cannot?

• How do you prevent people from copying movies & music illegally?

Networking

• How can you transmit data reliably and at high-speed using ordinary power-line cables?

(this is important in developing countries without telephone infrastructure)

Machine Learning

• Can computers learn to recognize the same kinds of patterns in nature that humans can?

• Example: where are the faces in the video?

Bioinformatics and Computational

Biology

• How can we analyze human DNA sequences to determine the risk factors of certain diseases?

Complexity Theory

• How can we use quantum computers to solve mathematical problems more quickly than with conventional machines?

Computer Science:Sub-areas (many)

• These are all the ones I can think of off-hand:• Graphics

VisionMachine learningDatabasesCompilersNetworkingOperating systemsDistributed systemsSoftware engineeringRoboticsNumerical computingHuman-computer interactionGraphicsComputer architectureSecurityCryptographyAlgorithmsImage & video processingBioinformaticsComplexity theoryComputational geometry

• By the way - some of these areas also fit into Electrical Engineering (EE).

Some of these are mostly empirical, some are mostly theoretical, and some are both.

Theoretical vs. Empirical Research

• Theoretical research - use mathematics/logical reasoning to prove what you are saying is definitely true.

• Empirical research - research through observation - run many experiments to show that what you are saying is probably true.

Theoretical vs. Empirical Research

• Example:

• Theoretical proof that a particular casino gambling strategy will give you the highest possible winnings.

• Demonstration over many experiments that a particular strategy works better than others.

• Important: It’s not always easy/possible to prove something mathematically.

Algorithms - the foundation of

computer science

Algorithms

•The notion of algorithm is fundamental to all of computer science.

•An algorithm is a step-by-step procedure (“recipe”) for how to complete a particular task.

•Computers can perform tasks very quickly, but they require a very precise description of what to do.

Algorithms

• Example:

• INPUT: flour, eggs, milk, butter, baking powder, salt, chocolate chips

• PROCEDURE:

1.Beat eggs.2.Add all other ingredients.3.Stir everything until smooth.4.Drop in 5” circles on hot frying pan.5.Fry on both sides until brown.

Algorithms

• Example (continued):

• OUTPUT: chocolate chip pancakes

Algorithms

• More interesting algorithms:

• If you type a phrase into Google, what is the fastest way of finding all of the webpages (on the whole Internet!) that contain that phrase?

• How do you automatically find the faces in a digital video?

• How do you animate Shrek’s mouth to match his speech?

Algorithms

•Algorithm descriptions are typically written in a computer programming language (e.g., C, Java, Python). (This is where programming comes in.)

•Algorithm descriptions are called code.

•They are then converted (by a compiler or interpreter) into machine language (the computer’s native language, written in 0’s and 1’s).

Algorithms

• Example: counting from 1 to 100.

• PROCEDURE (in English):

1.Start with the number 1.2.Print the number.3.Add 1 to the number.4.Go back to Step (2) until we’ve reached 100.

Algorithms


• PROCEDURE (in C, a programming language):

1.int number = 1;while (number <= 100) {

printf(“%d\n”, number);number += 1;

}

Algorithms


• PROCEDURE (in PowerPC G5 machine language, binary):

1. 111111101110110111111010110011100000000000000000000000000001001000000000000000000000000000001010000000000000000000000000000000100000000000000000000000000000101000000000000000000000010000010100000000000000000000000000100001010000000000000000000000000000000100000000000000000000000000111000010111110101111101010000010000010100011101000101010110100100010101010010010011110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000

...

Algorithms•Several sub-areas of computer science

study the science of algorithms:

•Algorithm design

•Come up with new fundamental algorithms.

•Complexity & computability theory

•Study the kinds of algorithms that can be written, and how fast they are.

Computability Theory

•There are some tasks which are fundamentally impossible to solve.

•One of the most famous is the “halting problem”.

•It is possible to write a program that never ends. Consider:

1.Print out “hello”.2.Go back to step 1.

The “Halting Problem”

•It would be useful to know whether a particular computer program will ever finish.

•What we would like is the following:

•INPUT: any computer program.

•OUTPUT: either Yes or No according to whether the specified computer program will (eventually) come to an end.

The “Halting Problem”

•It is provably impossible to create such an algorithm.

•Hence, no such algorithm can ever be created, no matter how sophisticated computers ever become.

•This research result comes from the field of computability theory.

•Determining whether any computer program will terminate is incomputable.

Complexity Theory

• Some computer science tasks, even if there exists algorithms to solve them, may take prohibitively long to finish - for instance, a google-google-google years (this is a long time).

• Other algorithms may finish their work in less than a second.

• Which tasks can be solved quickly, and which take more time?

• The amount of time that an algorithm takes to finish is called the complexity of that algorithm.

Two Algorithms -Which takes longer to

finish?

•“Stable Marriage”: How can we pair n men with n women so that divorces their spouse for someone else?

•“Graph 3-Coloring”: How can we color the circles in a graph so that connected circles have different colors?

Stable Marriage Algorithm

•Suppose we have 10 men and 10 women who wish to marry each other (heterosexual).

•Each person ranks all 10 persons of the opposite gender in order of preference.

•We want to pair people so that no one will divorce their current spouse to marry someone else. Show on board.

Graph 3-Coloring•A “graph” has circles and lines.

•Some circles are connected to others by lines.

•If two circles are connected by a line, then the circles cannot have the same color.

•Can we color all the circles in the graph using just 3 different colors?

Algorithmic Complexity

•Which problem is harder? “Stable Marriage”, or “Graph 3-Coloring”?

•These algorithms may seem contrived, but they actually find many uses in computer science.


• It turns out that the problem of “Stable Marriage” has a very efficient algorithm.

•Finding marriages for, say, 1000 people will take a fraction of a second.

•Graph 3-Coloring is a hard problem - no known efficient algorithm exists (although slow ones do exist).

•For a graph with 1000 circles, finding a 3-coloring will take longer than the universe will continue to exist!


•Let n describe the “size” of the problem.

•n men/women who wish to marry.

•n circles in the graph we wish to color.

•The Stable Marriage algorithm is quadratic in n.

•The best known Graph 3-coloring algorithm is exponential in n.

Machine Learning

My Research:Machine Learning

• “Machine learning” - the study of algorithms that enable computers to “learn” the same things that humans can do easily (and not so easily).

•Machine learning is similar to pattern recognition.


•Examples:

• How to hear & understand speech.

• How to see & recognize people.

• How to determine whether an email contains a virus.

• The science involved in machine learning is coming up with ways to perform such tasks (or to improve their performance).


•My work:

•Automatic facial expression recognition (and other kinds of face processing).

•Automated teaching systems.

Automatic Facial Expression Recognition

•INPUT: an image/video containing faces.

•OUTPUT: the locations of all faces in the image, along with their facial expressions.

•PROCEDURE: unknown (but we’re working on it)


•Let’s define “facial expression” more clearly:

•One definition is the emotion - happy, sad, angry, etc.

•Another definition (ours) is the kinds of facial muscle movements that occur in the face.

Facial Muscle Movements

•There are 46 independent muscle groups in the face. Here are a few examples:

Source: http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm

Automatic Facial Expression Recognition•Why would we want to recognize facial

muscle movements?

• Psychological research - how do people respond to certain stimuli?

•Computer games - make the game responsive to how the user is feeling.

• Lie detection.

• Perceptual computer interfaces (more later).

•Remote control using the face (more later).


•The basic flow of how things work:

1.Find all the faces in the image (face detection).

2.For each face you find, determine its facial expression (expression recognition).

Face Detection

• Contrast face detection and face recognition.• Detection - where are the faces in

the image?• Recognition - whose face is that?

Face Detection• In 2001, Paul Viola and Michael Jones

(from Mitsubishi Electric Research Labs) revolutionized the field of face detection.

• Their design is now called the Viola-Jones Face Detector.

• PROCEDURE:1.Divide the image into thousands of

different square regions.2.For each square region, check if it

contains a face. (illustrate on board)

Does the region contain a face?

• We must classify each square region as either a face, or a non-face. This is performed by a classifier.

• Classifiers: examines a square region of an image, and outputs either 1 (“face”) or 0 (“non-face”).

• Must operate extremely quickly!

Classifying a region as face/non-

face•We must “train” a face classifier using many “training examples”.

•~10,000 examples of faces.

•~1,000,000,000 examples of non-faces.

•The classifier training algorithm compares the faces to the non-faces and finds the image properties which distinguish these two kinds of images. (illustrate on board)

Classifying a region as face/non-

face•The classifier training algorithm automatically finds “features” of the image that distinguish faces from non-faces.

•Once trained, the classifier can decide (very quickly) whether the square region contains a face.

Expression Recognition

•Now that we’ve found the faces, we must determine their facial expressions.

•We examine each facial muscle movement separately:

• Is the person wrinkling their nose? Yes/No.• Is the person smiling? Yes/No.• Is the person blinking? Yes/No.• ...

Expression Recognition

• To detect expressions, we use a similar approach as for face detection:

• Train a classifier to distinguish between smile/non-smile, frown/non-frown, etc.

• For each classifier, we need many examples for both the presence, and the absence, of the expression.

• The classifier examines the face and determines whether a particular expression occurred or not.

Our System•Our lab (MPLab) has developed a state-

of-the-art facial expression recognition system.

•“CERT” = Computer Expression Recognition Toolbox.

•CERT rates each expression in terms of intensity(e.g., not smiling at all ==> very smiley).

•Demo.

Unexpected Application: Art

Exhibit•We deployed CERT’s smile detector to “force” people to smile for 1.5 hours.

•Whenever the people stopped smiling, CERT would beep at them.

Unexpected Application: Art

Exhibit

QuickTime™ and aSorenson Video 3 decompressorare needed to see this picture.

Automated Teaching Systems•Facial expression recognition (and

other forms of “machine perception”) have many applications.

•In my research, I’m interested in one particular application: automated teaching systems.

•I define automated teaching systems very generally - any machine that teaches, or helps to teach, a human.

Automated Teaching Systems

•Why would we want to do this?

• There are many parts of the world in which good teachers (especially in specialized subjects) are scarce.

• The current model of education is one teacher for many students. Automated teaching systems could help reduce this imbalance.

Automated Teaching Systems• Some aspects of teaching (e.g., drilling of

key concepts) are highly repetitive and could be automated.

• (My research): Online learning has become very popular in higher education; students watch pre-recorded lectures on video streamed over the Internet. Can we make these video lectures more responsive to the individual student?

• Facial expression recognition may be useful here.

Automated Teaching Systems•“Smart Video Lecture Player” - the basic

idea:

•Some parts of a lecture are already familiar to you, or easy to understand. You can watch these faster than normal and still follow.

•Other parts are harder to understand; slow these down!

•We want to adjust the playback speed automatically using facial expression.

Smart Lecture Video Player

•Why not just control the speed manually (with the keyboard)?

•The student may be taking notes with his/her hands.

•May be more natural or convenient to use the face, similarly to how students interact (subconsciously) with their teacher.

Smart Lecture Video Player: Experiment•Question/hypothesis: Does the student’s

facial expression tell us how difficult he/she finds the lecture to be?

•Experimental procedure:

1.Each student watched a short video lecture (3 mins). He/she could adjust the speed (1-10) with the keyboard.

2.Each student’s facial expressions were recorded in real-time using the CERT software.

3.Quiz.4.Each student then watched the video again, this

time giving his/her assessment of how easy/difficult (1-10) the lecture was to understand at each moment in time.

Smart Lecture Video Player: Experiment

•Experimental procedure:

5.We employed statistical regression algorithms to convert the student’s facial expressions into:

The preferred viewing speed of the student.

The perceived difficulty of the lecture.

(Note: the true “computer science” research in this project is developing the algorithm to map from facial expression to Difficulty and Preferred Viewing Speed.)

Smart Lecture Video Player: Experiment•Question:

•How accurately can our automated computer algorithm predict Difficulty and Preferred Viewing Speed from the facial expression information?

•We already collected this data, so we can determine this quantitatively.

Smart Lecture Video Player: Experiment

Difficulty prediction:0.42 mean Pearson correlation (validation).Speed prediction: 0.29 mean Pearson correlation (validation).

Results (in English):

•Using automatic facial expression recognition, we can get an approximate sense of how Easy/Difficult the student finds the lecture to be.

•Can also infer how Fast/Slow the student wants to watch the video.

•Applications:

•Smart Video Lecture Player (as intended).

•Robotic teachers (we have one).

Smart Video Lecture Player

•Demo.

Other Work on Automated Teaching Systems at Our Lab

•The RUBI project: deploy a robot among pre-school children at UCSD’s Early Childhood Education Center.

•Children are between 18-24 months old.

•Teach them basic shapes, letters, vocabulary (English, Finnish).

RUBI in action

RUBI 1.0 RUBI 2.0

Science as a Career

Quick Blurb About Me

• Graduated from Stanford (2001); majored in computer science.

• Worked as a software engineer (two years) and college lecturer (two years) after finishing college.

• Been working at UCSD as a researcher & graduate student since 2005.

• Aim to finish my PhD in computer science by 2011.

• Hope to eventually become a college professor.

Scientific Research: My

Personal Perspective• In research, you will never stop

learning, by definition.

•Usually lots of freedom: the emphasis is on getting results.

•You will expand the frontiers of human knowledge (but usually only in small ways).

•You will become an “expert” in something.

Scientific Research: My

Personal Perspective

•Greed for money is replaced by greed for prestige.

•Research can be lonely - you will (often) spend lots of time in the lab.

•You won’t get rich (unless you start a company).

From High Schooler to Researcher•Attending a prestigious research

university (e.g., MIT, Harvard) is always helpful.

•BUT - going to a lesser known school by no means shuts you out. You just have to take more initiative to get involved in research projects during college.

•UC schools are all excellent (in certain fields) and are much cheaper (for CA residents) than private schools.

From High Schooler to Researcher• In my opinion, the #1 most important thing

to do in college is:GET TO KNOW YOUR PROFESSORS.

• Go to office hours.

• Ask them questions.

• Ask them about their research.

• Ask them how you can get involved in one of their research projects.

• Professors are the reason why famous schools are famous.

• A strong letter of recommendation from a professor is worth its weight in platinum.

The End

PPT slides - MACHINE PERCEPTION LABORATORY

Documents

Transcript of PPT slides - MACHINE PERCEPTION LABORATORY