Intro to Machine LearningRecitation 1: Intro, KNN, K-Means

1

● Name: Yossi Adi and Felix Kreuk ● Number of assignments: 6

○ Programming (python) and theoretical exercises ○ Checked with the “Submit” system.

● % assignments, % test ● Office hours: ?? ● Piazza: ? ● Project ?

Administrative Stuff

2

Machine Learning is Everywhere

3

INTERNET & CLOUD

Image Classification Speech Recognition Language Translation Language Processing

Sentiment Analysis Recommendation

MEDICINE & BIOLOGY

Cancer Cell Detection Diabetic Grading Drug Discovery

MEDIA & ENTERTAINMENT

Video Captioning Video Search

Real Time Translation

SECURITY & DEFENCES

Face Detection Video Surveillance Satellite Imagery

AUTONOMOUS MACHINES

Pedestrian Detection Lane Tracking

Recognize Traffic Sign

Machine Learning

Herbert Alexander Simon:

“Learning is any process by which a system improves performance from experience.”

“Machine Learning is concerned with computer programs that automatically improve their performance through experience. “

4

Why use Machine Learning?

● Develop systems that can automatically adapt themselves ○ Personalised systems (facebook feed, google search)

● Discover new knowledge from large databases (cluster users) ● Market basket analysis (e.g. diapers and beer) ● Mimic human abilities in mundane tasks

○ Recognising handwritten characters ○ Speech recognition ○ Image annotation

● Develop systems that are too difficult to construct manually (if & else)

5

Why Now?

● Flood of data ● Computational power ● Research Boost (academic and industry)

6

What is Learning?

Learning = Improving with experience at some task

Improve over task - T

With respect to performance measure - P

Based on experience - E

7

Example - image classification

{cat, dog, …, car}

*Adapted from Stanford cs231n course presentations. 8

Example - image classification

{cat, dog, …, car}

*Adapted from Stanford cs231n course presentations. 9

Challenges: viewpoint

All pixels change when the camera moves!

*Adapted from Stanford cs231n course presentations. 10

Challenges: illumination

*Adapted from Stanford cs231n course presentations. 11

Challenges: Deformation

*Adapted from Stanford cs231n course presentations. 12

Challenges: Occlusion

*Adapted from Stanford cs231n course presentations. 13

Challenges: Background Clutter

*Adapted from Stanford cs231n course presentations. 14

Challenges: Intraclass variation

*Adapted from Stanford cs231n course presentations. 15

Learning Schemes

● Supervised Learning ● Examples are labeled

● Unsupervised ● No labels

● Semi-Supervised ● Partially labeled

● Reinforcement Learning ● Reward

16

Outputs

● Classification: one over K classes ● Object classification, text classification, etc.

● Regression: predict real valued vector ● Stock values, height prediction, etc.

● Structured: predict complex outputs with structure ● Parsing trees, ASR, translation, etc.

17

Separators and Generalisation

Which one would you choose?

18

Separators and Generalisation

19

Separators and Generalisation

Linear case - two mistakes

Non Linear case - zero mistakes

20

Terminology

● Target function: ● In classification: ● In regression:

● Hypothesis: A proposed function , an approximation of . ● Hypothesis space: The space of all hypotheses that can, in principle, be

output by the learning algorithm. ● Model: A function . The output of our learning algorithm. ● Loss: an evaluation metric:

Y =�1, 2, · · · , k

<latexit sha1_base64="y4mvVQD7XNlEXlh4MjctA2YSyXc=">AAACB3icbVBNS8NAEN34WetX1KMHF4vgoZSkCOpBKHrxWMHYShPKZrNtl252w+5GKKFHL/4VLx5UvPoXvPlv3KY5aOuDgcd7M8zMCxNGlXacb2thcWl5ZbW0Vl7f2Nzatnd275RIJSYeFkzIdogUYZQTT1PNSDuRBMUhI61weDXxWw9EKir4rR4lJIhRn9MexUgbqWsf3MML6Ie072duFdar0MeR0KoKh7k47toVp+bkgPPELUgFFGh27S8/EjiNCdeYIaU6rpPoIENSU8zIuOyniiQID1GfdAzlKCYqyPJHxvDIKBHsCWmKa5irvycyFCs1ikPTGSM9ULPeRPzP66S6dxZklCepJhxPF/VSBrWAk1RgRCXBmo0MQVhScyvEAyQR1ia7sgnBnX15nnj12nnNuTmpNC6LNEpgHxyCY+CCU9AA16AJPIDBI3gGr+DNerJerHfrY9q6YBUze+APrM8fyGeXgQ==</latexit><latexit sha1_base64="y4mvVQD7XNlEXlh4MjctA2YSyXc=">AAACB3icbVBNS8NAEN34WetX1KMHF4vgoZSkCOpBKHrxWMHYShPKZrNtl252w+5GKKFHL/4VLx5UvPoXvPlv3KY5aOuDgcd7M8zMCxNGlXacb2thcWl5ZbW0Vl7f2Nzatnd275RIJSYeFkzIdogUYZQTT1PNSDuRBMUhI61weDXxWw9EKir4rR4lJIhRn9MexUgbqWsf3MML6Ie072duFdar0MeR0KoKh7k47toVp+bkgPPELUgFFGh27S8/EjiNCdeYIaU6rpPoIENSU8zIuOyniiQID1GfdAzlKCYqyPJHxvDIKBHsCWmKa5irvycyFCs1ikPTGSM9ULPeRPzP66S6dxZklCepJhxPF/VSBrWAk1RgRCXBmo0MQVhScyvEAyQR1ia7sgnBnX15nnj12nnNuTmpNC6LNEpgHxyCY+CCU9AA16AJPIDBI3gGr+DNerJerHfrY9q6YBUze+APrM8fyGeXgQ==</latexit><latexit sha1_base64="y4mvVQD7XNlEXlh4MjctA2YSyXc=">AAACB3icbVBNS8NAEN34WetX1KMHF4vgoZSkCOpBKHrxWMHYShPKZrNtl252w+5GKKFHL/4VLx5UvPoXvPlv3KY5aOuDgcd7M8zMCxNGlXacb2thcWl5ZbW0Vl7f2Nzatnd275RIJSYeFkzIdogUYZQTT1PNSDuRBMUhI61weDXxWw9EKir4rR4lJIhRn9MexUgbqWsf3MML6Ie072duFdar0MeR0KoKh7k47toVp+bkgPPELUgFFGh27S8/EjiNCdeYIaU6rpPoIENSU8zIuOyniiQID1GfdAzlKCYqyPJHxvDIKBHsCWmKa5irvycyFCs1ikPTGSM9ULPeRPzP66S6dxZklCepJhxPF/VSBrWAk1RgRCXBmo0MQVhScyvEAyQR1ia7sgnBnX15nnj12nnNuTmpNC6LNEpgHxyCY+CCU9AA16AJPIDBI3gGr+DNerJerHfrY9q6YBUze+APrM8fyGeXgQ==</latexit>

Y = Rn<latexit sha1_base64="9LkLj5qkmtpvoPesQVkZbXaqbmI=">AAAB+HicbVBNS8NAFHypX7V+RT16WSyCp5KIoB6EohePVYyttLFstpt26WYTdjeFEvpPvHhQ8epP8ea/cdPmoK0DC8PMe7zZCRLOlHacb6u0tLyyulZer2xsbm3v2Lt7DypOJaEeiXksWwFWlDNBPc00p61EUhwFnDaD4XXuN0dUKhaLez1OqB/hvmAhI1gbqWvbj+gSdSKsB0GQ3U2ejFR1as4UaJG4BalCgUbX/ur0YpJGVGjCsVJt10m0n2GpGeF0UumkiiaYDHGftg0VOKLKz6bJJ+jIKD0UxtI8odFU/b2R4UipcRSYyTyjmvdy8T+vnerw3M+YSFJNBZkdClOOdIzyGlCPSUo0HxuCiWQmKyIDLDHRpqyKKcGd//Ii8U5qFzXn9rRavyraKMMBHMIxuHAGdbiBBnhAYATP8ApvVma9WO/Wx2y0ZBU7+/AH1ucPCZeS1A==</latexit><latexit sha1_base64="9LkLj5qkmtpvoPesQVkZbXaqbmI=">AAAB+HicbVBNS8NAFHypX7V+RT16WSyCp5KIoB6EohePVYyttLFstpt26WYTdjeFEvpPvHhQ8epP8ea/cdPmoK0DC8PMe7zZCRLOlHacb6u0tLyyulZer2xsbm3v2Lt7DypOJaEeiXksWwFWlDNBPc00p61EUhwFnDaD4XXuN0dUKhaLez1OqB/hvmAhI1gbqWvbj+gSdSKsB0GQ3U2ejFR1as4UaJG4BalCgUbX/ur0YpJGVGjCsVJt10m0n2GpGeF0UumkiiaYDHGftg0VOKLKz6bJJ+jIKD0UxtI8odFU/b2R4UipcRSYyTyjmvdy8T+vnerw3M+YSFJNBZkdClOOdIzyGlCPSUo0HxuCiWQmKyIDLDHRpqyKKcGd//Ii8U5qFzXn9rRavyraKMMBHMIxuHAGdbiBBnhAYATP8ApvVma9WO/Wx2y0ZBU7+/AH1ucPCZeS1A==</latexit><latexit sha1_base64="9LkLj5qkmtpvoPesQVkZbXaqbmI=">AAAB+HicbVBNS8NAFHypX7V+RT16WSyCp5KIoB6EohePVYyttLFstpt26WYTdjeFEvpPvHhQ8epP8ea/cdPmoK0DC8PMe7zZCRLOlHacb6u0tLyyulZer2xsbm3v2Lt7DypOJaEeiXksWwFWlDNBPc00p61EUhwFnDaD4XXuN0dUKhaLez1OqB/hvmAhI1gbqWvbj+gSdSKsB0GQ3U2ejFR1as4UaJG4BalCgUbX/ur0YpJGVGjCsVJt10m0n2GpGeF0UumkiiaYDHGftg0VOKLKz6bJJ+jIKD0UxtI8odFU/b2R4UipcRSYyTyjmvdy8T+vnerw3M+YSFJNBZkdClOOdIzyGlCPSUo0HxuCiWQmKyIDLDHRpqyKKcGd//Ii8U5qFzXn9rRavyraKMMBHMIxuHAGdbiBBnhAYATP8ApvVma9WO/Wx2y0ZBU7+/AH1ucPCZeS1A==</latexit>

h<latexit sha1_base64="lt6MdrejMYgtNBMJ4G/0FERlp4M=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaw16l6tbcGcgy8QpShQKNXuWr209YFqM0TFCtO56bmiCnynAmcFLuZhpTykZ0gB1LJY1RB/ns0Ak5tUqfRImyJQ2Zqb8nchprPY5D2xlTM9SL3lT8z+tkJroKci7TzKBk80VRJohJyPRr0ucKmRFjSyhT3N5K2JAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA8HYy/</latexit><latexit sha1_base64="lt6MdrejMYgtNBMJ4G/0FERlp4M=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaw16l6tbcGcgy8QpShQKNXuWr209YFqM0TFCtO56bmiCnynAmcFLuZhpTykZ0gB1LJY1RB/ns0Ak5tUqfRImyJQ2Zqb8nchprPY5D2xlTM9SL3lT8z+tkJroKci7TzKBk80VRJohJyPRr0ucKmRFjSyhT3N5K2JAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA8HYy/</latexit><latexit sha1_base64="lt6MdrejMYgtNBMJ4G/0FERlp4M=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaw16l6tbcGcgy8QpShQKNXuWr209YFqM0TFCtO56bmiCnynAmcFLuZhpTykZ0gB1LJY1RB/ns0Ak5tUqfRImyJQ2Zqb8nchprPY5D2xlTM9SL3lT8z+tkJroKci7TzKBk80VRJohJyPRr0ucKmRFjSyhT3N5K2JAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA8HYy/</latexit>

f<latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit>

` : R⇥ R ! R+<latexit sha1_base64="CwMi5ZjwYsMTbYGBxpegtchb5sc=">AAACG3icbVDLSsNAFL2pr1pfUZduBosgCCVRwceq6MZlFWMLTSyT6aQdOnkwMxFK6Ie48VfcuFBxJbjwb5y0XfThgYEz597Lvef4CWdSWdavUVhYXFpeKa6W1tY3NrfM7Z0HGaeCUIfEPBYNH0vKWUQdxRSnjURQHPqc1v3edV6vP1EhWRzdq35CvRB3IhYwgpWWWuaJSzm/RG6IVdf3s7sBchULqZxW4onv41HLLFsVawg0T+wxKcMYtZb57bZjkoY0UoRjKZu2lSgvw0Ixwumg5KaSJpj0cIc2NY2wPsDLhuYG6EArbRTEQr9IoaE6OZHhUMp+6OvO/EY5W8vF/2rNVAXnXsaiJFU0IqNFQcqRdpsnhdpMUKJ4XxNMBNO3ItLFAhOl8yzpEOxZy/PEOa5cVKzb03L1apxGEfZgHw7BhjOowg3UwAECz/AK7/BhvBhvxqfxNWotGOOZXZiC8fMH26yhcQ==</latexit><latexit sha1_base64="CwMi5ZjwYsMTbYGBxpegtchb5sc=">AAACG3icbVDLSsNAFL2pr1pfUZduBosgCCVRwceq6MZlFWMLTSyT6aQdOnkwMxFK6Ie48VfcuFBxJbjwb5y0XfThgYEz597Lvef4CWdSWdavUVhYXFpeKa6W1tY3NrfM7Z0HGaeCUIfEPBYNH0vKWUQdxRSnjURQHPqc1v3edV6vP1EhWRzdq35CvRB3IhYwgpWWWuaJSzm/RG6IVdf3s7sBchULqZxW4onv41HLLFsVawg0T+wxKcMYtZb57bZjkoY0UoRjKZu2lSgvw0Ixwumg5KaSJpj0cIc2NY2wPsDLhuYG6EArbRTEQr9IoaE6OZHhUMp+6OvO/EY5W8vF/2rNVAXnXsaiJFU0IqNFQcqRdpsnhdpMUKJ4XxNMBNO3ItLFAhOl8yzpEOxZy/PEOa5cVKzb03L1apxGEfZgHw7BhjOowg3UwAECz/AK7/BhvBhvxqfxNWotGOOZXZiC8fMH26yhcQ==</latexit><latexit sha1_base64="CwMi5ZjwYsMTbYGBxpegtchb5sc=">AAACG3icbVDLSsNAFL2pr1pfUZduBosgCCVRwceq6MZlFWMLTSyT6aQdOnkwMxFK6Ie48VfcuFBxJbjwb5y0XfThgYEz597Lvef4CWdSWdavUVhYXFpeKa6W1tY3NrfM7Z0HGaeCUIfEPBYNH0vKWUQdxRSnjURQHPqc1v3edV6vP1EhWRzdq35CvRB3IhYwgpWWWuaJSzm/RG6IVdf3s7sBchULqZxW4onv41HLLFsVawg0T+wxKcMYtZb57bZjkoY0UoRjKZu2lSgvw0Ixwumg5KaSJpj0cIc2NY2wPsDLhuYG6EArbRTEQr9IoaE6OZHhUMp+6OvO/EY5W8vF/2rNVAXnXsaiJFU0IqNFQcqRdpsnhdpMUKJ4XxNMBNO3ItLFAhOl8yzpEOxZy/PEOa5cVKzb03L1apxGEfZgHw7BhjOowg3UwAECz/AK7/BhvBhvxqfxNWotGOOZXZiC8fMH26yhcQ==</latexit>

f<latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit><latexit sha1_base64="pqxdpHFpGkgp3bPBHwon3SFgUdA=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaUa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwA5F4y9</latexit>

21

t : X ! Y<latexit sha1_base64="DJMpgH1mE0VRzKAYzxwVQroWcWo=">AAAB8HicbVDLSgMxFL1TX7W+qi7dBIvgqkxF8LEqunFZwbHVdiiZNNOGZjJDckcoQ//CjQsVt36OO//GtJ2Fth4IHM65l9xzgkQKg6777RSWlldW14rrpY3Nre2d8u7evYlTzbjHYhnrVkANl0JxDwVK3ko0p1EgeTMYXk/85hPXRsTqDkcJ9yPaVyIUjKKVHvGStEgHY/LQLVfcqjsFWSS1nFQgR6Nb/ur0YpZGXCGT1Jh2zU3Qz6hGwSQflzqp4QllQ9rnbUsVjbjxs+nFY3JklR4JY22fQjJVf29kNDJmFAV2MqI4MPPeRPzPa6cYnvuZUEmKXLHZR2EqiY04iU96QnOGcmQJZVrYWwkbUE0Z2pJKtoTafORF4p1UL6ru7WmlfpW3UYQDOIRjqMEZ1OEGGuABAwXP8ApvjnFenHfnYzZacPKdffgD5/MHfquPrw==</latexit><latexit sha1_base64="DJMpgH1mE0VRzKAYzxwVQroWcWo=">AAAB8HicbVDLSgMxFL1TX7W+qi7dBIvgqkxF8LEqunFZwbHVdiiZNNOGZjJDckcoQ//CjQsVt36OO//GtJ2Fth4IHM65l9xzgkQKg6777RSWlldW14rrpY3Nre2d8u7evYlTzbjHYhnrVkANl0JxDwVK3ko0p1EgeTMYXk/85hPXRsTqDkcJ9yPaVyIUjKKVHvGStEgHY/LQLVfcqjsFWSS1nFQgR6Nb/ur0YpZGXCGT1Jh2zU3Qz6hGwSQflzqp4QllQ9rnbUsVjbjxs+nFY3JklR4JY22fQjJVf29kNDJmFAV2MqI4MPPeRPzPa6cYnvuZUEmKXLHZR2EqiY04iU96QnOGcmQJZVrYWwkbUE0Z2pJKtoTafORF4p1UL6ru7WmlfpW3UYQDOIRjqMEZ1OEGGuABAwXP8ApvjnFenHfnYzZacPKdffgD5/MHfquPrw==</latexit><latexit sha1_base64="DJMpgH1mE0VRzKAYzxwVQroWcWo=">AAAB8HicbVDLSgMxFL1TX7W+qi7dBIvgqkxF8LEqunFZwbHVdiiZNNOGZjJDckcoQ//CjQsVt36OO//GtJ2Fth4IHM65l9xzgkQKg6777RSWlldW14rrpY3Nre2d8u7evYlTzbjHYhnrVkANl0JxDwVK3ko0p1EgeTMYXk/85hPXRsTqDkcJ9yPaVyIUjKKVHvGStEgHY/LQLVfcqjsFWSS1nFQgR6Nb/ur0YpZGXCGT1Jh2zU3Qz6hGwSQflzqp4QllQ9rnbUsVjbjxs+nFY3JklR4JY22fQjJVf29kNDJmFAV2MqI4MPPeRPzPa6cYnvuZUEmKXLHZR2EqiY04iU96QnOGcmQJZVrYWwkbUE0Z2pJKtoTafORF4p1UL6ru7WmlfpW3UYQDOIRjqMEZ1OEGGuABAwXP8ApvjnFenHfnYzZacPKdffgD5/MHfquPrw==</latexit>

Terminology: Empirical Risk Minimization

Goal: to minimize:

S = {(x1,y1), . . . , (xm,ym)}

Rho is unknown, so we use a set of examples:

E(x,y)⇠⇢ [`(y, y(x))]

We would like to minimize:

w⇤ = argminw

1

m

mX

i=1

`(yi, yw(xi)) +�

2⌦(w)

22

Terminology

● Training set ● Used to train the classifier

● Validation set ● Used to tune hyper-parameters

● Test set ● Not seen during training, used to evaluate our classifier

● Epoch: one “pass” over the training set

23

Training Process

● Training loop: ● Modify predictor function according to the training set ● Evaluate the loss on the validation set

● Evaluate the loss on test set ● Output predictor function

24

(which performs “well” on the test set according to the loss function)

Terminology: overfitting and underfitting

25# Iterations

Loss

K Nearest Neighbours

26

K Nearest Neighbours (KNN)

● Our first machine learning algorithm ● One of the simplest algorithms ● Algorithm:

● Given a test example x ● Define a metric d:

● Non-negative ● Identity ● Symmetry ● Triangle inequality

● Find the k closest examples to x according to d ● Predict the majority class

27

K Nearest Neighbours (KNN) - Properties

● Supervised ● Lazy ● Need to specify K ● Non-Linear ● Different metrics will output different boundaries ● New examples directly change the classifier ● Every dimension contributes equally ● Sensitive to outliers

28

K Nearest Neighbours (KNN) - Demo

● Demo

29

K Means

30

K Means

● The k-means algorithm is an iterative method for clustering a set of N points in to K clusters

● Algorithm: ● Define a metric d ● Initialise K centroids ● Repeat until convergence:

● Assign each point to the closest centroid according to d

● Update each centroid to be the mean of the points in its group

31

K Means - Properties● Unsupervised ● Need to specify K ● Different centroids and metrics will output different boundaries ● Assumptions: spherical, same size and density ● Sensitive to outliers

32

K Means - Demo

● Demo

33

Questions?

34