Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files ›...
Transcript of Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files ›...
![Page 1: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/1.jpg)
Chelsea Finn, Pieter Abbeel, Sergey Levine
Lifelong Few-Shot Learning
![Page 2: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/2.jpg)
Lifelong Learning Goal: learn many tasks, typically in sequence
Key Elements: - avoid forgetting previously learned tasks - reuse prior experience for fast learning of new tasks
Chelsea Finn, UC Berkeley
![Page 3: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/3.jpg)
Lifelong Learning Goal: learn many tasks, typically in sequence
Key Elements: - avoid forgetting previously learned tasks - reuse prior experience for fast learning of new tasks
Typical Approach: Run optimization algorithm (e.g. policy gradient)a.k.a. continual fine-tuning
Chelsea Finn, UC Berkeley
![Page 4: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/4.jpg)
Fine-tuning for lifelong learning
Suppose, we have trained a model on N tasks.
Now, we want to learn a new task with little data
Why is fine-tuning not effective?
None of these are satisfactory :(
Chelsea Finn, UC Berkeley
…1 2 3 N
Option 0: collect a lot more data Option 1: Finetune on (N+1)’th task Option 2: Replay old task data, including new data
![Page 5: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/5.jpg)
Chelsea Finn, UC Berkeley
Lifelong Meta-Learningcontinuously learn to learn sequence of tasks
[D1 for learning, D2 for meta-learning]
non-stationary task distribution
Sample a task
Sample N datapoints from
Result: Can reuse prior learning experience to quickly learn new tasks
meta-learning
, split into datasets D1 and D2
Update model so that learning using D1 enables effective performance on D2
How can we do better? Learn how to learn efficiently
![Page 6: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/6.jpg)
Key idea: Train over many tasks, to learn parameter vector θ that transfers
Fine-tuning:task
pretrained parameters
Our method:
[test-time]
Background: Model-Agnostic Meta-Learning fine-tuning from ImageNet-trained features (Deng et al. ’09, Donahue et al. ’14)
+ simple, works well, same learning rule- no ImageNet for non-vision domains, won’t extend to few-shot setting
Chelsea Finn, UC Berkeley
![Page 7: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/7.jpg)
One slice of lifelong learning:
Preliminary Investigation into Lifelong Meta-Learning
Chelsea Finn, UC Berkeley
- given experience with N tasks - evaluate ability to efficiently learn (N+1)’th
task from different distribution
Illustrative Regression Example
Adaptation in Reinforcement Learning
![Page 8: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/8.jpg)
RegressionQuickly learn a new real-valued function
meta-learning
training data evaluation data
… …
Chelsea Finn, UC Berkeley
Tasks: sine function with varied amplitude & frequency
x y
new task
![Page 9: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/9.jpg)
Comparison: task-conditioning
Chelsea Finn, UC Berkeley
x
y
Condition model on task information
Train with standard supervised learning on training tasks. Fine-tune on new task.
![Page 10: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/10.jpg)
In-distribution task performance
Chelsea Finn, UC Berkeley
10 datapoints used for all fine-tuning steps
![Page 11: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/11.jpg)
Out-of-distribution task performance
Chelsea Finn, UC Berkeley10 datapoints used for all
fine-tuning steps
![Page 12: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/12.jpg)
Task representation out-of-distribution
Chelsea Finn, UC Berkeley
10 datapoints used for all fine-tuning steps
the importance of task representation
![Page 13: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/13.jpg)
Meta-Learning: Half-Cheetah
Chelsea Finn, UC Berkeley
Training task distribution:
Quickly adapt behavior to run at a goal velocity
sa
Comparison to task-conditioning
Evaluate on:
![Page 14: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/14.jpg)
Half-Cheetah in-distribution performance
Chelsea Finn, UC Berkeley
20 trajectories used for each fine-tuning step
![Page 15: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/15.jpg)
Half-Cheetah out-of-distribution performance
Chelsea Finn, UC Berkeley
20 trajectories used for each fine-tuning step
![Page 16: Lifelong Few-Shot Learning - Artificial Intelligence › ~cbfinn › _files › icml2017_llworkshop.pdf · Preliminary Investigation into Lifelong Meta-Learning Chelsea Finn, UC Berkeley-given](https://reader033.fdocuments.us/reader033/viewer/2022060501/5f1b5524379fff0d8544ea4a/html5/thumbnails/16.jpg)
Conclusion:- lifelong meta-learning enables faster learning of new tasks
Chelsea Finn, UC Berkeley
CollaboratorsSergey Levine Pieter Abbeel