About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that...
Transcript of About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that...
![Page 1: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/1.jpg)
About Understanding and ValuesAbout Understanding and Values
Bas R. SteunebrinkBas R. Steunebrink
Swiss AI lab IDSIA (postdoc)Swiss AI lab IDSIA (postdoc)NNAISENSE (co-founder)NNAISENSE (co-founder)
![Page 2: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/2.jpg)
Bas's Super Pessimistic Value Learning MethodBas's Super Pessimistic Value Learning Method
Bas R. SteunebrinkBas R. Steunebrink
Swiss AI lab IDSIA (postdoc)Swiss AI lab IDSIA (postdoc)NNAISENSE (co-founder)NNAISENSE (co-founder)
![Page 3: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/3.jpg)
What if...
• … we fail to come up with the perfect utility function.
• … we can't axiomatize the agent or the environment.
• … the agent won't have enough resources to do the optimal thing.
• What ingredients are needed to get such agents to develop a robust value system?
![Page 4: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/4.jpg)
Scope
• This is intended as a line of reasoning in parallel to formal methods
• No intention to show results; I would like you to think with me about necessary ingredients
• Content of talk is necessarily informal– Like a moderated discussion
![Page 5: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/5.jpg)
Overview of Key Ingredients
• Architectural details– Knowledge representation– Goals & constraints– Learning & control
• Methodological details– Teaching– Testing– Growth & stabilization
![Page 6: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/6.jpg)
PREDICTIONS
GOALSGRANULESSENSE
ACT
PATTERN EXTRACTOR
PERFORMANCE RATING
COMPILER
ADD
DELETE
COMPRESS
= LEARNING
REQUIREMENTSUPDATE
HUMAN
![Page 7: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/7.jpg)
Key ImplementationDetails
• Requirements specifiable as goals and constraints• Simulation before commitment• Knowledge decoupled from goals• Controller dynamically couples knowledge and
goals to obtain actions• Requirements must be updatable on the fly
![Page 8: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/8.jpg)
Ingredients of Knowledge• Knowledge represented as granules– Functionality of forward and inverse models– Tentative, additive, reversible, very fine-grained– No reasoning about self-modifications– Experience-based vindication & falsification
• The controller is architecturally shielded from self-modifications
• Granules grow from a “seed”
![Page 9: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/9.jpg)
Experience-based AI
• I call this class of systems “EXPAI”• Not hot air: ≥1 implementation exists (AERA)
![Page 10: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/10.jpg)
Recursive Self-Improvement
• The ability to leverage current know-how to make increasingly better self-modifications, continuing over the system’s entire lifetime
• Caveat: quality of eventual behavior depends on imposed requirements and experienced phenomena
![Page 11: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/11.jpg)
Proof that EXPAI does RSI
1. The world has exploitable regularities and is not too deceptive and adversarial
2. Knowledge is represented homogeneously and hierarchically by “granules”
3. Learning by 3 separate types of processes: additive, subtractive, and compressive
4. Curiosity is realized through a simple analysis of granules’ performance ratings
![Page 12: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/12.jpg)
Proof that EXPAI does RSI
5. From (2) and (3) we conclude that learning entails comprehensive self-modification
6. From (1) and (4) we conclude that good experience is gathered continually
7. From (5) and (6) we conclude that an EXPAI performs self-improvement
8. In a life-long learning setting, an EXPAI performs recursive self-improvement
![Page 13: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/13.jpg)
Key Methodological Details
• Having a seed ≠ having a mature AGI• Put agent through curriculum-test cycles• Levels of understanding must be rigorously
tested during growth• Bridge the gap between underspecified
requirements and the agent's knowledge• Adjust requirements when necessary• How does an agent develop a value system?
![Page 14: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/14.jpg)
Understanding a Table
![Page 15: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/15.jpg)
About Understanding
• An account of understanding is sorely lacking• The level of understanding of a phenomenon
Φ is determined by the completeness and accuracy of granules relating elements of Φ (both within Φ and to other phenomena)
![Page 16: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/16.jpg)
About Understanding
• Understanding of a phenomenon Φ requires ≥4 capabilities (ordered by strength):
1. To predict Φ2. To achieve goals with respect to Φ3. To explain Φ4. To (re)create Φ
![Page 17: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/17.jpg)
About Values
• Values are anchored in human-imposed constraints
• These constraints may be underspecified• Therefore the agent must build an
understanding relating these constraints to phenomena in the environment
• But... understanding is not enough!
![Page 18: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/18.jpg)
About Values
• Values requires an agent to be compelled to adhere to them
• Recall that we specified the controller to be unmodifiable, so we should be safe?
• 2 more ingredients needed:–Meta-values: value persistence of values– Ensuring the understanding of each value
stabilizes
![Page 19: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/19.jpg)
Values Must Be Stable
• Values must be robust against influence• Must be compelled to adhere to its values• Must be compelled to protect its values• Interference may come from other agents
(human or artificial), environmental forces (radiation), and from itself
![Page 20: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/20.jpg)
Values Stabilization
• Recall key architectural ingredients:– simulation before commitment– cannot delete constraints– acquire and maintain knowledge on the
principle of effective utility and parsimony– knowledge representation is defeasible
![Page 21: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/21.jpg)
Knowledge Stabilization
• Logically inconsistent courses of action can by necessity not be effective
• Pools of knowledge that turn out to be effective will be more logically consistent
• Thus an EXPAI will tend towards progressive logical consistency
![Page 22: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/22.jpg)
Value Stabilization
• Interconnectedness and cross-reliance of understanding with other knowledge “protects” against corruption– Changes lead to logical inconsistencies
• Stabilization happens during a sensitivity period which makes an agent open to constraint injection early, and becoming less open over time
• As the knowledge pertaining to a constraint stabilizes, it turns into a value
![Page 23: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/23.jpg)
About Testing
• The stabilization values must be tested for• A test must consists of:– set of requirements specifying a task– an agent– pressure– a stakeholder– consequences
![Page 24: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/24.jpg)
Artificial Pedagogy
• This whole approach places a lot of importance on the teacher during the sensitivity period of an AGI
• There should probably be laws on who's allowed to teach “baby AGIs”, with regulations about checking test results
• Once stabilized, the agent should be robust even against humans (un)intentionally specifying tasks that violate its values
![Page 25: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/25.jpg)
Credit: SMBC comics
![Page 26: About Understanding and Values · Proof that EXPAI does RSI 5. From (2) and (3) we conclude that learning entails comprehensive self-modification 6. From (1) and (4) we conclude that](https://reader036.fdocuments.us/reader036/viewer/2022071010/5fc7a4211aa18432dc78cd0e/html5/thumbnails/26.jpg)