CSC411CSC411 Artificial IntelligenceArtificial Intelligence 11
Chapter 10
Machine Learning: Symbol-Based
ContentsContentsA FrameworkVersion Space SearchID3: Decision Tree
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 22
Machine LearningMachine LearningAI systems grow from a minimal amount of AI systems grow from a minimal amount of knowledge by learningknowledge by learningHerbert Simon (1983):Herbert Simon (1983):– Any change in a system that allows it to Any change in a system that allows it to
perform better the second time on repetition of perform better the second time on repetition of the same task or on another task drawn from the same task or on another task drawn from the same populationthe same population
Machine learning issues:Machine learning issues:– Generalization from experienceGeneralization from experience
InductionInductionInductive biasesInductive biases
– Performance change: improve or degradePerformance change: improve or degrade
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 33
Machine Learning CategoriesMachine Learning CategoriesSymbol-based learningSymbol-based learning– Inductive learning -- learning by examplesInductive learning -- learning by examples– Supervised learning/unsupervised learningSupervised learning/unsupervised learning
Concept learning –- classificationConcept learning –- classification
Concept formation -- clusteringConcept formation -- clustering
– Explanation-based learningExplanation-based learning– Reinforcement learningReinforcement learning
Neural/connectionist networksNeural/connectionist networks
Genetic/evolutionary learning Genetic/evolutionary learning
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 44
A general model of the learning process
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 55
Learning ComponentsLearning ComponentsData and goals of learning taskData and goals of learning task– What are given – training instancesWhat are given – training instances– What are expectedWhat are expected
Knowledge representationKnowledge representation– Logic expressionsLogic expressions– Decision treesDecision trees– RulesRules
OperationsOperations– Generalization/specializationGeneralization/specialization– Heuristic rules Heuristic rules – Weight adjustsWeight adjusts
Concept spaceConcept space– Search space: representation, formatSearch space: representation, format
Heuristic searchHeuristic search– Search control in the concept spaceSearch control in the concept space
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 66
Learning By ExamplesLearning By ExamplesPatrick Winston (1975)Patrick Winston (1975)– Given a set of positive and a set of negative Given a set of positive and a set of negative
examplesexamples– Find a concept representationFind a concept representation– Semantic network representationSemantic network representation
ExampleExample– Learn a general definition of structural Learn a general definition of structural
concept, say “concept, say “archarch””– Positive examples: examples of Positive examples: examples of archarch
What an arch looks like, to define the archWhat an arch looks like, to define the arch
– Negative examples: near missesNegative examples: near missesWhat an arch doesn’t look like, to avoid the over-What an arch doesn’t look like, to avoid the over-coverage of archcoverage of arch
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 77
Examples and near misses for the concept “arch.”
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 88
Generalization of descriptions to include multiple examples.
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 99
Generalization of descriptions to include multiple examples (cont’d)
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1010
Specialization of a description to exclude a near miss. In c we add constraints to a so that it can’t match with b.
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1111
Version Space SearchVersion Space SearchInductive learning as search through a Inductive learning as search through a concept spaceconcept spaceGeneralization imposes an ordering on the Generalization imposes an ordering on the concepts in the space and uses the concepts in the space and uses the ordering to guide the searchordering to guide the searchGeneralizationGeneralization– PrinciplesPrinciples
Extend the coverage of instances Extend the coverage of instances Shorten/shrink the constrainsShorten/shrink the constrains
– Operations Operations Replacing constant with variablesReplacing constant with variablesDropping conditions from a conjunctive expressionDropping conditions from a conjunctive expressionAdding a disjunct to an expressionAdding a disjunct to an expressionReplacing a concept with one of its parent conceptsReplacing a concept with one of its parent concepts
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1212
A concept space: • Initial state obj(X, Y, Z) might cover all instances: too general• As more instances are added, X, Y, Z will be constrained
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1313
Version Space Search AlgorithmsVersion Space Search AlgorithmsCharacteristics of these algorithmsCharacteristics of these algorithms
– Data-drivenData-drivenPositive examples to generalize the conceptPositive examples to generalize the conceptNegative examples to constrain the concept (avoid Negative examples to constrain the concept (avoid overgeneralization)overgeneralization)
– Procedure:Procedure:Starting from whole space Starting from whole space Reducing the size of the space as more examples included Reducing the size of the space as more examples included Finding regularities (rules) in the training dataFinding regularities (rules) in the training data
– Generalization on these regularities (rules)Generalization on these regularities (rules)
Three algorithmsThree algorithms– Reducing the size of the version space in a Reducing the size of the version space in a specific to specific to
generalgeneral direction direction– Reducing the size of the version space in a Reducing the size of the version space in a general to general to
specificspecific direction direction– Combination of above: Combination of above: candidate elimination algorithmcandidate elimination algorithm
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1414
The role of negative examples in preventing overgeneralization by forcing the learner to specialize concepts in order to exclude negative examples
Negative ExamplesNegative Examples
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1515
Specific to General SearchSpecific to General SearchMaintains a set S of candidate concepts, Maintains a set S of candidate concepts, the maximally specific generalizations the maximally specific generalizations from the training instancesfrom the training instances
A concept c is maximally specific if it A concept c is maximally specific if it – covers all positive examples, non of the covers all positive examples, non of the
negative examples, and negative examples, and – for any other concept c’ that covers the for any other concept c’ that covers the
positive examples, c≤c’positive examples, c≤c’
The algorithm usesThe algorithm uses– Positive examples to generalize the candidate Positive examples to generalize the candidate
conceptsconcepts– Negative example to avoid overgeneralizationNegative example to avoid overgeneralization
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1616
For hypothesis set S:Specific to General Search AlgorithmSpecific to General Search Algorithm
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1717
Specific to general search of the version space learning the concept “ball.”
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1818
General to Specific SearchGeneral to Specific SearchMaintains a set G of maximally general Maintains a set G of maximally general concepts concepts A concept c is maximally general if it A concept c is maximally general if it – covers non of the negative training examples, covers non of the negative training examples,
and and – for any other concept c’ that covers no for any other concept c’ that covers no
negative training examples, cnegative training examples, cc’c’
The algorithm usesThe algorithm uses– negative examples to specialize the candidate negative examples to specialize the candidate
concepts concepts – Positive examples to eliminate Positive examples to eliminate
overspecializationoverspecialization
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 1919
General to Specific Search AlgorithmGeneral to Specific Search Algorithm
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2020
General to specific search of the version space learning the concept “ball.”
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2121
Candidate Elimination AlgorithmCandidate Elimination Algorithm
Combination of above two algorithms into a bi-direction searchMaintains two sets of candidate concepts– G, the set of maximally general candidates– S, the set of maximally specific candidates
The algorithm specializes G and generalizes S until they converge on the target concept.
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2222
Candidate Elimination AlgorithmCandidate Elimination Algorithm
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2323
The candidate elimination algorithm learning the concept “red ball.”
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2424
Converging boundaries of the G and S sets in the candidate elimination algorithm.
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2525
Decision TreesDecision TreesLearning algorithms of inducing concepts Learning algorithms of inducing concepts from examplesfrom examples
CharacteristicsCharacteristics– A tree structure to represent the concept, A tree structure to represent the concept,
equivalent to a set of rulesequivalent to a set of rules– Entropy and information gain as heuristics for Entropy and information gain as heuristics for
selecting candidate conceptsselecting candidate concepts– Handling noise dataHandling noise data– Classification – supervised learningClassification – supervised learning
Typical systems: ID3, C4.5, C5.0Typical systems: ID3, C4.5, C5.0
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2626
Data from credit history of loan applications
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2727
A decision tree for credit risk assessment.
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2828
A simplified decision tree for credit risk assessment.
CSC411CSC411 Artificial IntelligenceArtificial Intelligence 2929
The induction algorithm begins with a sample of correctly classified members of the target categories.
Decision Tree Construction AlgorithmDecision Tree Construction Algorithm
Top Related