Meta-Learning : t he future of data mining
-
Upload
rooney-lucas -
Category
Documents
-
view
25 -
download
1
description
Transcript of Meta-Learning : t he future of data mining
![Page 1: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/1.jpg)
Meta-Learning:the future of data mining
Meta-Learning:the future of data mining
Włodzisław Duch & Co
Department of Informatics, Nicolaus Copernicus University, Toruń, Poland
School of Computer Engineering, Nanyang Technological University, Singapore
Google: Duch W
University of Lausanne, 4/2012
![Page 2: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/2.jpg)
Toruń
![Page 3: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/3.jpg)
Toruń
Nicolaus Copernicus: born in 1473 in Toruń
![Page 4: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/4.jpg)
DI NCU Projects:NCI
Neurocognitive Informatics – a new branch of science.
• Computational creativity, insight, intuition, imagery.• Imagery agnosia, especially imagery amusia.• Neurocognitive approach to language, word games. • Medical information retrieval, analysis, visualization. • Comprehensive theory of autism, ADHD, phenomics. • Visualization of high-D trajectories, EEG signals, neurofeedback.• Brain stem models and consciousness in artificial systems.• Geometric theory of brain-mind processes. • Infants: observation, guided development. • Neural determinism, free will, and social consequences.
![Page 5: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/5.jpg)
Interdisciplinary Center of Innovative Technologies
Why am I interested in this? CS curriculum + ICIT in construction, working - end of 2012.
Neurocognitive lab, 5 rooms, many projects requiring experiments.
Funding: national/EU grants.
Pushing the limits of brain plasticity and understanding brain-mind relations.
![Page 6: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/6.jpg)
![Page 7: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/7.jpg)
Norbert Tomek Marek Krzysztof
![Page 8: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/8.jpg)
PlanPlan
• Problems with Computational intelligence (CI)
• Problems with current approaches to data mining/pattern recognition.
• Meta-learning as search in the space of all models.• First attempts: similarity based framework for metalearning
and heterogeneous systems. • Hard problems and support features, k-separability and the
goal of learning.• Transfer learning and more components to build algorithms:
SFM, aRMP, LOK, ULM, QPC-PP, QPC-NN, C3S, cLVQ.• Implementation of meta-learning - algorithms on demand.
![Page 9: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/9.jpg)
What is there to learn?What is there to learn?Brains ... what is in EEG? What happens in the brain?
Industry: what happens with our machines? Cognitive robotics: vision, perception, language. Bioinformatics, life sciences.
![Page 10: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/10.jpg)
What can we learn?What can we learn?What can we learn using pattern recognition, machine lerning, computational intelligence techniques? Everything?
Neural networks are universal approximators and evolutionary algorithms solve global optimization problems – so everything can be learned? Not at all! All non-trivial problems are hard, need deep transformations.
Duda, Hart & Stork, Ch. 9, No Free Lunch + Ugly Duckling Theorems: • Uniformly averaged over all target functions the expected error for all
learning algorithms [predictions by economists] is the same. • Averaged over all target functions no learning algorithm yields
generalization error that is superior to any other. • There is no problem-independent or “best” set of features.“Experience with a broad range of techniques is the best insurance for solving arbitrary new classification problems.”In practice: try as many models as you can, rely on your experience and intuition. There is no free lunch, but do we have to cook ourselves?
![Page 11: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/11.jpg)
Data mining packagesData mining packages
• No free lunch => provide different type of tools for knowledge discovery: decision tree, neural, neurofuzzy, similarity-based, SVM, committees, tools for visualization of data.
• Support the process of knowledge discovery/model building and evaluating, organizing it into projects.
• Many other interesting DM packages of this sort exists: Weka, Yale, Orange, Knime ... >170 packages on the-data-mine.com list!
• We are building Intemi, radically new DM tools.
GhostMiner, data mining tools from our lab + Fujitsu: http://www.fqspl.com.pl/ghostminer/
• Separate the process of model building (hackers) and knowledge discovery, from model use (lamers) => GM Developer & Analyzer
![Page 12: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/12.jpg)
What DM packages do?What DM packages do?
Hundreds of components ... transforming, visualizing ...
Rapid Miner 5.2, type and # components: total 712 (March 2012)Process control 38Data transformations 114Data modeling 263Performance evaluation 31Other packages 266 Text, series, web ... specific transformations, visualization, presentation, plugin extensions ... ~ billions of models! Keel has >450 components.
Visual “knowledge flow” to link components, or script languages (XML) to define complex experiments.
![Page 13: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/13.jpg)
With all these tools, are we
really so good?
Surprise!
Almost nothing can be learned using such tools!
![Page 14: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/14.jpg)
May the force be with youMay the force be with you
Hundreds of components ... billions of combinations ...Our treasure box is full! We can publish forever! Specialized transformations are still missing in many packages.Data miners have a hard job … what to select?
What would we really like to have?Press the button, and wait for the truth!
Computer power is with us, meta-learning should replace data miners in find all interesting data models = sequences of transformations/procedures.
Many considerations: optimal cost solutions, various costs of using feature subsets; simple & easy to understand vs. optimal accuracy; various representation of knowledge: crisp, fuzzy or prototype rules, visualization, confidence in predictions ...
![Page 15: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/15.jpg)
Meta-learningMeta-learning
Meta-learning means different things for different people.Some will call “meta” learning of many models, ranking them, boosting, bagging, or creating an ensemble in many ways , so heremeta optimization of parameters to integrate models.Landmarking: characterize many datasets and remember which method worked the best on each dataset. Compare new dataset to the reference ones; define various measures (not easy) and use similarity-based methods. Regression models: created for each algorithm on parameters that describe data to predict expected accuracy, ranking potentially useful algorithms.Stacking, ensambles: learn new models on errors of the previous ones.Deep learning: DARPA 2009 call, methods are „flat”, shallow, build a universal machine learning engine that generates progressively more sophisticated representations of patterns, invariants, correlations from data. Success in limited domains only …Meta-learning: learning how to learn.
![Page 16: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/16.jpg)
Principles: information compressionPrinciples: information compression
Neural information processing in perception and cognition: information compression, or algorithmic complexity. In computing: minimum length (message, description) encoding.
Wolff (2006): all cognition and computation is information compression! Analysis and production of natural language, fuzzy pattern recognition, probabilistic reasoning and unsupervised inductive learning.Talks about multiple alignment, unification and search, but so far only models for sequential data and 1D alignment have been demonstrated. Information compression: encoding new information in terms of oldhas been used to define the measure of syntactic and semantic information (Duch, Jankowski 1994); based on the size of the minimal graph representing a given data structure or knowledge-base specification, thus it goes beyond alignment; real info = what model cannot predict. “Surprise” and curiosity measures; Schmidhuber, Baldi and others, are based on the same idea.
![Page 17: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/17.jpg)
Similarity-based frameworkSimilarity-based frameworkSearch for good models requires frameworks characterizing models. p(Ci|X;M) posterior classification probability or y(X;M) approximators,models M are parameterized in increasingly sophisticated way. Similarity-Based Methods (SBMs) may be organized in such framework.
(Dis)similarity: • more general than feature-based description, • no need for vector spaces (structured objects), • more general than fuzzy approach (F-rules are reduced to P-rules), • includes nearest neighbor algorithms, MLPs, RBFs, separable function
networks, SVMs, kernel methods, specialized kernels, and many others!
A systematic search (greedy, beam, evolutionary) in the space of all SBM models is used to select optimal combination of parameters and procedures, opening different types of optimization channels, trying to discover appropriate bias for a given problem.
Results: several candidate models are created, even very limited version gives best results in 7 out of 12 Stalog problems.
![Page 18: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/18.jpg)
SBM framework componentsSBM framework components• Pre-processing: objects O => features X, or (diss)similarities D(O,O’). • Calculation of similarity between features d(xi,yi) and objects D(X,Y).• Reference (or prototype) vector R selection/creation/optimization. • Weighted influence of reference vectors G(D(Ri,X)), i=1..k.• Functions/procedures to estimate p(C|X;M) or y(X;M). • Cost functions E[DT;M] and model selection/validation procedures. • Optimization procedures for the whole model Ma.• Search control procedures to create more complex models Ma+1.• Creation of ensembles of (local, competent) models.
• M={X(O), d(.,.), D(.,.), k, G(D), {R}, {pi(R)}, E[.], K(.), S(.,.)}, where:• S(Ci,Cj) is a matrix evaluating similarity of the classes;
a vector of observed probabilities pi(X) instead of hard labels.
The kNN model p(Ci|X;kNN) = p(Ci|X;k,D(.),{DT}); the RBF model: p(Ci|X;RBF) = p(Ci|X;D(.),G(D),{R}), MLP, SVM and many other models may all be “re-discovered” as a part of SBF.
![Page 19: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/19.jpg)
Meta-learning in SBM schemeMeta-learning in SBM scheme
Start from kNN, k=1, all data & features, Euclidean distance, end with a model that is a novel combination of procedures and parameterizations.
k-NN 67.5/76.6%
+d(x,y); Canberra 89.9/90.7 %
+ si=(0,0,1,0,1,1); 71.6/64.4 %
+selection, 67.5/76.6 %
+k opt; 67.5/76.6 %
+d(x,y) + si=(1,0,1,0.6,0.9,1); Canberra 74.6/72.9 %
+d(x,y) + selection; Canberra 89.9/90.7 %
k-NN 67.5/76.6%
+d(x,y); Canberra 89.9/90.7 %
+ si=(0,0,1,0,1,1); 71.6/64.4 %
+selection, 67.5/76.6 %
+k opt; 67.5/76.6 %
+d(x,y) + si=(1,0,1,0.6,0.9,1); Canberra 74.6/72.9 %
+d(x,y) + sel. or opt k; Canberra 89.9/90.7 %
k-NN 67.5/76.6%
+d(x,y); Canberra 89.9/90.7 %
+ si=(0,0,1,0,1,1); 71.6/64.4 %
+ranking, 67.5/76.6 %
+k opt; 67.5/76.6 %
+d(x,y) + si=(1,0,1,0.6,0.9,1); Canberra 74.6/72.9 %
+d(x,y) + selection; Canberra 89.9/90.7 %
![Page 20: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/20.jpg)
Meta-learning in SBM schemeMeta-learning in SBM scheme
Start from kNN, k=1, all data & features, Euclidean distance, end with a model that is a novel combination of procedures and parameterizations.
k-NN 67.5/76.6%
+d(x,y); Canberra 89.9/90.7 %
+ si=(0,0,1,0,1,1); 71.6/64.4 %
+selection, 67.5/76.6 %
+k opt; 67.5/76.6 %
+d(x,y) + si=(1,0,1,0.6,0.9,1); Canberra 74.6/72.9 %
+d(x,y) + selection; Canberra 89.9/90.7 %
![Page 21: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/21.jpg)
Heterogeneous systemsHeterogeneous systemsNext step: use components from different models. Problems requiring different scales (multiresolution).
2-class problems, two situations:
C1 inside the sphere, C2 outside.MLP: at least N+1 hyperplanes, O(N2) parameters. RBF: 1 Gaussian, O(N) parameters.
C1 in the corner defined by (1,1 ... 1) hyperplane, C2 outside.MLP: 1 hyperplane, O(N) parameters. RBF: many Gaussians, O(N2) parameters, poor approx.
Combination: needs both hyperplane and hypersphere!
Logical rule: IF x1>0 & x2>0 THEN C1 Else C2
is not represented properly neither by MLP nor RBF!
Different types of functions in one model, first step beyond inspirations from single neurons => heterogeneous models.
![Page 22: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/22.jpg)
Heterogeneous everythingHeterogeneous everythingHomogenous systems: one type of “building blocks”, same type of decision borders, ex: neural networks, SVMs, decision trees, kNNsCommittees combine many models together, but lead to complex models that are difficult to understand.
Ockham razor: simpler systems are better. Discovering simplest class structures, inductive bias of the data, requires Heterogeneous Adaptive Systems (HAS).
HAS examples:NN with different types of neuron transfer functions.k-NN with different distance functions for each prototype.Decision Trees with different types of test criteria.
1. Start from large network, use regularization to prune.2. Construct network adding nodes selected from a candidate pool.3. Use very flexible functions, force them to specialize.
![Page 23: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/23.jpg)
Taxonomy - TFTaxonomy - TF
![Page 24: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/24.jpg)
HAS decision treesHAS decision treesDecision trees select the best feature/threshold value for univariate and multivariate trees:
Decision borders: hyperplanes.
Introducing tests based on La Minkovsky metric.
or ; ,i k k i i ki
X T W X X W
Such DT use kernel features!
For L2 spherical decision border are produced.
For L∞ rectangular border are produced.
For large databases first clusterize data to get candidate references R.
1/; , R i i R
i
T X R
X R X R
![Page 25: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/25.jpg)
SSV HAS DT exampleSSV HAS DT example
SSV HAS tree in GhostMiner 3.0, Wisconsin breast cancer (UCI)699 cases, 9 features (cell parameters, 1..10)Classes: benign 458 (65.5%) & malignant 241 (34.5%).
Single rule gives simplest known description of this data:
IF ||X-R303|| < 20.27 then malignant
else benign coming most often in 10xCV
Accuracy = 97.4%, good prototype for malignant case!
Gives simple thresholds, that’s what MDs like the most!
Best 10CV around 97.5±1.8% (Naïve Bayes + kernel, or opt. SVM)
SSV without distances: 96.4±2.1%
C 4.5 gives 94.7±2.0%
Several simple rules of similar accuracy but different specificity or sensitivity may be created using HAS DT. Need to select or weight features and select good prototypes.
![Page 26: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/26.jpg)
Maximization of margin/regularization
Among all discriminating hyperplanes there is one defined by support vectors that is clearly better.
![Page 27: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/27.jpg)
LDA in larger spaceSuppose that strongly non-linear borders are needed.
Use LDA, but add new dimensions, functions of your inputs!Add to input Xi
2, and products Xi Xj, as new features.
Example: 2D => 5D case Z={z1…z5}={X1, X2, X12, X2
2, X1X2}
The number of such tensor products grows exponentially – no good.
Fig. 4.1Hasti et al.
![Page 28: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/28.jpg)
Kernels = similarity functionsGaussian kernels in SVM: zi (X)=G(X;XI ,s) radial features, X=>ZGaussian mixtures are close to optimal Bayesian errors. Solution requires continuous deformation of decision borders and is therefore rather easy.
Support Feature Machines (SFM): construct features based on projections, restricted linear combinations, kernel features, use feature selection.
Gaussian kernel, C=1.In the kernel space Z decision borders are flat, but in the X space highly non-linear!
SVM is based on quadratic solver, without explicit features, but using Z features explicitly has some advantages: Multiresolution (Locally Optimized Kernels): different s for different support features, or using several kernels zi (X)=K(X;XI ,s). Use linear solvers, neural network, Naïve Bayes, or any other algorithm, all work fine.
![Page 29: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/29.jpg)
Thyroid screening, network solutionThyroid screening, network solution
Garavan Institute, Sydney, Australia
15 binary, 6 continuous
Training: 93+191+3488 Validate: 73+177+3178
•Determine important clinical factors
•Calculate prob. of each diagnosis.
Hiddenunits
Finaldiagnoses
TSH
T4U
Clinical findings
Agesex……
T3
TT4
TBG
Normal
Hyperthyroid
Hypothyroid
Poor results of SBL and SVM … needs decision borders with sharp corners due to the inherent logic based on thresholding by medical experts.
![Page 30: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/30.jpg)
Hypothyroid dataHypothyroid data2 years real medical screening tests for thyroid diseases, 3772 cases with 93 primary hypothyroid and 191 compensated hypothyroid, the remaining 3488 cases are healthy; 3428 test, similar class distribution. 21 attributes (15 binary, 6 continuous) are given, but only two of the binary attributes (on thyroxine, and thyroid surgery) contain useful information, therefore the number of attributes has been reduced to 8.
Method % train % test error
SFM, SSV+2 B1 features ------- 0.4 SFM, SVMlin+2 B1 features ------- 0.5 MLP+SVNT, 4 neurons 0.2 0.8 Cascade correlation 0.0 1.5MLP + backprop 0.4 1.5 SVM Gaussian kernel 0.2 1.6 SVM lin 5.9 6.7
![Page 31: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/31.jpg)
Hypothyroid dataHypothyroid data
![Page 32: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/32.jpg)
How much can we learn?Linearly separable or almost separable problems are relatively simple – deform or add dimensions to make data separable.
How to define “slightly non-separable”? There is only separable and the vast realm of the rest.
![Page 33: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/33.jpg)
Linear separabilityLinear separability
QPC projection used to visualize Leukemia microarray data.
2-separable data, separated in vertical dimension.
![Page 34: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/34.jpg)
Approximate separabilityApproximate separability
QPC visualization of Heart dataset: overlapping clusters, information in the data is insufficient for perfect classification, approximately 2-separable.
![Page 35: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/35.jpg)
Easy problemsEasy problems• Approximately linearly separable problems in
the original feature space: linear discrimination is sufficient (always worth trying!).
• Simple topological deformation of decision borders is sufficient – linear separation is then possible in extended/transformed spaces.
This is frequently sufficient for pattern recognition problems (more than half of UCI problems).
• RBF/MLP networks with one hidden layer also solve such problems easily, but convergence/generalization for anything more complex than XOR is problematic.
SVM adds new features to “flatten” the decision border:
( )1 2( , ,... ); ,in ix x x z K X X X X
achieving larger margins/separability in the X+Z space.
![Page 36: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/36.jpg)
Neurons learning complex logicBoole’an functions are difficult to learn, n bits but 2n nodes => combinatorial complexity; similarity is not useful, for parity all neighbors are from the wrong class. MLP networks have difficulty to learn functions that are highly non-separable.
Projection on W=(111 ... 111) gives clusters with 0, 1, 2 ... n bits;
easy categorization in (n+1)-separable sense.
Ex. of 2-4D parity problems.
Neural logic can solve it without counting; find a good point of view.
![Page 37: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/37.jpg)
Easy and difficult problemsEasy and difficult problemsLinear separation: good goal if simple topological deformation of decision borders is sufficient.
Linear separation of such data is possible in higher dimensional spaces; this is frequently the case in pattern recognition problems.
RBF/MLP networks with one hidden layer solve such problems.
Difficult problems: disjoint clusters, complex logic.
Continuous deformation is not sufficient; networks with localized functions need exponentially large number of nodes.
Boolean functions: for n bits there are K=2n binary vectors that can be represented as vertices of n-dimensional hypercube.
Each Boolean function is identified by K bits.
BoolF(Bi) = 0 or 1 for i=1..K, leads to the 2K Boolean functions.Ex: n=2 functions, vectors {00,01,10,11},
Boolean functions {0000, 0001 ... 1111}, ex. 0001 = AND, 0110 = OR,
each function is identified by number from 0 to 15 = 2K-1.
![Page 38: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/38.jpg)
Boolean functionsBoolean functionsn=2, 16 functions, 12 separable, 4 not separable.
n=3, 256 f, 104 separable (41%), 152 not separable.
n=4, 64K=65536, only 1880 separable (3%)
n=5, 4G, but << 1% separable ... bad news!
Existing methods may learn some non-separable functions, but in practice most functions cannot be learned !
Example: n-bit parity problem; many papers in top journals.
No off-the-shelf systems are able to solve such problems.
For all parity problems SVM is below base rate!
Such problems are solved only by special neural architectures or special classifiers – if the type of function is known.
But parity is still trivial ... solved by
1
cosn
ii
y b
![Page 39: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/39.jpg)
Goal of learningGoal of learningIf simple topological deformation of decision borders is sufficient linear separation is possible in higher dimensional spaces, “flattening” non-linear decision borders, kernel approaches are sufficient. RBF/MLP networks with one hidden layer solve the problem. This is frequently the case in pattern recognition problems.
For complex logic this is not sufficient; networks with localized functions need exponentially large number of nodes.
Such situations arise in AI reasoning problems, real perception, object recognition, text analysis, bioinformatics ...
Linear separation is too difficult, set an easier goal. Linear separation: projection on 2 half-lines in the kernel space:
line y=WX, with y<0 for class – and y>0 for class +.
Simplest extension: separation into k-intervals, or k-separability.
For parity: find direction W with minimum # of intervals, y=W.X
![Page 40: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/40.jpg)
NN as data transformationsNN as data transformations
Vector mappings from the input space to hidden space(s) and to the output space + adapt parameters to improve cost functions.
Hidden-Output mapping done by MLPs:
T = {Xi} training data, N-dimensional.
H = {hj(T)} X image in the hidden space, j =1 .. NH-dim.
... more transformations in hidden layers
Y = {yk(H )} X image in the output space, k =1 .. NC-dim.
ANN goal:
data image H in the last hidden space should be linearly separable; internal representations will determine network generalization.
But we never look at these representations!
![Page 41: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/41.jpg)
T-based meta-learningT-based meta-learning
To create successful meta-learning through search in the model space fine granulation of methods is needed, extracting info using support features, learning from others, knowledge transfer and deep learning.
Learn to compose, using complexity guided search, various transformations (neural or processing layers), for example:
• Creation of new support features: linear, radial, cylindrical, restricted localized projections, binarized … feature selection or weighting.
• Specialized transformations in a given field: text, bio, signal analysis, ….• Matching pursuit networks for signal decomposition, QPC index, PCA or ICA
components, LDA, FDA, max. of mutual information etc. • Transfer learning, granular computing, learning from successes: discovering
interesting higher-order patterns created by initial models of the data.• Stacked models: learning from the failures of other methods.• Schemes constraining search, learning from the history of previous runs at
the meta-level.
![Page 42: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/42.jpg)
8-bit parity solution8-bit parity solution
QCP solution to 8-bit parity data: projection on W=[1,1…1] diagonal.k-separability is much easier to achieve than full linear separability.
![Page 43: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/43.jpg)
Network solutionNetwork solutionCan one learn a simplest model for arbitrary Boolean function?
2-separable (linearly separable) problems are easy; non separable problems may be broken into k-separable, k>2.
Blue: sigmoidal neurons with threshold, brown – linear neurons.
X1
X2
X3
X4
y=W.
X
+1
-1
+1-1
s(by+q1)
s(by+q2)
+1
+1+1+1
s(by+q4)
Neural architecture for k=4 intervals, or 4-separable problems.
![Page 44: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/44.jpg)
QPC Projection PursuitQPC Projection PursuitWhat is needed to learn data with complex logic?• cluster non-local areas in the X space, use W.X• capture local clusters after transformation, use G(W.X-q) SVMs fail because the number of directions W that should be considered grows exponentially with the size of the problem n.What will solve it? Projected clusters!
1. A class of constructive neural network solution with G(W.X-q) functions combining non-local/local projections, with special training algorithms.
2. Maximize the leave-one-out error after projection: take some localized function G, count in a soft way cases from the same class as Xk.
Grouping and separation; projection may be done directly to 1 or 2D for visualization, or higher D for dimensionality reduction, if W has d columns.
k k
k kC C
Q A G A G
X X X
W W X X W X X
![Page 45: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/45.jpg)
Parity n=9Parity n=9
Simple gradient learning; QCP quality index shown below.
![Page 46: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/46.jpg)
Learning hard functionsLearning hard functions
Training almost perfect for parity, with linear growth in the number of vectors for k-sep. solution created by the constructive neural algorithm.
![Page 47: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/47.jpg)
Real dataReal data
On simple data results are similar as from SVM (because they are almost optimal), but c3sep models are much simpler although only 3-sep. assumed.
![Page 48: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/48.jpg)
RulesRules
QPC visualization of Monks artificial symbolic dataset, => two logical rules are needed.
![Page 49: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/49.jpg)
Complex distributionComplex distribution
QPC visualization of concentric rings in 2D with strong noise in remaining 2D; transform: nearest neighbor solutions, combinations of ellipsoidal densities.
![Page 50: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/50.jpg)
Example: aRPMExample: aRPM
Almost Random Projection Machine (with Hebbian learning): generate random combinations of inputs (line projection) z(X)=W.X, find and isolate pure cluster h(X)=G(z(X)); estimate relevance of h(X), ex. MI(h(X),C), leave only good nodes;continue until each vector activates minimum k nodes.Count how many nodes vote for each class and plot.
![Page 51: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/51.jpg)
Knowledge transfer
Brains learn new concepts in terms of old; use large semantic network and add new concepts linking them to the known.
Knowledge should be transferred between the tasks. Not just learned from a single dataset.
Need to discover good building blocks for higher level concepts/features.
![Page 52: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/52.jpg)
Learning from others … Learning from others …
Learn to transfer knowledge by extracting interesting features created by different systems. Ex. prototypes, combinations of features with thresholds … => Universal Learning Machines.
Example of feature types:
B1: Binary – unrestricted projections b1 B2: Binary – complexes b1 ᴧ b2 … ᴧ bk
B3: Binary – restricted by distance
R1: Line – original real features ri; non-linear thresholds for “contrast enhancement“ s(ri-bi); intervals (k-sep).
R4: Line – restricted by distance, original feature; thresholds; intervals (k-sep); more general 1D patterns. P1: Prototypes: general q-separability, weighted distance functions or specialized kernels. M1: Motifs, based on correlations between elements rather than input values.
1 1 1 2 2 20 , , ...ib r r r r r r
![Page 53: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/53.jpg)
B1/B2 FeaturesB1/B2 Features
Dataset B1/B2 Features
Australian F8 < 0.5 F8 ≥ 0.5 ᴧ F9 ≥ 0.5
Appendicitis F7 ≥ 7520.5 F7 < 7520.5 ᴧ F4 < 12
Heart F13 < 4.5 ᴧ F12 < 0.5 F13 ≥ 4.5 ᴧ F3 ≥ 3.5
Diabetes F2 < 123.5 F2 ≥ 143.5
Wisconsin F2 < 2.5 F2 ≥ 4.5
Hypothyroid F17 < 0.00605 F17 ≥ 0.00605 ᴧ F21 < 0.06472
Example of B1 features taken from segments of decision trees.These features used in various learning systems greatly simplify their models and increase their accuracy. Convert Decision Tree to Distance Functions!Almost all systems reach similar accuracy!
![Page 54: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/54.jpg)
Dataset Classifier
SVM (#SV) SSV (#Leafs) NB
Australian 84.9±5.6 (203) 84.9±3.9 (4) 80.3±3.8
ULM 86.8±5.3(166) 87.1±2.5(4) 85.5±3.4
Features B1(2) + P1(3) B1(2) + R1(1) + P1(3) B1(2)
Appendicitis 87.8±8.7 (31) 88.0±7.4 (4) 86.7±6.6
ULM 91.4±8.2(18) 91.7±6.7(3) 91.4±8.2
Features B1(2) B1(2) B1(2)
Heart 82.1±6.7 (101) 76.8±9.6 (6) 84.2±6.1
ULM 83.4±3.5(98) 79.2±6.3(6) 84.5±6.8
Features Data + R1(3) Data + R1(3) Data + B1(2)
Diabetes 77.0±4.9 (361) 73.6±3.4 (4) 75.3±4.7
ULM 78.5±3.6(338) 75.0±3.3(3) 76.5±2.9
Features Data + R1(3) + P1(4) B1(2) Data + B1(2)
Wisconsin 96.6±1.6 (46) 95.2±1.5 (8) 96.0±1.5
ULM 97.2±1.8(45) 97.4±1.6(2) 97.2±2.0
Features Data + R1(1) + P1(4) R1(1) R1(1)
Hypothyroid 94.1±0.6 (918) 99.7±0.5 (12) 41.3±8.3
ULM 99.5±0.4(80) 99.6±0.4(8) 98.1±0.7
Features Data + B1(2) Data + B1(2) Data + B1(2)
![Page 55: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/55.jpg)
Support Feature MachinesSupport Feature Machines
General principle: complementarity of information processed by parallel interacting streams with hierarchical organization (Grossberg, 2000).Cortical minicolumns provide various features for higher processes.Create information that is easily used by various ML algorithms: explicitly build enhanced space adding more transformations.
• X , original features• Z=WX, random linear projections, other projections (PCA< ICA, PP)• Q = optimized Z using Quality of Projected Clusters or other PP techniques.• H=[Z1,Z2], intervals containing pure clusters on projections. • K=K(X,Xi), kernel features.• HK=[K1,K2], intervals on kernel features
Kernel-based SVM is equivalent to linear SVM in the explicitly constructed kernel space, enhancing this space leads to improvement of results. LDA is one option, but many other algorithms benefit from information in enhanced feature spaces; best results in various combination X+Z+Q+H+K+HK.
![Page 56: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/56.jpg)
Locally Optimized KernelsLocally Optimized Kernels
![Page 57: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/57.jpg)
Universal Learning MachinesUniversal Learning Machines
![Page 58: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/58.jpg)
Real meta-learning!Real meta-learning!Meta-learning: learning how to learn, replace experts who search for best models making a lot of experiments.
Search space of models is too large to explore it exhaustively, design system architecture to support knowledge-based search.
• Abstract view, uniform I/O, uniform results management.
• Directed acyclic graphs (DAG) of boxes representing scheme
• placeholders and particular models, interconnected through I/O.
• Configuration level for meta-schemes, expanded at runtime level.
An exercise in software engineering for data mining!
![Page 59: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/59.jpg)
Intemi, Intelligent MinerIntemi, Intelligent Miner
Meta-schemes: templates with placeholders.
• May be nested; the role decided by the input/output types.
• Machine learning generators based on meta-schemes.
• Granulation level allows to create novel methods.
• Complexity control: Length + log(time)
• A unified meta-parameters description, defining the range of sensible values and the type of the parameter changes.
![Page 60: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/60.jpg)
Advanced meta-learningAdvanced meta-learning• Extracting meta-rules, describing search directions.• Finding the correlations occurring among different items in
most accurate results, identifying different machine (algorithmic) structures with similar behavior in an area of the model space.
• Depositing the knowledge they gain in a reusable meta-knowledge repository (for meta-learning experience exchange between different meta-learners).
• A uniform representation of the meta-knowledge, extending expert knowledge, adjusting the prior knowledge according to performed tests.
• Finding new successful complex structures and converting them into meta-schemes (which we call meta abstraction) by replacing proper substructures by placeholders.
• Beyond transformations & feature spaces: actively search for info.
Intemi software (N. Jankowski and K. Grąbczewski) incorporating these ideas and more is coming “soon” ...
![Page 61: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/61.jpg)
Meta-learning architectureMeta-learning architecture
Inside meta-parameter search a repeater machine composed of distribution and test schemes are placed.
![Page 62: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/62.jpg)
Generating machinesGenerating machines
Search process is controlled by a variant of approximated Levin’s complexity: estimation of program complexity combined with time. Simpler machines are evaluated first, machines that work too long (approximations may be wrong) are put into quarantine.
![Page 63: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/63.jpg)
Pre-compute what you canPre-compute what you can
and use “machine unification” to get substantial savings!
![Page 64: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/64.jpg)
Complexities on vowel dataComplexities on vowel data
……………
![Page 65: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/65.jpg)
Simple machines on vowel dataSimple machines on vowel data
Left: final ranking, gray bar=accuracy, small bars: memory, time & total complexity, middle numbers = process id (models in previous table).
![Page 66: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/66.jpg)
Complex machines on vowel dataComplex machines on vowel data
Left: final ranking, gray bar=accuracy, small bars: memory, time & total complexity, middle numbers = process id (models in previous table).
![Page 67: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/67.jpg)
SummarySummary
• Challenging data cannot be handled with existing DM tools. • Similarity-based framework enables meta-learning as search in the
model space, heterogeneous systems add fine granularity.• No off-shelf classifiers are able to learn difficult Boolean functions.• Visualization of hidden neuron’s shows that frequently perfect but
non-separable solutions are found despite base-rate outputs.• Linear separability is not the best goal of learning, other targets that
allow for easy handling of final non-linearities should be defined.• k-separability defines complexity classes for non-separable data. • Transformation-based learning shows the need for component-
based approach to DM, discovery of simplest models and support features.
• Meta-learning replaces data miners automatically creating new optimal learning methods on demand.
Is this the final word in data mining? Only the future will tell.
![Page 68: Meta-Learning : t he future of data mining](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813409550346895d9afcb6/html5/thumbnails/68.jpg)
Exciting times are coming!
Thank you for lending your ears!
Google: W. Duch => Papers & presentations;
Book: Meta-learning via search in model spaces (in prep ...).