Data Science Process - HDC : Health Data...

Data Science with KNIME

Amornthep Thongchiw 25-29 July 2016

www.reinforcebi.com

Data Science with KNIME (2016) A.Amornthep Thongchiw

www.reinforcebi.com Hot-line 08 1491 0909

Course outline * Data Science Process p.3 * Impact of big data & data sci p.9 * Cyclic nature of data mining p.50 * Data Mining Strategies p.59 * A machine learning perspective p.61 * Introduction to KNIME p.85 * Workshop 1 Data Preprocess p.112 * Workshop 2 Read & write p.120 * Workshop 3 Data Manipulate & View p.127 * Workshop 4 Data Preparation p.163 * Workshop 5 Classification model p.173 * Workshop 6 Clustering p.195 * Workshop 7 Associate rule p.205 * Workshop 8 Data Science Case p.211

2

http://www.reinforcebi.com/



Data Science Process

3




Stages of a data science project

Nina Zumel and John Mount, Practical Data Science with R, March 2014 4




Cathy O'Neil, Rachel Schutt, Doing Data Science ,2013

5



www.reinforcebi.com Hot-line 08 1491 0909 Cathy O'Neil, Rachel Schutt, Doing Data Science ,2013 6




The data science process

Cathy O'Neil, Rachel Schutt, Doing Data Science ,2013 7




A Data Scientist’s Role in This Process

Cathy O'Neil, Rachel Schutt, Doing Data Science ,2013 8




Impact of big data & data sci

http://www.kdnuggets.com/2015/03/interview-beena-ammanath-ge-data-driven-innovation.html 9

Today is industrial 4.0




The Big Data Story map

5

4

3 2

1

6

10




Human brain’s ability

11 http://www.redcentricplc.com/resources/infographics/byte-size/




Big Data at Big Companies

12




How Does Big Data Affect Our Daily lives?

13




The 4V of Big Data

http://www.ibmbigdatahub.com/sites/default/files/infographic_file/4-Vs-of-big-data.jpg 14




5V's: Turning Big Data into Value

http://www.datasciencecentral.com/m/blogpost?id=6448529%3ABlogPost%3A266056 15




Big Data is much bigger than you think

http://www.platfora.com/blog-post/iceberg/ 16




Predictive Analytics Transforms Insight into Action

• Source: RapidMiner official website 17




KNIME & R integration

18




Using R from Tableau

19




QlikView and R Integration

20




Why analytics?

21




Venn diagram of data science

22 http://advanceddataanalytics.net/images/



www.reinforcebi.com Hot-line 08 1491 0909 23

http://www.datasciencecentral.com/profiles/blogs/the-data-science-venn-diagram-revisited




Data & Social Sciences

24




Data Scientist Profile

25 Cathy O'Neil, Rachel Schutt, Doing Data Science, 2013

Stat Domain Expertise ML




Clustering of Data skill

26 Cathy O'Neil, Rachel Schutt, Doing Data Science, 2013




Data Science Compared With Different Analytics Disciplines

https://www.dezyre.com/article/data-science-compared-with-different-analytics-disciplines/175

27




What makes a data scientist?

28




Anatomy of a data scientist.

29




Data Scientist

30




Data Scientist

31




Categories of data scientists

• Those strong in statistics

• Those strong in mathematics

• Those strong in data engineering

• Those strong in machine learning

• Those strong in business

• Those strong in software engineering

• Those strong in visualization • Those strong in GIS, spatial data

32 http://www.datasciencecentral.com/profiles/blogs/six-categories-of-data-scientists




16 analytic disciplines compared to data science

• Quant

• Artificial intelligence

• Computer science

• Econometrics

• Data engineering

• Business intelligence

• Data analysis

• Business analytics

• Machine learning

• Data Mining

• Statistics

• Mathematical optimization

• Actuarial sciences

• High performance computing

• Operations research

• Six sigma

33

http://www.datasciencecentral.com/profiles/blogs/17-analytic-disciplines-compared




Data engineering.

• Performed by software engineers (developers) or architects (designers) in large organizations (sometimes by data scientists in tiny companies), this is the applied part of computer science (see entry in this article), to power systems that allow all sorts of data to be easily processed in-memory or near-memory, and to flow nicely to (and between) end-users, including heavy data consumers such as data scientists. A sub-domain currently under attack is data warehousing, as this term is associated with static, siloed conventational data bases, data architectures, and data flows, threatened by the rise of NoSQL, NewSQL and graph databases. Transforming these old architectures into new ones (only when needed) or make them compatible with new ones, is a lucrative business.

34 http://www.datasciencecentral.com/profiles/blogs/17-analytic-disciplines-compared




Difference Between Data Scientist and Data Analyst

35




22 skills of Data Scientist • Algorithms (ex: computational complexity, CS theory) DD,DR

• Back-End Programming (ex: JAVA/Rails/Objective C) DC, DD

• Bayesian/Monte-Carlo Statistics (ex: MCMC, BUGS) DD, DR

• Big and Distributed Data (ex: Hadoop, Map/Reduce) DB, DC, DD

• Business (ex: management, business development, budgeting) DB

• Classical Statistics (ex: general linear model, ANOVA) DB, DC, DR

• Data Manipulation (ex: regexes, R, SAS, web scraping) DC, DR

• Front-End Programming (ex: JavaScript, HTML, CSS) DC, DD

• Graphical Models (ex: social networks, Bayes networks) DD, DR

• Machine Learning (ex: decision trees, neural nets, SVM, clustering) DC, DD

• Math (ex: linear algebra, real analysis, calculus) DD,DR

• Optimization (ex: linear, integer, convex, global) DD, DR

• Product Development (ex: design, project management) DB

• Science (ex: experimental design, technical writing/publishing) DC, DR

• Simulation (ex: discrete, agent-based, continuous) DD,DR

• Spatial Statistics (ex: geographic covariates, GIS) DC, DR

• Structured Data (ex: SQL, JSON, XML) DC, DD

• Surveys and Marketing (ex: multinomial modeling) DC, DR

• Systems Administration (ex: *nix, DBA, cloud tech.) DC, DD

• Temporal Statistics (ex: forecasting, time-series analysis) DC, DR

• Unstructured Data (ex: noSQL, text mining) DC, DD

• Visualisation (ex: statistical graphics, mapping, web-based data‐viz) DC, DR 36




Comparison : Jobs in Data Science

37

http://www.analyticsvidhya.com/blog/2015/10/job-comparison-data-scientist-data-engineer-statistician/





38





39




The Data Science Ecosystem

40





41




https://blog.pivotal.io/pivotal/p-o-v/disruptive-data-science-transforming-your-company-into-a-data-science-driven-enterprise




Sectors that use big data

43




Health care Analytics: Case study

44

http://www.pwc.com/gx/en/issues/data-and-analytics/big-decisions-survey/industry/healthcare.html




http://www.pwc.com/gx/en/issues/data-and-analytics/big-decisions-survey/industry/healthcare.html




http://hin.com/blog/2015/12/21/infographic-10-healthcare-analytics-trends-for-2016/




http://www.healthcareitnews.com/blog/future-clinical-business-intelligence-healthcare




49 https://www.healthcatalyst.com/healthcare-analytics-summit-day-one-recap


Reinforce business intelligence Co.,Ltd.

www.reinforcebi.com

50

Cyclic nature of data mining.

Source: togaware


www.reinforcebi.com

51

Six steps of CRISP-DM

Source: togaware

Data mining is a small Process.


www.reinforcebi.com

52


1.Business Understanding We had better make sure we are addressing a real business

problem.

Initial phase focuses on understanding project objectives and requirements from a business perspective.

This knowledge is converted into a data mining problem definition.

Develop a preliminary plan designed to achieve the objectives.

Source: togaware


www.reinforcebi.com

53


2. Data Understanding Understand what data is available and its semantics.

Initial data collection

Familiarisation with the data Identify data quality problems.

Discover first insights into the data

Detect interesting subsets to from hypotheses for hidden information

Source: togaware


www.reinforcebi.com

54


3. Data Preparation. Bring together the data – get it into shape for

mining.

Construct the mining dataset

Derived from the initial raw datasets

Data preparation tasks: Table, record, and attribute selection

Generation of derived features

Data transformation

Data cleaning Source: togaware


www.reinforcebi.com

55


3. Data Preparation (con’t) Issues to be dealt with include:

Data Quality Missing data

Noisy data

Lead to inconsistent or too general / specific discoveries.

Data Cleaning Duplicates

Inconsistencies

Identify and merge the same entities. Source: togaware


www.reinforcebi.com

56


4. Modeling

Now the “data mining” begins!! Select various modeling techniques

Apply and calibrate modeling techniques

Typically there are several techniques for the same data mining problem

Some techniques have specific requirements on the form of data and require stepping back to the data preparation phase

Source: togaware


www.reinforcebi.com

57


5. Evaluation

How do we know we have a useful outcome? Evaluate the model and review the steps executed to

construct the model

Does the model properly achieve the business objectives?

Is there some important business issue that has not been sufficiently considered?

Decide on the use of the data mining results

Source: togaware


www.reinforcebi.com

58


6. Deployment

No point to data mining unless we action the outcomes. Deployment may be:

Generate a report of the discoveries made

Implement changes in the processes of the organization

Implement a repeatable data mining process

For successful deployment the customer must understand the actions to be carried out in order to actually make use of the created models.

Source: togaware


www.reinforcebi.com

59

Data Mining Strategies Supervised Learning

Process of building classification models using data instances of known origin to learning with predefined classes

Build models by using input attributes (independent variables) to predict output or dependent variables (class label)

Unsupervised Learning

To discover natural grouping or concept structures in data

Without a dependent variable to guide learning process

Rather, it builds knowledge structures by using some measures of cluster quality to group instances into classes


www.reinforcebi.com

60

Data mining techniques Supervised Learning

Classification: response is categorical

Regression: response is continuous

Time Series: dealing with observations across time

Optimization: minimize or maximize some characteristic


Principal Component Analysis : feature reduction

Clustering: grouping like objects together

Association and link Analysis: descriptive approach to exploring data that helps identify relationships among values in a database, e.g. Market basket analysis, those customers that buy hammers also buy nails. Examine conditional probabilities.


www.reinforcebi.com

61

A machine learning perspective

Artificial Intelligence to model the world


Cluster Analysis

Association Analysis

Supervised Learning

Decision Trees

Random forest, Ensembles

Support Vector Machines

Neural Networks


www.reinforcebi.com

62

Data mining Applications (1 of 3)

Database analysis and decision support Market analysis and management

Target marketing

Find clusters of “model” customers who share the same characteristics: interest, income level, spending habits, etc.

Market basket analysis

Find the relationship between product sales

Customer profiling

Data mining can tell you what types of customers buy what products (clustering or classification)

Identifying customer requirements

Identifying the best products for different customers

Use prediction to find what factors will attract new customers


www.reinforcebi.com

63


Database analysis and decision support Risk analysis and management

Resource planning

Competition Monitor competitors and market directions

Set pricing strategy in a highly competitive market

Finance planning and asset evaluation Time series analysis

Fraud detection and management Auto insurance

Money laundering

Medical insurance

Credit card fraud

Detecting telephone fraud


www.reinforcebi.com

64


Other Applications Text mining (news group, email, documents) and Web analysis. Medicine: disease outcome, effectiveness of treatments

Analyze patient disease history: find relationship between diseases

Sports IBM Advanced Scout analyzed NBA game statistics (shots blocked, assists,

and fouls) to gain competitive advantage for New York Knicks and Miami Heat

Astronomy JPL and the Palomar Observatory discovered 22 quasars with the help of

data mining

Web site/store design and promotion Find affinity of visitor to pages and modify layout

Molecular/Pharmaceutical Identify new drugs


www.reinforcebi.com

65

Data mining/analytic tools.

The little known Open Source.



Machine learning

1

http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/ 66




Machine learning

2





Machine learning

3





Machine learning





Machine learning

http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/

76




Machine learning


77




Machine learning


78




Machine learning


79




Machine learning


80




http://www.lauradhamilton.com/machine-learning-algorithm-cheat-sheet

Machine Learning Algorithm Cheat Sheet

81





82





http://www.lauradhamilton.com/machine-learning-algorithm-cheat-sheet

83




Machine learning map

http://scikit-learn.org/stable/tutorial/machine_learning_map/ 84




Introduction to KNIME

85

http://www.kdnuggets.com/2016/02/gartner-2016-mq-analytics-platforms-gainers-losers.html




Download KNIME

86




KNIME installation

87




KNIME Versions

88




Select analytics platform

89




KNIME installation

90




KNIME installation

91




KNIME Quick Start จาก docs

92




Node from KNIME

93




KNIME Uninstalling

94





95




KNIME Quick Start

96




File Reader

97




Color Manager

98





99




Node icons

100




Workflow Import

101




Export KNIME workflow(s)

102




Yours Workspace

103




Installing KNIME Extensions

104




KNIME workflow

105




Node & Status

106




Anatomy of KNIME

107




KNIME Hotkeys

111




Workshop 1 Data Preprocess

112

Preprocess & Views




ARFF Reader

113




Pie Chart (interactive)

114




Sorter

115




Row Sampling

116




Bootstrap Sampling

117




Shuffle

118




Equal Size Sampling

119




Workshop 2 Read & write

120

Read -> Select -> Write




File Reader

121




Column Filter (Manual Selection)

122




Column Filter (Wildcard/Regex Selection)

123




Column Filter (Type Selection)

124




Row Filter

125




CSV Writer

126




Workshop 3 Data Manipulate & View

127

Data Manipulate & View.




Workshop 3 (set 1)

128




File Reader

129




Column Rename

130




Case Converter

131




Rule Engine

132




Workshop 3 (Set 2)

133




Column Combiner

134




Cell Splitter

135




Cell Splitter by Position

136




Cell Splitter

137




String Replacer

138




Database Writer

139




Workshop 3 (Set 3)

140




Column Resorter

141




Number To String

142




String To Number

143




Double To Integer

144




CSV Writer

145




Workshop 3 (Set 4)

146




Numeric Binner

147




Histogram (Interactive)

148




Histogram (Interactive)

149




Color Manager

150




Scatter Plot

151




Shape Manager

152




Scatter Plot after shape assign

153




Size Manager

154




View Box Plot

155

Right - click




Statistics from box plot

156

Right - click




Conditional Box Plot

157




Parallel Coordinates

158




Interactive Table

159




Line Plot

160




Scatter Matrix

161




Pie chart

162




Workshop 4 Data Preparation

163




File Reader

164




Missing Value treatment

165




Row Sampling

166




Partitioning

167




Concatenate example

168




Shuffle

169




Normalizer

170




Normalizer (Apply)

171




CSV Writer

172




Workshop 5 Classification model

173




ARFF Reader

174




Naïve Bayes Learner

175




Testing set

176




Naïve Bayes Predictor

177




What is Entropy?

178




CSV writer (prediction)

179




ROC Curve

180




View ROC Curve

181




Interactive Table

182




Decision Tree Learner

183




Decision Tree Predictor

184




ROC Curve

185




Normalizer

186




Missing Value treatment

187




MLP Learner

188




ROC Curve

189




Most important variables

190




Classification evaluation

191

http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/




Workshop 6 Clustering

195




CSV Reader

196




Conditional Box Plot

197




Missing Value Treatment

198




K-Means setting

199




View clustering data

200




Testing with ANOVA

201




CSV Writer (cluster assign)

203




Hierarchical clustering

204




Workshop 7 Associate rule

205




CSV Reader

206




Rules Result

207




Rule Evaluation

208

http://www.listendata.com/2015/12/market-basket-analysis-with-r.html




Rule Evaluation

209




Workshop 8 Data science case

211




Q & A

212


Data Science Process - HDC : Health Data...

Documents

Transcript of Data Science Process - HDC : Health Data...