New Teaching conceptual modelling: how to guide the conceptual...
Transcript of New Teaching conceptual modelling: how to guide the conceptual...
Teaching conceptual modelling: how to guide the conceptual
modelling process of students?
Monique Snoeck LIRIS, FEB@KULeuven
1
4
1.Conceptual
Modelling (CM)is difficult... to learn... to teach
Scaffolding(instructional
design)
Feedback(automated)
Simulation(model execution)
Learning Analytics(does it help?)
5
1.Conceptual
Modelling (CM)is difficult... to learn... to teach Why?
Case Study: Conceptual Modelling
• Conceptual modelling is a complex learning task • requires rigorous analytical skills• many solutions to a problem, many routes to finding a good solution
• Current standard tooling is not adequate• Many tools do not provide any form of feedback• Some do focus on model validation as a form of feedback• Simulation tools focus on code (skeleton) generation• Lack of logging functionalities
• Research• Specific tool for educational purposes including logging, predictive analytics
and providing personalized feedback in real-time
6
Conceptual modelling…is a complex learning task analyzing business requirements constructing a semantically correct conceptual model
that reflects the structural and dynamic views of a given domain description
17:40
7
problem : skill transferability
• Teaching “experience” (domain specific knowledge) is difficult
• Some aspects cannot be gained with reading and lecturing alone, e.g. dynamic representation of a system-to-be
• Lack of technical insights
16:36
8
novice expert
Contributing factorsCM requires combining competences from different areas
Each contributing factor introduces own challenges
17:42
9
Domain knowledge
Modeling knowledge
Language knowledge
Skill
Teaching complex knowledge
• Compartmentalization• Atomistic domains of learning
• Fragmentation• Sequential teaching
• Transfer Paradox• Missing generalization skill to integrate
knowledge into unfamiliar aspects of the tasks
16:36 14
16
1.Conceptual
Modelling (CM)is difficult... to learn... to teach
Scaffolding(instructional
design)
Learning outcomes in CM Each learning outcome can be categorized by the following parameters:
• Knowledge level• Cognitive level• Content area (which can be defined by means of related keywords)
Example:
LO1: The student should be able to interpret a given UML diagram (translate from a model to a text)
Knowledge level: Conceptual Cognitive process: UnderstandingContent area: Models
17
Content areas tree
Bogdanova & Snoeck, 2018
18
Content areas and scaffolding levels
Adapted from Bogdanova & Snoeck, 2017
19
ClassObject Attribute
Class level
Inheritance
Relationships level: generalization
Binary Association
Recursive Association
Aggregation
N-aryAssociation
Association Class
Relationships level: association
Simple Model
Model level: simple
Complex Model
Model level: complex
Pattern
GENERAL
KNOWLEDGE
Revised Bloom’s Taxonomy
CreateEvaluateAnalyzeApply
Understand
Remember Define, Recall, Identify
Discuss, Explain, Match
Use, Practice, Execute
Examine, Analyze, Compare
Check, Verify, Critique
Design, Build, Improve
Cognitive dimension
Adapted from Krathwohl, 2002
20
Revised Bloom’s Taxonomy
Create
EvaluateAnalyzeApply
Understand
Remember
Cognitive dimension
Adapted from Krathwohl, 2002
21
Revised Bloom’s Taxonomy
CreateEvaluateAnalyzeApply
Understand
Knowledge dimension
Remember
Adapted from Krathwohl, 2002
22
Knowledge levels
Factual Terminology, notation and basic elements
Revised Bloom’s Taxonomy
Sources of information on modelling (textbooks, standards and thematic websites)
Modelling notation(s)
General modelling conventions – how to name classes, attributes, associations, etc.
Terms and their definitions according to the literature
23
Knowledge levels
Terminology, notation and basic elements
Conceptual Interrelationships between elements, principles and theories
Revised Bloom’s Taxonomy
Conceptual differences between various terms
Classification of information
Fundamental principles of conceptual modelling
Commonly used modelling patterns
24
Knowledge levels
Terminology, notation and basic elements
Interrelationships between elements, principles and theories
Procedural How to do it: Subject-specific techniques and skills
25Revised Bloom’s Taxonomy
Step-by-step guidelines or algorithms for performing a modelling task at any stage
Criteria for use of a method or technique when solving a particular kind of modelling task
25
Knowledge levels
Terminology, notation and basic elements
Interrelationships between elements, principles and theories
How to do it: Subject-specific techniques and skills
Metacognitive Awareness about your own knowledge and learning/thinking strategies
Revised Bloom’s Taxonomy
Knowledge of cognition as such
Strategic knowledge for learning the subject
Self-knowledge
26
Revised Bloom’s TaxonomyCognitive levels
Define, Recall, IdentifyRemember
Identify a false statement in the list of term definitions. (Remember – Factual – General)
Define properties characterizing inheritance (overlapping/disjoint, etc.)(Remember – Factual/Conceptual – Generalization)
Examples from educational resources:
27
Revised Bloom’s TaxonomyCognitive levels
Discuss, Explain, MatchUnderstand
Examples from educational resources:
Give an example of an instance belonging to more than one class(Understand – Conceptual – Class, Generalization)
Interpret cardinalities of the given simple class diagram(Understand – Factual/Conceptual – Model (Simple))
28
Revised Bloom’s TaxonomyCognitive levels
Use, Practice, ExecuteApply
Examples from educational resources :
Use a given textual analysis technique to identify all the relevant classes in the text(Apply – Procedural – Class)
Modify the attributes for a new kind of subclass(Apply – Conceptual – Generalization)
29
Revised Bloom’s TaxonomyCognitive levels
Examine, Analyse, CompareAnalyse
Examples from educational resources:
Compare main approaches to conceptual model design(Analyse – Conceptual/Procedural – General)
Identify the similarities between the given class diagrams(Analyse – Conceptual – Model (Simple))
30
Revised Bloom’s TaxonomyCognitive levels
31
Check, Verify, CritiqueEvaluate
Examples from educational resources:
Find the mistakes in given class diagrams and justify the answer(Evaluate – Conceptual – Model (Complex))
Find which of the given object diagrams is valid/invalid, according to the given model(Evaluate – Conceptual – Model (Simple)
Current state: what do we teach?
Bogdanova & Snoeck, 2017
“Remember” and “Evaluate” are heavily underrepresentedMetacognitive knowledge level is not represented at all, factual – almost not represented
36
Current state: what do we teach?
Bogdanova & Snoeck, 2017
The majority of tasks address model and relationships levels; MOOCs “like” simple models, while books and exams “like” complex ones
37
Current state: what do we teach?
UnderstandCreate
Conceptual
RememberEvaluateAnalyze
FactualMetacognitive
Apply
Procedural
Result: uneven scaffolding; lack of constructive approach
VS
38
39
1.Conceptual
Modelling (CM)is difficult... to learn... to teach
Scaffolding(instructional
design)
Simulation(model execution)
Modelling & Simulation• Understanding a model (e.g. for validating it against requirements)
requires “picturing” the resulting artefact
16:36 40
????
Specific Problem of CM• Complexity associated with ANALYZING & EVALUATING a CM
• Experienced modellers • Can mentally picture and simulate a model• They can analyse & evaluate its impact
• Junior modellers• lack experience to achieve these skills
• Common solutions• Explanations & Demo’s• Case Studies• Lot’s of exercises with individual feedback
41
Create
EvaluateAnalyzeApply
Understand
Remember
Simulation of requirements?• is considered as an effective instrument
allowing to achieve complex learning goals by
• Visualizing design choices into concrete forms• Learning by experiencing (knowledge that emerges from own practice)• Successful transfer of the skills to real-world environments
16:36
42
CM Simulation environment
• Adapted to learning context makes it suitable for research goals
• the use of a restricted part of UML as proposed by the MERODE methodology (Snoeck et al., 1998): creating executable PIMs (EPIM)
• Structural view: UML class diagram• Behavioural view: multiple interacting statecharts• Interaction model : CRUD + collaboration rules
• starting from a high-level PIM allows removing or hiding details irrelevant for a conceptual modelling view.
16:36
43
Requires three clicks1 2 3
JMermaid Simulation
16:36
44
Current State
• Simulation by means of 3 clicks
2
“Order” is not required to create an instance of an invoice
3
1
45
Built in data storage with query builder
16:36
46
CM Simulation environment
• Fully functional (single click) prototype• Teaching with JMermaid
• single conceptual model generate simulate• test and early defect detection revisit/refine model
18:20 47
Test -> early detection of
defects
Revisit / refine
For more details, see EMMSAD 2012, EMMSAD 2013, and ModelsWard 2013, papers by Sedrakyan, Snoeck
Reflect on
48
1.Conceptual
Modelling (CM)is difficult... to learn... to teach
Scaffolding(instructional
design)
Feedback(automated)
Simulation(model execution)
Feedback in CM
• Personal feedback • + immediate = most demanded form of feedback by students• is the most “expensive” in terms of teacher time• automated “just in time” or “anytime” feedback
• Group feedback• for home works• for group assignment
• peer feedback
49
Feedback in JMermaid
• Observed problem: students cannot interpret the behaviour of the prototype
• Solution: Augment generated prototype with a feedback feature that links results of a model test to its causes in the corresponding part of the model [EMMSAD 2012]
18:01 50
Feedback - Current State• Feedback in the modelling
environment:• Model to text feature• Model validation checks
51
Feedback - Current State• “To buy wholesale products customers need to place an order.
However ordering is not required for buying a retail product.”
18:21
52
2“Order” is required to create an instance of an invoice
3
1
Feedback - Current state
18:23
53
Examples of other Artefacts for Feedback• Online tests (Automated !)
• small exercises per topic• individual feedback
• Homework/Assignments• small cases per topic• individual feedback + group feedback
• Self-study exercises / lab exercises• exercises archived from previous home works + lab exercises• model solutions + commented student solutions• individual feedback + group feedback
• Group work• larger, integrated case• Peer, group/individual feedback
• Automated feedback in simulation environment• Individual immediate/on-demand
54
Online Test Feedback Example 55
56
1.Conceptual
Modelling (CM)is difficult... to learn... to teach
Scaffolding(instructional
design)
Feedback(automated)
Simulation(model execution)
Learning Analytics(does it help?)
Research question 1
• Does a feedback-enabled prototyping (simulation) improve modeling knowledge of a novice modeler in terms of his/her capability of assessing a model’s semantic quality ?
• Method empirical evaluation• Assessing the effects on learning outcomes by comparing the test scores with
and without a use of proposed simulation technique
18:01
57
Experimental study
• 5 studies : with participation of 169 final year master-level students from Leuven and Brussels campuses, spanning 3 academic years (2012-2013-2014-2015)
• Dependent variable = model quality
• Experimental design : pre/post test control group design• 4-group factoral with cycle rotation• Extraneous variables: personal characteristics such as gender, previous knowledge and skills [Venkatesh, 2003],
computer self-efficacy• Intended utility: perceived utility and ease of use [Technology Acceptance Model, Davis 89].
• More in [Sedrakyan, G., Snoeck, M., Poelmans, S. (2014)]
18:30
58
ConclusionsH1: Feedback-enabled simulation significantly improves model validation capabilities of a novice business analyst.
confirmed : magnitude of the effect = 2.33 – 4 ( out of 8 )
H2: The use of the prototype has a persisting learning effect on student’s test scores when is no longer used.
confirmed : no significant diff. withoutPT cycle following withPT cycle
H3 : The test scores are not influenced by any particular personal characteristics of users (previous knowledge and gender).
confirmed : no impact of variables on the relative advantage (with PT – without PT) was identified
H4 : The proposed simulation method is suitable for novice business analysts (user acceptance).
confirmed : no impact of acceptance variables on the relative advantage (with PT – without PT) was identified
User preferences: 4.5 – 5.58 (6 - point Likert scale)18:32
59
Research question 2
• Can modeling and validation process affect its outcomes?• How we can adapt teaching guidance to achieve process-oriented guidance (how to
do it right ?) based on learning process observations vs. outcome feedback (is the solution correct? Why(not)?)
16:36
60
Process Mining - Current State• Process Mining of the modelling process• Log of user activities in JMermaid for the group task during a whole
semester’s duration.TIMESTAMP
GROUP ID
SESSION ID
SESSION TYPE
SCORE
ORIGINAL ACTIVITY
ABSTRACTED ACTIVITY
MODELING VIEW
DIAGRAMMING TYPE
19/11/2013 1:54:00 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:54:16 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:55:55 1 Session1 EARLY 6 CREATE DEPENDENCY CREATE S EDG 19/11/2013 2:08:03 1 Session1 EARLY 6 CREATE ATTRIBUTE CREATE S EDG 19/11/2013 2:08:36 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:37:28 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:40:05 1 Session1 EARLY 6 DELETE EVENT EDIT B OET 19/11/2013 4:40:18 1 Session1 EARLY 6 UNDO DELETE EVENT EDIT B OET 19/11/2013 5:09:53 1 Session1 EARLY 6 REDO DELETE EVENT EDIT B OET 19/11/2013 5:10:58 1 Session1 EARLY 6 EDIT ATTRIBUTE EDIT S EDG 10/12/2013 11:04:18 1 Session2 LATE 6 CREATE METHOD CREATE B OET 10/12/2013 11:04:23 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:05 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:23 1 Session2 LATE 6 CREATE TRANSITION CREATE B FSM 12/12/2013 11:02:34 1 Session3 LATE 6 DELETE DEPENDENCY EDIT S EDG 12/12/2013 11:02:41 1 Session3 LATE 6 CREATE DEPENDENCY CREATE S EDG 12/12/2013 11:02:44 1 Session3 LATE 6 CREATE ATTRIBUTE CREATE S EDG 12/12/2013 11:02:47 1 Session3 LATE 6 DELETE STATE EDIT B FSM 12/12/2013 11:03:00 1 Session3 LATE 6 EDIT TRANSITION EDIT B FSM
61
Analysis of logs: example of findings
• Process Mining:• More iteration in the better performing groups (R),
whereas worst performing groups (L) show a more linear pattern
• Better students simulate more• Better students also verify behavioural part of the model through simulation
62 62
Worse Better
Worse
Better
Process Mining of the modelling process63
TIMESTAMP
GROUP ID
SESSION ID
SESSION TYPE
SCORE
ORIGINAL ACTIVITY
ABSTRACTED ACTIVITY
MODELING VIEW
DIAGRAMMING TYPE
19/11/2013 1:54:00 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:54:16 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:55:55 1 Session1 EARLY 6 CREATE DEPENDENCY CREATE S EDG 19/11/2013 2:08:03 1 Session1 EARLY 6 CREATE ATTRIBUTE CREATE S EDG 19/11/2013 2:08:36 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:37:28 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:40:05 1 Session1 EARLY 6 DELETE EVENT EDIT B OET 19/11/2013 4:40:18 1 Session1 EARLY 6 UNDO DELETE EVENT EDIT B OET 19/11/2013 5:09:53 1 Session1 EARLY 6 REDO DELETE EVENT EDIT B OET 19/11/2013 5:10:58 1 Session1 EARLY 6 EDIT ATTRIBUTE EDIT S EDG 10/12/2013 11:04:18 1 Session2 LATE 6 CREATE METHOD CREATE B OET 10/12/2013 11:04:23 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:05 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:23 1 Session2 LATE 6 CREATE TRANSITION CREATE B FSM 12/12/2013 11:02:34 1 Session3 LATE 6 DELETE DEPENDENCY EDIT S EDG 12/12/2013 11:02:41 1 Session3 LATE 6 CREATE DEPENDENCY CREATE S EDG 12/12/2013 11:02:44 1 Session3 LATE 6 CREATE ATTRIBUTE CREATE S EDG 12/12/2013 11:02:47 1 Session3 LATE 6 DELETE STATE EDIT B FSM 12/12/2013 11:03:00 1 Session3 LATE 6 EDIT TRANSITION EDIT B FSM
63
Log of user activities in JMermaid for the group task.
Analysis of logs: example of findings
• Analysis of types of activities in early versus late modelling sessions
64
37%43%
8%12%
0%10%20%30%40%50%60%
PHASE1 PHASE2
create edit
31% 33%
13%23%
0%10%20%30%40%50%60%
PHASE1 PHASE2
create edit
35%22%
18%25%
0%10%20%30%40%50%60%
PHASE1 PHASE2
create edit
Worse Satisfactory Best
Analysis of logs: example of findings
• Dotted charts: analysis of distribution of modelling effort over time:
• Students are deadline driven ...• Better students start earlier• Better students simulate more often
65
BB
B
B
B
W
S
S
W
W
W
WW
AA D1 PD& T LC D2
S
SS
66
1.Conceptual
Modelling (CM)is difficult... to learn... to teach
Scaffolding(instructional
design)
Feedback(automated)
Simulation(model execution)
Learning Analytics(does it help?)
1.Conceptual
Modelling (CM)can be made
easier... to learn
By Scaffolding anddeveloping the analysis
& evaluation skills
Feedback is required to let students understand
simulation results(automated)
Simulation is anexcellent way of developing these
skills
Learning Analytics helpsunderstanding & proved the
effect of interventions
Publications• See http://feb.kuleuven.be/monique.snoeck
• Selected Articles, Presentations & Conferences
• Daria Bogdanova, Monique Snoeck. Domain Modelling in Bloom: Deciphering How We Teach It. The Practice of Enterptise Modelling (2017), pp.3-17.
• Estefanía Serral, Monique Snoeck. Conceptual Framework for Feedback Automation in SLEs. 20th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES-SEEL2016). Tenerife, Spain. 2016.
• Sedrakyan G, De Weerdt J, Snoeck M, Process-mining enabled feedback: “tell me what I did wrong” vs. “tell me how to do it right”, 2016, Computers in Human Behavior, vol. 57, pp. 352 - 376.
• Sedrakyan G, Snoeck M, Poelmans S, Assessing the effectiveness of feedback enabled simulation in teaching conceptual modeling, 2014, Computers and Education, vol. 78, pp. 367 - 382.
• ....
Software• merode.econ.kuleuven.be
Contact: [email protected]
67