ESCO Mapping Pilot workshop 4 - ec.europa.eu · ESCO Mapping Pilot workshop 4 03/11/2015 Brussels....

31/07/2015

ESCO - Eurpean Skills, Competences and Occupations

1/

ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015

ESCO Mapping Pilotworkshop 4

03/11/2015Brussels

31/07/2015


2/


Agenda

Time Agenda item Time

10:00 Opening of the meeting 12:45 General conclusions mapping classifications

10:20 Status update mapping pilot

13:45 Coffee break

11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching

11:15 Country specific conclusions mapping classifications

14:45 Next steps & planning

12:00 Lunch 15:00 End of the meeting

31/07/2015


3/


Opening of the meeting

31/07/2015


4/


Approval of meeting minutes July 2nd

• Meeting minutes of the July 2nd workshop• Feedback of Julius de Zeeuw on section Dutch data

• Feedback of Alain Dupuch on section French data

• Feedback of María José Arias Fernández on section Spain data

• Other corrections

31/07/2015


5/


Agenda




13:45 Coffee break





31/07/2015


6/


Status update mapping pilot

31/07/2015


7/


Mapping pilot objectives

• Define a straightforward process to create a mapping

• To prototype tooling to support the mapping process

• To create draft mappings for NL, FR, CZ, and ES (Hospitality and Tourism)

• To validate the draft mappings to learn

• What type of information is retained and what is lost

• What is going well, and what needs improvement

• To test the mappings with some CVs and Job Vacancies

31/07/2015


8/


Mapping process

31/07/2015


9/


Actions from previous workshopTask Who

1 Provide an updated mapping tool• User validation interface• Enhanced ROME occupation proposals based on appélations• Enhanced skill mapping proposals based on skill/occupation

relations

TenForce

2 Participants provide feedback on mappings PESs

3 Select & translate curriculum vitae and job vacancies for matching TenForce

4 Process curriculum vitae & job vacancies PESs

5 Process cross-border curriculum vitae and job vacancy matching TenForce

31/07/2015


10/


Updated mapping tool –strategies to improve proposed mapping

1. SolR based text matching[new things below this line]

2. Automatically add synonyms

3. Better filtering using ISCO-08

4. Match concepts based on context

5. Use hierarchy

6. Use skill relations

31/07/2015


11/


Updated mapping tool- subject based clustering

• Using open source Mallet NLP engine (UMASS)

• Use extracted subjects from Wikipedia to increase number of mapping proposals

• Subject based clustering provides

• Extra context, not solely based on textual similarity

• Boost clarity on mapping proposals

• Can increase the number of false positives

31/07/2015


12/


Update mapping tool- combining different solutions

• Need a normalized total score, so used weighted approach.

• Weights should be configurable• Should depend which metric performs best, e.g. text based matching is better

than context based matching and receives a higher weight…

• But need expert knowledge to make that call

31/07/2015


13/


Updated mapping tool- Concept relations and flooding (explained)

1. For every proposed mapping, e.g. between two occupations

2. These occupations have relations, like hierarchical or occupation-skill relations

3. Look at ‘nearby mappings’ of concepts connected through these relations

4. If these ‘nearby mappings’ have a high value, ‘flood’ some of this value to the target mapping being considered

5. This is iterative, if the user accepts or rejects a proposal, the certainty of the suggestion changes to 1 (accepted) or 0 (rejected) and the flooding can be redone.

31/07/2015


14/


Concept relations and flooding❶

31/07/2015


15/


Concept relations and flooding❷

31/07/2015


16/


Concept relations and flooding❸

31/07/2015


17/


Concept relations and flooding❹

31/07/2015


18/


Concept relations and flooding❺

31/07/2015


19/


Updated mapping tool –enhanced clarity on mapping proposals

31/07/2015


20/



31/07/2015


21/



Reduce uncertainty for proposed mappings

31/07/2015


22/


Mapping tool – workflow

2 Step approach: first basic mapping, second refined mapping

+ Faster to functional result

+ Earlier gap analysis

+ Allow perfecting the mapping at a later stage

- Good proposals visited twice

• Experts can manually add/correct mappings

31/07/2015


23/


Mapping tool - general

• Two step approach towards mapping

• Focus on efficiency in the workflow

• Need for usability improvements

• Features to be added (lessons learned):• Function to add additional mappings

• Filter and search capabilities

• Side-by-side concept browsing

• Ignore stop words/words without meaning

• Configuration of scoring parameters

• Automatically check mappings on logical errors

31/07/2015


24/


Feedback received from participants on data and mapping tool• On the data : expert involvement facilitates

• Validate the mapping outcome

• Assure the quality of the translations (not an objective of the mapping pilot)

• On the process: no comments

• On the mapping tool: overall positive, however:• Approach by the tool differs from current practice

• Need more functionalities in validation interface

• Persist user status in workflow

• Noise level in mapping proposals is sometimes too high

• ISCO filtering: differences due to differences in how concepts are classified (irrelevant when mapping complete classifications)

31/07/2015


25/


Agenda




13:45 Coffee break





31/07/2015


26/


Country specific conclusionclassifications mapping

31/07/2015


27/


NOC specific results

• Detailed analysis per NOC

• Numbers represent what can be done purely using tool

• No manual addition of mappings

• TEG completed mappings

• Noise reduction by tool: 90%

31/07/2015


28/


Common vocabularyParameter Description

Number NOC & NSC Concepts Total number of OCC and KSC from PES to be mapped

Exact Matches Transitive, symmetric, interchangeable

Close Matches Similar, sometimes interchangeable

Broad Match Hierarchical relation

Narrow Match Hierarchical relation

No Match from ESCO (54 – 353) Absolute number of percentage of concepts without match

No Match from NOC Absolute number of percentage of concepts without match

Mandays @ PES # days to complete mapping HOSP

Mandays @ TEG # days to complete mapping HOSP

31/07/2015


29/


31/07/2015


30/


Mapping results – NL NOC

• Received feedback on the use of ISCO, different views on ISCO tagging resulted in need of manual adding of matches outside of HOSP

• Note this is due to the restriction of the mapping pilot to the HOSP sector, the effect of this restriction on a full mapping should be of lesser impact.

31/07/2015


31/


• More NOC concepts, finer granularity

• 50% of ESCO occupations has exact match

• many of NOC occupations are not matched:• Different approach towards cross-sectoral

occupations

• Different interpretation on the scope of the sector (94 are not ESCO HOSP)

• ISCO filtering

• 50% of NOC skills are not matched

• Occupations are broader while skills are narrower

ESCO: 54 OCC & 353 KSCOCC KSC

Number of NOC and NSC Concepts

181 605

Exact Matches 28 94

Close Matches 19 114

Broad Match (ESCO>NOC) 37 41

Narrow Match (ESCO<NOC) 16 285

No Match from ESCO 7 123

No Match from NOC 113/19 362

Mandays @ PES

Mandays @ TEG 0.5 1

31/07/2015


32/


31/07/2015


33/


Mapping results – FR NOC

Granularity mismatch Granularity mismatch

“fiche métier” (Occupation group) “appélations” (occupations)

Many terms Few terms

Significant words decreased in value Mapping base too small

a balance of both?

31/07/2015


34/


• Granularity difference with ESCO• No NPTs at level of appélations• ISCO filtering (364 out of 534

unmatched NOC appélations are not ESCO HOSP)

• 33% additional mapping because subject based clustering (7k)

ESCO: 54 OCC & 353 KSCFROCC

FRKSC


652 374

Exact Matches 2 8

Close Matches 134 64

Broad Match 64 95

Narrow Match 56 507


No Match from NOC 534/170 187

Mandays @ PES

Mandays @ TEG 1 2

31/07/2015


35/


31/07/2015


36/


• Notes:• Missing relation between skills and

occupations

• Missing non-preferred terms

• Most ESCO occupation mapped, many with exact match

• 70% of NOC skills are mapped, but many ESCO skills are not

ESCO: 54 OCC & 353 KSC OCC KSC


115 108

Exact Matches 34 9

Close Matches 40 72

Broad Match (ESCO>NOC) 37 111

Narrow Match (ESCO<NOC) 13 44


No Match from NOC 69 39

Mandays @ PES

Mandays @ TEG 0.5 1

31/07/2015


37/


31/07/2015


38/


• High number of unmatched concepts• ISCO filtering

• Solr rules to be improved

• Mappings done by PES, more restrictive

• Low compatibility of skill descriptions

CZ OCC CZ KSC

No. Nat.Class. Concepts

45 303

Exact Matches 0 101

Close Matches 17 63

Broad Match 4 93

Narrow Match 12 79

No Match from ESCO (54 OCC– 353 KSC)

31 182

No Match from NOC 22 157

Mandays @ PES

Mandays @ TEG 0 0

31/07/2015


39/


31/07/2015


40/


Agenda




13:45 Coffee break





31/07/2015


41/


General conclusionsclassification mapping

31/07/2015


42/


Agenda




13:45 Coffee break





31/07/2015


43/


Status update job vacanciesand curriculum vitae matching

31/07/2015


44/


• Tagging

• Scenario’s

• Evaluation grid

CV-JV Matching

31/07/2015


45/


Task Who Status

Selection of CV/JV TenForce Done

Translation of CV/JV TenForce Done

Annotation of CV/JV PES In progress

Evaluation grid TenForce To do

31/07/2015


46/


31/07/2015


47/


Matching scenario’s

• + additional trials to investigate effects and parameters

1 Input Actor Output MATCH Input Actor Output

CV language A PES A CV tagged PES A JV language B PES B JV tagged PES B

CV tagged PES A MAP CV tagged ESCO

CV tagged ESCO MAP CV tagged PES B


CV language A PES A CV tagged PES A JV language B PES B JV tagged PES B

CV tagged PES A MAP CV tagged ESCO JV tagged PES B MAP JV tagged ESCO


CV language A TF CV tagged ESCO JV language B TF JV tagged ESCO

31/07/2015


48/


Agenda




13:45 Coffee break





31/07/2015


49/


Next steps and planning

31/07/2015


50/


•Homework: processing the CVs and JVs • Before Fri 13 November 2015

•Workshop Tuesday 15 December 2015• Mapping• Matching results : grid• Actions• Conclusions

Next steps and planning

31/07/2015


51/


Thanks for your attendance and participation !

and

your collaboration towards workshop 5 !

ESCO Mapping Pilot workshop 4 - ec.europa.eu · ESCO Mapping Pilot workshop 4 03/11/2015 Brussels....

Documents

Transcript of ESCO Mapping Pilot workshop 4 - ec.europa.eu · ESCO Mapping Pilot workshop 4 03/11/2015 Brussels....