Learner selection - Bridge1. High achievers. In general the high achievers at selection, remain high...

Learner selectionFindings and reflections on learner

selection from a few evaluation studies.

Paul Hobden, PhD & Sally Hobden, [email protected] [email protected]

School of Education University of KwaZulu-Natal

1

Presentation to BRIDGE focus group May 2012

Overview

1. Purpose of selection

2. Example projects

3. Predicting achievement from maths tests

4. Looking deeper at schools and individuals

5. Some factors that cause blips in trends

©Hobden - BRIDGE talk May 20122

Selection for what?

Learners are given the status of being “selected’. The question is • what have they been selected for ?• who are the learners from the selectors viewpoint? • what is the learners viewpoint?

Learner• Bursary to new school with good teachers and resources• Financial freedom from paying fees• Special academic support• Recognition of my talent

Project• Someone who is underachieving at present and has potential • Someone who will take-up science or maths careers• Someone who will not achieve without our help• Someone who is disadvantaged and needs our help.• Someone who deserves a quality education


3

Some notes on the results and graphs The small numbers of learners in each school make robust statistics at

school and individual level impossible. They are only indicators.

Individual learner high and low scores (outliers) have a big effect on means etc. when there are only a few learners.

A statistical significant difference does not always give a meaningful difference

These results are preliminary, are in process of being validated and are open to alternative interpretations.

Please do not quote without permission of authors.

©Hobden - BRIDGE talk May 2012 4

Do initial tests predict NSC success?

Example A: Independent NGO School selected top 12 learners in grade for extra Maths and Science and English tuition. Voluntary and attendance approximately 80%.Total of 130 sessions over three years of grade 10, 11 & 12. Table indicates number of learners who sat the exam (did not drop out) and level of pass.

No significant difference between control and project group. Those who were good at start passed and those poor failed in both cases.

Control Project

Quintile Sat Pass Sat PassUpper 12 11 25 21

1 19 12 6 3

2 20 6 4 2

Lower 8 0 2 0


0

10

20

30

40

50

60

0 2 4 6 8 10 12 14

Total -Science

Average(All schools)

Average( All Schools -Academy)

2009 T 1

2009 T 2

2009T 3

2009 T 4

2010 T 1

2010 T 2

2010 T 3

2010 T 4

2011 T 1

2011 T 2

2011 prelim

s Matric

Mean Schools 38.9 34.1 50.4 38.6 34.8 35.0 44.3 38.7 47.5 42.4 40.0 41.0Mean After school group 53.5 43.7 56.1 47.4 46.8 43.8 48.8 47.2 51.9 49.7 49.0 53.7

Example B: Good School extension programme

Private school with excellent results. Learners bussed in to school three time a week for M, S & E. No significant difference. Those good at start remained good.


Example 3 : Learners given bursaries to excellent schoolsIf all learners in a year group are taken together there is not a strong correlation between maths marks used for selection and final NSC mark of learners.

There is a weak positive correlation between the Mathematics entrance test mark and the final NSC mark (r=0.314; n= 70,) p<0.05).

There is a positive correlation between the English entrance test mark and the final NSC mark (r=0.586; n= 69, p<0.00).

1009080706050403020100

Mat

hs N

SC

100

90

80

70

60

50

40

30

20

10

0

R Sq Linear = 0.098


Fine grain analysis In general it appears as if little change or intervention effects.

The selection tests don’t appear to have strong correlations with NSC results

These are aggregated results for full cohorts of learners.

However, learners come from different schools, different backgrounds and go to different schools etc.

Need to disaggregate marks and look at school and learner level.


Maths selection

Mark in NSC

Maths selection test over 60% N = 49 Mean = 61.86 S.D. 16.19

Maths selection test beween 40 and 50% N = 63 Mean = 50.92 S.D. 12.09

1. High achievers.In general the high achievers at selection, remain high achievers. They have already shown they are good in under resourced schools so they continue in new schools with added advantage of better quality resources.

High achievers: If the learners score very well on the entrance then good chance that they will do well at NSC.

Low achievers:If the learners have low score at start then they generally have a low chance of success.

Those in the middle either increase, decrease or stay the same! No pattern or relationship.


2. Educationally disadvantaged:Where selected learners are entering from genuinely disadvantaged schools, the better predictor of their success could be their command of English (the LOLT of their new school).

Maths NSC English NSCEnglish entrance test

Pearson Correlation .303 .469Sig. (2-tailed) .027 .000N 53 53

Maths entrance test

Pearson Correlation .030 .104Sig. (2-tailed) .831 .457N 53 53

These schools takes in many learners some of whom come from disadvantaged backgrounds and have low entrance scores.

In this case, the Maths entrance test does not correlate significantly with either of the final Grade 12 scores However the English selection test correlates weakly but significantly with both Grade 12 English and mathematics scores.


3. Already in schoolWhere a general group (high and low achievers) learners already have strong existing links with the school they continue as before.

Learners had already done Grade 8 and 9 at the school so the selection tests are more a reflection of their trajectory in the school programme than a true starting point. The selection tests are reasonably strongly and significantly correlated with the final marks in the NSC in both Maths and English.


Pearson Correlation

.600 .502

Sig. (2-tailed) .000 .000N 73 73

Maths entrance test

Pearson Correlation .517 .272*

Sig. (2-tailed) .000 .025N 68 68


4. Economic disadvantage but not educational disadvantageLearners drawn from existing ex-model c schools or good township schools where they have been doing well.


Pearson Correlation

.495 .682

Sig. (2-tailed) .259 .092N 7 7

Maths entrance test

Pearson Correlation .902 .867

Sig. (2-tailed) .006 .012N 7 7

In this case, the English entrance test correlated strongly (but not significantly) with the English NSC mark.

The Maths entrance test mark correlated very strongly and significantly with both the English and Maths NCS marks.


5. Some don’t follow the patternsSome students overcome a very poor Maths selection test result and achieve. Refer to top left quadrant. These 13 students obtained less than 50% for selection but passed NSC with over 60% in Mathematics


OUTLIERS

Looking at a whole cohort of learners some do very well. For example two learners who were above position 100 in selection moved up to about 6th in the NSC.

MathematicsEntrance test QPiE Gr 10 Gr 12 final

8098 39 55 1

8112 27 13 2 Top 15 in entrance test

8019 1 3

8109 31 1 4 Top 15 in first QPiE test

8025 39 10 6

8139 8 6

8119 103 55 6 Up 97 places

8084 109 2 8 Up 101 places

8043 13 3 10

8095 18 4 10

8028 8 12

8107 31 17 12

8115 43 13 12

8027 109 66 15

8056 43 25 15

8102 4 72 16

8121 43 17 17

8011 9 10 19

8076 39 34 19

8071 35 7 20

8087 82 13 22

8120 69 13 22

8138 74 112 23

8142 20 55 24

8021 48 25 26

8004 82 66 26

8096 95 26

8104 35 88 29

8063 126 66 29 Up 97 places

8134 24 23 31

8111 69 21 31 Up 38 places

8150 8 72 33

8030 55 33

8040 74 19 33


6. READINESSSuccessful selection does not mean learners are ready.

Project & Classmate Total percent Area: Life Area: Energy Area: Earth Area: Matter Basic itemsOutcome:

InvestigationsOutcome:

KnowledgeProject n =76

Mean 62.20 69.74 62.94 54.84 62.06 78.12 60.04 56.25

Classmaten =309

Mean 50.85 59.90 54.64 43.48 45.99 67.84 45.12 47.71

Total N =385

Mean 53.09 61.84 56.28 45.73 49.16 69.87 48.07 49.40

Science readinessTesting the cohort at selection reveals areas of deficit. Where this is done with existing learners in school, it also reveals that classmates also have major deficits. Consequently teachers don’t recognize and sort out the gaps as the selected project learners are similar to the norm.

Science readiness


Mathematics readiness

N Minimum Maximum Mean S.D

Grade 5 level work 75 38 100 79.59 14.474Grade 6 level work 75 17 100 64.78 20.462Grade 7 level work 75 20 100 61.49 18.794Numbers & operations 75 14 100 71.40 22.027Patterns & Algebra 75 0 100 71.93 18.501Shape & Space 75 0 100 71.05 22.668Measurement 75 0 100 52.19 19.502Data Handling 75 0 100 70.79 22.965High School MathsReadiness score

75 31 92 68.82 14.807

Example of baseline test revealing areas of weakness and deficit e.g. Measurement.

Basically the learners selected do not have 30% of the basic skills and knowledge required to access and successful take advantage of grade 8 teaching.


Difficult to have one treatment plan Readiness is not the same across schools or within schools.

Histograms show number of learners in each mark category


FACTOR The school: Same project but school effect

Many dramatic gains Started off with very high marks, mostly unsustained NOTE: Light green bar is mark at selection and dark blue bar indicates the difference to NSC mark. If below line indicates mark has gone down by the indicated amount.

The intakes at schools were very different. Aggregating to obtain project means hides interesting relationships to selection.

Progress at one school has been significant compared to other school.

English marks in two different schools


FACTOR Learning area: Improvement could depend on a teacher or learning area (same learners)

The majority of the selected learners have done well in ENGLISH and have improved on their scores at selection.

The majority of learners in MATHS have actually regressed and scores have decreased.

Same school, same intervention but different teacher and learning area.

English Maths

Same school, same learners


FACTOR : Unrealistic expectations

In general, individual learners marks are stable across the years. They maintain their relative position in class in a learning area. However, in some schools the grade 11 mark peaks (for bursary applications?) and then there is a dip at the NSC. Maybe learners are under false impressions that they will do well.


FACTOR Where learner comes from A1 and A2 : Been in school for a few years but have low

selection test marks B Learner who is new to the school but from a good feeder

school and was selected because of good selection test marks. C Learner from a disadvantaged school with low selection test

results.


21

Learner B is doing English FL and is maintaining her good marks.

The others are doing English FAL.

This has allowed them to achieve well and reach marks around 70%.

Learner C from a relatively disadvantaged school has shown similar improvement to Learners

A1 and A2 who have had the benefit of a good school for longer.


What can be learnt about selecting learners for programmes ?

What you want from the learner at the endpoint should determine the selection criteria.

The playing field is not level. Depends where the learners comes from and where the learner is going.

Maths tests are good predictors of success for high achievers i.e. if doing really well now, the chances are they will do really well later.

If learner is from disadvantaged background and poor school then language is better predictor.

Some learners are unpredictable.

All learners appear to have gaps and areas in which they are not ready for the grade related curriculum.

We do not have a predictor of undetected talent and potential. Current tests do not help as they simply confirm what we already know. Time for new tests such as aptitude and reasoning tests.

The current interventions are not able to make significant changes to learners trajectories. They maintain them but rarely make significant impacts. Time to change the way we do things.


Learner selection - Bridge1. High achievers. In general the high achievers at selection, remain high...

Documents

Transcript of Learner selection - Bridge1. High achievers. In general the high achievers at selection, remain high...