Kusuma Ratna Fellowship Programme Evaluation

EXECUTIVE SUMMARY

July  2014  

1. Introduction

The Kusuma Ratna Fellowship Programme supports high achieving students from under-privileged backgrounds to progress to higher education. The programme was launched in Sambalpur in Odisha in 2009. The programme comprises a combination of mentoring support, a financial stipend and access to a network of other fellows. Fellowships are awarded on the basis of financial need and academic attainment: only students from families with an annual income less than INR 75,000 are eligible to apply. In addition, students at Grade 10 need to have achieved a score of at least 60% in all subjects. At Grade 12, scores of 60% for arts subjects or 70% for science and commerce subjects are required. Students may apply for a fellowship at two points: following completion of Grade 10 or Grade 12. The rationale for targeting the fellowship programme is informed by research evidence, which shows that these years represent a critical period for students in their progression through the secondary school system. Most students leave school with a Secondary School Certificate at the end of Grade 10 and many do not participate in education beyond this point (ASER, 2012). Obtaining a Higher School Certificate at Grade 12 facilitates access to higher or vocational education. Since its inception, one hundred students have been awarded fellowships each year. The evaluation, which was undertaken between August and December 2013 by the London School of Economics, focused on the first four cohorts (2009-2012).

 

2. Objectives The overall purpose of the evaluation was to capture the broad impact of the fellowship programme in order to inform its future development. The evaluation explored the direct and indirect impact of the fellowship programme on a) beneficiaries; b) unsuccessful applicants; and c) the social networks of successful and unsuccessful applicants. More specifically, the evaluation aimed to measure: • The direct impact of the fellowship programme on fellows’ academic performance, subjective well-being, expectations

about future careers and wages, and interactions with social networks about education and career choices; • The indirect impact of the fellowship programme on households (e.g. time spent by parents and children on chores) and

parental attitudes to education; and • The indirect influence of the programme on fellows’ social networks, including friends and younger siblings, particularly in

relation to views about education, jobs and career choices. 3. Methodology 3.1 Study design A key challenge for the evaluation was to produce reliable data on the impact of the fellowship programme in the absence of a control group. Ideally, it would be good to know how well fellows would have done without the support of a fellowship. Against this background, researchers selected an evaluation design that compared students who received a fellowship with a comparable group of students who were eligible but were not successful because their scores fell just below a pre-selected cut-off point1. Comparing successful with unsuccessful applicants around the cut-off point allowed for any differences in outcome to be attributed to the fellowship programme. This method is referred to as a regression discontinuity design. 3.2 Sample Findings are based on data collected from three cross-sectional surveys and administrative data on four cohorts of applicants to the fellowship programme between 2009 and 2012. Administrative data was held by SIDI, a non-governmental organisation funded by Kusuma to deliver the fellowship scheme. (i) Student sample From a random sample of all students (n=1,595) who applied to the fellowship programme in the first four years, outcomes were compared between a sample of successful and unsuccessful applicants around the cut-off point described above. This produced a sample of 289 applicants (of which 133 were above the cut-off, and 97 were below). (ii) Parent sample A further survey was conducted of 453 parents of 563 applicants to the fellowship programme. The parent sample was almost equally divided between parents of successful applicants (n=224) and parents of unsuccessful applicants (n=229). (iii) Social network sample A survey was conducted of neighbours (n=402), friends (n=415) and siblings (n=91) of applicants (successful and unsuccessful) to the fellowship programme (producing a total peer sample of n=908). 4. Key findings 4.1 Selection of fellows Overall, the academic and financial criteria were effectively implemented. On average, successful applicants scored 7.7 points higher in the Secondary School Certificate and Higher Secondary Certificate compared with unsuccessful candidates. Similarly, the average family income of fellows was INR 3,203 lower than their unsuccessful counterparts. 4.2 Drop-outs The main predictor for dropping out of the programme was the academic stream chosen (science, commerce or arts). 137 students dropped out of the fellowship programme (a drop out rate of 34%) due to failure to meet the minimum grades required for continued participation (60% for commerce and arts subjects, and 70% for science subjects). Of the 137 fellows who dropped out, the vast majority were Science fellows (91% N=125). This suggests that the higher score required for science subjects may be too strict compared with the other two streams.   1 Scores between 0 (low) and 100 (high) capture Secondary School Certificate scores, financial need, and performance in a written test and interview. The top 100 applicants who meet the criteria for selection are admitted into the fellowship programme and the score of the 100th candidate in any given batch represents the ‘cut off’ point.  

 

Findings also show that students who drop out of the fellowship programme are not more likely to drop out of school. Consequently, termination of the fellowship does not appear to have an adverse impact on student trajectories through the secondary school system. 4.3 Impact on Fellows Educational attainment Based on Grade 12 examination results and results of a PISA-based test administered to assess mathematical reasoning, findings show that fellows and non-recipients performed equally well. One possible explanation for this unexpected finding relates to students’ use of their time following the award of a fellowship. Post-award, fellows spend fewer hours on household chores, allocate more time to leisure pursuits (reading books, newspapers and magazines). Nevertheless, it could be argued that, as a result of these leisure activities, students gain a more rounded education that goes beyond narrow measures of attainment. Alternatively, this finding could be interpreted as lending some support to the notion that fellowships can reduce students’ incentive to study harder. There is, at least, no evidence to suggest that fellowships reduce student attainment. Income and career expectations Fellows have higher expectations about their future careers and likely salaries following completion of higher education compared with unsuccessful applicants. Evidence suggests that these expectations are not unrealistic. The fellowship award appears to foster the view that participation in higher education represents a worthwhile investment. Well-being Fellows do not report more satisfaction with life compared with non-recipients. There was, however, a significant difference in respondents’ views about their anticipated satisfaction with life in 5 or 10 years’ time. On average, and on a scale of 1 (low) to 10 (high), fellows reported higher expected levels of satisfaction (by 1.3 points) compared with unsuccessful applicants. This suggests that, compared with non-recipients, fellows feel more optimistic about their future. 4.4 Parents Household chores and time-use The award of a fellowship appears to have a differential impact on the use of time within households. While fellows spent less time on chores and more time on reading books, newspapers and magazines, parents appear to compensate by spending more time on chores and less time on leisure pursuits, particularly time spent watching television. Attitudes to education Having a child who has been awarded a fellowship has a positive impact on parents’ attitudes to the benefits of education. Parents of fellows are significantly more likely to agree with the statement that ‘if the oldest child is successful, the younger children should follow’, and that ‘children should postpone marriage until they have completed their education’. To the extent that these attitudes translate into action, the positive impact of the fellowship programme may cascade to younger siblings. Similarly, parents of fellows are more likely to engage with their peer group in terms of discussing their children’s education and career choices, and encouraging their neighbours’ children to apply for a fellowship. This peer group activity mirrors that of fellows, and highlights the important role of fellows and their parents in providing information about, and encouraging access to, the fellowship programme. 4.5 Social networks Influence on peer group networks Fellows have an important role to play in passing on information and encouraging others to apply to the fellowship programme. Compared with unsuccessful applicants, they are more likely to tell others, particularly friends, about the fellowship programme, and to encourage them to apply. Those encouraged by fellows were also more likely to actually submit an application.Students in the networks of fellows scored significantly higher on the PISA-based test for mathematical reasoning, compared with students in the networks of non-recipients. The fellowship programme therefore has positive spillover effects on the educational attainment of fellows’ social networks. 2 The Programme for International Student Assessment (PISA) is a worldwide study of 15-year-old students to evaluate knowledge and skills.

 

 

 

Kusuma Trust UK

Mezzanine Level l 48-49 St James’s Street l London SW1A 1JT T: +44 (0)20 7569 1920 l www.kusumatrust.org

5. Conclusions Overall, the Fellowship programme has had a positive direct and indirect impact as follows: Fellows are more likely to place a higher value on education in terms of their career aspirations and anticipated incomes. Fellows anticipate achieving a higher income as a result of completing higher education (as compared with secondary education). Overall, fellows are more likely to view higher education as a high-return investment. Fellowships did not lead to an increase in attainment as measured by school marks and an administered test but neither did it lead to a reduction in attainment. However, it did lead to a reallocation of time use: fellows tended to spend less time on household chores and more time on reading books, newspapers and other leisure pursuits. This impact could be interpreted as positively contributing to a more rounded education for fellows, or as a sign that fellowships may reduce students’ incentive to study harder. Fellows have an important role to play in disseminating information about, and facilitating access to, the fellowship programme. They were more likely to encourage friends and others in their social networks to apply for a fellowship, and this encouragement was more likely to result in a completed application. The fellowship award has a positive impact on parental attitudes to the benefits of education: they are more likely to view education as a valuable investment, and to agree that younger siblings should follow the success of older siblings. Parents of fellows are also more likely to agree that children should postpone marriage until they have completed their education. The fellowship programme may therefore exert an important and indirect influence on future decision-making in families about children’s education trajectories. Fellowships have an indirect and positive impact on academic attainment among fellows’ peer group networks. Although the mechanism that might explain this impact is unclear, evidence suggests that exposure to fellows led to an increase in performance, as measured by a PISA-based test for mathematical reasoning. 6. Implications In terms of the future development of the fellowship programme, evidence on reasons for drop-out indicates that it may be fairer to introduce a minimum 60% score for all subjects. The fellowship programme is successful in reaching its target population and any changes to the criteria should be carefully considered. Such changes may need to be introduced to take into account changes in average local incomes. Care should be taken to ensure that agreed selection criteria continue to be implemented in a fair and transparent way. There is a need for further research to explore the long-term impact of the fellowship programme on the education and career trajectories of fellows and fellows’ younger siblings, and on parents and wider social networks. References ASER (2012) Understanding Access to Education, Sursa Block (Hardoi, Uttar Pradesh), New Delhi: ASER Centre.