Enhancing fairness through a social contract

Spann (1990)

By Mahsa Farahanynia

PhD student

Allameh Tabataba'I University

2015

Fairness Generally speaking, fairness is a man-made concept, a

philosophical ideal which does not exist in the natural world.

An ideal fair test would measure something accurately (be unbiased and free of error) consistently, and completely (possess construct and content validity, and obtain a sufficient sampling of the domain). It would always be used for its appropriate, intended purpose and be easily and correctly interpreted.

Obstacles to having fair tests

1. A test is a representative sampling of the complex real world domain

2. Examinees have different language and educational background which may advantaged or disadvantaged in L2 test.

3. Different performance might be due to L1 distance from L2 or to unfamiliar format, or to emphasis on a skill area which is not emphasized in the educational system of the examinee’s home country.

Test developers cannot be fair but try to be equitable.

Equitability Equitability in language testing implies the joint responsibility of the test

developer, the test user, and the examinee, in a sort of social contract, in which

the developers promise to maximize validities (construct, content, etc.),

reliability, and practicality of application.

Test developers

Test usersExaminees

What is the role of test developers? In this social contract, test developers: 1. educate the users by providing readable, understandable

interpretations and guidelines for use of their products, including the provision of norms and/or scoring criteria.

2. must be able to justify any claims made about the test3. must encourage test users and examinees to cooperate

with use at the very first stages of test development4. solicit feedback from users and examinees.5. bring users and examinees back into the test

development circle in every aspect, including grading methods.

What is the role of test developers? (Cont.)5. inform users and examinees about the test level, method, and

content, and to standardize test delivery as much as possible so as various forms of test abuse are discouraged.

6. seek from examinees for information about their tests asking following questions: what was their impression of the test? Did it accurately measure what they knew? Did it accurately reflect the kinds of tasks they would be required to perform? Did it accurately relFect the kinds of tasks they would be required to performs? Did it seem to use an appropriate elicitation method? Was it engaging?

7. provide extremely detailed information about scoring methods, criteria, and rater training and built-in safeguards such as multiple ratings.

What is the role of test users?

In this social contract, users 1. do not abuse test by applying the wrong test to the

wrong population, or by removing parts from one or more tests and inserting them into new tests. If they erroneously think the parts are equal to the whole, they may think they retain validity.

What is the role of examinees?

In this social contract, examinees: 1. inform themselves about the test content and method

beforehand to avoid negative task effect2. try to match the level of test to their skill level3. Just like teachers and test users, must be able to

question the perceived content validity or grading methods, doubting about the reliability and accuracy of evaluative system (vs. objective scoring)

Goal of this social contract Promotion of synergy between language test developers

and users to enhance essentials of good language testing (according to ILTA code of practice), namely, validity, reliability and practicality.

It means test developers bring the users and examinees into the process of test development before actual product is on the market to know what are the specific skills they will need to perform what tasks?

Advantages of this social contract Moving from a skills model (Speaking, Listening,

Writing, Reading) towards a competencies model (linguistic, pragmatic, sociolinguistic, and strategic competencies)

Moving towards matching test content and tasks more closely to the specified examinee population

Moving from general proficiency tests to task-based specialized tests

Move from NRT to CRT

Why are cries of unfair still heard?

Problems arise and cries of unfair are heard when this synergy breaks down and a communication gap occurs.

For example:

1. Pupil fail some tests and pass some other: due to a) mismatch between two tests (having different levels of difficulty) or b) mismatch between understanding of methods, goals, and expected performance criteria for the two tests.

2. Students working hard on specific areas (writing) can not pass it: due to different understanding of expected performance criteria on the part of the test developer and the teacher.

Solution of ILTA Code of Practice: Establishing a direct and logical relationship between performance criteria and the manner in which they are operationally represented in the test.

Conclusion

Enhancing social contract (a synergistic, cooperative commination cycle among the test developers, users, and examinees) lead to the improvement of the validity, reliability, and interpretability result in the enhancement of fairness which is the goal of all these developments

Thanks for your attention