FACTORS RELEVANT TO THE ELTS EVALUATION Background
Educational (including the psychometric) factors
ELTS was a proficiency test used in an educational setting, that is at entrance to a British university. The educational questions that needed investigation were: to what extent the test was being used at undergraduate and postgraduate levels, and for temporary attachment by senior academics; and whether it was suitable at these different levels; to what extent it was in use (and found suitable) in different faculties, and whether it was (as had been intended) being used differentially so that test scores deemed proficient in one faculty (or university) were rated not sufficiently proficient in any others. Again, many universities had provision for remedial English tuition for their foreign students; what was the extent of this provision and what use was made by those involved of the ELTS results? Both sensitivity to the ways in which tertiary institutions operated and an understanding of their role and complexity was called for in the project.
In addition, there were other skills that were required. One of these was the analysis of the ELTS test in terms of its relevance to the content of the specialist areas it purported to test. For example, what connection, if any, was there between the test materials in the life sciences module and the textbooks, lectures and so on in university departments of biology, bio-chemistry, botany and so on? And how far were these likely to be constant at the different levels, under- and postgraduate? For analyses of this kind it was necessary to draw on the expertise of those skilled in teaching life sciences at the different levels. The applied linguist could not be expected to carry out the content analysis but he/she would expect to interpret the results.
Another essential skill, which appears to be unique to the testing area but in practice is relevant everywhere in applied linguistics research, is that of familiarity with measurement (the psychometric aspect of educational studies) and with elementary statistics, to which must also be added the skill of data entry and of statistical analysis by computer. Those applied linguists who are consulted about testing projects (as with this ELTS project) are likely to have all three skills, the psychometric, the statistical and computer literacy. If not, then they must call on colleagues or external consultants.
But for the purposes of understanding the working of any test these skills are all essential since they provide the necessary evidence about the test’s workings, often called its validity and reliability. Using these skills it was possible to examine the extent to which ELTS gave its users the decisions they needed, in particular whether it enabled them to make the right choices of candidates, those with adequate English for their studies.
It was also important to investigate whether there was even-handedness across modules, so that one module (say the life sciences module) was no harder than any other (say the social studies module). If it was harder, then it was possible it was right that it should be, but this was a matter for empirical investigation: if the English of life sciences is truly more difficult than the English of social studies, then it is appropriate that the tests should mirror this distinction. If not, and one was in fact more difficult than the other, then we had established a case of irrelevant bias which should be corrected.
ELTS contained four parts, tests of the four skills: reading, writing, listening and speaking. The results offered a profile score (skill by skill) and a global score, combining the four skills. It was important to determine the extent to which the profile scores provided useful (and usable) information to the test users (university departments etc.). Here again what was needed was a statistical analysis across a wide range of candidates of the various scores and a survey (through interview, questionnaire etc.) of the actual use of these different modes of reporting to the institutions.
Since ELTS information was already being used in a large number of tertiary institutions, it was necessary to check how the information was being used. Were the so-called profile scores (separate scores for the four skills of reading, writing, listening and speaking) being used or were institutions interested only in the overall global score?
By the time of the evaluation teachers of English as a foreign language (EFL) were already beginning to consider the needs of those candidates intending to take the test. Inevitably, courses in EFL, both the more general and those specifically designed to provide pre-ELTS tuition, were taking account of what the test involved. The question for the evaluators was whether such washback from the test was beneficial in terms of its influence on EFL teaching or whether it was likely to exert a largely negative influence, for example by overstressing the practice of multiple-choice test questions.
