The Common European Framework in its political and educational context What is the Common European Framework?

The DIALANG self-assessment scales
Source
Most of the self-assessment statements used in DIALANG were taken from the English
version of the Common European Framework (Draft 2, 1996). In this respect, DIALANG
is a direct application of the Framework for assessment purposes.
Qualitative development
The DIALANG Working Group on Self-Assessment
1
reviewed all CEF statements in 1998
and chose those which appeared to be the most concrete, clear and simple; North’s
(1996/2000) empirical results on the statements were also consulted. More than a
hundred statements were selected for reading, listening and writing. In addition,
statements about speaking were chosen but as speaking is not part of the present
DIALANG system, they were not included in the validation study described below and
are thus not presented in this appendix.
The wording of the statements was changed from ‘Can do’ to ‘I can’ because they were
to be used for self-assessment rather than teacher assessment purposes. Some of the
statements were modified to simplify them further to suit the intended users; a few new
statements were also developed where there was not enough material in the CEF to draw
on (the new statements are in italics in the tables). All statements were audited by Dr
Brian North, the originator of the statements in the CEF, and by a group of four language
testing and teaching experts before the final wording of the statements was agreed.
Translation
Because DIALANG is a multilingual system, the self-assessment statements were then
translated from English into the other thirteen languages. The translation followed an
agreed procedure. Guidelines for translation and negotiation were agreed;
comprehensibility to learners was a prime quality criterion. Initially, two to three
experts per language translated the statements into their language independently and
then met to discuss differences and to agree a consensus wording. The translations
were forwarded to the Self-Assessment Group whose members had the linguistic
proficiency to additionally cross-check the quality of the translations in nine
languages. The translators were contacted and any questions related to wording were
discussed and modifications agreed.
Calibration of the self-assessment statements
So far, the DIALANG project has carried out one calibration study on the self-
assessment statements. (Calibration is a procedure in which the level of difficulty of
Appendix C: The DIALANG scales
228
1
The group consisted of Alex Teasdale (chair), Neus Figueras, Ari Huhta, Fellyanka Kaftandjieva, Mats Oscarson, and
Sauli Takala.

---

items, statements, etc. is determined statistically and a scale is constructed of them.)
The calibration was based on a sample of 304 subjects (complete test design) who also
took a number of DIALANG tests in Finnish. The SA-statements were presented to them
either in Swedish (for 250 subjects whose mother tongue was Swedish) or in English. In
addition, most subjects could consult the Finnish language version of the statements.
2 
The data was analysed with the OPLM programme (Verhelst et al. 1985; Verhelst and
Glass 1995).
3
The results of the analysis were very good: over 90% of the statements
could be scaled (i.e. they ‘fitted’ the statistical model used). The three self-assessment
scales which were constructed on the basis of the calibration of the statements were
very homogeneous, as indicated by the high reliability indices (Cronbach’s alpha): .91
for reading, .93 for listening and .94 for writing.
4
Similar calibration studies will be carried out when the other 13 languages are
piloted, following the approach developed by the Data Analysis Group. They will show
to what extent the excellent results of the first study can be replicated and whether
there is any tendency for some statements to be consistently better than the others, for
self-assessment purposes.
Although the first calibration study is only one study, it is important to note that it
tells about the quality of more than one language version of the SA statements in
DIALANG. This is because most of the learners studied could choose any, even all, of
the three versions (Swedish, English or Finnish) when completing the self-assessment
part, although most of them probably relied on the Swedish one. Because of the careful
translation procedure, we can safely assume that the SA statements are largely
equivalent across the languages – an assumption which will obviously be tested as part
of the other calibration studies.
Additional evidence for the quality of the DIALANG self-assessment scales – and for
the CEF scales – was obtained by Dr Kaftandjieva by correlating the difficulty values of
the statements in this study with the values for the same statements obtained by
North (1996/2000) in a different context. The correlation was found to be very high
(.83), or even .897, if one strangely behaving statement is excluded.
Document C1 presents the 107 self-assessment statements for reading, listening and
writing which survived the calibration study based on Finnish data. The statements in
each table are ordered in terms of difficulty from the easiest to the hardest. Statements
which were not taken from the Framework are in italics.