Evaluating course books checklists what makes a good checklist the textbook selection in primary and secondary schools

Participation Number of textbooks Percentage
Participated 1852 73.4%
Not participated 671 26.6%
Total 2523 100%
In other items we asked the respondents to describe the situation in their schools –
who took part in textbook selecting process in their schools and who decided the
textbooks to be selected (see table 2).
Caught in the Web or lost in the Textbook 343
Table 2. Decision-making on textbook selection in schools
All responses
only response
Variant of response
frequency percentage frequency percentage
Every individual teacher 121 15.4% 43 5.5%
Teachers together 655 83.6% 429 54.7%
Headmaster 160 20.4% 30 3.8%
Subject head teacher 155 19.8% 10 1.3%
Not know 8 1.0% x x
Most of the teachers took part in textbook selection in their schools: 73% textbooks
the respondents were working with had been selected with their participation. Group
decision-making was the most frequent way of textbook selection in schools in our
sample (84% of respondents); 15% of respondents confirmed the individual choice.
The role of headmasters was important as well. First of all they approved of the
chosen textbooks - almost 50% of respondents confirmed the fact, and not only in a
formal way; was formal approval indicated by 23% of respondents. Furthermore,
20% of teachers claimed that headmasters had taken part right in the textbook
selection process. Nevertheless, an authoritative decision-making was not typical -
only 5% of respondents asserted that the headmasters alone made the decisions. The
differences between primary and lower-secondary schools on the one hand and
upper-secondary schools on the other were proved: primary and lower-secondary
school headmasters took part in selection more frequently (30%) than their
colleagues in upper-secondary schools.
Statistic analysis showed that teacher’s opportunity to select the textbook for his/her
teaching had varied depending on
– the length of teacher’s practice: two groups of younger teachers with (0-5)
and (6-10) years of teaching practice participated in textbook selection less
frequently than other groups with longer practice;
– the school type: the primary school teachers participated in textbook
selection less frequently than teachers from other 4 groups – more than
44% of textbooks they used had been selected without their participation;
– the teacher’s qualification: it was no surprise that fully qualified teachers
had taken part in textbook selection more frequently than non-qualified
respondents.
The most important issue we dealt with in this investigation was to verify if there is
a relationship between ‘teacher’s opportunity to participate in textbook selection’
and ‘teacher’s satisfaction with this textbook’. The hypothesis was formulated:
H 1: The teacher’s satisfaction with the textbook he/she uses in the lessons
depends on his/her opportunity to participate in the textbook selection.
In the course of the research an additional hypothesis was elaborated:
344 Workshop 3: Quality, Evaluation and Selection
H 2: The teachers who selected the textbook individually are more satisfied
with it than the teachers who selected it as members of groups.
The respondents expressed the degree of their satisfaction in 5-point scale, where ‘1’
meant ‘not satisfied at all’ and ‘5’ ‘fully satisfied’. The teachers evaluated the
textbooks they were using in given school year 2003/2004 quite highly: mean scale
value was 3.68 (standard deviation = 1.06, median = 4, number of evaluations =
2523). Both null hypotheses were tested with one-way analysis of variance (H 1: the
value of test statistic F
0.05
(1;2422) = 69; probability p= 0.0001; H 2: F
0.05
(1;1771) =
13.53; p= 0.00024). Both hypotheses have been proved.
Table 3. Teachers’ satisfaction with textbooks
in relation to participation in their selection
Teacher’s participation in textbook selection
The influence of external factors
‘External factors’ in our work meant social, economic and other backgrounds, the
circumstances that accompany the textbook selection while the textbook
characteristics were considered to be ‘internal factors’. We offered 8 external factors
and the respondents had to score each of them in the 5-point scale: ‘1’ meant
minimal influence, ‘5’ maximal influence.
Table 4. The influence of external factors
Factor order arithmetic mean
1. Ministry of Education approval clause 3.63
2. Sufficiency of information regarding books 3.39
3. State of school finances 3.21
4. Commercial leaflets from publishers 3.03
5. Socio-economic status of students families 2.81
6. Parents’ willingness to share costs 2.78
7. Students’ opinions on textbooks 2.18
8. Opportunity to sell used textbooks 2.06
The most important external factor influencing the textbook selection was ‘approval
clause’ from the Ministry of Education. It is understandable in primary and lower-
secondary schools because the state subventions are allocated only to approve
textbooks. But it was the most influencing factor for general secondary school
teachers and secondary technical school teachers, too. The approval clause seems to
be regarded as a sort of quality guarantee or at least as a guarantee of
correspondence with obligatory curricular documents.
Caught in the Web or lost in the Textbook 345
The orders of factors according to primary and lower-secondary school teachers
were the same and their correlations with the orders according to general secondary
and secondary technical school teachers were quite high, too (Pearson’s coefficient
But vocational school teachers rated the influence of external factors in an
entirely different way. The two most influencing factors were according to them ‘the
socio-economic status of students’ families’ and ‘the willingness of parents to share
the costs’. It is not surprising if we consider the fact that students often come from
socially disadvantaged backgrounds. Nevertheless, it is important not to subordinate
the textbook quality to its price and to search for some ways of help. It is also
symptomatic that ‘students’ opinions on textbooks’ took the 7 or the last place:
30% of respondents rated it with minimal score ‘1’ and other 20% score ‘2’; the
mean value was 2.18. The users, who the textbooks are intended for first of all, have
almost no influence on the textbooks they will be working with for the whole year.
Teachers’ needs and interests concerning textbook selection
One of the research purposes was to find out what needs concerning textbook
selection the teachers had, that means:
– if they had enough information available and where this information came
from;
– if they would welcome any help, possibly what kind of help could be useful
for them.
Sufficiency of information on textbooks
It can be said that the respondents regarded themselves as well-informed: 67% of
them chose the answer ‘yes’, moreover 17% ‘definitely yes’, only 11% answered
‘no’, 1% ‘no, not at all’. Statistically important differences were proved between
younger and older teachers. The groups of respondents with (0-5) and (6-10) years
of practice regarded themselves as less informed than the groups with longer
practice.
This begs the question of what kind of information the teachers have got. In our
opinion teachers should have both an overview of newly edited titles for their
subjects and some ideas of basic qualities of these textbooks. Almost never they
have an opportunity to try out the textbooks in their lessons and mostly they don’t
have enough time and energy to analyse e.g. 5 parallel textbooks in detail. That is
why we considered the issue of information resources to be the key problem and
asked where the teachers obtained the data on textbooks from (see table 5).
Workshop 3: Quality, Evaluation and Selection . Information resources concerning textbooks
Information resource
frequency percentage
1. Commercial leaflets from publishers 589 75.1%
2. Visits to bookstores 508 64.8%
3. Colleagues’ opinions 389 49.6%
4. Textbook exhibitions 324 41.3%
5. Dealers from publishing houses 304 38.8%
6. Educational journals 284 36.2%
7. Professional teachers organisations 198 25.3%
8. Other resources 43 5.5%
‘Commercial materials on textbooks’ from publishers were the most important
information resources for teachers from all 5 types of schools: for about 67% of
respondents from primary and lower-secondary schools, 82% from upper-secondary
schools. ‘Textbook exhibitions’ organised by publishing houses are almost as
important as ‘commercial leaflets’ for primary and lower-secondary teachers (62%).
‘Dealers’ who personally attend the schools took theplace in order. It is quite
clear from these data what influence the publishing houses exert upon the textbooks
that are actually used at schools.
In the Czech Republic there is no institution or organisation for systematic
evaluation of textbooks and other teaching materials. Some reviews are published in
professional and educational journals. Maybe that is why only 36% of respondents
used the ‘educational journals’ when looking for the information.
The need for help in textbook selection
Most of the respondents (61%) would welcome some kind of help in the process of
textbook evaluation and selection, about 33% did not need any help. It begs the
question to what extent the teachers are able to evaluate and select the textbooks and
whether they have the necessary knowledge and skills. In the Czech Republic there
are no special courses. Teacher students can gain some knowledge on textbooks
during their studies; teachers in-service can be naturally experienced in evaluating
textbooks. The problem is that even an experienced teacher need not to have an
overview of all the textbooks available and may not have time enough to explore the
textbook thoroughly. We believe the teachers do need some kind of help in form of
the reviews from independent evaluators available.
Some differences were proved among various groups as to the need of help: e.g.
more primary school teachers as well as younger teachers would welcome help than
teachers in other groups. Rather surprisingly the relation between ‘need for help’
and ‘satisfaction with the textbook’ was not proved.
I’m currently studying a module in Materials Development through NILE online. It’s a really worthwhile course so far!
Unit two talked about evaluating materials, specifically course books. We were introduced to a range of checklists that could be used for evaluating a course book, and discussed the pros and cons of each. I can’t imagine everyone would find this topic interesting, but it was really topical for me – in the same week I was given a checklist to evaluate our new course book for teen classes. It gave me the opportunity to reflect on our own evaluation process and suggest some changes if necessary…
What makes a good checklist
We looked at about six different checklists that were listed in McGrath (2002). In some of my jobs I’ve completed evaluations like this one from Harmer (1991:281)

They’re alright. I mean, the fact that a school is bothering to evaluate a course book in the first place is a positive thing. I’ve worked in other jobs where this wasn’t standard practice and the course books used seemed unsuitable.
Reflecting on pitfalls/considerations when making a checklist, McGrath mentioned these points:
Selection of criteria – if you use an existing checklist rather than make your own can you guarantee that all the criteria are relevant to your context? Do you need to adapt it?
Item format – how do you want teachers to evaluate? Will you use open ended questions, tick boxes, a mixture, etc.
Weighting of criteria – are all items equally important? Should some be given greater importance than others?
Value-laden responses – this comes down to subjectivity from teachers based on their own beliefs. It’s tough to control/address I think
Are the learners involved in the evaluation process?
The last point is one of the most important, but the checklists I’ve looked at are meant for teachers.
In addition to McGrath’s list, I think wording is really important. I noticed in the checklists we looked at that the criteria used can be ambiguous or a bit leading, plus they can be too time consuming for teachers. For example, in Harmer’s example above:
‘Is there are sufficient amount of communicative output in the materials under consideration?’
What constitutes a ‘sufficient amount’? Harmer does provide a description, but that involves more investment as a teacher to read up on this. I don’t often have time, so I’d probably just use my own judgement on what a sufficient amount of communicative output might be. But I’d likely supplement a course book anyway to provide more communicative tasks – so is the question actually whether I have to supplement too much?
Weighting and rating
I’d never come across a ‘weighting and rating’ checklist before. Here’s an example from McGrath (2002):
Different items are given a ‘weighting’ depending on how important they are in the evaluation. Then they are given a ‘rating’ depending on how effective the course book is in this area. Weighting multiplied by rating is the evaluation score for that particular item. I prefer this to YES/NO questions, it gives a bit more information and prioritises the most important features of the materials.
Our checklist – is it any good
It turns out that our checklist at the BC is pretty much like Harmer’s:
Pros:
It’s comprehensive. It covers methodology, language, skill, practice, variety, ease of use, exploitability, suitability, assessment and more
Most items in the list are clear and measurable
It’s not that time consuming to complete and comments are optional
Cons:
Some items are still a bit vague or open to interpretation
You’re forced to choose between YES/NO, although you can elaborate on viewpoint with comments
Possible improvements:
We could use a rating system, it might tell us more about how effective the material is
Teachers could devise the checklist together rather than using the existing one, but we don’t get the time to do this
The checklist could be accompanied by an evaluation for learners to complete – this might be coming soon though!
Anyway, this course is certainly making me think more about our approaches. I hope I can put some of what I learn to good use.
Do you evaluate the materials you use? Do you use a check list to do so? Do you think it’s effective?
All comments welcome 
All examples taken from McGrath, I. (2002). Materials evaluation and design for language teaching. Oxford: OUP
WHICH COURSEBOOK TO USE (IF ANY)?
March 18, 2018
Some weeks ago I shared a post about the decision of using a coursebook (or not). If you haven’t read it yet, here is the link To use or not to use coursebooks? Suppose you have decided (or are compelled) to use a coursebook and you are free to decide which coursebook to use, or you have been using a coursebook for some time and you want to change it because it is out-of-date or you are just bored, what issues should you consider to make the best choice? There are thousands of coursebooks in the ELT market, which one will be the best fit for your students and your own teaching approach? Well, this post intends to help you in that choice.
If you have been teaching for some time, you probably know some of the coursebooks, especially those published by the most famous ELT editorials. Maybe some of your colleagues have recommended the ones they are most comfortable with (or the ones you should never consider!). Maybe, there are some limitations in the availability of some of the coursebooks and there are only certain books you and your students can access. Taking these considerations into account, choose two or three coursebooks to evaluate (of course you can evaluate as many coursebooks as you want, but the decision will be harder if you have to evaluate many options).
Remember that after choosing a coursebook and telling your students (and sometimes their parents) which one/ones they will need to buy, it will be troublesome to convince them later that they will have to buy another book (coursebooks are not cheap for some educational contexts) because the first one was a bad choice (your bad choice!). So, take time to evaluate your options so that you end up with the best coursebook.
EVALUATION OF ELT COURSEBOOKS
Tomlinson and Masuhara’s (2004:1) definition of materials evaluation is: “Materials evaluation involves measuring the value (or potential value) of a set of learning materials by making judgements about the effect of materials on people using it”.
Teachers interested in the evaluation of ELT materials can find many frameworks and criteria developed by researchers and coursebook authors (Byrd, 2004; Cunningsworth, 1995; Harmer, 2007; McGrath, 2002; Sheldon, 1988; Tomlinson & Masuhara, 2004; Williams, 1983). However, as McDonough, Shaw & Masuhara (2013: 52) state “(…) there does not seem as yet an agreed set of criteria or procedures for evaluation”.
Advertisements
REPORT THIS AD
In this post I will discuss two frameworks: McDonough, Shaw & Masuhara’s (2013) which attempts to provide a comprehensive framework which might be applied in the majority of ELT situations worldwide; and, Littlejohn’s (2011) framework which aims to evaluate the materials ‘as they are’, not the ‘materials-in-action’ (i.e. as the teacher thinks the material should be used).
COURSEBOOK EVALUATION IN TWO STAGES
McDonough, Shaw & Masuhara (2013) examine materials in two stages: an external evaluation (cover, introduction, table of contents) and a more detailed internal evaluation.
The external evaluation “(…) aims at examining the organization of the material as stated explicitly by the author/publisher by looking at: the ‘blurb’, or the claims made on the cover of the d/students’ book, and the introduction and table of contents” (op. cit.: 54). To achieve this the following information should be gathered (op. cit.: 55-58):
target audience
the proficiency level
the context in which the material will be used
how the language is organized into units, modules, etc.
the authors’ views on language, methodology and the relationship between the language, the language process and the learner
whether the material will be used as the ‘core’ course
whether it is locally available
visuals, layout and presentation
presence of vocabulary lists or appendixes
cultural bias, representation of minority groups
the inclusion of digital materials (CDs, DVDs, downloadable materials), and the inclusion of a teacher’s book and tests
According to these authors, after this stage and having in mind the profile of the learners, we will have enough information to identify if the material is potentially appropriate and is worth a deeper inspection. 
The internal evaluation seeks to find information about:
the presentation of the skills (coverage, proportion, integration)
grading and sequencing (the type of progression, principle underlying progression, levels)
whether discourse skills are included
the ‘authenticity’ of the listening materials
the nature of interaction in oral dialogues (natural or artificial dialogues?)
the relationship of tests and activities to learners’ needs and the content of the book
suitability for different learning styles and access to self-study
the possibility of engagement for learners and teachers in terms of needs, goals, skills and beliefs.
Then, an overall evaluation can be made considering:
the usability factor (possibility of integration to the syllabus)
the generalizability factor (whether the whole coursebook can be used or only a part of it)
based on the previous factor, the adaptability factor
the flexibility factor (how rigid is the sequencing and grading?).
However, these authors as well as Tomlinson (2004), state that the success or failure of a material can only be fully determined after a while and post-use evaluation.
2 COURSEBOOK EVALUATION BASED ON METHODOLOGY AND LINGUISTIC ASPECTS
Littlejohn (2011) does not take into account the ‘superficial aspect’ of materials or their content, his framework focuses on the methodology and the linguistic nature of the coursebook.
The author identifies three levels of analysis: objective description, subjective description and subjective inference.
In level 1 ‘objective description’, we will find the information about:
publication date
intended audience
type of material (general, specific, main course, etc.)
the amount of classroom time required and type of use (self-study, order, etc.)
published form, number of pages, use of colour
components (teacher’s book, student’s book, CDs, etc.)
the division into sections, access (indexes, detailed content, hyperlinks, etc.)
how the sections are distributed between teachers and students, length of sections and any pattern in them.
In the ‘subjective analysis’ in level 2, we analyse what teachers and learners will have to do in each task to test the claims made by the material (a task is defined by this author as any proposal made to students whose aim is bringing about the learning of the L2). For each task we need to identify:
the process, including turn-take (the learners’ role in classroom discourse),
focus: on meaning? form? or both?
mental operations: the mental processes required, like repetition, deducing, hypothesizing
type of classroom participation: alone? pair work? in groups?
and the content of the input and of the learners’ output (written or oral? individual sentences or discourse?), source (from the material? the teacher? or the students?) and nature (grammar explanation? fiction? or personal information?).
Based on the previous levels of analysis we can determine the aims of the material and the basis for the selection and sequencing, the following step is to identify the teacher’s and the learners’ roles implied in it. Finally, a conclusion about the material as a whole can be done (subjective inference).
Littlejohn proposes a further step which is to analyse the teachers/students/institutions situation and their expectations from the material to decide its rejection, adoption, adaptation or supplementation.
MATERIAL ANALYSIS
What aspect should you consider to evaluate the coursebook itself? Littlejohn summarizes them with the following image

With the analysis of the material and the analysis of your teaching context in mind, you might have enough information to reject, adopt, adapt, supplement or use the material with its pros and cons and discuss them with your students (it would be a great source of discussion at least).
References
Littlejohn, a. 2011. The analysis of language teaching materials: inside the trojan horse. In materials development in language teaching (2nd edition). Cambridge university press.
Mcdonough, j., shaw, c., and masuhara, h. 2013. Materials and methods in elt. A teacher’s guide (3rd edition). Wiley-blackwell, uk.
Advertisements
Report this ad
Related
To use or not to use coursebooks?
February 19, 2018
In "English language learning"
Desire to learn: motivation in L2 education
April 18, 2018
In "English language learning"
I Jornada de Investigación sobre la Enseñanza Secundaria
November 14, 2019
In "Demanding teaching"
Posted in: Coursebooks evaluation, English language learning, L2 learning, Teaching English to Speakers of Other LanguagesTagged: Coursebook evaluation, Coursebooks, ELT materials, English language learning, English Language Teaching, TESOL
PUBLISHED BY MARIANGEL CARREÑO
I am a teacher of English in Uruguay. I hold an MA TEFL, a postgraduate degree in Virtual Learning Environments, one in Curriculum and another one in Assessment. I am researching English language learning beyond the class as part of my thesis for the Master in Human Sciences, option Language, culture and society. I am also interested in materials design and the integration of ICT in language learning.