Conceptual review and meta-analysis of school effectiveness

The Concept of School Effectiveness

School A can be called more effective than school B when school A does better in achieving its core objectives. This is the general definition of school effectiveness that emerges from about three decades of research conducted under this label.

It is a definition that requires more precision, and, moreover, even when further clarified, remains debatable.
First of all, the comparison in the general definition should be “fair”, which means that goal-attainment measures should be adjusted for possibly diverging entrance characteristics of the units (i.e. the students) on which these measures are taken. In research practice this means that outcome measures that reflect goal-attainment are to be corrected for prior achievement, proxy’s like scholastic aptitude or social-economic status or both (Bosker, 1995). This is also known as the “value-added” perspective in determining school effectiveness. The fact that the determination of school effectiveness is generally conceptualized as a comparative endeavour should also be explicitly underlined. Schools are compared among themselves on value-added effectiveness criteria rather than being judged by employing absolute standards.
Secondly, “goal attainment” in schooling could mean many different things. What goals? Being the obvious question to be answered. Cheng (1996) illustrates the complexity of this question by referring to various functions of schooling (technical/economic functions, human/ social functions, political functions, cultural functions and educational functions), each of which likely to emphasize different categories of educational objectives. Goal-attainment of schooling can thus be defined in terms of varying long-term societal effects and in terms of more direct attainment categories at the end of a fixed period of schooling. But also with respect to these more direct attainment categories there are various possibilities and priorities to be set among them: cognitive vs. non-cognitive outcomes and, within the cognitive domain, various types of knowledge and skills, varying from basic subject-matter mastery to higher order problem solving skills.
Thirdly, from an organization-theoretical perspective the concept of school effectiveness could be defined in even broader terms. According to typologies on organizational effectiveness (Cameron & Whetten, 1983; Scheerens, 1992; Cheng, 1996). The “goal-attainment” model, implied in the above general definition, is just one of several models of organizational effectiveness. The goal-attainment model uses “productivity” of the organization’s primary process as the central effectiveness criterion. Other models, like the resource-input model and the organization process model emphasize other criteria, namely procured resources and student intake and smooth internal functioning respectively.
Finally, it should be noted that in school effectiveness research, rather than in some more applied contexts of determining school effects, the inquiry does not stop by, for example rank-ordering schools on the basis of their value-added performance but, in addition, aims to answer the question to which specific characteristics of school organization or instruction, such differences could be attributed to. School effectiveness is intrinsically a causal concept, in which the black box of “a school” is opened in order to reveal specific variables that relate to the effect criterion. Gradually, as will be explained in subsequent sections school effectiveness research has lead to the development of causal models in which these various characteristics are related to each other and the effect criterion.
These four points show that in order to use school effectiveness in more operational terms in a research context, there are various choices to be made:

1. 
   choice of an effect criterion, where one could choose among indicators of the more long term outcomes of schooling, between goal-attainment or alternative criteria of organizational effectiveness and between various domains of goal-attainment;
   
2. 
   choice of one or more variables to adjust effect measures;
   
3. 
   choice of hypothetical effectiveness enhancing conditions as possible attributes of differences in effect measures and a particular causal model in which these conditions are interrelated.
   

Choices with respect to (a) are of a substantive nature, whereas choices with respect to (b) and (c) are of a more technical nature. In the various strands of educational effectiveness research in which school effectiveness research has become integrated, a rather limiting choice with respect to the effectiveness criterion is evident. In the large majority of educational effectiveness studies, achievement in basic school subjects, reading and writing in the native language and mathematics, is used as effect-criterion. It should also be noted that the bulk of educational effectiveness research is carried out at the level of primary and lower secondary schools. In this context a discussion on the educational significance of this limiting choice of effect criteria in educational effectiveness research will be left out (see Scheerens, 1992; Cheng, 1996). The implicit position in continuing this review, however, being that achievement in basic school subjects is sufficiently important to figure out how such outcomes are best accomplished.

The choice of hypothetical effectiveness enhancing conditions depends on the disciplinary background from which research is conducted. The evolvement of different traditions of educational effectiveness research in what shall be referred to as “integrated school effectiveness research” and the implications for conceptual modelling will be addressed in the next section.

Before describing this development, however, it is necessary to briefly refer to a set of critical questions about the scope of the concept of school effectiveness, some of which were raised in an article by Ralph and Fenessey (1983). Can a school be called effective on the basis of achievement results measured only at the end of a period of schooling, or should such a school be expected to have high performance at all grade levels? Can school effectiveness be assessed by examining results in just one of two school subjects, or should all subject matter areas of the curriculum be taken into account? And: shouldn’t one restrict the qualification of a school being effective to consistently high performance over a longer period of time, rather than a “one shot” assessment at just one point in time?

Fortunately all of these questions are amenable to empirical research, This type of studies that have to do with the consistency of school effects over grade-levels, teachers, subject-matter areas and time have sometimes be referred to as “foundational studies” (Scheerens, 1993), because they are aimed at resolving issues that bear upon the scope and “integrity” of the concept of school effectiveness.
A recent review of such foundational studies is given in Scheerens & Bosker, 1997 (ch. 3). Their results concerning primary schools are presented in Table 1. Consistency is expressed in terms of the correlation between two different rank orderings of schools. Results are based on arithmetic and language achievement.

Table 1: Consistency of school effectiveness (primary level); from Scheerens & Bosker, 1997 (ch. 3).

Type of consistency	Average correlation
across time (stability) (1 or 2 years)	r = .70 (.34 - .87)
across grades	r = .50 (.20 - .69)
across subjects	r = .70 (.59 - .83)

The results summarized in Table 1 indicate that there is a reasonable consistency across cohorts and subjects, while the consistency across grades is only average. Results measured at the secondary level likewise show reasonably high stability coefficients (consistency across cohorts), somewhat lower coefficients for stability across grades (for example in a French study (Grisay, 1996) coefficients based on value-added results were .42 for French language and .27 for mathematics. The average consistency between subjects at the secondary level was somewhat lower than in the case of primary schools (r about .50). This phenomenon can be explained by the fact that, at the secondary level, different teachers usually teach different subjects, so that inconsistency is partly due to variation between teachers.
The few studies in which factor analysis was used to examine the size of a stable school factor relative to year specific and subject specific effects have shown results varying from a school factor explaining 70% of the subject and cohort specific (gross) school effects (Bosker, 1990), to 39% (Van der Werf & Guldemond, 1995) and 25% (Luyten, 1994).

The picture that emerges from these studies on the stability and consistency of school effects is far from being undifferentially favourable with respect to the unidimensionality of the school effects concept. Consistency is fair, when effects at the end of a period of schooling are examined over a relatively short time interval. When grade-level and subject-matter area are brought into the picture consistency coefficients tend to be lower, particularly when different teachers teach different grades or subjects. Especially at the secondary school level school effects are to a considerable amount to be seen as teacher effects.

The message from these “foundational studies” is that one should be careful not to overgeneralize the results of school effectiveness studies when only results in one or two subject matter areas at one point in time are measured. Another implication is that hypothetical antecedent conditions of effects are not only to be sought at the school organizational level, but also at the level of teaching and the teaching and learning process.

Turning back now to the definition of school effectiveness that was given in the first sentence of this section, the results of the subsequent conceptual analysis can be summarized as a set of qualifications of this general characterization:

• 
  the comparison of schools is to be based on value-added effect measures;
  
• 
  the choice of just a few core objectives as a basis for the effect criterion indicates a limitation of most empirical school effectiveness studies with respect to more encompassing perspectives on organization effectiveness;
  
• 
  not just the effectiveness of school “as a whole”, but also specific school and instructional characteristics are the antecedent conditions in school effectiveness research;
  
• 
  as empirical studies have shown that the unidimensionality of school effects is somewhat uncertain, care should be taken not to overgeneralize the results of studies involving just a few subject-matter areas, sub-systems of the school and one point in time when effects are measured.