Handbook of psychology volume 7 educational psychology

Additional Forms of Contemporary Intervention

Research Evidence

In this section we single out for critical examination three

other methods of empirical inquiry, along with their resulting

forms of evidence, which are thriving in psychological and

educational intervention research today. These are the case

study, the demonstration study, and the design experiment.

The Case Study

Case study research—consisting of the intensive (typically

longitudinal) study and documentation of an individual’s

“problem” of interest, along with the (typically unsystematic)

introduction of various intervention agents designed to ad-

dress the problem—is not a new methodology in psychology

and education. Examples can be observed throughout the

history of these disciplines. Although limitations of the case

study have been known for some time (see Kazdin, 1981;

Kratochwill, 1985), it continues to flourish in intervention re-

search. It is not the use of case study research that is prob-

lematic, but rather the claims and generalizations for practice

that result from this methodology. An illustration of its appli-

cation in research on treatment of children’s disorders will

alert the reader to case study concerns.

Considerable research has been conducted focusing on the

treatment of posttraumatic stress disorder (PTSD) in adults,

but treatment of children has not been as extensive. Never-

theless, children are very likely to experience PTSD; the dis-

order can be diagnosed in young children and needs to be

treated. Although several treatments might be considered,

eye movement desensitization and reprocessing (EMDR)

therapy (Cocco & Sharp, 1993) seems to be getting increased

attention as a particularly effective treatment for stress-

related problems in children (see Greenwald, 1999). But does

EMDR qualify as an evidence-based treatment of PTSD in

children? In a recent review of research in that area (Saigh, in

press), only one study could be found that offered empirical

support for this form of therapy.

EMDR typically involves asking the person to imagine a

traumatic experience while at the same time visually tracking

566

Educational / Psychological Intervention Research

the finger movements of the psychologist. While this activity

is going on, the child may be instructed to state negative and

positive statements about him- or herself, with an emphasis

on coping. In a case study in this area, Cocco and Sharpe

(1993) used a variant of EMDR through an auditory proce-

dure for treatment of a 4-year, 9-month-old child who was as-

saulted. Following the assault, the child was reported to

experience nightmares, bed-wetting, carrying a toy gun, and

sleeping in his parents’ bed. During the therapy the child was

told to imagine the event and track the therapist’s finger

movements. In addition, the child was asked to draw a picture

of the assailants and a picture of a superhero for the treatment

sessions. An auditory procedure was used in which the thera-

pist clicked his finger at the rate of 4 clicks per second for

12 s. At the same time, the child was asked to look at the pic-

ture he had drawn and then to verbalize what the hero was

doing to the assailants as the therapist clicked his fingers. It

was reported that the child stabbed the picture while verbal-

izing that he was killing the assailants. The treatment was

considered successful in that the child no longer experienced

nightmares, no longer wet his bed, and did not need to sleep

with his parents or carry a toy gun. At a six-month follow-up,

however, the child was reported to wet the bed and sleep in

his parents’ bed. 

What can be concluded from this case study? In our opin-

ion, very little, if anything. In fact, the desperate clinician

looking for an effective treatment might be misled into

assuming that EMDR is an effective procedure for this child-

hood disorder when, in fact, more tightly controlled and

replicated (i.e., CAREful) research would suggest effective

alternatives. Among the variety of treatment procedures

available for children experiencing PTSD, behavior-therapy

techniques have emerged as among the most successful, based

on clinical research (Saigh, Yasik, Oberfield, & Inamdar,

1999). In particular, flooding therapy, a procedure investi-

gated in a series of controlled single-participant research

studies with replication of findings across independent partic-

ipants (e.g., Saigh, 1987a, 1987b, 1987c, 1989), has emerged

as an effective treatment for this serious disorder.

Again, our negative view of case studies is related to the

generalizations that are often made for practice. Within a

proper context, case-study research may be useful in generat-

ing hypotheses for future well-controlled investigations

(Kratochwill, Mott, & Dodson, 1984). Moreover, not all case

studies are alike on methodological dimensions, and the re-

searcher using these methods has available options for improv-

ing the inference that can be drawn from such studies.

Table 22.2 shows some of the methodological features that

suggest levels of inference (varying from high to low) that can

be applied to both design of case studies and interpretation

of data from these investigations (see also Kazdin, 1998).

Nevertheless, case studies fall into the “demonstration study”

category (to be discussed next) and differ from another often-

confused “single case” design, the systematically implemented

and controlled single-participant study, in which replication

and (in many instances) intervention randomization are central

features (see Kratochwill & Levin, 1992).

The Demonstration Study

Two ubiquitous examples of demonstration studies in educa-

tional contexts include (a) an instructional intervention that is

introduced within a particular classroom (with or without a

nonintervention comparison classroom) and (b) an out-of-

classroom special intervention program that is provided to a se-

lected group of students. The critical issue here (which will be

revisited shortly) is that with only one classroom receiving spe-

cial instruction or only one group participating in a special pro-

gram, it is not possible to separate the effects of the intervention

or the program from the specific implementation of it.

TABLE 22.2

Levels of Inference Generally Associated With Various Research Methodology and Outcome Features

Characteristics

Low Inference

High Inference

Types of data

Subjective data

Objective data

Assessment occasions

Single point measurement

Repeated measurement

Planned vs. ex post facto

Ex post facto

Planned

Projections of performance

Acute problem

Chronic problem

Effect size

Small

Large

Effect impact

Delayed

Immediate

Number of participants

N

ϭ 1

N

Ͼ 1

Heterogeneity of participants

Homogeneous

Heterogeneous

Standardization of treatment

Nonstandardized treatment

Standardized treatment

Integrity of treatment

No monitoring

Repeated monitoring

Impact of treatment

Impact on single measure

Impact on multiple measures

Generalization and follow-up assessment

No formal measures

Formal measures

The Concept of Evidence

567

Levin and Levin (1993) discussed interpretive concerns

associated with the evidence derived from a demonstration

study in the context of evaluating the outcomes of an acade-

mic retention program. They are encompassed in three

CAREful-component questions rolled into one: Was the pro-

gram effective? With an emphasis on “effective,” one can ask,

“Relative to what?” for in many program evaluation studies

frequently lacking is an appropriate Comparison (either with

comparable nonprogram students or with participants’ pre-

program data). With an emphasis on “the,” one can ask, “Do

you mean this single implementation of the program?” for

generalization to other program cohorts or sites is not possible

without an Again and again replication component. Finally,

with an emphasis on “program,” one can ask, “Can other, non-

program-related, factors account for the observed outcomes?”

for without program randomization and control, one cannot

readily Eliminate other potential contributors to the effects.

Levin, Levin, and Scalia’s (1997) report of a college retention

program for academically at-risk minority students provides

an example of a demonstration study. Like our previous case

study example, because of the uncontrolled nature of the

study and the one-time implementation of the program, any of

the documented positive outcomes associated with program

participants cannot be regarded as either scientifically credi-

ble or generalizable to other implementations of the program.

In that sense, then, and as Levin et al. (1997, pp. 86–87)

pointed out, a report of their particular program and its out-

comes can indicate only what happened under a unique and

favorable set of circumstances. It clearly is not an indication

of what to expect if a similar program were to be implemented

by others with other college students elsewhere.

The Design Experiment

Also considered here is the classroom-based design experi-

ment, originally popularized by Collins (1992) and by Brown

(1992) and welcomed into the educational research commu-

nity by Salomon (1995, p. 107) and by various research-

funding agencies (e.g., Suter, 1999). In design experiments

research is conducted in authentic contexts (e.g., in actual

classrooms, in collaboration with teachers and other school

personnel), and the experiment is not fixed in the traditional

sense; rather, instructional-design modifications are made as

desired or needed.

On closer inspection, one discovers that from a strict ter-

minological standpoint, design experiments neither have a

design nor are experiments. In particular, in conventional re-

search usage, design refers to a set of pre-experimental plans

concerning the specific conditions, methods, and materials to

be incorporated in the study. In a design experiment, how-

ever, any components may be altered by the researcher or

teacher as the investigation unfolds, as part of “flexible de-

sign revision” (Collins, 1992): “It may often be the case that

the teacher or researchers feel a particular design is not work-

ing early in the school year. It is important to analyze why it

is not working and take steps to fix whatever appears to be the

reason for failure” (p. 18).

Similarly, in conventional research terminology, experi-

ment refers to situations in which participants are randomly

assigned to the two or more systematically manipulated and

controlled conditions of a study (e.g., Campbell & Stanley,

1966). In a design experiment, however (and as will be ex-

panded upon shortly), appropriate randomization and control

are conspicuously absent, which, in turn, does not permit a

credible attribution of outcomes to the intervention proce-

dures under investigation. Take, for example, Collins’s

(1992) description of a hypothetical design experiment (with

numbers in square brackets added for identification in the

subsequent paragraph):

Our first step would be to observe a number of teachers, and to

choose two who are interested in trying out technology to teach

students about the seasons, and who are comparably effective [1],

but use different styles of teaching: for example, one might work

with activity centers in the classroom and the other with the entire

class at one time [2]. Ideally, the teachers should have compara-

ble populations of students [3]. . . . Assuming both teachers teach

a number of classes, we would ask each to teach half her classes

using the design we have developed [4]. In the other classes, we

would help the teacher design her own unit on the seasons using

these various technologies [5], one that is carefully crafted to fit

with her normal teaching style [6]. (Collins, 1992, p. 19).

From this description, it can be seen that in a design ex-

periment there are numerous plausible alternative explana-

tions for the observed outcomes that compete with the

intervention manipulation of interest. Consider the following

components of Collins’ hypothetical study: 

Regarding [1], how can “comparably effective” teachers

be identified, let alone defined? In [2], teachers differing in

“teaching style” differ in countless other ways as well; one,

for example, might have brown hair and the other gray, which

could actually be an age or years-of-experience proxy. Re-

garding [3], again, how are student populations “compara-

ble,” and how are they defined to be so? For [4] through [6],

assuming that the two teachers could both teach their respec-

tive classes in precisely the prescribed manner (a tall assump-

tion for a within-teacher instructional manipulation of this

kind) and that individualized teacher-style “crafting” could be

568

Educational / Psychological Intervention Research

accomplished (another tall assumption), any result of such a

study would represent a confounding of the intervention ma-

nipulation and specific teacher characteristics (as alluded to in

[2]), so nothing would be learned about the effects of the in-

structional manipulations per se. Even worse, in the rest of

Collins’s (1992, p. 19) example, the described instructional

manipulation contains no less than seven sequentially intro-

duced technology components. Consequently, even if teacher

effects could be eliminated or accounted for, one would still

have no idea what it was about the intervention manipulation

that produced any outcome differences. Was it, for example,

that students became more engaged by working on the com-

puter, more attuned to measurement and data properties and

accuracy by collecting information and entering it into a

spreadsheet, more self-confident by interacting with students

from other locations, more proficient writers through book

production, and so on? There is no way of telling, and telling

is something that a researcher-as-intervention-prescriber

should want, and be able, to do.

The design experiment certainly has its pros and cons.

Those who regard intervention research’s sole purpose as

improving practice also often regard research conducted in

laboratory settings as decontextualized and irrelevant to nat-

ural contexts (see Kazdin, 1998). In contrast, the design

experiment is, by definition, classroom based and classroom

targeted. On the other side of the ledger, design experi-

ments can be criticized on methodological grounds, as well

as on the basis of design experimenters’ potential to subordi-

nate valuable classroom-instructional time to the (typically

lengthy and incompletely defined) research agenda on the

table. In our view, design experiments can play an informa-

tive role in preliminary stages of intervention research as

long as the design experimenter remembers that the research

was designed to be “preliminary” when reporting and specu-

lating about a given study’s findings. For a true personal

anecdote of how researchers sometimes take studies of this

kind and attempt to sneak them “through the back door”

(Stanovich, 1999) into credible-research journals, see Levin

and O’Donnell (2000).

In fact, design experiments and other informal classroom-

based studies are incorporated into the model of intervention

research that we propose in a later section. On a related note,

we heartily endorse Brown’s (1992, pp. 153–154) research

strategy of ping-ponging back and forth between classroom-

based investigations and controlled laboratory experiments

as a “cross-fertilization between settings” (p. 153) for devel-

oping and refining contextually valid instructional theories

(see also Kratochwill & Stoiber, 2000, for a similar view of

research in school psychology). The reader must again be

reminded, however, that scientifically credible operations

(chiefly, randomization and control) are not an integral part

of a design experiment, at least not as Collins (1992) and

Brown (1992) have conceptualized it.

Summary

For much intervention research as it is increasingly being

practiced today, we are witnessing a movement away from

CAREful research principles, and even away from prelimi-

nary research models principally couched in selected obser-

vations and questionable prescriptions. Rejection of the

scientific method and quantitative assessment may be leading

to inadequate graduate training in rigorous research skills

that are valued by many academic institutions and funding

agencies. At the same time, it should not be forgotten that

even qualitatively oriented researchers are capable of engag-

ing in mindless mining of their data as well. Vanessa Siddle

Walker (1999) recently distinguished between data and good

data, which, in our current terminology, translates as, “Not all

evidence is equally credible.”

Just as in other fields informed by bona fide empirical in-

quiry, in psychology and education we must be vigilant in dis-

missing “fantasy, unfounded opinion, ‘common sense,’

commercial advertising claims, the advice of ‘gurus,’ testimo-

nials, and wishful thinking in [our] search for the truth”

(Stanovich, 1998, p. 206). Case studies, demonstration studies,

and design experiments have their place in the developmental

stages of intervention research, as long as the researchers

view such efforts as preliminary and adopt a prescription-

withholding stance when reporting the associated outcomes.

We cannot imagine, for example, well-informed researchers

and consumers taking seriously instructional prescriptions

from someone who proudly proclaims: “Let me tell you about

the design experiment that I just conducted. . . .”

In the next section we offer some additional reflections on

the character of contemporary intervention research. In so

doing, we provide suggestions for enhancing the scientific

integrity of intervention research training and the conduct of

intervention research.

ENHANCING THE CREDIBILITY OF

INTERVENTION RESEARCH

Psychological/Educational Research Versus

Medical Research

Very high standards have been invoked for intervention out-

come research in medicine. The evidence-based intervention

movement was initiated in medical research in the United

Kingdom and embraced more recently by clinical psychology

Enhancing the Credibility of Intervention Research

569

(Chambless & Ollendick, 2001). An editorial in the New Eng-

land Journal of Medicine spelled out in very clear and certain

terms the unacceptability of admitting anecdotes, personal

testimony, and uncontrolled observations when evaluating

the effectiveness of a new drug or medical treatment:

If, for example, the Journal were to receive a paper describing a

patient’s recovery from cancer of the pancreas after he had in-

gested a rhubarb diet, we would require documentation of the

disease and its extent, we would ask about other, similar patients

who did not recover after eating rhubarb, and we might suggest

trying the diet on other patients. If the answers to these and other

questions were satisfactory, we might publish a case report—not

to announce a remedy, but only to suggest a hypothesis that

should be tested in a proper clinical trial. In contrast, anecdotes

about alternative remedies (usually published in books and mag-

azines for the public) have no such documentation and are con-

sidered sufficient in themselves as support for therapeutic

claims. Alternative medicine also distinguishes itself by an ide-

ology that largely ignores biologic mechanisms, often disparages

modern science, and relies on what are purported to be ancient

practices and natural remedies. . . . [H]ealing methods such as

homeopathy and therapeutic touch are fervently promoted de-

spite not only the lack of good clinical evidence of effectiveness,

but the presence of a rationale that violates fundamental scien-

tific laws—surely a circumstance that requires more, rather than

less, evidence. (Angell & Kassirer, 1998, p. 839)

Angell and Kassirer (1998) called for scientifically based

evidence, not intuition, superstition, belief, or opinion. Many

would argue that psychological research and educational inter-

vention research are not medical research and that the former

represents an inappropriate analog model for the latter. We dis-

agree. Both medical research and psychological/educational

research involve interventions in complex systems in which it

is difficult to map out causal relationships. Reread the Angell

and Kassirer (1998) excerpt, for example, substituting such

words as “child” or “student” for “patient,” “amelioration of a

conduct disorder or reading disability” for “recovery from can-

cer of the pancreas,” “ingested a rhubarb diet” for “ingested a

rhubarb diet,” and so on. Just as medical research seeks pre-

scriptions, so does psychological and educational research;

and prescription seeking should be accompanied by scientifi-

cally credible evidence to support those prescriptions (see,

e.g., the recent report of the National Research Council, 2001).

Furthermore, as former AERA president Michael Scriven

poignantly queried in his contemplation of the future of educa-

tional research, “Why is [scientifically credible methodology]

good enough for medical research but not good enough for

educational research? Is aspirin no longer working?” (Scriven,

1997, p. 21).

Moreover, the kinds of researchable questions, issues, and

concerns being addressed in the medical and psychological/

educational domains are clearly analogous: Is one medical

(educational) treatment better than another? Just as aspirin

may have different benefits or risks for different consumers,

so may an instructional treatment. And just as new medical

research evidence may prove conventional wisdom or tradi-

tional treatments incorrect (e.g., Hooper, 1999), the same is

true of educational research evidence (e.g., U.S. Department

of Education, 1986; Wong, 1995). Citing the absence, to date,

of research breakthroughs in psychology and education (in

contrast to those that can be enumerated in medicine) is, in

our view, insufficient cause to reject the analogy out of hand.

It is possible that many people’s initial rejection of the

medical model of research as an apt analogue for psychologi-

cal/educational research results from their incomplete under-

standing of what constitutes medical research. In the

development of new drugs, clinical trials with humans pro-

ceed through three phases (NIH, 1998). In Phase I clinical tri-

als research is conducted to determine the best delivery

methods and safe dosage levels (including an examination of

unwanted side effects) of a drug. Phase II clinical trials address

the question of whether the drug produces a desired effect.

Phase III trials compare the effects of the new drug against the

existing standards in the context of carefully controlled ran-

domized experiments. Thus, although medical research in-

cludes various forms of empirical inquiry, it culminates in a

randomized comparison of the new drug with one or more al-

ternatives to determine if, in fact, something new or better is

being accomplished (see, e.g., the criteria from the Clinical

Psychology Task Force for a similar view). A recent example

of this work is the evaluation of the effects of Prozac on de-

pression in comparison to other antidepressants. The phases of

clinical trials described here roughly parallel the stages in the

model of educational research that we now propose.