Way of the turtle

•
179
•
twelve
ON SOLID GROUND
Trading with poor methods is like learning to juggle while 
standing in a rowboat during a storm. Sure, it can be done, but it is
much easier to juggle when one is standing on solid ground.
N
ow that you are aware of some of the major ways in which you
can get inaccurate results from backtests, you may be won-
dering: “How can I determine what I might actually be able to
achieve?” or “How do I avoid the problems described in Chapter
11?” or “How do I test the right way?” This chapter will discuss the
general principles of doing proper backtesting. A thorough under-
standing of the underlying causes of the backtesting predictive
errors discussed in Chapter 11 is an important prerequisite for this
chapter, and so it is a good idea to reread that chapter carefully if
you only skimmed it the first time.
At best, you can get a rough sense of what the future holds by
looking at the results of historical simulation. However, fortunately,
even a rough idea can provide enough edge to enable a good trader
to make a lot of money. To understand the factors that affect the
margin of error, or degree of roughness, for your ideas, you need to
look at a few basic statistical concepts that provide the basis for his-
Copyright © 2007 by Curtis M. Faith. Click here for terms of use. 

---

torical testing. Since I am not a big fan of books filled with formu-
las and lengthy explanations, I will try to keep the math light and
the explanations clear.
Statistical Basis for Testing
Proper testing takes into account the statistical concepts that affect
the descriptive ability of the tests as well as the limitations inher-
ent in those descriptions. Improper testing can make you overcon-
fident when in reality there is little or no assurance the test results
have any predictive value. In fact, bad tests may provide the wrong
answer entirely. 
Chapter 11 covered most of the reasons why historical simula-
tions are at best rough estimates of the future. This chapter shows
how to increase the predictive ability of testing and get the best
rough estimate possible. 
The area of statistics relating to inference by sampling from a
population is also the basis for the predictive potential of testing
through the use of historical data. The basic idea is that if you have
a sufficiently large sample, you can use measurements made on
that sample to infer likely ranges of measurements for the entire
group. So, if you look at a sufficiently large sample of past trades
for a particular strategy, you can draw conclusions about the likely
future performance of that strategy. This is the same area of statis-
tics that pollsters use to make inferences about the behavior of a
larger population. For example, after polling 500 people drawn ran-
domly from every state, pollsters draw conclusions about all the vot-
ers in the United States. Similarly, scientists assess the effectiveness
of a particular drug for the treatment of a particular disease on the