Way of the turtle

180
•
Way of the Turtle

---

basis of a relatively small study group because there is a statistical
basis for that inference.
The two major factors that affect the statistical validity of the
inferences derived from a sample of a population are the size of the
sample and the degree to which the sample is representative of the
overall population. Many traders and new system testers understand
sample size at a conceptual level but believe that it refers only to
the number of trades in a test. They do not understand that the sta-
tistical validity of tests can be lessened even when they cover thou-
sands of trades if particular rules or concepts apply to only a few
instances of those trades.
They also often ignore the necessity that a sample be represen-
tative of the larger population as this is messy and hard to measure
without some subjective analysis. A system tester’s underlying
assumption is that the past is representative of what the future is
likely to bring. If this is the case and there is a sufficiently large sam-
ple, we can draw inferences from the past and apply them to the
future. If the sample is not representative of the future, the testing
is not useful and will not tell us anything about the likely future
performance of the system that is being tested. Thus, this assump-
tion is critical. If a representative sample of 500 people is sufficient
to determine who the new president is likely to be with a 2 percent
margin of error using a representative sample, would polling 500
attendees at the Democratic National Convention tell us anything
about the voting of the overall population? Of course it wouldn’t
because the sample would not be representative of the population
—it only includes Democrats whereas the actual U.S. voting pop-
ulation includes many Republicans who were not included in the
sample; Republicans probably will vote for candidates different
On Solid Ground
•
181

---

from the ones your poll indicated. When you make a sampling mis-
take like this, you get an answer, perhaps even the one you want,
but it is not necessarily the right answer.
Pollsters understand that the degree to which a sample truly
reflects the population it is intended to represent is the key issue. Polls
conducted with samples that are not representative are inaccurate,
and pollsters get fired for conducting inaccurate polls. In trading,
this is also the key issue. Unfortunately, unlike pollsters, who gen-
erally understand the statistics of sampling, most traders do not.
This is perhaps most commonly seen when traders paper trade or
backtest over only the very most recent history. This is like polling
at the Democratic convention.
The problem with tests conducted over a short period is that dur-
ing that period the market may have been in only one or two of the
market states described in Chapter 2, perhaps only in a stable and
volatile state, in which reversion to the mean and countertrend
strategies work well. If the market changes its state, the methods
being tested may not work as well. They may even cause a trader
to lose money. So, testing must be done in a way that maximizes
the likelihood that the trades taken in the test are representative of
what the future may hold.