Way of the turtle

Sample Size
The concept of sample size is simple: You need a large enough
sample to make valid statistical inferences. The smaller the sam-
ple, the rougher the guess provided by those inferences; the larger
the sample, the better the guess provided by those inferences.
There is no magic number; there is only larger is better, smaller is
worse. A sample size of less than 20 will produce a large degree of
error. A sample size of more than 100 is much more likely to have
predictive value. A sample size of several hundred is probably suf-
ficient for most testing. There are specific formulas and method-
ologies that will give you specific answers to the question of how
large a sample is required, but unfortunately, those formulas are
not designed for the types of data encountered in trading, where
we do not have a nice neat distribution of potential outcomes such
as the distribution of women’s height in Figure 4-3. 
However, the real challenge does not lie in deciding exactly how
many samples you need. The difficulty arises in assessing the infer-
ences from past data when one is considering particular rules that
do not come into effect very often. So, for these types of rules there
is no way to get a large enough sample. Take the behavior of mar-
kets at the end of large price bubbles. You can come up with some
specific rules for those market conditions and even test them, but
you will not have a very large sample on which to base your deci-
sions. In these cases we need to understand that the tests do not tell
us anywhere near as much as they would if we had a much larger
194
•
Way of the Turtle

---

sample. The seasonal tendencies I outlined earlier are another area
where this problem arises.
In testing a new rule for a system, you have to try to measure
how many times that particular rule affected the results. If a rule
made a difference only four times during the course of the test,
you do not have a statistical basis for deciding whether that rule
is helping. It is too easy for the effects you see to be random. One
solution to this problem is to find ways to generalize the rule 
so that it comes into play more often; that will increase the sam-
ple size and therefore the statistical descriptive value of tests for
that rule.
There are two common practices that compound the problem
of small sample sizes: single-market optimization and the building
of overly complex systems. 
• Single-market optimization: Optimization methods that
are performed separately for each market are much more
difficult to test with a sufficient sample size because a single
market offers much less trading opportunity. 
• Complex systems: Complex systems have many rules, and it
becomes very difficult at times to determine how many
times a particular rule may have come into effect or the
degree of that effect. Therefore, it is harder to be confident
in the statistical descriptive value of tests that are run using a
complex system.
For these reasons, I do not recommend optimizing for single
markets and prefer simple ideas that have stronger statistical 
meaning.
On Solid Ground
•