Way of the turtle

Overfitting or Curve Fitting
Scammers also use other methods to generate historical results that
are unrealistic. The most unscrupulous ones intentionally overfit
or curve fit their systems. Overfitting often is confused with the opti-
mization paradox, but they concern different issues.
172
•
Way of the Turtle

---

Overfitting occurs when systems become too complex. It is pos-
sible to add rules to a system that will improve its historical per-
formance, but that happens only because those rules affect a very
small number of important trades. Adding those rules can create
overfitting. This is especially true for trades that occur during crit-
ical periods in the equity curve for the system. For example, a rule
that lets you exit a particularly large winning trade close to the peak
certainly would improve performance but would be overfit if it did
not apply to enough other situations.
I have seen many examples where system vendors have used this
technique to improve results of their systems after a period of relatively
poor performance. They sometimes sell the new improved systems as
plus or II versions of their original systems. Anyone contemplating a
purchase of a system “improved” in this matter would do well to inves-
tigate the nature of the rules which constitute the improvements to
make sure that they have not benefited from overfitting.
I often find it useful to look at examples of a phenomenon taken
to the extreme to understand it better. Here I will present a system
that does some pretty egregious things that overfit the data. We will
start with a very simple system, the Dual Moving Average system,
and add rules that start to overfit the data.
Remember that this system had a very nasty drawdown in the last
six months. Therefore, I will add a few new rules to fix that draw-
down and improve performance. I am going to reduce my positions
by a certain percentage when the drawdown reaches a particular
threshold and then, when the drawdown is over, resume trading at
the normal size.
To implement this idea, let’s add a new rule to the system with
two new parameters for optimization: the amount to be reduced and
Lies, Damn Lies, and Backtests
•
173

---

the threshold at which that reduction occurs. Looking at our simu-
lation’s equity curve, I decide that reducing positions by 90 percent
when I reach a drawdown of 38 percent will limit the drawdowns.
The addition of this rule improves the returns, which go from 41.4
percent without the rule to 45.7 percent with it, and the drawdown
drops from 56.0 percent to 39.2 percent, with the MAR ratio going
from 0.74 to 1.17. One might think, “This is a great rule; the system
is now much better.” However, this is completely incorrect!
The problem is that there is only one time during the entire test
when this rule comes into play. It happens at the very end of the
test, and I’ve taken advantage of my knowledge of the equity curve
to construct the rules, and so the system has been fitted intention-
ally to the data. “What’s the harm?” you ask. Consider the shape of
the graph in Figure 11-6, where we vary the threshold for the draw-
down where a reduction kicks in.
You may notice the rather abrupt drop in performance if we use
a drawdown threshold of less than 37 percent. In fact, a 1 percent
change in the drawdown threshold makes the difference between
earning 45.7 percent and losing 0.4 percent per year. The reason
for the drop in performance is that there is an instance in August
1996 where this rule kicks in and we cut back the position size so
much that the system does not earn enough money to dig out of
the hole created by the drawdown. Perhaps this is not such a good
rule. It worked in the first instance only because the drawdown was
so close to the end of the test.
Traders call this phenomenon a cliff. The presence of cliffs—
large changes in results for a very small change in parameter val-
ues—is a good indication that you have overfit the data and can
expect results in actual trading that are wildly different from those