Way of the turtle

Existing Measures Are Not Robust
In testing, you are trying to determine relative performance, assess
potential future performance, and determine whether a particular
idea has merit. One of the problems with this process is that the
generally accepted performance measures are not very stable—they
are not robust. This makes it difficult to assess the relative merits
182
•
Way of the Turtle

---

of an idea because small changes in a few trades can have a large
effect on the values of these nonrobust measures. The effect of the
instability in the measures is that it can cause one to believe that
an idea has more merit than it actually has or discard an idea
because it does not appear to have as much promise as it might
when examined using more stable measures.
A statistic is robust if changing a small part of the data set does
not change that statistic significantly. The existing measures are too
sensitive to changes in the data, too jumpy. This is one of the rea-
sons that in doing historical simulation for trading system research,
slight differences in parameter values cause relatively large differ-
ences in some of the measures; the measures themselves are not
robust (i.e., they are too sensitive to small portions of the data). Any-
thing that affects those small portions can affect the results too
greatly. This makes it easy to overfit and to fool yourself with results
that you will not be able to match in real life. The first step in test-
ing the Turtle Way is to address this issue by finding performance
measures that are robust and not sensitive to small changes in the
underlying data.
One of the questions that Bill Eckhardt asked me during my
initial interview for the Turtle position was: “Do you know what
a robust statistical estimator is?” I stared blankly for a few seconds
and admitted: “I have no idea.” (I now can answer that question.
There is a branch of mathematics that tries to address the issue
of imperfect information and poor assumptions; it is called robust
statistics.)
It is clear from the question that Bill had respect for the imper-
fect nature of testing and research based on historical data as well
as knowledge of the unknown that was rare at that time and is still
On Solid Ground
•
183

---

rare. I think this is one of the reasons Bill’s trading performance has
held up so well over the years.
This is yet another example of how far ahead of the industry Rich
and Bill’s research and thinking were. The more I learn, the deeper
my respect for their contribution to the field becomes. I am also
surprised at how little the industry has advanced beyond what Rich
and Bill knew in 1983.