Way of the turtle

•
151
•
eleven
LIES, DAMN LIES,
AND BACKTESTS
Charlatans and scoundrels lurk in dark corners, 
awaiting the unsuspecting. Don’t be their prey.
The Stonehenge Plus system TURNED $5,000 INTO $1,000,000 
in JUST 5 YEARS. Stonehenge Plus was invented by Stupendus
Magnificus, a NASA scientist who discovered a way to use the same
process used to launch the Mars Rover for trading currencies. 
OVER 90% ACCURATE, it hasn’t had a losing month 
in 10 years. It is so good we are only going to sell 100 COPIES. 
Get yours now while there is still time for only $1,999.
—Advertisement from a system vendor
A
nyone who has been around trading for any amount of time
has seen ads like this one, as has anyone who has his or her
name on a mailing list and is interested in trading. But buyer
beware: There are charlatans who use irresponsible marketing tac-
tics and unrealistic backtest results to tout their newest inventions.
Copyright © 2007 by Curtis M. Faith. Click here for terms of use. 

---

Many of those vendors build systems that they know will never yield
returns at the levels they are advertising. Many of them intention-
ally alter the tests to make their systems look better than they really
are. However, not all vendors are that unscrupulous. There are some
who sell systems that they believe will work well without realizing
that their basic methods are flawed or knowing the limitations of
historical testing and the pitfalls of using historical test results to pre-
dict future performance. Of course, there are those who are skilled
at avoiding the pitfalls of historical testing. Unfortunately, these ven-
dors are very few in number and it is extremely difficult for an inex-
perienced trader to distinguish between the systems that have been
developed using good testing methods and those that have not.
Even highly experienced traders often do not know the reasons
why their systems perform much more poorly in actual trading
than they do in historical simulations. They know this phenom-
enon exists and compensate for it but do not understand its
causes. There are four major sources of the discrepancies traders
often find between historical test results and what is encountered
in actual trading:
• Trader effects: The fact that a particular method has made a
lot of money in the recent past increases the likelihood that
other traders will have noticed it and will start using similar
ideas, increasing the chances that the method will not work
as well as it did initially.
• Random effects: The historical test may have demonstrated
better performance than the underlying edge normally
would indicate as a result of purely random chance.