Way of the turtle
Overfitting or Curve Fitting
Download 6.09 Mb. Pdf ko'rish
|
Way Of The Turtle
Overfitting or Curve Fitting
Scammers also use other methods to generate historical results that are unrealistic. The most unscrupulous ones intentionally overfit or curve fit their systems. Overfitting often is confused with the opti- mization paradox, but they concern different issues. 172 • Way of the Turtle Overfitting occurs when systems become too complex. It is pos- sible to add rules to a system that will improve its historical per- formance, but that happens only because those rules affect a very small number of important trades. Adding those rules can create overfitting. This is especially true for trades that occur during crit- ical periods in the equity curve for the system. For example, a rule that lets you exit a particularly large winning trade close to the peak certainly would improve performance but would be overfit if it did not apply to enough other situations. I have seen many examples where system vendors have used this technique to improve results of their systems after a period of relatively poor performance. They sometimes sell the new improved systems as plus or II versions of their original systems. Anyone contemplating a purchase of a system “improved” in this matter would do well to inves- tigate the nature of the rules which constitute the improvements to make sure that they have not benefited from overfitting. I often find it useful to look at examples of a phenomenon taken to the extreme to understand it better. Here I will present a system that does some pretty egregious things that overfit the data. We will start with a very simple system, the Dual Moving Average system, and add rules that start to overfit the data. Remember that this system had a very nasty drawdown in the last six months. Therefore, I will add a few new rules to fix that draw- down and improve performance. I am going to reduce my positions by a certain percentage when the drawdown reaches a particular threshold and then, when the drawdown is over, resume trading at the normal size. To implement this idea, let’s add a new rule to the system with two new parameters for optimization: the amount to be reduced and Lies, Damn Lies, and Backtests • 173 the threshold at which that reduction occurs. Looking at our simu- lation’s equity curve, I decide that reducing positions by 90 percent when I reach a drawdown of 38 percent will limit the drawdowns. The addition of this rule improves the returns, which go from 41.4 percent without the rule to 45.7 percent with it, and the drawdown drops from 56.0 percent to 39.2 percent, with the MAR ratio going from 0.74 to 1.17. One might think, “This is a great rule; the system is now much better.” However, this is completely incorrect! The problem is that there is only one time during the entire test when this rule comes into play. It happens at the very end of the test, and I’ve taken advantage of my knowledge of the equity curve to construct the rules, and so the system has been fitted intention- ally to the data. “What’s the harm?” you ask. Consider the shape of the graph in Figure 11-6, where we vary the threshold for the draw- down where a reduction kicks in. You may notice the rather abrupt drop in performance if we use a drawdown threshold of less than 37 percent. In fact, a 1 percent change in the drawdown threshold makes the difference between earning 45.7 percent and losing 0.4 percent per year. The reason for the drop in performance is that there is an instance in August 1996 where this rule kicks in and we cut back the position size so much that the system does not earn enough money to dig out of the hole created by the drawdown. Perhaps this is not such a good rule. It worked in the first instance only because the drawdown was so close to the end of the test. Traders call this phenomenon a cliff. The presence of cliffs— large changes in results for a very small change in parameter val- ues—is a good indication that you have overfit the data and can expect results in actual trading that are wildly different from those Download 6.09 Mb. Do'stlaringiz bilan baham: |
Ma'lumotlar bazasi mualliflik huquqi bilan himoyalangan ©fayllar.org 2024
ma'muriyatiga murojaat qiling
ma'muriyatiga murojaat qiling