THE PROBLEM OF CURVE FITTING.
The amazing computer power we have at our hands nowadays makes it feasible to backtest limitless ideas, optimize them and find which ones (and which parameters of them) worked best in the past and so “fitting” by a great amount the systems’ parameters onto the past “curve” of price data. Of course you can’t altogether eliminate the curve fitting of a system to historical data because the systems are in the first place designed by examining what worked well in the past.
Overfitting, however, is a problem. This is because the greater the fitting of a system to the past, the greater its historical performance but also the greater the risk that it will not perform well in the future because the future price movements are not exactly the same as the past ones.
OVEROPTIMIZED OR NOT? ROBUSTNESS CHECK PROVIDES THE ANSWER.
The greatest fear of all systematic traders is that their trading system will stop working when the technical conditions change. This is because the systems are designed on past data and there is the danger of what is called “Curve Fitting”. The use of so-called robustness checks provides information about how the trading strategy is “immune” towards changes in the price data. Apart from using as few parameters as possible for a system and extensively testing it on Out-Of-Sample data (that is, on data which was not used to find its best set of parameters) there are also three common tests you can do to increase the odds that a system is not overly curve fitted and thus it is robust to changes in future price behavior.
The first test regards examining the system’s performance in symbols of the same nature as the one your system is designed to trade. So, for example, suppose that you designed a system to trade commodity A. When it performed well on A (In-sample and even Out-Of-Sample) but failed on a commodity B with the same characteristics as A then this is a cautionary flag that the system may be prone to future losses if you use it to trade commodity A since it failed to capture the primary forces which move the prices of B. You can’t expect a system to be profitable in all kind of markets but at least it should be able to have a decent performance in all symbols of the same nature because such symbols are usually traded by the same market participants and share common traits.
The second (and most popular in technical analysis software) robustness test regards checking the performance of a system when its parameters are changed a bit (for more information download How To 03 “Stresstesting”). Tradesignal’s optimization feature does a pretty good job in giving the user sophisticated means to perform such a test. When for example a simple moving average (SMA) crossover system performs pretty well using a short term SMA of 5 bars and a long term SMA of 35 bars but performs miserably for a short term SMA of 6 bars and a long term SMA of 36 bars it is an indirect indication that the 5/35 combination is curve fitted and thus the odds it will fail in the future are quite increased.
The third robustness test (an example of which we will focus below) regards checking the performance of a system when the price data are changed. This is a more direct test and more sophisticated than the previous ones since there are numerous ways for price data to be changed. To perform it you need to conduct a vast number of backtests using Monte Carlo simulations.
MONTE CARLO SIMULATION.
Broadly speaking, Monte Carlo simulations (a.k.a Monte Carlo methods or Monte Carlo experiments) refer to repeated simulations to obtain numerical results. For conducting the third robustness test this translates to defining what general kind or price changes you want your system to be tested upon, make as many such changes as possible (from very small to very big) to the price data, backtest your system on these changed data and finally examine the magnitude of change of your system’s performance in relation to the magnitude of price changes. In the sequence I will show you an example of such a test in Tradesignal.
ADDING A FRACTION OF ATR AS NOISE TO THE PRICE.
Consider a trading symbol named SYMBOL. We want to create a new artificial symbol (let’s name it SYMBOLnew) by changing the OPEN, HIGH, LOW, CLOSE values of each price bar of the SYMBOL. The new values for them (namely OPENnew, HIGHnew, LOWnew and CLOSEnew respectively) will be the same as the old ones plus or minus a random fraction of the Average True Range (ATR) provided that all new values are positive numbers and the logical order is maintained (that is, the LOWnew is positive and equals their minimum and the HIGHnew equals their maximum). To create the SYMBOLnew in Tradesignal all you need to do is write an indicator which will construct the OPENnew, HIGHnew, LOWnew, CLOSEnew and it will draw a Candlestick chart based on these values. I have created such an indicator (namely Noisy Symbol). By dragging and dropping the Noisy Symbol onto the chart of a trading symbol a new sub chart showing the SYMBOLnew is created. The indicator has three input parameters:
Its default value is 1 and defines the period of the ATR which will be used. The native AvgTrueRange function of Tradesignal is used to calculate the ATR.
Its default value is 0.3 and defines the maximum fraction of the ATR which will be used to generate the noise. A zero value for this parameter results in zero noise, that is, the Noise Symbol will be the same as the initial symbol. For example, if this parameter is 0.3 then the random numbers which will be added or subtracted from the OPEN, HIGH, LOW, CLOSE to produce their new values won’t be higher than 0.3 times the ATR.
This parameter determines whether the initial order of Open and Close should be maintained or not (default: false). If set to TRUE it maintains the initial order of OPEN and CLOSE.
This is a dummy input which will be used in the Optimizer to perform the Monte Carlo simulation. When the indicator is applied in a chart the value of this input doesn’t affect anything.
In Figure 1 (next page) you can see the Noisy Symbol applied in a daily chart of ADIDAS with its default values. Each time the Noisy Symbol is calculated or recalculated (for example when you change one of its parameters or you press the Reload button) a new “noisy” version of ADIDAS is created.
Since the Noisy Symbol is not a static symbol but an indicator which changes each time it is calculated you can now use the Tradesignal’s optimization feature to perform a Monte Carlo simulation for a trading strategy and see how the performance of the strategy changes when an ATR noise is added to the price.