TRADESIGNAL HOW TO 03.

STRESS TESTING – HOW ROBUST IS YOUR STRATEGY?

Properly optimization instead of curve fitting.

With the right software, testing and optimizing a trading strategy is very easy these days. It is precisely this simplicity, however, that allows users – consciously or unconsciously to fall into the curve fitting trap. In this issue we want to show how optimization leads to over-fitting and also how to identify if your strategy is robust by using a stress test.

WATCH VIDEO NOW. (Invest 6 minutes in Know-How.)

By downloading you accept the terms of use below.

TABLE OF CONTENTS.

  • Back test and optimization – a holy grail?
  • Optimization versus curve fitting
  • Using the stress test for creating a robust strategy
  • The stress test indicator
  • Application in practice
  • Robustness – the key to success

BACK TEST AND OPTIMIZATION – A HOLY GRAIL?

Algorithmic trading strategies are based purely on objectives and clear rules:

Getting into a position, the stop, exit and the position size are all subject to unique requirements. Besides this, another advantage is that any emotional traps are easily avoidable. The use of quantitative trading strategies also provides a decisive advantage: Any strategy that can be transferred into a source code and as such converted it into a machine language, can be back tested – in other words, simulated based on historical data – before it is applied to real funds.

By using the gained risk and return metrics the characteristics and quality of a strategy can be analyzed in detail and compared with alternatives. Combined with an optimization function, many powerful tools to aid the development of trading strategies are available to the user.


OPTIMIZATION VERSUS CURVE FITTING.

Used incorrectly, however, the tools just mentioned can have undesirable consequences, namely ‘curve fitting’. The following anecdote, which reiterates a conversation between scientists, Freeman Dyson and Enrico Fermi, describes this phenomenon:

It is 1953. Freeman Dyson questions Fermi in Chicago to discuss with him his own results for the meson-proton distribution. Fermi was however visibly unimpressed by the results and asks Dyson, how many selectable parameters he had used for the calculations. Dyson answers that he had used four parameters. Fermi’s retort: “I remember how my friend John von Neumann used to say, with four parameters I can adjust an elephant, and with five I can make him wiggle his trunk.”

This quote describes, what is at stake in curve fitting – to over-fitting a trading strategy to the underlying data series. During the optimization process, individual parameters are modified (eg, length of the moving average, stop or target size) in an iterative process until, finally, from a large number of different combinations those are picked, which promise “optimal” results – but only on the basis the specific historical data that was used for the calculation.

So before a back test and optimization are performed at all, the trader should have a solid idea for the trading strategy and attempt to define meaningful parameter ranges to be tested. In short, the software should validate a trading idea, not discover it.

The more parameters a trading strategy has and the higher the number of combinations tested, the higher the curve fitting effect.

The result is that the trading strategy delivers wonderful results in the back test, but fails miserably in real trading. Simulated scientific studies show that the stronger the strategy was adapted to the historical data, the poorer the results in real trading.*

A further study entitled “The probability of backtest overfitting” can be found here.

For this reason, a division of the representative historical market data in at least two sub-periods is necessary. This allows for the development and optimization of a trading strategy based on the period A (In-sample back test) in order to test it on the unknown data-terrain, period B (Out-of-sample back test).

The results of the sub-periods should not show any significant deviations. If this were the case, it is highly likely to be evidence of an over-fitted strategy.

A special form of this test method in which the optimal parameters of a sub-period for the subsequent out-of-sample period are used,it is called the walk forward test. How you can use it in practice and what features are provided by Tradesignal? We discuss this in greater detail in a future How To issue.


USING THE STRESS TEST FOR CREATING A ROBUST STRATEGY.

For a trading strategy to work, not only in the back test, but also in real trading, it must be robust. With Tradesignal the back testing and optimization function allows you to check how stable your trading strategy behaves on different markets and in different time frames – even using an entire portfolio. The testing of the strategy at different time frames can help identify changes in performance behavior, which would be due to differing market phases. The use of other securities in the back test shows, however, whether the algorithm actually identifies general patterns that can be observed in other underlying assets or if it has been trimmed on specific patterns within the underlying data series.

An easy way to test this is to monitor the change in the risk and return characteristics while the parameter values are being modified. The more varied the results, the less robust is the strategy. An extreme negative example is shown in figure 1. The graph shows the profit factor of a trading strategy in response to an entry and exit parameter; red areas represent unprofitable, green for profitable values. As can be seen, of all parameter combinations only one is profitable. If exactly this “optimal” combination of parameters is the basis for a trading strategy, one thing is inevitable in real trading – its failure.

FIGURE 1: OPTIMIZATION HEATMAP.

Profitable parameter combinations being only available in a small area signal curve fitting in its purest form.


THE STRESS TEST INDICATOR.

As already mentioned, it should be examined in a stress test; or how the chosen strategy behaves on other markets and different time frames. With the stress test indicator we want to show you a simple and convenient way to perform such a stress test.

As already mentioned, it should be examined in a stress test; or how the chosen strategy behaves on other markets and different time frames. With the stress test indicator we want to show you a simple and convenient way to perform such a stress test.

Meta:

 ​​ ​​ ​​ ​​ ​​ ​​​​ Synopsis("Example indicator that is designed to be the parent of a trading

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ strategy for which the strategy author would like to 'stress test' by

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ varying the indicators input parameters during optimization."),

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ Author("Tradesignal GmbH");

 

Inputs:

 ​​ ​​ ​​ ​​ ​​ ​​​​ Security(SIE,​​ ALVG,​​ DAI,​​ EUR,​​ DAX),​​ // Change the security being tested

 ​​ ​​ ​​ ​​ ​​ ​​​​ Period(Minutes1,​​ Minutes5,​​ Minutes60,​​ Daily,​​ Weekly,​​ Monthly)​​ =​​ Daily;​​ // Change the period being tested

 ​​ ​​ ​​ ​​ ​​ ​​​​ 

Instruments:

 ​​ ​​ ​​ ​​ ​​ ​​​​ inst( ​​ IFF(Security​​ =​​ SIE,​​ "SIEGn.DE",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Security​​ =​​ ALVG,​​ "ALVG.DE",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Security​​ =​​ DAI,​​ "DAIGn.DE",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Security​​ =​​ EUR,​​ "EUR=",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Security​​ =​​ DAX,​​ ".GDAXI",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ "")))​​ +​​ " "​​ +

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Period​​ =​​ Minutes1,​​ "1m",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Period​​ =​​ Minutes5,​​ "5m",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Period​​ =​​ Minutes60,​​ "60m",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Period​​ =​​ Daily,​​ "daily",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Period​​ =​​ Weekly,​​ "weekly",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ IFF(Period​​ =​​ Monthly,​​ "monthly",

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ "")))))))) ); ​​ 

 ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​ ​​​​ 

DrawBar(Open​​ of​​ inst,​​ High​​ of​​ inst,​​ Low​​ of​​ inst,​​ Close​​ of​​ inst);

FIGURE 2: STRESS TEST INDICATOR EQUILLA CODE.

With the stress test indicator trading strategies can be checked for robustness by defining several securities and time frames as inputs.

As you can see, the securities included in this indicator are stored as so-called inline instruments and defined as inputs. Since inputs of indicators can be optimized, this function offers a clever way to test any number of securities against a trading strategy to examine their stability.

To enable you to test the Stress Test Indicator on your Tradesignal platform, we provide you with the corresponding Equilla Code.

You can apply the indicator by saving the code above by copying and pasting it into the Equilla editor. In the sample code, the RICs from Thomson Reuters have been used.


APPLICATION IN PRACTICE.

An application of the stress test indicator requires the following steps:

  • 01. Copy the Equilla code and paste it in Equilla editor;
  • 02. Modify and expand the securities included in the code if required;
  • 03. Save the indicator on your Tradesginal platform;
  • 04. Insert the indicator into an empty chart;
  • 05. Choose a trading strategy of your choice and add it into the chart (eg Moving Average Crossover Single);
  • 06. Select the toolbar button “Optimize”
    • a. Select “Brute Force
    • b. Parameters to optimize:
      • i. Trading strategy period (5 to 200, step 5);
      • ii. Period of the stress test indicator (eg, monthly, weekly, daily, 60min, 5min);
    • c. Select a data field for the back test

FIGURE 3: OPTIMIZATION ASSISTANT.

The stress test indicator provides the possibility using different time frames as parameters within an optimization.

After these steps, the user gets an overview that displays the period of the selected parameter of the trading strategy on the x-axis, while the selected time levels are shown on the y-axis.

A tabulation (not shown) is also included for detailed analysis. If you want to check the robustness of the strategy – for example, measured on profit factor – to analyze the dependence of these two factors, then selecting „profit factor“ is sufficient in the upper input window of the optimizer and one obtains the following chart analysis.

FIGURE 4: ROBUSTNESS CHECK (PARAMETER VS. TIME FRAMES).

The coloured representation of the optimization results shows profitable and unprofitable combinations of two parameters. In this example it is the period of a simple crossover strategy and a total of five different time frames.

Using the color gradation it is possible to detect profitable areas in no time. Green and yellow areas represent profit factor above 1, while orange and red coloured cells represent unprofitable areas. As expected, the exemplarily chosen crossover strategy provides good results on a monthly and weekly time frame (row 1 and 2). On a daily level, positive results can be seen only around the period 200.

On 60- and 5-minute-planes (bottom two rows), however, the results are negative across the board.

The stress test indicator can be further used to analyze the robustness of a trading strategy as a function of the individual securities and a given period. To do this, not only the period of the SMA, but also the desired securities (parameter selected is “Security Strategy (stress test indicator)” as the comparison parameter (see step 6b and graph below). A few moments later, the optimization process is complete and the visual presentation just shows all combinations. As you can see, the trading strategy generates positive results basis in all four securities. Particularly high profit factor values are found in the range from 200 to 250 days.

FIGURE 5: ROBUSTNESS CHECK (PARAMETERS VS. SECURITIES).

The colored representation of the optimization results shows profitable and unprofitable combinations of two parameters. In this example, it involves the period value of a simple crossover strategy and four securities.


ROBUSTNESS – THE KEY TO SUCCESS.

Tradesignal provides all the professional tools, the trader and portfolio manager need for the back test and optimization. The presented stress test indicator provides one of several ways to check the stability and robustness of an algorithmic trading strategy to reduce curve fitting-effects to a minimum. Besides the robustness check, the consideration of realistic costs (commissions, slippage) and the use of a rule-based, dynamic position control size are essential prerequisites for the use of a trading strategy in practice.

Finally, we would like to use a famous quote from Albert Einstein that can be transferred very well to Algorithmic Trading. Namely, it provides the easiest and most effective recommendation that needs to be considered when developing quantitative trading strategies:

“Everything should be made as simple as possible, but no simpler.” Albert Einstein

For questions about Tradesignal we are always happy to help. If you are not yet a Tradesignal customer, we will gladly provide you with a trial version.

That’s it for today. Take care, take profit and “auf Wiedersehen”.

DISCLAIMER.

© Copyright Tradesignal Ltd., London.
Distribution allowed under a Creative Commons Attribution-Noncommercial license
 http://creativecommons.org/licenses/by-nc/3.0/

Tradesignal® is a registered trademark of Tradesignal GmbH. Unauthorized use or misuse is specifically prohibited. All other protected brands and trademarks mentioned in this document conform, without restriction, to the provisions of applicable trademark law and the copyrights of the respective registered owners.

Tradesignal Ltd. obtains information from sources it considers reliable, but does not guarantee the accuracy or completeness of its information contained therein. Tradesignal Ltd. and its affiliates make no representation or warranty, either express or implied, with respect to the information or analysis supplied herein, including without limitation the implied warranties of fitness for a particular purpose and merchantability, and each specifically disclaims any such warranty. In no event shall Tradesignal Ltd. or its affiliates be liable to for any decision made or action taken in reliance upon the information contained herein, lost profits or any indirect, consequential, special or incidental damages, whether in contract, tort or otherwise, even if advised of the possibility of such damages. This material does not constitute an offer or a solicitation of an offer or a recommendation to buy or sell securities. All expressions of opinion are subject to change without notice.

This code is provided free of charge and “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of fitness for a particular purpose and noninfringement. In no event, shall Tradesignal Ltd. be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, or of or in connection with this code or the use of this code.

.