Reading Assignment: Common Backtesting Mistakes

TA & Algo Trading Course
#1

In this reading assignment we will dive deeper into the most important issue when it comes to building a profitable trading strategy - Backtesting. We will look at some of the common mistakes, over-optimization and curve-fitting and how you can avoid them. Read through this blog post and answer the following questions in this forum thread. Use the knowledge you have learned so far as well.

  1. What is so dangerous about over-optimization?
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
  3. Why should you avoid asymmetric trading signals?
10 Likes
#2

  1. curve fitting; it maybe over adjusted or tailored to past events (back testing) and may not be suitable for the future markets or new future data sets.

  2. snapshot 6 months; with a comparison to the longest possible (>5 years ideally, 9 to 11 years) data set humanly possible!

Focus on the development of a trading system aiming for adaptability, broad optimizations, robust profitability and large periods of testing data increasing better chance or probability of achieving a Ferrari.

  1. Asymmetric system; Asymmetric information can lead to adverse selection, incomplete markets and is a type of market failure.

New Algo Team; join the A-Team.

Serious about Algo optimisation keep in touch & say hello; [email protected]

6 Likes
#3

  1. Over optimization can lead to “curve-fitting”, which is the unwanted tuning of the strategy to fit specific past data.

  2. According to the article, approx 10yrs on higher time frames for accuracy.

  3. Asymmetric signals provide more complexity which tends to lean towards tuning the strategy for specific datasets… curve-fitting.

Funny I haven’t heard of this term “curve-fitting” before, but I’ve certainly been victim of it while fine tuning strategies in the past. Interesting ;o)

2 Likes
#4

What is so dangerous about over-optimisation?

  • There is a danger of over-optimisation is to produce a trading strategy with absolutely astonishing results that will not be achievable going forward.

How long should a testing period be if you are serious about building a profitable trading strategy?

  • Time frames should be greater than 30 minutes
  • Ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.

Why should you avoid asymmetric trading signals?

  • Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitting solutions.
2 Likes
#5

  1. What is so dangerous about over-optimization? Over-optimization is dangerous because it trasforms the strategy in a curve-fitting of past data that will be never identic to the future data. So the very good results on the past data will not reply in the future.

  2. How long should a testing period be if you are serious about building a profitable trading strategy? In the traditional market a testing period of 9-11 years should be used.

  3. Why should you avoid asymmetric trading signals? Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

2 Likes
#6

  1. What is so dangerous about over-optimization?
    Creating a false sense that plan will work just customized for past data.

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    About 10 years

  3. Why should you avoid asymmetric trading signals?

You end up matching up with previous market cycles.

4 Likes
#7
  1. What is so dangerous about over-optimization?
  • the strategy will be perfect for past data but not the future trading.
  1. How long should a testing period be if you are serious about building a profitable trading strategy?
  • ten years data
  1. Why should you avoid asymmetric trading signals?
  • It will lead to curve fitting as the asymmetric signals is work for the past data but we cannot sure about whether it works for the future.
1 Like
#8

1. What is so dangerous about over-optimisation?
It will only be optimised for the specific data set. This is not a clear indication of what will happen in the future. Your algorithm becomes more of a curve-fitting program; this will mean losses in the future as it eliminates the probabilistic nature of the market.

2. How long should a testing period be if you are serious about building a profitable trading strategy?
Ideally 9 -11 years if possible. Author also states a minimum of 5 years to avoid curve fitting: https://mechanicalforex.com/2010/06/five-common-mistakes-in-system.html

3. Why should you avoid asymmetric trading signals?
These trading signals should be avoided as they are generally based around the economic variables at that time period for the particular currency (interest rates etc). A good example is over optimisation of previous data sets. This can create two separate sets of criteria in the market for “shorts” and “longs”. This strategy becomes excessively prone to curve-fitting.

3 Likes
#9
  1. What is so dangerous about over-optimization?

A strategy performing extremely well on a given dataset is likely to be “unready” to other market situations and conditions. Meaning, it is not general enough. Which might lead to missed profits or even loss of the funds.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?
  • About 10 years to train the strategy
  • About 2 years outside that period to check how it performs on new unknown data.
  1. Why should you avoid asymmetric trading signals?

More probability of making a mistake. And more space for over-optimizing due to “more parameters to tweak”.

1 Like
#10
  1. Over-optimization will most likely cause curve fitting, in which the past data will have “astonishing results that will not be achievable going forward”
  2. Ideally 9-11 years of data should be used
  3. Asymmetric trading signals should be avoided because having them increases the degrees of freedom of the trading strategy, which makes curve fitting more likely to happen, KISS (keep it simple stupid)
1 Like
#11

  1. Curve-fitting - the strategy is optimized for events that happened in the past, but won’t be able to produce positive results in the future.

  2. Ideally 9 to 11 years of data should be used to make the strategy more reliable.

  3. With asymmetric trading signals you have separate criteria for long and short trades which makes the strategy prone to curve-fitting.

1 Like
#12

  1. Over-optimization yields a result that is not generally applicable, but only optimized for that particular dataset. Therefore, it can’t be reliably used for other sets of data, which is directly the goal of optimization in the context of trading.

  2. The testing period should be on the order of a decade, because this provides enough market data to account for both micro- and macroeconomic cycles, and other cyclic events in the economy. In the context of cryptocurrency trading, since the market is less than a decade old, one must test through a least one bull and bear market, and we’ve seen a couple of those already.

  3. Asymmetric trading signals create bias toward a certain trend. This partially invalidates the model, since it makes an assumption that the general behavior of the tested dataset will remain the same in the future.

1 Like
#13

(1) What is so dangerous about over-optimization? This can lead to curve fitting which is dangerous because it can give a false sense of predictability of future data.
(2) How long should a testing period be if you are serious about building a profitable trading strategy? Time frame should be greater than 30 minutes. Ideally 9 to 11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.
(3) Why should you avoid asymmetric trading signals? This can increase the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

1 Like
#14

  1. If you over-optimize you are fitting your algorythm to specific circumstances that have happened in one section in the past. Here you risk making it too specific to those and it could make it miss fire as the data in the future is unlikely to be exactly the same.

  2. About 9-11 years from the post… However this could be difficult with bitcoin trading as its newish.

  3. Asymmetric systems rely on macro economic variables that change throughout the ecconomic cycle.

1 Like
#15

  1. Over-optimization can lead to the curve-fitting issue in which your algorithm gonna lost predictive power. Remember you want to develop an algo to predict the future, not to perfectly fit with past data.

  2. The testing period should be long enough. In crypto world, it should be more than 4 years.

  3. We should avoid asymmetric trading signals because the asymmetric can eventually lead to curve-fitting in the long run.

1 Like
#16

1.What is so dangerous about over-optimization?
The danger is curve-fitting, that is, our trading strategy will be suitable for past data but not for future data.

2.How long should a testing period be if you are serious about building a profitable trading strategy?

The testing period should ideally be around 10 years of data. The time frame should be larger than 30 min.

3.Why should you avoid asymmetric trading signals?
Because they increase the risk of curve-fitting.

1 Like
#17
  1. It will perfectly fit the historic data, but will be irrelevant in the current market conditions
  2. 9-11 years of data is to be used
  3. Asymmetric trading signals usually results from the past market data, which is dependent upon other variables that tend to change.
2 Likes
#18
  1. Dangerous is curve -fitting your strategy in the past events.
  2. Around 10 years.
  3. Because then it leading to curve fitting.
1 Like
#19

image
image1640Ă—1326 478 KB

It occurs to me that part of the over-optimization problem is one of pricing frequency. I’m currently a developer maintaining CIFER, wherein we can fit models to data across specific frequencies within the data, ignoring high frequency noise and low-frequency disturbances. Hmmm…

I’m also thinking forget deep learning techniques as they’d over-optimize by nature. However genetic algorithms may work for combining populations of simple, randomly-tuned indicators into strategies of limited complexity, and then forcing evolution towards the dominant strategies. Hmmm…

2 Likes
#20

  • What is so dangerous about over-optimization?
    A past behavior of price is never 100% identical to the future behavior . over optimizing might lead to wrong trades.

  • How long should a testing period be if you are serious about building a profitable trading strategy?
    9-11 years

  • Why should you avoid asymmetric trading signals?
    It will increase the potential for curve-fitting/

1 Like