|
|
|
Quantifying Robustness
Within the world of quantitative finance, robustness has become a well known buzzword and as happens with most other buzzwords, its meaning has become a bit blurred around the edges from overuse. From context, the word can mean anything from effective to bulletproof, but the measure is always a qualitative one. For the sake of bringing some clarity back to the situation, this article will propose a quantitative meaning for robustness.
|
As systems traders know, all trading systems require some sort of initial parameter selection. These parameters may represent the period for a moving average or the number of standard deviations used in calculating a volatility band or a variable used in calculating any of countless indicators. In nearly every case, the performance of the system is directly tied in with the selected parameter. For example, some systems perform very well with a parameter set to 60 but perform disappointingly when the parameter used is set to 10. Examine the table at right, which lists returns at varying parameters for a hypothetical trading system.
|
|
| Parameter | Performance |
|---|
| 40 | -3% |
| 50 | 18% |
| 60 | 7% |
| 70 | 23% |
| 80 | 16% |
| 90 | 48% |
|
|
Although the system above clearly has the potential to be rather profitable, the wise trader would probably shy away from it due to the fact that parameter selection is a potential minefield. Now examine the results of another hypothetical system in the next table. Although the average return across the different parameter values is the same as in the system above, this system is far less vulnerable to underperformance due to poor parameter selection. With single parameter systems, a visual comparison is all that's required to determine which one is superior. However, any realistic trading system will have more than one parameter.
|
|
| Parameter | Performance |
|---|
| 40 | 12% |
| 50 | 17% |
| 60 | 18% |
| 70 | 21% |
| 80 | 19% |
| 90 | 22% |
|
In real world trading a system will typically have a handful of parameters, for example, position size, maximum acceptable portfolio risk, variables relating to entry and exit indicators, etc. There may be a total of more than a thousand combinations of parameters. Testing all of these combinations by hand is merely time consuming but visually evaluating a system's vulnerability to poor parameter selection in such circumstance is practically impossible. As a solution to this problem, we present the following formula:
Robustness = (Average Return / Standard Deviation of Returns)
where the sample group is the set of returns for all parameter combinations. This quantitative measure of robustness ensures that a trader can make a wise comparison between systems, even when the total parameter space is very large.
In the case of our systems above, the robustness score for the first system is 1.052 while the robustness score for the second system is 5.125 which is far superior. With more complex systems, this formula for robustness can be quite telling as to whether or not the system has been over optimized (curve fit), or is otherwise too dependent on the selection of initial parameters. Our solution: Order a system which was designed with robustness in mind from the beginning.
| |
