Navigation: Running Tests >

Multiple Tests and Optimization

1

Parameter List

Shows each of your optimization variables along with the count of iterations for that variable and the range of values. The checkboxes within the list can be used to exclude any variable from optimization. When a variable is excluded, its default value will be used.

2

Optimization Mode

Specifies the type of optimization to be performed:

•Combinatorial mode is the traditional exhaustive nested loop covering every possible parameter combination. If you have more than 2 or 3 variables with more than a few values for each, you will see that the total test count quickly becomes very large. Due to the speed of RealTest, it’s quite practical to run portfolio-level optimizations involving thousands of tests across thousands of stocks across multiple years of time. Whether this approach is likely to discover a strategy that will be profitable in the future is another matter.

•Sequential mode loops over each variable in order. At the end of a loop, that variable keeps whichever value produced the highest score. Running 2 or 3 iterations of a sequential optimization will often be a quicker way to find areas of good parameter values than the combinatorial approach.

•Genetic mode is not a true genetic optimizer, but the concept is similar. Before each test, a random subset of the parameters is selected and then the values of those variables are selected at random from their value lists. (How many parameters to change each time is determined by the "mutate" option.) If the score from the test is higher than the prior best score, the new values are kept, otherwise the prior best values are restored. Genetic mode will generally converge on the best combinatorial result within approximately the square root of the total combination count. Note that to use genetic mode, you must specify the number of test iterations to be run.

•Random mode is similar to genetic mode except that it randomly selects a value for every variable before every test, and (therefore) completely ignores the score of each test. A good use of random mode is to select a reasonable value ranges for each parameter, run 100 test iterations (random combinations), and look at the median result. This might provide a reasonable estimate of how the system would perform in the future, given how arbitrary parameter selection can be. Note that to use random mode, you must specify the number of test iterations to be run.

•Defaults Only mode is equivalent to clearing all the check boxes.

3

For Each Strategy

This option causes a separate test (or multi-test optimization) to be run for each strategy in the script, as if all the other strategy definitions had been commented out.

4

For Each Symbol

This option causes a separate test (or multi-test optimization) for each symbol in the data file, as if that was the only symbol in the data.

5

Test Iterations

Use this setting to do any of the following:

•run many iterations of the same test repeatedly using the Random function in one or more strategy formulas, to obtain a range of potential outcomes

•run a parameter optimization in Genetic mode, to specify how many iterations to run

•run a parameter non-optimization in Random mode, to specify how many iterations to run

6

Score

Allows you to select any column from your results window to use as your "fitness function". Since the results columns are all formula-based, this means you have unlimited possibilities for what to use here.  The value returned by the selected formula is used to rank the results after an optimization run if "Sort After Each Test" is checked. This is most useful with the Sequential or Genetic optimization modes, and when creating a Walk-Forward test.

7

Results Window Options

The results window options choices enable you to clear the results window at the start of the test (assuming it was already open, otherwise a new one is created), and optionally be partially cleared and sorted during the test as well. While watching an optimization run, it is often desirable to just see the best 10 results, for example. Keep No Trade Tests overrides the default behavior of not saving test runs that produced zero trades.

8

Date Intervals

The Date Intervals panel is used primarily to generate a walk-forward test but can also be used to simply produce a series of results for different time periods. (It is OK to run a script in Optimize mode even if it contains no optimization parameters.) In particular, it can often be useful to set the time unit to "Years", simply to run a strategy for each year of a date range separately.

By default each date interval is a sliding window of the same length. The Anchor setting lets you optionally lock the start or end date.

Anchoring the start date makes each interval larger than the preceding one, and anchoring the end date does the opposite.

9

Create Walk-Forward

See Walk-Forward Tests for a detailed description of this feature.

10

Multi Save As CSV

Causes SaveStatsAs and/or SaveTradesAs to be performed at the end of the optimization, creating separate files for each test. These settings are ignored when running multiple tests unless this box is checked.

11

Summary Log

Creates a log window summarizing the optimization.

12

Maximum Test Count

At the lower-right corner of the dialog is the test count. Notice how it changes as you check and uncheck the checkboxes and other options.