Navigation: Running Tests >

Multiple Tests and Optimization

 

 

 

 

When you run a test in Optimize mode, the optimization dialog appears before the test starts running.

Parameter List

Shows each of your optimization variables along with the count of iterations for that variable and the range of values. The checkboxes within the list can be used to exclude any variable from optimization. When a variable is excluded, its default value will be used.

Optimization Mode

Specifies the type of optimization to be performed.

Combinatorial mode is the traditional exhaustive nested loop covering every possible parameter combination. If you have more than 2 or 3 variables with more than a few values for each, you will see that the total test count quickly becomes very large. Due to the speed of RealTest, it’s quite practical to run portfolio-level optimizations involving thousands of tests across thousands of stocks across multiple years of time. Whether this approach is likely to discover a strategy that will be profitable in the future is another matter.

Sequential mode loops over each variable in order. At the end of a loop, that variable keeps whichever value produced the highest score. Running 2 or 3 iterations of a sequential optimization will often be a quicker way to find areas of good parameter values than the combinatorial approach.

Genetic mode is not a true genetic optimizer, but the concept is similar. Before each test, a random subset of the parameters is selected and then the values of those variables are selected at random from their value lists. (How many parameters to change each time is determined by the "mutate" option.) If the score from the test is higher than the prior best score, the new values are kept, otherwise the prior best values are restored. Genetic mode will generally converge on the best combinatorial result within approximately the square root of the total combination count. Note that to use genetic mode, you must specify the number of test iterations to be run.

Random mode is similar to genetic mode except that it randomly selects a value for every variable before every test, and (therefore) completely ignores the score of each test. A good use of random mode is to select a reasonable value ranges for each parameter, run 100 test iterations (random combinations), and look at the median result. This might provide a reasonable estimate of how the system would perform in the future, given how arbitrary parameter selection can be. Note that to use random mode, you must specify the number of test iterations to be run.

For Each Symbol

This option can be used whether or not the script contains any parameters. If checked, rather than running a single test (or multi-test optimization) which scans all symbols each day, RealTest will run a separate test (or a separate multi-test optimization) for each symbol in the data file. This makes it easy, for example, to find the best parameters for each symbol and then look at the combined results to get a feel for the robustness of those parameters. (For a per-symbol optimization, the settings Sort After Each Test, Keep Best 1, and group By Symbol are recommended.)

Test Iterations

Use this setting to do any of the following:

run many iterations of the same test repeatedly using the Random function in one or more strategy formulas, to obtain a range of potential outcomes

run a parameter optimization in Genetic mode, to specify how many iterations to run

run a parameter non-optimization in Random mode, to specify how many iterations to run

Score

Allows you to select any column from your results window to use as your "fitness function". Since the results columns are all formula-based, this means you have unlimited possibilities for what to use here.  The value returned by the selected formula is used to rank the results after an optimization run if "Sort After Each Test" is checked. This is most useful with the Sequential or Genetic optimization modes, and when creating a Walk-Forward test.

Results Window Options

The results window options choices enable you to clear the results window at the start of the test (assuming it was already open, otherwise a new one is created), and optionally be partially cleared and sorted during the test as well. While watching an optimization run, it is often desirable to just see the best 10 results, for example.

Date Intervals

The Date Intervals panel is used primarily to generate a walk-forward test but can also be used to simply produce a series of results for different time periods. (It is OK to run a script in Optimize mode even if it contains no optimization parameters.) In particular, it can often be useful to set the time unit to "Years", simply to run a strategy for each year of a date range separately.

 

By default each date interval is a sliding window of the same length. The Anchor setting lets you optionally lock the start or end date.

 

 

Anchoring the start date makes each interval larger than the preceding one, and anchoring the end date does the opposite.

Create Walk-Forward

See Walk-Forward Tests for a detailed description of this feature.

Maximum Test Count

At the lower-right corner of the dialog is the test count. Notice how it changes as you check and uncheck the checkboxes.

 

See also Tutorial 2 - Optimization for a specific usage example.

 

 

 

 

Copyright © 2020-2024 Systematic Solutions, LLC