|
A common way to guard against curve fitting is to develop and tune a strategy using one date range (the in-sample period) while reserving a later range (the out-of-sample period) to confirm the results on data that development never touched. RealTest can run both periods with a single Test command, adding a separate, clearly labeled row to the Results window for each.
Setting It Up
Add a WalkForward section to the script containing a single Dates line with two dates: the start of the in-sample period and the start of the out-of-sample period.
WalkForward:
Dates: 1/1/2000, 1/1/2018
Running the script in Test mode then runs two tests instead of one:
•TestName #WF-IS -- the in-sample test, run from the first date through the second date
•TestName #WF-OOS -- the out-of-sample test, run from the second date through the EndDate setting
Both ranges are limited to the StartDate and EndDate settings, and a WalkForward section is ignored when a script is run in Orders mode.
With Parameters
No Parameters section is required. If the script does define parameters, the WalkForward section can list one value for each; both tests then use those values, which also appear in the parameter columns of the Results window.
Parameters:
MALen: from 20 to 50 step 10
WalkForward:
Dates: 1/1/2000, 1/1/2018
MALen: 30
Selecting the Test Period Per Run
During development you may want to run only the in-sample period, later only the out-of-sample period, or the whole range as a single test -- without disturbing the locked split dates. Conditional comments make the choice a one-line edit at the top of the script:
// pick one (or none = full range as one test):
//#define IS
//#define OOS
#define SPLIT
Settings:
DataFile: is_oos_example.rtd
#ifdef IS
TestName: "RSI Pullback IS"
StartDate: 2000-01-01
EndDate: 2017-12-31
#else
#ifdef OOS
TestName: "RSI Pullback OOS"
StartDate: 2018-01-01
EndDate: Latest
#else
TestName: "RSI Pullback"
StartDate: 2000-01-01
EndDate: Latest
#endif
#endif
// ... Data and Strategy sections as usual ...
#ifdef SPLIT
WalkForward:
Dates: 2000-01-01, 2018-01-01
#endif
In command line mode, the scope can be selected without editing the script at all:
RealTest -define OOS -test myscript.rts
Comparing Every Combination in Both Periods
To compare in-sample and out-of-sample results for every parameter combination -- a model-level robustness check, not a parameter selection step -- run the same optimization once per period into the same results file. With OptimizeMode: Combinatorial and OptKeepBest: 0 (keep every test), each combination gets its own results row with its parameter values in the results columns. Reusing the scheme above:
RealTest -define IS -optimize myscript.rts
RealTest -define OOS -optimize myscript.rts
Add a ResultsFile setting so both runs accumulate in one file, and let the conditional blocks tag TestName with the period so the rows are labeled. Rows then pair up by the parameter columns (combinatorial order is deterministic, so row N of each sweep is the same combination). Export with SaveTestListAs to compare the periods in a spreadsheet or script. See also Overfitting Analysis, which does this comparison in a single optimize run: its Surface view pairs every combination’s in-sample and out-of-sample Sharpe at a chronological split and reports their correlation, and its Selection view evaluates the in-sample winner across thousands of symmetric partitions.
A WalkForward section with more than two dates defines a full walk-forward playback, in which parameter values change from interval to interval. Sections like that are normally generated by a walk-forward optimization rather than written by hand; see Walk-Forward Tests.
|
