|
Every item in a script's Data Section is normally calculated once, up front, and stored in memory -- one value per bar per symbol. References during a test are then simple array reads, and an item that several formulas share is never computed twice. The cost is storage: memory use grows with symbols × bars × items, and a multi-market, multi-system script over a long history can need more memory than the machine has.
The Data on demand option (on the Calculation tab of the Program Options dialog, default off) trades calculation time for memory. When it is enabled, eligible Data items are not stored at all; each reference evaluates the item's formula on the fly, the same way a Library formula would be. A test that needed tens of gigabytes of Data storage can run in almost none -- it just runs more slowly, since values are recomputed at every bar where they are referenced.
The option applies automatically to every eligible item, so the same script runs unchanged on a large-memory and a small-memory machine. There is no need to maintain a separate low-memory copy of a script with its Data formulas moved to Library.
Which Items Stay Stored
Some kinds of Data items only work as stored arrays, and RealTest keeps them stored even when the option is enabled:
•Cross-sectional (breadth) items -- breadth tags such as #Rank, #PctRank, #Count, #Avg and #Sum are computed across all symbols, date by date, and must be stored. (Their input formulas can still be other, on-demand items.)
•Self-referential items -- cumulative formulas that refer to their own prior values, e.g. cum: cum[1] + ...
•DataValueFile items -- their values come from an external file, not a formula.
•#OnePerDate and #OnePerSym items -- these already store only one value per date or per symbol, so there is little to save.
•Items whose formulas contain Random() -- the stored draws must persist so that every reference sees the same value.
•DLL data-calc items, and any items of the same bar size defined before one -- the user DLL is handed pointers to the stored arrays.
•Items built on exponential smoothing or other recursive calculations -- the EMA-family functions (EMA, ESD, AEMA, AESD, DEMA, TEMA, KAMA, RsiF, SarF) and the indicators built on them (RSI, RRSI, CRSI, ATR, ADX, PDI, MDI, MACD, MACDS, MACDH, KBTop, KBBot, SuperTrend and the Ehlers filters). Their stored calculation carries exact running state across the whole history, which a per-reference evaluation could only approximate, so they stay stored to guarantee identical results (see below).
Everything else -- simple moving averages and other windowed calculations, comparisons and combinations, items that reference other Data items, items at a different bar size than the test -- is evaluated on demand.
Results Match Stored Calculation
An on-demand item returns the same value a stored read would have returned, so test results do not change when the option is toggled. The recursive EMA-family items listed above stay stored for precisely this reason: evaluated on the fly they would use the bounded lookback that the EMA precision setting defines, which is normally indistinguishable from the full-history calculation but can diverge sharply in special cases (after a long one-directional run, an on-the-fly RSI can read 0 where the full-history value is near 100).
The memory cost of this exclusion is smaller than it may appear. When an EMA-family function is part of a larger formula, the "Extract one-pass items" optimization normally splits it out into its own hidden stored item first, and only that piece stays stored -- the containing item can still evaluate on demand. (An EMA-family call written directly in a strategy formula, where it is not extracted, continues to use the bounded-lookback evaluation it always has; that behavior is independent of this option.)
Performance Notes
•The cost of a reference is proportional to the formula's lookback window (e.g. Avg(C, 200) scans 200 bars each time it is referenced). Items that reference other on-demand items multiply this cost, so keep deeply nested chains in mind if a test is unexpectedly slow.
•Breadth items evaluate their input formulas once per symbol per date during calculation, so scripts that rank complex on-demand factors with #Rank feel the cost most.
•A per-thread cache absorbs repeated references to the same item at the same bar (multiple strategies, entry plus exit formulas), which is the most common pattern.
•With "Log calculation details" (same Calculation tab) enabled, on-demand items are marked *** in the data calculation log, so you can see exactly which items were stored and which were not.
The Out-of-Memory Check
Independently of this option, RealTest computes the projected size of the Data section before allocating it and compares it to the amount of memory that can currently be committed (available RAM plus page file). A projection that merely exceeds physical RAM is allowed to run — the operating system uses the page file as needed, which is slow but works.
If the projection exceeds what can currently be committed, the Maximum Memory setting (on the General tab of the Program Options dialog) decides what happens next. When projected memory use stays within that limit, RealTest asks whether to continue, since a system-managed Windows page file can grow to make up the difference; in command line mode, a warning goes to the log and the run continues. When projected use would exceed the limit, the run stops immediately with a message reporting how many gigabytes the calculated data items require and how many can be committed, and suggesting that you enable Data on demand if it is not already on, reduce the amount of data loaded, or raise Maximum Memory. In command line mode, the message goes to the error log and the run exits with the memory error code (4).
|
