Backtesting Test Plan: Define Rules Before You Run It
Create a backtesting test plan before viewing results. Define the hypothesis, rules, data, costs, benchmark, holdout period, metrics, and pass/fail criteria.
Quick answer: Before running a backtest, write a test plan that fixes the strategy hypothesis, entry and exit rules, market scope, data source, execution assumptions, benchmark, validation split, metrics, and pass/fail criteria. This prevents you from changing the rules after seeing the result and makes the final report easier to audit.
A backtest can produce a detailed report even when the original question was vague. That is the problem this page solves. It is not another complete guide to running a backtest, and it is not the final reliability audit after the results arrive. It is the specification you write before pressing Run.
Use the complete backtesting guide for manual and automated execution steps. Use the backtesting validation checklist after a report exists and you need to test whether it is trustworthy.
What Is a Backtesting Test Plan?
A backtesting test plan is a written record of what you intend to test and how you will judge the result. It should be completed before optimization or result review.
The plan answers eight questions:
- What market behavior is the strategy trying to capture?
- What exact information is allowed at each decision point?
- What triggers an entry, exit, or no-trade decision?
- Which instruments, sessions, timeframes, and dates are included?
- What costs and execution assumptions are applied?
- What benchmark will the strategy be compared with?
- Which data is reserved for validation?
- What evidence would make you keep, revise, or reject the idea?
The goal is not to predict whether the strategy will succeed. The goal is to make the test reproducible and difficult to manipulate after the fact.
Why Write the Plan Before Viewing Results?
Without a written plan, the strategy can change every time the equity curve looks disappointing. A filter gets added, a weak year gets removed, a stop rule changes, or the best-performing parameter is selected after many trials. The final backtest may look precise while no longer testing the original idea.
Writing the plan first creates a boundary between research decisions and result-driven adjustments.
| Before the test | After the test |
|---|---|
| State the hypothesis | Compare the evidence with the hypothesis |
| Freeze the initial rules | Record any proposed rule change as a new version |
| Choose the dataset and holdout period | Review whether the chosen data was sufficient |
| Define costs and fill assumptions | Check how sensitive results are to those assumptions |
| Select metrics and a benchmark | Interpret the metrics in context |
| Define pass, revise, and reject conditions | Apply the predefined decision rule |
A changed rule is not automatically wrong. It simply creates a new strategy version that should receive a new test rather than being blended into the original result.
Step 1: State the Strategy Hypothesis
Begin with a short explanation of the market behavior the strategy is designed to capture.
A useful hypothesis contains:
- the condition being observed;
- the expected market response;
- the reason the response might persist;
- the conditions under which the idea should fail.
For example:
After a directional move, a controlled pullback that holds above a previously broken boundary may indicate that the prior direction remains intact. The test will examine whether a rules-based continuation entry behaves differently from a simple benchmark after costs.
This statement is more useful than “buy strong charts” because it can be translated into observable rules and challenged by evidence.
Step 2: Freeze the Information and Rules
Write the initial rules before opening the test results.
At minimum, define:
- setup conditions;
- entry trigger;
- exit and invalidation logic;
- position-sizing method;
- whether short positions are allowed;
- whether signals can overlap;
- how missing data is handled;
- what happens at the end of the test period.
Also define what information is not allowed. A rule cannot use a future close, a revised data value that was unavailable at the time, or an indicator that repaints historical signals.
Subjective strategies can still be tested manually, but the observation labels must be written clearly enough that another reviewer could classify the same chart without knowing the outcome.
Step 3: Define the Market and Data Scope
The test plan should identify the exact data universe rather than using phrases such as “several years of data.”
Record:
| Field | What to specify |
|---|---|
| Instrument universe | Symbols or the rule used to select them |
| Asset type | Stocks, forex, futures, crypto, or another market |
| Timeframe | The bar interval used for signals and execution assumptions |
| Session | Regular session, extended hours, or a defined market window |
| Test dates | Exact start and end dates |
| Data source | Vendor, broker feed, exchange source, or exported dataset |
| Corporate actions | Split and dividend adjustment policy when relevant |
| Delisted instruments | Whether failed and delisted symbols remain in the universe |
| Missing values | Exclusion, repair, or no-trade policy |
| Timezone | The clock used to align bars and events |
Data depth varies by symbol and timeframe. TradingView, for example, documents that available historical depth can differ across symbols and intervals, while MetaTrader's tester can download additional history and may shift an effective start date when the required pre-start data is unavailable. Check the actual dataset rather than assuming the requested dates were fully tested.
Step 4: Write the Execution Assumptions
A price chart does not prove that an order could have been filled at the displayed value. Define the model used for entries and exits.
The plan should cover:
- signal calculation timing;
- order submission timing;
- market, limit, stop, or other order logic;
- bid, ask, last, or midpoint convention;
- spread treatment;
- commission and fee treatment;
- slippage assumption;
- partial-fill policy;
- position and leverage limits;
- treatment of gaps and locked markets.
Use assumptions that match the strategy's holding period and market. A daily strategy and a short-horizon intraday strategy do not have the same sensitivity to spread, intrabar sequencing, or latency.
TradingView notes that historical strategy calculations may rely on assumptions about intrabar movement because complete tick-by-tick sequencing is not always available. MetaTrader likewise offers multiple modeling modes with different levels of detail. The selected mode belongs in the test record because it can materially change the result.
Step 5: Choose a Benchmark
A positive return does not automatically show that the strategy added value. Define the comparison before seeing the outcome.
Possible benchmarks include:
- buy and hold for the same instrument and period;
- cash or no-position exposure;
- an equal-weight universe;
- a simpler version of the same strategy;
- the same signal without the newest filter;
- a relevant market index.
The benchmark should use compatible dates, costs, and exposure assumptions. Otherwise, the comparison may reward the strategy for taking more risk rather than making better decisions.
Step 6: Reserve Validation Data
Do not optimize on every available observation and then call the same data a validation test.
Before testing, record:
- which period is used for development;
- which period is held out;
- whether parameters remain locked during the holdout test;
- whether walk-forward testing will be used;
- how many strategy variations or parameter sets will be tried;
- whether a later paper-trading or forward-testing stage is planned.
MetaTrader's official tester includes a forward option that separates part of the selected period for checking parameters outside the optimization segment. The important principle is the separation itself: validation data should not become another optimization surface after the first disappointing result.
Step 7: Select Metrics Before the Report Exists
Choose metrics that match the strategy's objective and risk profile.
A compact initial set can include:
- trade count and exposure;
- net and gross result after modeled costs;
- average result per trade or per unit of risk;
- maximum drawdown and recovery behavior;
- distribution of wins and losses;
- turnover and cost sensitivity;
- performance by time period, instrument, and market condition;
- benchmark-relative result;
- out-of-sample result.
Do not set universal thresholds simply because another article labels a certain win rate, profit factor, or Sharpe ratio “good.” The acceptable range depends on the test design, sample, market, turnover, concentration, and intended use.
For detailed metric interpretation, use Backtesting Results: What They Can and Cannot Tell You.
Step 8: Define Pass, Revise, and Reject Conditions
The decision rule should be written before the final equity curve is visible.
Use conditions tied to the original objective, such as:
Pass for further validation
- the result remains useful after relevant costs;
- performance is not concentrated in one isolated period or instrument;
- the strategy compares reasonably with its benchmark;
- the holdout result does not invalidate the original hypothesis;
- drawdown and exposure are compatible with the intended use.
Revise and create a new version
- the underlying behavior appears plausible, but one documented assumption is unsupported;
- a rule is too ambiguous to reproduce;
- execution modeling is materially incomplete;
- the test reveals a narrower market condition than the original hypothesis.
Reject or archive
- performance depends on future information or unavailable data;
- small parameter changes destroy the result;
- modeled costs remove the apparent advantage;
- the result exists only in the development segment;
- the strategy cannot be described consistently enough to reproduce.
“Revise” does not mean adjusting settings until the report turns positive. It means recording a new hypothesis and rerunning the research as a separate version.
Copyable Backtesting Test Plan Template
| Section | Your specification |
|---|---|
| Strategy name and version | |
| Research question | |
| Market hypothesis | |
| Instruments and selection rule | |
| Timeframe and session | |
| Test start and end dates | |
| Data source and adjustment policy | |
| Entry rule | |
| Exit and invalidation rules | |
| Position-sizing rule | |
| Order and fill model | |
| Spread, fees, and slippage | |
| Benchmark | |
| Development segment | |
| Holdout or walk-forward design | |
| Metrics selected in advance | |
| Pass conditions | |
| Revise conditions | |
| Reject conditions | |
| Number of variants to be tested | |
| Change-log location |
Save this plan with the code, spreadsheet, chart notes, or exported report used for the test. A future reviewer should be able to connect the written specification to the exact result.
Manual Replay and Automated Backtesting Need the Same Discipline
The same planning logic applies whether the test is coded or performed candle by candle.
| Manual replay | Automated backtest |
|---|---|
| Define how visual setups are labeled | Define the signal in code |
| Hide future candles | Prevent look-ahead and repainting |
| Record every decision, including no-trades | Preserve the complete trade log |
| Use the same chart and data settings | Preserve the data and tester configuration |
| Separate rule changes into new sessions | Version code and parameters |
| Reserve unseen dates for later review | Hold out data or use walk-forward validation |
ChartMini is suitable for lightweight manual replay and decision-process review. It can help test whether discretionary rules are clear enough to apply without seeing future candles. It is not an automated strategy engine, does not optimize parameters, and does not reproduce broker execution, spreads, slippage, or order-book conditions.
How This Page Fits the Backtesting Cluster
Use each page for a different stage:
- Before running the test: use this test-plan template.
- To execute the test: follow the complete backtesting workflow.
- To choose between methods: compare market replay, backtesting, and paper trading.
- To interpret the report: review what backtesting results can and cannot tell you.
- To challenge reliability: apply the backtesting validation checklist.
This separation prevents one page from trying to own the test plan, execution tutorial, metrics guide, method comparison, and final audit at the same time.
Frequently Asked Questions
What should be written before running a backtest?
Write the market hypothesis, exact rules, data scope, execution assumptions, benchmark, validation split, selected metrics, and pass/revise/reject conditions before viewing the result.
Is a backtesting test plan the same as a backtesting checklist?
No. A test plan specifies the research before the test runs. A validation checklist examines whether an existing result is reliable after the report has been generated.
Why should pass and fail conditions be defined in advance?
Predefined conditions reduce the temptation to reinterpret weak results, remove unfavorable periods, or change rules only because the original report was disappointing.
Do manual backtests need a written specification?
Yes. Manual replay still requires consistent setup labels, entry and exit rules, chart settings, data boundaries, and a complete decision log. Otherwise, hindsight can change how each historical chart is judged.
Should all available data be used for strategy development?
No. Reserve a separate period or use a walk-forward structure so the final strategy can be checked on data that was not used to choose its rules or parameters.
Can ChartMini validate an automated trading strategy?
No. ChartMini is a manual historical chart replay tool for price-action and decision practice. Automated strategy validation requires a testing engine that can execute coded rules and model the relevant data, costs, and order assumptions.
Official Sources
- TradingView: What are strategies, backtesting and forward testing?
- TradingView: Why historical and real-time strategy results can differ
- MetaTrader 5: Strategy Testing
- Fidelity: About Strategy Testing
- CFTC Rule 4.41: Limitations of hypothetical or simulated performance
