Backtesting & Strategy Optimization

In the Strategy Tester, change the mode from "Single" to "Optimization."

In the Inputs tab, check the parameters to optimize and set Start, Step, and Stop values. For example: Fast MA from 5 to 30 step 5, Slow MA from 30 to 100 step 10.

Optimize by: Maximum profit, Maximum Sharpe Ratio, Minimum drawdown, or custom criterion. Sharpe Ratio is generally the best choice as it balances profit and risk.

Go: open a terminal in a folder you can write to — cd ~/Desktop on macOS or Linux, cd %USERPROFILE%\Desktop on Windows.

Do: save this as modelling.py and run python3 modelling.py.

"""The tester's modelling mode changes the answer -- but only for some rules."""
import random

random.seed(5)

def make_bar_ticks(n=40):
    ticks, p = [], 100.0
    for _ in range(n):
        p += random.uniform(-0.25, 0.25)
        ticks.append(round(p, 2))
    return ticks

BARS = [make_bar_ticks() for _ in range(200)]

def bar_ohlc(ticks):
    return ticks[0], max(ticks), min(ticks), ticks[-1]

THRESHOLD = 0.15

# RULE A: "did price ever get 0.15 above the open?" -- order does not matter.
def count_entries(mode):
    n = 0
    for ticks in BARS:
        o, h, l, c = bar_ohlc(ticks)
        points = [c] if mode == "open prices only" else \
                 [o, h, l, c] if mode == "1 minute OHLC" else ticks
        if any(p - o > THRESHOLD for p in points):
            n += 1
    return n

# RULE B: "was the stop or the target hit FIRST?" -- order is everything.
def stop_first(mode):
    stop_wins = target_wins = neither = 0
    for ticks in BARS:
        o, h, l, c = bar_ohlc(ticks)
        target, stop = o + 0.30, o - 0.30
        if mode == "1 minute OHLC":
            # The tester must GUESS the order. A reasonable guess is that price
            # visits the NEARER extreme first.
            seq = [o, l, h, c] if (o - l) <= (h - o) else [o, h, l, c]
        elif mode == "optimistic OHLC":
            # A careless guess: always visit the TARGET side first.
            seq = [o, h, l, c]
        else:
            seq = ticks
        hit = None
        for p in seq:
            if p >= target: hit = "target"; break
            if p <= stop:   hit = "stop";   break
        if   hit == "stop":   stop_wins += 1
        elif hit == "target": target_wins += 1
        else:                 neither += 1
    return stop_wins, target_wins, neither

print("200 bars, 40 real ticks each = 8,000 price points")
print()
print("RULE A -- 'did price get 0.15 above the open?'  (order irrelevant)")
print("%-22s %10s" % ("MODELLING MODE", "ENTRIES"))
print("-" * 34)
for mode in ("open prices only", "1 minute OHLC", "every tick"):
    print("%-22s %10d" % (mode, count_entries(mode)))
print()
print("OHLC and every-tick AGREE, because the bar's high is one of the four")
print("points -- so if any tick reached the level, the high did too.")
print()
print("RULE B -- 'stop or target first?'  (order is everything)")
print("%-22s %8s %8s %8s" % ("MODELLING MODE", "STOP", "TARGET", "NEITHER"))
print("-" * 50)
for mode in ("optimistic OHLC", "1 minute OHLC", "every tick"):
    s, t, n = stop_first(mode)
    print("%-22s %8d %8d %8d" % (mode, s, t, n))

s_ohlc, t_ohlc, _ = stop_first("1 minute OHLC")
s_opt,  t_opt,  _ = stop_first("optimistic OHLC")
s_tick, t_tick, _ = stop_first("every tick")
print()
print("a reasonable guess is wrong on  : %d of %d bars (%.0f%%)"
      % (abs(s_ohlc - s_tick), len(BARS), 100.0 * abs(s_ohlc - s_tick) / len(BARS)))
print("a careless guess is wrong on    : %d of %d bars (%.0f%%)"
      % (abs(s_opt - s_tick), len(BARS), 100.0 * abs(s_opt - s_tick) / len(BARS)))
print("and it errs in ONE direction    : %+d stop-outs vs reality"
      % (s_opt - s_tick))
print()
print("THAT is where modelling mode matters. Four points per bar cannot say")
print("whether the high or the low came first, so the tester ASSUMES -- and")
print("any strategy whose result depends on which of your two exits was hit")
print("first is being scored on that assumption, not on the market.")

You should see: two modes agreeing on one rule and disagreeing badly on another:

200 bars, 40 real ticks each = 8,000 price points

RULE A -- 'did price get 0.15 above the open?'  (order irrelevant)
MODELLING MODE            ENTRIES
----------------------------------
open prices only               90
1 minute OHLC                 166
every tick                    166

OHLC and every-tick AGREE, because the bar's high is one of the four
points -- so if any tick reached the level, the high did too.

RULE B -- 'stop or target first?'  (order is everything)
MODELLING MODE             STOP   TARGET  NEITHER
--------------------------------------------------
optimistic OHLC              64      136        0
1 minute OHLC                99      101        0
every tick                  100      100        0

a reasonable guess is wrong on  : 1 of 200 bars (0%)
a careless guess is wrong on    : 36 of 200 bars (18%)
and it errs in ONE direction    : -36 stop-outs vs reality

THAT is where modelling mode matters. Four points per bar cannot say
whether the high or the low came first, so the tester ASSUMES -- and
any strategy whose result depends on which of your two exits was hit
first is being scored on that assumption, not on the market.

Rule A is the surprise. “One-minute OHLC” and “every tick” give identical answers, because the bar's high is one of the four points — so if any tick reached the level, the high did too. For a rule that only asks whether price ever reached somewhere, the cheap mode is exact, and spending hours on tick modelling buys nothing.

Rule B is where it matters. Four points per bar cannot say whether the high or the low came first, so the tester assumes — and any strategy with both a stop and a target is scored on that assumption. A reasonable assumption (visit the nearer extreme first) is wrong on 1 bar in 200. A careless one that always visits the target side first is wrong on 36 of 200, and every one of those errors flatters you: 136 targets reached where reality gives 100.

So the question to ask of your own strategy is not “which mode is best” but “does my result depend on the order of events inside a bar?” If it does — stop and target both in range, an intrabar trailing stop, anything scalping — use real tick data and treat cheaper modes as meaningless. If it does not, the fast mode is fine.

If not: python3: command not found on Windows means Python was installed without “Add python.exe to PATH”; try py modelling.py. The figures are seeded and will match this page exactly.

Go: the same folder.

Do: save this as spread.py and run python3 spread.py.

"""The tester uses one spread. Real markets use several."""

# A day's trading, by hour, with the spread that was actually available.
HOURS = [
    ("00:00 thin overnight",      2.8), ("03:00 Asia",            2.1),
    ("08:00 London open",         0.9), ("10:00 London",          0.8),
    ("13:30 US data release",     7.5), ("14:00 US session",      1.0),
    ("17:00 London close",        1.4), ("22:00 rollover",       11.0),
]

TRADES_PER_HOUR = 6
LOT_VALUE_PER_POINT = 1.0

fixed = 1.0                      # what a tester is usually configured with
avg   = sum(h[1] for h in HOURS) / len(HOURS)

print("%-28s %10s" % ("WHEN", "SPREAD (points)"))
print("-" * 42)
for label, s in HOURS:
    print("%-28s %10.1f" % (label, s))

print()
print("tester's fixed spread      : %5.1f" % fixed)
print("simple average of the real : %5.1f" % avg)
print()

cost_fixed = fixed * TRADES_PER_HOUR * len(HOURS) * LOT_VALUE_PER_POINT
cost_real  = sum(s * TRADES_PER_HOUR * LOT_VALUE_PER_POINT for _, s in HOURS)

print("trades in the day          : ", TRADES_PER_HOUR * len(HOURS))
print("cost at the fixed spread   : %8.2f" % cost_fixed)
print("cost at the real spreads   : %8.2f" % cost_real)
print("understated by             : %8.2f  (%.0f%%)"
      % (cost_real - cost_fixed, 100.0 * (cost_real - cost_fixed) / cost_fixed))

print()
print("And the two worst rows are not random. Spreads widen exactly when a")
print("breakout strategy wants to trade -- at a data release -- and around")
print("rollover, when many strategies check for a new day.")
print()
print("So the error is not merely large; it is CORRELATED with your entries.")
print("A strategy that trades at quiet times is barely affected. One that")
print("trades news is charged several times what the tester assumed.")
print()
print("Configure the tester with your broker's real variable spread if it")
print("offers that, and if not, use a fixed spread near the WORST you expect")
print("rather than the best. A backtest that survives a pessimistic spread is")
print("worth something; one that only survives an optimistic one is not.")

You should see: costs understated by 244%:

WHEN                         SPREAD (points)
------------------------------------------
00:00 thin overnight                2.8
03:00 Asia                          2.1
08:00 London open                   0.9
10:00 London                        0.8
13:30 US data release               7.5
14:00 US session                    1.0
17:00 London close                  1.4
22:00 rollover                     11.0

tester's fixed spread      :   1.0
simple average of the real :   3.4

trades in the day          :  48
cost at the fixed spread   :    48.00
cost at the real spreads   :   165.00
understated by             :   117.00  (244%)

And the two worst rows are not random. Spreads widen exactly when a
breakout strategy wants to trade -- at a data release -- and around
rollover, when many strategies check for a new day.

So the error is not merely large; it is CORRELATED with your entries.
A strategy that trades at quiet times is barely affected. One that
trades news is charged several times what the tester assumed.

Configure the tester with your broker's real variable spread if it
offers that, and if not, use a fixed spread near the WORST you expect
rather than the best. A backtest that survives a pessimistic spread is
worth something; one that only survives an optimistic one is not.

The size of the error is not the worst part. It is correlated with when strategies trade. Spreads widen at data releases — which is exactly when a breakout rule fires — and around rollover, which is when a great many strategies check for a new day.

So the tester's flat spread is not a random approximation that averages out. It is systematically too generous precisely at the moments your strategy chose to act, which means the error grows with how news-driven the strategy is.

The practical rule: if your platform can use your broker's recorded variable spread, use it. If not, set a fixed spread near the worst you expect rather than the typical. A strategy that survives a pessimistic spread has told you something; one that only survives an optimistic one has told you about the setting.

If not: the arithmetic is fixed, so the figures will match. If the understatement is much smaller, the two large spread rows were edited — they are there because those two moments are real and are when strategies most often trade.

Go: the same folder.

Do: save this as drawdown.py and run python3 drawdown.py.

"""Two strategies with the same final profit. Only one is holdable."""
import random

random.seed(2)

def equity_curve(trades):
    eq, curve = 10000.0, [10000.0]
    for t in trades:
        eq += t
        curve.append(round(eq, 2))
    return curve

def max_drawdown(curve):
    peak, worst, worst_at = curve[0], 0.0, 0
    for i, v in enumerate(curve):
        peak = max(peak, v)
        dd = (peak - v) / peak
        if dd > worst:
            worst, worst_at = dd, i
    return worst * 100.0, worst_at

def longest_losing_streak(trades):
    best = cur = 0
    for t in trades:
        cur = cur + 1 if t < 0 else 0
        best = max(best, cur)
    return best

# A: small steady wins, occasional large loss.
A = []
for i in range(200):
    A.append(-900.0 if i % 23 == 0 else 45.0)

# B: choppier, but the losses are bounded.
B = []
for i in range(200):
    B.append(random.choice([-120.0, -80.0, 150.0, 90.0, -60.0, 130.0]))

# Make the totals comparable.
B = [b + (sum(A) - sum(B)) / len(B) for b in B]

for name, trades in (("A: steady wins, rare crashes", A), ("B: choppy, bounded losses", B)):
    curve = equity_curve(trades)
    dd, at = max_drawdown(curve)
    print("%-32s" % name)
    print("   trades              : ", len(trades))
    print("   final equity        : ", "%.2f" % curve[-1])
    print("   total profit        : ", "%+.2f" % (curve[-1] - curve[0]))
    print("   max drawdown        :  %.1f%%  (worst at trade %d)" % (dd, at))
    print("   longest losing run  : ", longest_losing_streak(trades))
    print()

print("Identical profit. Very different experiences -- and note which way")
print("round it came out.")
print()
print("B never loses more than 120 on a trade, which SOUNDS safer than A's")
print("occasional 900. But B's losses cluster: an eight-trade losing run")
print("digs a deeper hole (17.9%) than A's isolated crashes (9.0%).")
print()
print("Per-trade loss size is not risk. What matters is how losses ARRIVE,")
print("because consecutive ones compound into a drawdown and isolated ones")
print("do not. That is exactly what a summary line of 'total profit' hides.")
print()
print("Max drawdown answers 'how bad did it get before it got better', and")
print("the longest losing run answers 'for how long'. Those two decide")
print("whether a real person leaves the system switched on -- which is the")
print("only way it ever reaches the final number.")

You should see: identical profit and very different journeys:

A: steady wins, rare crashes    
   trades              :  200
   final equity        :  10495.00
   total profit        :  +495.00
   max drawdown        :  9.0%  (worst at trade 1)
   longest losing run  :  1

B: choppy, bounded losses       
   trades              :  200
   final equity        :  10495.00
   total profit        :  +495.00
   max drawdown        :  17.9%  (worst at trade 166)
   longest losing run  :  8

Identical profit. Very different experiences -- and note which way
round it came out.

B never loses more than 120 on a trade, which SOUNDS safer than A's
occasional 900. But B's losses cluster: an eight-trade losing run
digs a deeper hole (17.9%) than A's isolated crashes (9.0%).

Per-trade loss size is not risk. What matters is how losses ARRIVE,
because consecutive ones compound into a drawdown and isolated ones
do not. That is exactly what a summary line of 'total profit' hides.

Max drawdown answers 'how bad did it get before it got better', and
the longest losing run answers 'for how long'. Those two decide
whether a real person leaves the system switched on -- which is the
only way it ever reaches the final number.

The result comes out the opposite way round from most people's intuition, which is what makes it worth running. Strategy B never loses more than 120 on a single trade, against A's occasional 900 — and B has the deeper drawdown, 17.9% against 9.0%, because its losses arrive consecutively. An eight-trade losing run compounds; isolated crashes do not.

Per-trade loss size is not risk. How losses arrive is risk, and “total profit” conceals it completely.

Two numbers to demand of any backtest, yours included: maximum drawdown, which says how bad it got, and longest losing streak, which says for how long. Those decide whether a real person leaves the system switched on — and a system switched off during its drawdown never reaches the final figure that made it look attractive.

If not: the trade sequences are seeded, so the figures will match. If both drawdowns are identical, the profit-equalising line that adjusts B is not running — the point is to compare two curves that end in the same place.

Go: the same folder.

Do: save this as cannot.py and run python3 cannot.py.

"""What no backtest models, however good the data."""

ITEMS = [
    ("your order moving the price",        "you are not in the tester's market"),
    ("a requote or a rejected order",      "the tester always fills you"),
    ("slippage on a fast move",            "usually modelled as zero"),
    ("the broker widening spreads on news","spread is a setting, not a reaction"),
    ("a platform or internet outage",      "the tester never disconnects"),
    ("the strategy stopping working",      "the tester has no future"),
    ("you switching the EA off in a loss", "the tester has no feelings"),
    ("you 'improving' it after a bad week","the tester runs the rule you gave it"),
]

print("%-40s %s" % ("WHAT HAPPENS IN REALITY", "WHY THE TESTER MISSES IT"))
print("-" * 88)
for what, why in ITEMS:
    print("%-40s %s" % (what, why))

print()
print("items listed        :", len(ITEMS))
print("caused by the market:", 5)
print("caused by YOU       :", 2)
print("caused by machinery :", 1)
print()
print("The last two rows are the ones nobody puts in a spreadsheet, and they")
print("are the most common reason a profitable system produces a losing")
print("account. The tester ran your rule for two years without once deciding")
print("that this time was different.")
print()
print("None of this makes backtesting useless. It makes it a FILTER: it can")
print("tell you an idea definitely does not work, which is worth knowing")
print("cheaply. It cannot tell you one does.")

You should see: eight things that happen in reality and in no simulation:

WHAT HAPPENS IN REALITY                  WHY THE TESTER MISSES IT
----------------------------------------------------------------------------------------
your order moving the price              you are not in the tester's market
a requote or a rejected order            the tester always fills you
slippage on a fast move                  usually modelled as zero
the broker widening spreads on news      spread is a setting, not a reaction
a platform or internet outage            the tester never disconnects
the strategy stopping working            the tester has no future
you switching the EA off in a loss       the tester has no feelings
you 'improving' it after a bad week      the tester runs the rule you gave it

items listed        : 8
caused by the market: 5
caused by YOU       : 2
caused by machinery : 1

The last two rows are the ones nobody puts in a spreadsheet, and they
are the most common reason a profitable system produces a losing
account. The tester ran your rule for two years without once deciding
that this time was different.

None of this makes backtesting useless. It makes it a FILTER: it can
tell you an idea definitely does not work, which is worth knowing
cheaply. It cannot tell you one does.

Five of these belong to the market and one to your equipment. Two belong to you, and those two are the most common reason a profitable system produces a losing account: the tester ran the rule for two years without once switching it off during a bad month or deciding this time was different.

None of this makes backtesting worthless, and it would be a mistake to conclude that. A backtest is a filter: it can establish cheaply and quickly that an idea definitely does not work, which saves months. What it cannot do is establish that one does.

Which is why the sequence that actually works is: backtest to reject, forward-test on a demo account to observe, then trade the smallest size that is real. The demo step is the one people skip, and it is the only one that includes rows six to eight of that table.

If not: this prints a fixed list and cannot fail; the counts beneath it are literals matching the table, so editing a row means editing them too.