DataForge exports MT5 bars to CSV — and, more importantly, it tells you what is really inside them.
The mark is the product: two readings of the same instrument, and one column where they disagree — the declared value dashed, the measured one short.
Most machine-learning failures in trading are not model failures. They are data failures that nobody looked for: a field that silently reads zero, a contract spec that differs between the server you trained on and the server you trade on, a feature that your Python code and your MQL5 code compute almost the same way.
DataForge is two MQL5 scripts that make those failures visible before you train, not after you lose money.
It never trades. No orders, no positions, no account access. It reads history and writes a text file.
# server=FundedNext-Server 3
# symbol=XAUUSD
# digits=2
# point=0.01
# contrat=100
# tick_value=1
# lot_min=0.01
# lot_step=0.01
# lot_max=3
Export the same symbol from two servers and diff the two headers. That is the whole idea.
★ Why it matters — a real result. Exporting XAUUSD from three brokers on the same day gave:
| field | MetaQuotes-Demo | Alpari-MT5-Demo | FundedNext-Server 3 |
|---|---|---|---|
| digits / point / contract | 2 / 0.01 / 100 | 2 / 0.01 / 100 | 2 / 0.01 / 100 |
| tick_value | 0.1 | 0.1 | 1 |
| lot_max | 100 | 100 | 3 |
Same symbol, same contract size, same tick size — and a tenfold difference in what one tick is worth. (Contract 100 × tick 0.01 = $1, so the funded server is the one that is right and the two demo servers report a tick_value inconsistent with their own spec.) A backtest built on one and deployed on the other is off by a factor of ten in money, and nothing in the platform warns you.
VOLUME OF THE BARS, PER YEAR (read via CopyRates, NOT the tester)
2019 : 23475 bars | tick_volume non-zero 23475 (100.0 %) | real_volume non-zero 0 ( 0.0 %)
2020 : 23611 bars | tick_volume non-zero 23611 (100.0 %) | real_volume non-zero 0 ( 0.0 %)
★ Why it matters — the second real result. The Strategy Tester's "history quality" bar describes the generated tick stream. It says nothing about the real_volume stored in the bars. Those are two different things, and confusing them means opening the volume door too early — or closing it for nothing.
Measured across three brokers:
real_volumeis zero on every source from 2019 to today;- where it does exist (2012–2018) the sources contradict each other — for 2015, one reports 100 % coverage and another 9.6 %; for 2017, 99.3 % against 0 %;
tick_volumeis present at 100 % everywhere — but it is a tick count, entirely broker-dependent, usable only in relative terms.
If you were about to build volume features, that table is the difference between a model and a fiction.
ForgeExport also writes the feature columns as MQL5 computes them (f0…fN). ForgeParity writes them again after normalisation (z0…zN). Your training code recomputes both and compares.
★ Why it matters — the third real result, and the sharpest one. In one project, the same feature code was checked by transliterating the MQL5 into Python and diffing the two Python versions. It reported a maximum error of 2.1e-10. When the actual MQL5 output was finally compared:
P1 raw features 8.882e-16 machine precision ✓
P2 after normalisation 1.339e+00 threshold 1e-5 ✗
Ten orders of magnitude apart. A check that never crosses the language boundary does not check the language boundary. Without it, an ONNX model would have gone live receiving inputs on a scale it had never seen — and it would have kept returning perfectly normal-looking probabilities.
- Open MetaEditor from MetaTrader 5 (F4).
- Copy the folders into your terminal's
MQL5directory, keeping the layout:Find that directory from the terminal: File → Open Data Folder.MQL5/Scripts/DataForge/ForgeExport.mq5 MQL5/Scripts/DataForge/ForgeParity.mq5 MQL5/Include/DataForge/ForgeFeatures.mqh - Compile each
.mq5with F7. Expect0 errors, 0 warnings.
Getting the repository:
git clone https://github.com/Sjrazaviebra/DataForge.git- Tools → Options → Charts → Max bars in chart → set it to Unlimited.
CopyRateswill never return more than this cap, and it raises no error when it truncates. - Open the chart, click on it, and hold
Homeuntil it stops scrolling back. A script reads what the terminal has cached, not what the server holds.
Then: Navigator (Ctrl+N) → Scripts → DataForge → ForgeExport, drag it onto an M15 chart.
| input | default | meaning |
|---|---|---|
EX_Depuis |
2004.01.01 |
start date; leave it wide — the script publishes the real bounds it found |
EX_MaxBarres |
600000 |
safety cap on the number of bars |
EX_AvecFeatures |
true |
also write f0…fN, the parity reference |
The volume table and the real bounds are printed to the Experts tab. The file lands in MQL5/Files/ and its name carries the server, so three sources give three files and nothing is overwritten.
ForgeParity does not recompute anything. It reads the raw features out of the export file and writes them back normalised, so the only thing under test is the one thing that should be: your scaler, applied by MQL5.
★ That is a deliberate design choice, and it came from a failure. An earlier version did recompute them — with its own warm-up depth, and two hours after the export ran. The result was that the normalisation test silently re-tested the feature computation instead, at a different depth and a different instant. Both sides now read the same lines.
- Your training code writes a scaler CSV whose header carries
# etage=and# empreinte_cadre=. - Drop it in the terminal's common folder —
File → Open Data Folder, then up toTerminal/Common/Files. - Run ForgeParity on the same chart.
| input | default | meaning |
|---|---|---|
PA_Export |
(empty) | export CSV; empty → DataForge_<SYM>_<SERVER>_M15.csv |
PA_Scaler |
(empty) | scaler; empty → DataForge_scaler_<SYM>_e<STAGE>.csv |
PA_Etage |
1 |
expected stage — checked against the scaler's own header |
PA_Empreinte |
(empty) | expected frame fingerprint; empty → not checked |
PA_DossierCommun |
true |
read the scaler from the common folder |
PA_Barres |
1000 |
parity window: the last N populated lines of the export |
★ The stage and fingerprint are verified, not assumed. A scaler from the wrong stage is refused instead of silently testing the wrong normalisation — the filename is no longer the only safeguard.
# key=value ← passport + per-year volume table, one line each
time,open,high,low,close,spread,tickvol,realvol,f0,f1,…,fN
2024-01-01 21:00:00,2062.99,2063.44,2062.85,2063.12,29,41,0,nan,nan,…
- Timestamps are
YYYY-MM-DD HH:MM:SS— ISO order, on purpose. MetaTrader's ownTimeToStringwrites2025.01.02with dots, and a parser that guesses day/month would shift03.04by three months without raising anything. - The first bars come out
nan: the recursive indicators need a warm-up, and the file says how many. - Comment lines are written in the terminal's ANSI code page. Read them tolerantly (
encoding_errors="replace"in pandas) — the data rows themselves are pure ASCII.
Two real headers are in examples/, so you can see the diff before running anything.
- M15 only. The feature set is calibrated for it; other periods are untested.
- Order-book depth is empty on every CFD broker tested (
MarketBookAddis accepted, and nothing comes out). Verified on three. - No trade direction. No tick carries a buy/sell flag on any tested broker (
BUY 0.000 %,SELL 0.000 %), so no order-flow delta is measurable. Any "signed volume" you build rests on the tick rule (Lee-Ready) — an inference, not a measurement, and its error rate belongs next to it. ForgeParityneeds a scaler produced by your own training code. It ships no model.
Source-available for reference, evaluation and demonstration. See LICENSE.
Built by Javad Razavi — The Solution Maker · javadrazavi.fr
See also HorizonProbe — the same discipline applied to time: at what holding period does your instrument stop being unprofitable by construction?