Direct answer: what to measure
Execution quality for an NFA approach should be assessed by comparing planned trade execution details with actual outcomes, using measurable process factors rather than relying on future performance claims. Focus on items you can observe or reconstruct from your own records (order timestamps, fill timestamps, executed prices, commissions/fees, and reported spreads), and evaluate how those items change under different liquidity and volatility conditions.
Because market conditions and trading frictions vary, any evaluation should be framed as a test of execution mechanics—not a prediction of returns. Also, be careful with evidence quality: incomplete reporting, missing timestamps, or aggregations can make execution quality appear better or worse than it is.
Mechanism and definition: what “execution quality” means
Execution quality is the closeness between what an order intends and what the trading system actually delivers. For an NFA-style concept, you typically want to examine the relationship between:
- Order intent: direction, size, order type, and the time the order is submitted.
- Execution path: when the system routes the order, how long it waits, and whether it can match your request.
- Actual outcome: fill time(s), fill price(s), any partial fills, and the total transaction cost you pay.
To keep the assessment meaningful, separate stable mechanics (how fills are handled and reported) from variable conditions (spread changes, market volatility, and liquidity). Stable mechanics are about whether the system consistently produces fills near the intended price given similar conditions.
Key measurable metrics you can compute from records are:
- Latency: time between order submission and fill (or between key processing events you can observe).
- Slippage: the difference between an intended reference price (your planned price or a defined mid/quote reference at submission) and executed price.
- Total cost: explicit fees/commissions plus implicit costs such as execution away from reference.
- Fill reliability: rates of full fills vs partial fills, and the frequency of rejected or non-filled orders.
Evidence or example: a simple, assumption-driven test
A practical way to evaluate execution quality is to run an offline comparison between planned and actual details.
Example (with explicit assumptions):
- Assumption A: you define a reference price at submission time as the midpoint of the best available bid/ask you observed (or another consistent reference you can justify).
- Assumption B: you compute slippage as: executed price − reference midpoint for buys, and reference midpoint − executed price for sells.
- Assumption C: you include all observable transaction costs in total cost.
Then, for each order, record:
- submission timestamp
- fill timestamp(s)
- executed price(s)
- whether the order was fully filled
- the total cost
After collecting enough orders across different conditions, you can summarize:
- average and distribution of slippage (not just one number)
- typical latency range
- partial-fill frequency
- whether execution outcomes become worse as volatility rises
This helps you isolate whether execution quality degrades systematically due to mechanics (for example, frequent partial fills or large slippage under moderate conditions) versus simply reacting to the market being harder to trade.
Limitations and risks: material failure modes
Several limitations can undermine an execution-quality assessment:
- Reference mismatch: if your intended reference price is not consistent (for example, using different quote sources across time), slippage calculations become unreliable.
- Incomplete reporting: missing timestamps or aggregated fills can hide delay or routing effects.
- Selection bias: only analyzing “good” trades can overstate execution quality.
- Non-stationary relationships: a pattern observed historically (for instance, slippage being small in one period) does not guarantee the same behavior later.
- Failure modes: execution can fail in ways that look like “performance” elsewhere. Common examples include partial fills that change average price, order rejections, and timing gaps that increase slippage during fast moves.
Also note: even if execution quality looks consistent, outcomes still vary because the market moves and spreads change. So an execution-quality metric is best treated as a diagnostic about trade mechanics, not as proof of future results.
Verification and next question: what to check independently
To verify execution-quality conclusions, check whether your evidence includes the minimum observable elements: order submission time, fill time(s), executed price(s), and transaction costs. If any are missing, your analysis may be incomplete.