Direct answer
Execution quality for an ASIC (an automated market- or execution-related system) is assessed by comparing what actually happened to what was intended, using measurable metrics. Because you cannot assume future outcomes from past results, good assessment also requires documenting limitations, failure modes, and how much of the result comes from stable mechanics versus variable market and provider conditions.
This article focuses on general, verifiable methods that do not depend on real-time data or on promising outcomes.
Definition and core mechanics
“Execution quality” describes how well an order is translated into real transactions at costs and timings consistent with the order’s goals. In practice, assessment usually separates three parts:
- Intended parameters: what the system tried to do (order type, size, direction, time constraints, and target price or price protection).
- Observed execution outcomes: what happened (fill price(s), fill time(s), whether fills matched the intended quantity, and whether routing or handling changed the order).
- Cost and frictions: the components that move results away from the ideal (effective spread, commissions/fees, and slippage relative to a chosen reference).
A key concept is reference price. Slippage is only meaningful relative to a defined benchmark (for example, the quote observed right before the order, or the best available price at that moment). If the reference is unclear, “slippage” can be misleading even when the calculations look precise.
Evidence and a measurable assessment approach
To assess execution quality in a way you can independently verify, build a consistent worksheet that uses the same reference and the same event timestamps across trials.
1) Price execution quality (fill vs. reference)
- Compute effective fill deviation: for each fill, measure the difference between the actual executed price and your reference price.
- Summarize using average deviation and dispersion (for example, median and a percentile range). Dispersion matters because a system can sometimes achieve good average results while still failing in a meaningful fraction of orders.
Assumption for examples: choose one reference definition (e.g., “top-of-book at order submission”) and keep it fixed across the dataset.
2) Timing execution quality (delay and latency sensitivity)
- Measure time from order submission to first acknowledgement (if available) and to first fill.
- Compare execution delay against short-term price movement in the same window. If execution is consistently late relative to price changes, you should expect worse price deviation.
Assumption for examples: use a consistent timestamp source and clearly document which timestamp you mean (submission time, receipt time, or matching time).
3) Quantity and order-handling quality (fill completeness and correctness)
- Track whether the executed quantity matches the intended quantity, and whether partial fills occur frequently.
- Check whether rejections, amendments, cancellations, or route changes happen under conditions that can be explained by mechanics rather than by randomness.
Material limitation and failure mode to look for: partial fills and unintended order handling can inflate effective cost even if the first fill looks acceptable. Another common failure mode is delay caused by handling (for example, throttling, queueing, or validation steps) that makes the reference price stale.
Realistic scenario-impact checkpoints
- Scenario: A rapid price move happens shortly after submission.
- Possible impact: Effective slippage worsens because the executed price is compared to an earlier reference.
- Limitation: Without a stable reference definition and consistent timestamps, you cannot tell whether the system or the market caused the difference.
- Scenario: Liquidity thins and the same order size cannot be filled at the desired levels.
- Possible impact: Partial fills or worse fill prices occur.
- Control point: Compare executions across multiple market states and include the spread/liquidity regime as a contextual variable.
Limitations, risks, and what not to conclude
Even if you compute solid metrics, there are limits:
- Market and provider conditions change: spreads, liquidity, and microstructure vary. Relationships observed historically do not guarantee the same behavior later. - Hidden costs can be hard to measure: some venues or systems include costs not visible in the “headline” execution price. If your dataset does not include all relevant fee and routing details, you may misattribute cost. - Selection bias: testing only on favorable periods can make execution quality look better than it is.