Common misunderstandings that break performance metrics
Performance metrics are numbers used to describe how an activity went—often with the aim of comparing results across time, accounts, or methods. A common mistake is treating these metrics as objective truth rather than as outputs that depend on definitions, data choices, and assumptions.
Another frequent error is mixing stable mechanics with variable conditions. The mechanics are the formula and its inputs. The variable conditions are market movement, execution quality, transaction costs, and even changes in measurement settings. If you blur those, you can end up attributing differences to “performance” when they are really measurement differences.
Mechanics: what metrics usually rely on
A performance metric is only as correct as the information used to compute it. Common mistakes include:
- Changing the metric definition midstream (for example, switching between gross and net returns) without recording what changed.
- Using inconsistent time windows (daily vs. weekly), or mixing realized results with partial or estimated values.
- Calculating with the right formula but wrong inputs (for example, forgetting to include fees or using a proxy for price that differs from execution price).
- Comparing metrics across datasets that are not aligned (different start/end dates, different sampling frequency, or different coverage of trades).
When you include an example, state the assumptions explicitly: what inputs were used, how costs were handled, and whether results were realized or estimated. Without these assumptions, the example cannot be independently verified.
Evidence and examples of failure modes
Even when a metric is computed “correctly,” it can still mislead because of the failure mode:
-
Survivorship and selection bias If only the “successful” periods, accounts, or test runs are included, the metric can look stronger than the full history. The metric then reflects selection, not performance.
-
Small samples and unstable averages A metric based on few events (few trades, few days, few months) can swing dramatically. Averages or ratios may look meaningful but be mostly driven by randomness.
-
Performance vs. distribution shape Two series can share the same average outcome while one has many small gains and rare large losses. If you only track one summary metric, you may miss material differences in risk exposure.
-
Cost and execution invisibility Ignoring or estimating costs can create an overly optimistic picture. Execution conditions—like timing and fill quality—affect realized outcomes, so a metric based on assumptions rather than recorded execution can be internally inconsistent.
Limitations and risks to treat as baseline knowledge
Performance metrics are descriptive, not predictive guarantees. Historical relationships do not establish future results, and outcomes vary with market conditions, costs, and execution details. A material limitation is that metrics often compress complex behavior into one or a few numbers, which can hide drivers and make conclusions feel more certain than they are.
Neutral limitations-based checks:
- Verify the metric definition (inputs, units, inclusion rules) matches across the periods being compared.
- Check whether the metric uses realized values and recorded costs, or whether it relies on estimates.
- Confirm the dataset size and coverage (avoid mixing partial runs with full runs).
- Separate mechanics (formula correctness) from conditions (what changed in the environment).
Verification and next questions
If you want to independently verify performance metrics, you can start by writing down: the metric definition, the exact inputs, and the stated assumptions for any example or calculation. Then re-calculate using those same assumptions and confirm you obtain the same result.
A useful next question is whether your metric answers “what happened?” with clear measurement rules, or whether it is implicitly being used to argue “what will happen.” If it is the latter, you should treat the uncertainty as part of the metric’s limitations, not as an afterthought.