Repository navigation
Benchmark comparisons can pair returns covering different periods #584
Description
Summary
Matching date labels does not necessarily match the periods covered by the returns. This issue tracks the return-period matching decision separately from #557, following @arhancanli’s observation there.
For example, a strategy trading through the weekend earns +1% Saturday, −2% Sunday and +3% Monday: 1.9494% compounded. A benchmark tracking the same underlying growth but observed only on Friday and Monday reports 1.9494% on Monday. Comparing that with the strategy’s Monday-only 3% pairs different periods.
Holidays cause the same issue whenever the strategy continues trading while benchmark observations are unavailable. Weekly labels alone do not guarantee matching periods either: a Friday holiday can leave the benchmark valued on Thursday and the strategy on Friday.
In Test 1 below, the strategy and benchmark followed identical underlying performance. Observing the strategy daily but the benchmark only on weekdays changed beta/R-squared from 1.0000/1.0000 to 0.7186/0.5186. Compounding both series over matching intervals restored 1.0000/1.0000. Simply dropping weekend strategy returns did not fix the mismatch.
Two possible approaches
- Callers prepare matching intervals. Document how callers should compound both inputs over the same boundaries before calculating comparison statistics. This is the smaller, compatible change, but leaves the preparation to users.
- QuantStats prepares matching intervals. Combine returns over matching boundaries automatically. This removes that preparation burden, but changes the observations used and requires explicit rules for period boundaries, the first observation, rolling calculations and annualization.
Other libraries provide precedents for both approaches. Industry guidance supports comparable measurement periods, but does not determine which responsibility QuantStats should adopt.
Proposed implementation: PR #585 adds opt-in return-period matching. Its API, weekend and holiday handling, and alpha annualization rules await maintainer approval. The look-ahead correction remains separate in #557.
Reproduction evidence
These results are a historical investigation snapshot, tested against PR commit 09abf29; they are not a fresh verification of the latest PR head.
Test 1 isolates the calendar mismatch using identical underlying performance. Test 2 shows that the issue also affects a strategy that differs from its benchmark, including its information ratio.
Results and test setup
Test 1 — identical underlying performance
The strategy follows the market exactly, without noise or extra return. The first five rows use a market that changes every day; the last is a separate flat-weekend control.
| Input preparation | Beta | R-squared |
|---|---|---|
| Both series observed daily | 1.0000 | 1.0000 |
| Strategy daily; benchmark observed only on weekdays | 0.7186 | 0.5186 |
| Keep only weekday strategy returns, without compounding | 0.7186 | 0.7232 |
| Compound both series between benchmark observations | 1.0000 | 1.0000 |
| Compound both series to Friday-ended weeks | 1.0000 | 1.0000 |
| Separate flat-weekend control: strategy daily; benchmark observed only on weekdays | 1.0000 | 1.0000 |
The flat-weekend control avoids the mismatch because no growth is omitted between benchmark observations.
Test 2 — strategy with noise and an extra daily return
Daily beta is 1 by construction; R-squared is below 1 because the strategy differs from the market.
| Input preparation | Beta | R-squared | Information ratio |
|---|---|---|---|
| Both series observed daily | 1.0000 | 0.9168 | 0.1000 |
| Strategy daily; benchmark observed only on weekdays | 0.7179 | 0.4745 | 0.0372 |
| Keep only weekday strategy returns, without compounding | 0.7179 | 0.6609 | 0.0341 |
| Compound both series between benchmark observations | 0.9972 | 0.9161 | 0.1178 |
| Compound both series to Friday-ended weeks | 0.9978 | 0.9182 | 0.2659 |
Unlike Test 1, this differing strategy need not retain beta exactly 1 after compounding. Information ratio is unannualized and frequency-dependent: the daily, matched-interval and weekly values should not be interpreted as comparable improvements.
The accepted-upstream baseline 9ef4c6d also exhibited the issue. For Test 2’s daily-strategy/weekday-benchmark case, it produced beta 0.7094, R-squared 0.4633 and information ratio 0.0368.
Setup and boundaries
- Exercised
_prepare_benchmark,greeks,r_squared,information_ratioand equivalent pandas extension calls. Changing Monday’s benchmark return left earlier rows unchanged. - Both experiments used 2,000 deterministic weeks, seed 557, market draws at 1% scale and a shared initial zero-return observation. Benchmark prices were compounded from market returns; expected interval returns were independently compounded.
- Test 2 added mean-zero noise at 0.3% standard deviation, constructed to preserve daily beta exactly 1, plus a 0.03% daily increment.
- Environment: Python 3.11.16, pandas 3.0.6, NumPy 2.4.6 and SciPy 1.17.1.
- A new implementation should retain protections against future observations affecting earlier results, timezone errors, caller-input mutation and changes to intraday behavior. Existing regression expectations should change only after the contract is agreed.
Other systems and industry guidance
PerformanceAnalytics, Empyrical-reloaded, vectorbt and ffn generally leave comparable-period preparation to callers. QuantConnect LEAN’s Beta indicator instead pairs price observations before calculating returns.
GIPS guidance supports comparable measurement periods within its reporting scope. That supports matching the periods being compared, without choosing whether callers or QuantStats should perform the matching.
Sources and qualifications
| System | Relevant behavior |
|---|---|
| PerformanceAnalytics | Return calculation assumes regular observations and recommends converting irregular observations first. Guidance. |
| Empyrical-reloaded | Beta aligns return dates; compounded aggregation is a separate operation. Statistics, alignment. |
| vectorbt | Comparison statistics consume supplied returns; resampling is separate. Statistics, resampling. |
| ffn | Information ratio compares supplied returns on matching dates without combining their periods. Implementation. |
| QuantConnect LEAN | Beta calculates returns between consecutive matched price observations; missing asset bars leave the indicator unchanged. Pairing, Beta. |
These comparisons came from source/documentation inspection, not execution of our dataset in those systems. LEAN’s behavior depends on the supplied bars.
- GIPS: §2.A.18 requires the same measurement frequency for portfolio and benchmark additional risk measures; §2.A.22 addresses comparable annual valuation dates, including weekends and holidays. Firms handbook. Benchmark returns must cover the same reporting periods. Benchmark guidance.
- Information ratio: GIPS §32.A.32 specifies monthly composite relative returns for fiduciary management providers—a particular reporting context, not a universal library rule. FMP handbook.
- CFA Institute research: Return frequency can change measured beta. This is research, not an implementation prescription. Financial Analysts Journal paper.
- CQF: The material reviewed did not establish a specific weekend/holiday alignment rule. Conference material.
Applying reporting guidance to QuantStats is an inference, not a prescribed daily alignment algorithm.
AI assistance disclosure: OpenAI Codex executed the reproduction against committed source.
Activity
arhancanli commentedon Oct 27, 2026
Thanks for splitting this out, and for the tables: Test 1 matches what I saw in my own simulation on #557.
One point for the "QuantStats prepares matching intervals" option. Your Test 2 shows the information ratio going from 0.1000 (daily) to 0.1178 (matched intervals) to 0.2659 (weekly), unannualized. If the library resamples internally, annualization has to use the number of matched intervals per year, not 252 or 365. Otherwise the resampled IR looks like an improvement it isn't. The same applies to rolling windows, where a window of N would silently change from N days to N matched intervals.
That cost is an argument for the smaller option first: a documented helper or docstring that says
greeks,r_squaredandinformation_ratioassume both series observe the same calendar, and shows how to compound both to shared boundaries. Does the contract Igor picks need to cover strategies that are daily and benchmarks that are weekly, or only the weekday-versus-7-day case?JohnDReynolds commentedon Oct 30, 2026
Thanks, @arhancanli. Your annualization and rolling-window points helped shape the proposal in #585. It supports different input frequencies, including daily versus weekly, with the user explicitly choosing the comparison frequency. Alpha annualization is addressed separately and called out for maintainer approval. The proposal does not change rolling calculations, reports or plots. It takes the broader implementation approach rather than the documentation-only option you suggested, but that choice still needs maintainer approval.