Repository navigation
registry: sum_codes, and the equity full-size contracts added to the minis - #124
Merged
Merged
Conversation
…minis ES, NQ, YM and EMD each traded a full-size contract beside the mini for years and the registry read only the mini. In notional terms the big contract held 99% of S&P 500 open interest in 1997 to 1999, 82% in 2004 and 54% in 2008, so the stored positioning before roughly 2008 was a minority, retail-skewed slice of each market. New registry field sum_codes: parallel listings ADDED to the primary week by week in its contract units, with an optional from_date per scale (the big S&P was $500 x index until the 1997-11-04 report, 10 E-minis, then 5). get_cot sums OI and positions, takes the max of trader counts, recomputes Pct_of_OI_* and Change_in_* from the sums, keeps concentration ratios and metadata from the primary. Weeks only the listing reports extend the series: ES from 1986, NQ from 1996, YM from 1997. Verified against the CFTC Consolidated rows 2010-06 to 2021-09: OI and both commercial columns agree to rounding on 99.8% of NQ and YM weeks and 93% of ES weeks, the rest being weeks the dying big contract fell under the 20-trader publication threshold. Also fixes the hist_codes scale multiplying legacy trader counts (corn showed 72 traders on 1997-12-30 against 349 a week later). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sum_codes: parallel listings of the same index, ADDED to the primary week by week in the primary's contract units. Entries are[code, scale]or[code, scale, from_date](a dated entry takes over from that report date). Wherehist_codespicks one row per week,sum_codesadds rows, and weeks only the listing reports extend the series.138741,209741,124601,338741). The registry read only the mini, and in notional terms the big contract held 99% of S&P 500 open interest in 1997 to 1999, 82% in 2004, 54% in 2008 (NQ 97% in 1999, YM 84% in 2002). Scales: ES 5, and 10 before the 1997-11-04 report (CME halved the contract from $500 to $250 x index; OI doubles that week); NQ 5; YM 2; EMD 5.get_cotsums OI and every*_Positions_*column, takes the max ofTraders_*across listings, recomputesPct_of_OI_*andChange_in_*from the sums as floats, keeps concentration ratios and text metadata from the primary, and labels every row with the primary code. ES now starts 1986-01-15 (was 1997-09-16), NQ 1996-04-16 (was 1999-06-22), YM 1997-10-07 (was 2002-05-21).hist_codesscale applied to every numeric column, so legacyTraders_*were multiplied by lumber's 4.0 and the grains' 0.2 (corn showed 72 traders on 1997-12-30, 349 a week later). Scaling is now restricted to contract-denominated columns.registry.codes_for(sym)returns every code a producer must fetch; all four producers use it.Verification
Against the CFTC's own Consolidated rows (
13874+,20974+,12460+, which are big + mini/5 in big units, 2010-06 to 2021-09), after a real producer run into a scratch store:ES's remainder is weeks the dying big contract fell under the CFTC's 20-trader publication threshold: absent from the file, still inside Consolidated (up to 6% of OI in mid-2021). Not recoverable from published per-code rows. Non-commercial long/short do not reconcile by construction: the CFTC computes spreading per trader across both listings.
The 1997-11-04 scale switch is flat at the seam (ES OI 2.07M on 10-28, 2.02M on 11-04 in mini units, trader count 171 to 161).
Test plan
tests/test_cot_sum.py: scaled sum with a dated scale, Pct and Change recomputed, Traders max (including text-typed columns), metadata and code from primary, missing listing file leaves primary untouched, hist scale leaves trader counts aloneruff check src tests scriptscleanNot included
Micro contracts (
13874U,209747,124608,239747): same mechanism at scale 0.1, deliberately left out pending a decision on whether the retail micro pool belongs in the series.Deployment
The producers fetch the four new per-code files on their next run after the Windows box pulls. Until then the four symbols read exactly as before.
COTMETRICS_CACHEentries for ES, NQ, YM, EMD carry the mini-only series until rebuilt.🤖 Generated with Claude Code