The File Drawer // every test, the ones that found nothing too
Every validation test we have run
A seller of indicators shows the tests that worked. The tests that did not are what tell you whether the survivors were picked, and they usually go in a drawer. This page is that drawer, open: every test StockTools has run on its own methods and on claims it checked, with the control, the sample, how many variants were tried, the result, and what we did about it. Several found nothing, and a few killed the feature they were testing.
An entry exists only where a recorded run backs it, and each names the file or commit that holds the run. Every number in a sample or a result is copied from that record, and a check fails when one is not. 24 entries; the newest run is from Nov 2, 2026. Entries are added when a run is recorded, not on a schedule.
what the 24 tests found
4 measured, with no outside baseline
4 beat the method it replaced; shipped
4 matched the reference
3 made results worse; not shipped
1 no measurable effect
1 did worse than its baseline
1 could not tell it from its control
1 changed what we test
1 our published figure was overstated
1 claim withdrawn
1 the claim did not hold up
1 above its base rate; the gap is the claim
1 the first result was an artifact
Our methods, tested against a control or a baseline
Oct 5, 2026 · No measurable effect
Do resistance areas stop price?
Tested
Whether price that reached the nearest resistance area above the close, built from past swing highs only, then failed to close above it more often than chance.
Control
Random bands of the same width at a similar distance, overlapping no real area.
Sample
136 tickers, daily bars 2024-07-01 to 2026-10-02, evaluated every 25 sessions from session 60: 1,277 areas reached, 1,227 random bands reached.
Variants
One specification, fixed before the run and not tuned on results. Re-run the same day after the display cap rose from 10 to 25 areas: identical results.
Result
Held (no close above within 5 sessions of reaching it inside 20) 33.4%, 427 of 1,277, against 33.3%, 408 of 1,227, for random bands: +0.2 points, 95% bootstrap range over tickers -3.2 to +3.5.
Then
Shipped as a map of where price turned before, with this result printed on every resistance page; nothing on those pages says an area holds.
Does max pain land nearer the settle than no change?
Tested
Whether the max pain strike read before expiry lands within 0.5, 1 and 2 percent of the settlement price more often than a forecast of no change.
Control
The previous close: the spot at the last reading before expiry day.
Sample
613 settlements on 20 tickers, about 258 independent once settlements on the same expiry date are grouped.
Variants
Three tolerances, 0.5, 1 and 2 percent, all reported.
Result
The previous close landed within the tolerance 30, 52 and 77 percent of the time against 23, 44 and 66 for max pain, and the 95 percent interval on each gap excludes zero.
Then
Max pain is still published as a calculation, and its accuracy page prints the no-change baseline beside it.
Does price reach liquidation clusters more often than mirrors?
Tested
Whether price reaches a liquidation cluster on our map within a fortnight more often than a mirror band at the same distance on the other side of the price.
Control
A mirror band at the same distance on the opposite side, graded with each claim.
Sample
1,306 claims on 55 coins, grouped into 103 coin-fortnight blocks, because a coin’s daily claims share 13 of their 14 days.
Variants
Not recorded in the run note.
Result
Clusters minus mirrors: +3.4 points, 95% interval -1.1 to +8.3. The interval spans zero.
Then
The map still publishes; its pages say the record cannot yet tell clusters from mirrors, and the register keeps grading.
Can a claim about the largest cluster be graded at all?
Tested
Whether the largest cluster on each map sits close enough to the price to be reached inside the grading window.
Control
How far BTC actually travels in a fortnight, over the venue’s own candles.
Sample
The register’s first 67 claims; 900 days of candles.
Variants
One check.
Result
The largest cluster sat a median 37.5% from the mark and only 2 of 67 were closer than 15%. BTC travels 30% inside a fortnight 2% of the time.
Then
A register of largest clusters would have spent almost its whole sample on claims that could not resolve, so the nearest cluster above and below the price were registered as separate claims.
One candidate. The offset itself was never run, because the split ruled it out.
Result
Mean signed error by projected weekday: Monday -2.29, Tuesday +0.56, Wednesday +3.23, Thursday +0.79, Friday +1.70. A flat +1 day would have made Monday worse; the average was carried by Wednesday.
Replacing the weekday snap with a score that prefers the gap that followed the same quarter in earlier years, to fix projections that land a week early.
Control
The shipped estimator on the same predictions.
Sample
The walk-forward predictions of the estimator baseline (4,713).
Variants
One version; a second is the next entry.
Result
The +5 to +8 day cluster fell from 18.4% to 13.5%, but a -5 to -8 day cluster rose from 4.1% to 14.8%. Exact day fell from 46.8% to 40.0%, within 1 day from 61.6% to 53.4%, while the mean bias went from +1.08 to -0.01.
Then
Not shipped. The finding kept: a mean bias near zero can come with a worse model.
A rare override to the later week, firing only when a company’s same-quarter history was large, tight and disagreed with the projection by exactly 7 days.
Control
The shipped estimator on the same predictions.
Sample
4,713 predictions.
Variants
Four parameter settings, all reported.
Result
It fired on 5 to 11 predictions (0.11% to 0.23%); precision among the changed outcomes was 17% to 38%, below 50% at every setting, and the net exact-day change was negative at all four (-2 to -5).
Then
Not shipped. Two attempts bounded what reporting history alone can fix, and the work moved to reading the company’s own filings.
Jul 22, 2026 · Beat the method it replaced; shipped
Can the confirmation parser stop confirming replay dates?
Tested
A filter that rejects a date attached to a replay, archive or transcript, or to "available through", before the earnings confirmation parser can confirm it.
Control
The same harness and the same documents before the change; only the parser changed.
Sample
80 discovered scheduling documents.
Variants
One rule, tried on 8 sentences first, including a month-range trap.
Result
Replay or availability dates among confirmed dates fell from at least 1 to 0; 6 documents moved to a new availability-window bucket; confirmations went from 25 to 22, and the genuine all-dates-past count stayed at 35.
Then
Shipped. It can only reject a date, never newly confirm one. The parser’s precision against the dates companies actually reported has not been measured.
Oct 12, 2026 · Beat the method it replaced; shipped
Is every earnings 8-K the quarter’s report?
Tested
A classifier that anchors each Item 2.02 filing on the company’s own 10-Q or 10-K, so a pre-announcement no longer counts as the quarter’s report.
Control
The previous engine, walked forward on the same actual report dates.
Sample
The 142-ticker universe and 1,518 more calendar filers; 60 randomly drawn filings read by hand; 58,843 points compared with the prototype.
Variants
One rule, revised once: the hand check found 3 delinquent-filer releases wrong in the first cut, and they were fixed.
Result
Universe: within 7 days 87.4% to 90.9%, misses over 30 days 361 to 192, 178 projections improved and 5 harmed. Other filers: 85.2% to 89.0%, misses over 30 days 4,660 to 2,562, 2,573 improved and 105 harmed. Hand check: 60 of 60 correct after that fix.
Do our high, medium and low labels on earnings dates mean anything?
Tested
Whether projections labelled high land nearer the real date than those labelled medium or low, walked forward.
Control
The tiers against each other; no outside baseline.
Sample
4,778 scored points across 136 companies, 2005 to 2026: high 4,662, medium 116, low none.
Variants
One rule, the one the engine runs; it agreed with the live engine on 139 tickers, 0 disagreed.
Result
High: 69.0% within 3 days, 91.5% within 7. Medium: 62.9% and 87.9%. Low: no scored points, because scoring starts at a company’s fifth report. 97.6% of scored points carried a single label.
Then
Shipped saying the labels ordered correctly, with the single-label caveat printed as prominently. Withdrawn in part on 2026-10-21 (below).
Whether a 90% band drawn from a pool of 20 trades covers what the trader’s whole record produces, and how often real stretches of 20 fall outside it.
Control
Draws from the whole log, and its real 20-trade stretches.
Sample
One real log of 444 trades (dollar results, no stops); 22 stretches of 20.
Variants
One method.
Result
A band drawn from 20 trades held a median 63% of what the whole log produced, not 90%; real stretches fell outside it 4 times in 22, about twice what independent draws give.
Then
Shipped with the caveat printed in the panel, which leads with the middle half and says the band runs narrow.
Our own computations, checked against an independent reference
Jul 22, 2026 · The first result was an artifact
Why were most fair value bands unrated?
Tested
A sitewide audit of how many stocks the fair value band could rate, and why the rest failed.
Control
The data feed the audit depended on, checked before trusting the failures.
Sample
The sitewide audit, which read 110 of 152 as unrated.
Variants
One audit.
Result
105 of the 110 failures were the audit itself exhausting a rate-limited price feed. The real failures: 4 with no price and 1 lookup that found nothing.
Then
The audit’s figure was not reported as a finding, and a job that keeps the band’s inputs warm was added. The band itself has never been checked against realised prices.
Aug 8, 2026 · Beat the method it replaced; shipped
Rebuilding the fourth quarter from annual filings
Tested
Reconstructing a missing Q4 from the annual figure, with the nine-month figure checked against the quarters the company filed and an abstention when they disagree.
Control
The same 142 tickers before the change, in one run.
Sample
142 tickers.
Variants
One rule; a guard added after two filers broke the naive version.
Result
550 quarters recovered, 78 tickers gained a Q4, zero duplicate period-ends, and four-quarter coverage went from 12 to 57.
Then
Shipped.
Record
collab/DECISIONS.md:193
Oct 5, 2026 · Matched the reference
Are moves on an exact split ratio really splits?
Tested
Whether an overnight move within 2.5% of a split ratio is a split, since our daily bars are not split-adjusted.
Control
Each company’s announced split.
Sample
137 tickers with bars since 2024-07-01.
Variants
One rule; 5-for-4 left out because news gaps landed within 1% of it.
Result
Six overnight moves sat within 2.5% of a split ratio, and all six were real splits.
Then
Shipped: prices before a split are divided by its ratio.
Record
site/src/lib/bar-adjust.ts:8-24
Oct 8, 2026 · Beat the method it replaced; shipped
Which volume baseline tells a corporate action from news?
Tested
Which ordinary-volume baseline to compare a large down gap with, so that history is cut at corporate actions and not after a volume spike.
Control
Known events: NCRA after a spike, SITC’s special distribution, FDX’s disposition.
Sample
15 recent small-cap movers.
Variants
Five baselines: a plain 20-session mean, a median, leaving out the previous session, and leaving out sessions above 5x or above 3x the median.
Result
The plain mean cut 18 times, 12 of them behind a spike. The median lost SITC at 1.99x. Leaving out the previous session still cut NCRA at 1.71x. Leaving out sessions above 5x the median kept FDX at 1.44x and SITC at 1.49x and read NCRA at 24x; 3x cut the same events.
Then
Shipped leaving out sessions above 5x the median; the plain mean, the median and the previous-session rule were set aside.
Record
site/src/lib/bar-adjust.ts:25-42
Oct 13, 2026 · Matched the reference
Telling a reverse split from a news gap by volume
Tested
Whether share volume separates reverse splits from news moves among up gaps that the old dollar-volume test let through.
Control
The known reverse splits among them.
Sample
35 up gaps.
Variants
One threshold, share volume at 2x its 20-session average.
Result
The reverse splits traded at most 1.12x their usual shares; the news moves 2.64x or more.
Then
Shipped in the scans.
Record
site/src/lib/scans.ts:94-99
Sep 23, 2026 · Matched the reference
Where a company is incorporated: cover tag or directory?
Tested
Whether the state of incorporation in EDGAR’s company directory agrees with the one the filer tags on the cover of its annual report.
Control
The cover tag, which the filer writes itself.
Sample
142 ingested tickers: cover tag present 136, directory present 134, both present 129.
Variants
One comparison.
Result
Of the 129 with both, 3 disagree, and in all three the directory is wrong (DELL, SMCI, CCL).
Then
Shipped reading the cover, with each disagreement named on the page.
These are published without a recorded test, and the site says so where each one appears. They join the drawer when a run is recorded, whatever it finds.
The fair value band: Never checked against realised prices. (site/src/lib/provenance.ts:123)
F-Score: Computed from filings; not yet checked against a published reference set. (site/src/lib/provenance.ts:108)
M-Score: Computed from filings; its false-positive rate has not been measured. (site/src/lib/provenance.ts:118)
The earnings confirmation parser’s precision: Its harness measures the decision, not whether the confirmed date was right. (site/src/app/api/earnings-parser-eval/route.ts:109)
The AI Trade Planner’s filing-based parts: They cannot be tested historically without look-ahead, so they are not; only the price-trend part is backtested. (site/src/app/(legal)/disclaimer/page.tsx:124)
Records that keep grading
Some of the tests above became running records. These pages compute their figures from stored rows each time they are built, so their numbers move past the dated runs here:
Max pain accuracy: every settlement since the record began, beside the no-change baseline.