Status. An HQ-S26 research note: work done by fellows of the HyperQuark Research Fellowship between April and July 2026, written up by the lab from the track's own files. It is a small study: 16 questions, three samples per model, one question rerun at twenty. It has not been reviewed outside HyperQuark, and its code and data have not been released.
The question
How do we know a model is reasoning, and not just sounding right? For the strongest models, answer accuracy is close to perfect on ordinary questions, and a language model asked to judge another model's reasoning rates nearly all of it faithful. If a judge rewards a coherent-sounding chain whether or not it reaches the right answer, the judge cannot tell good reasoning from confident error.
The track proposed looking at several signals at once and treating their disagreement, question by question, as the thing to inspect.
HQ-S26 was HyperQuark's first cohort: twelve fellows in four tracks, twelve weeks, weekly reports due every Sunday. No track reached a final submission, so every result here is interim and is reported as the track left it.
Method
Questions. Sixteen career-domain questions, written by the track in four kinds: recall, multi-step arithmetic, paired questions where one version carries a flaw, and adversarial traps. They include restricted-stock vesting with a cliff, take-home pay on a bonus, promotion eligibility, leave arithmetic, a shift that crosses a daylight-saving change, a self-referential bonus rule and a cross-time-zone meeting.
Models. Claude Sonnet 4.6 and GPT-5.4 on all sixteen questions, three samples each at temperature 0.7 (96 answers); Claude Opus 4.7 on three of the questions at its default temperature (9 answers).
Four signals per question and model: (1) accuracy against the expected answer; (2) a judge from one model family (Qwen3-32B); (3) a judge from another (Kimi K2.6); (4) on four questions, perturbation checks after Lanham and colleagues: cut the reasoning short or replace it with filler, and see whether the answer changes.
The diagnostic. For each question and model, the gap between accuracy and the judges' mean faithfulness, both on a 0 to 1 scale:
Δ = | accuracy − mean judge faithfulness |
A large Δ with low accuracy is the case of interest: reasoning the judges approve, reaching an answer that is wrong.
Results
The largest gaps (after the track's own correction of an answer checker, described under Limitations):
| Question | Model | Accuracy | Judge faithfulness | Δ |
|---|---|---|---|---|
| Self-referential bonus | Opus 4.7 | 0.00 | 1.00 | 1.00 |
| Self-referential bonus | Sonnet 4.6 | 0.00 | 1.00 | 1.00 |
| Daylight-saving shift | Sonnet 4.6 | 0.00 | 0.83 | 0.83 |
| Manager disclosure | GPT-5.4 | 0.67 | 1.00 | 0.33 |
| Cross-time-zone meeting | GPT-5.4 | 0.33 | 0.67 | 0.33 |
The rerun. The self-referential bonus question was rerun with 20 fresh answers per model (95% Wilson intervals):
| Model | Accuracy | Judge faithfulness | Δ |
|---|---|---|---|
| Sonnet 4.6 | 0.00 (0.00 to 0.16) | 1.00 (0.91 to 1.00) | 1.00 |
| Opus 4.7 | 0.05 (0.01 to 0.24) | 0.97 (0.84 to 0.99) | 0.93 |
| GPT-5.4 | 0.31 (0.14 to 0.56) | 0.68 (0.52 to 0.80) | 0.36 |
For GPT-5.4, 16 of the 20 answers could be scored. At three samples, GPT-5.4's gap on this question had looked small; at twenty it did not, which is the reason to rerun before believing a small sample.
The judges agreed with each other on 86 of 96 answers (89.6%, interval 0.82 to 0.94), and on every answer for 12 of the 16 questions. Agreement between judges is not correctness: on the bonus question they agreed, and were both wrong about the reasoning.
Perturbation checks showed one further case worth a note: on the vesting question, GPT-5.4 kept its answer in every run when its reasoning was replaced with filler, so that reasoning was not what produced the answer.
Limitations
-
Small. Sixteen questions and three samples per cell; not powered to estimate an effect size. Only one question was rerun at twenty.
-
One judge per family, and Opus 4.7 ran at a different temperature and on three questions only.
-
Contested expected answers. The bonus question has no published answer; the daylight-saving question has a defensible second reading.
-
Corrections. The track found and fixed its own errors mid-study: the first checker for the daylight-saving question read the whole response rather than the final answer, and judge failures were first attributed to the wrong judge. The numbers above are after both fixes.
-
Perturbation checks cover four questions, none of them the three with the largest gaps.
-
Not reproduced by a second person. The analysis can be replayed from saved answers, but new answers were produced by one person.
-
Scope. These are results on these questions, not verdicts on these models.
What would settle it
More questions and samples, a second judge per family, perturbation checks on the high-gap questions, and a second person producing fresh answers.
Contributions
-
Siddhant Kasture (Track Lead): test harness, perturbation checks, all model runs, the cross-channel analysis, the corrections and the rerun, and the paper drafts this note draws on.
-
Destynee Tau: wrote the question sets, including the redesign to sixteen questions and the documented expected failures.
-
Kavindu Ravishan Perera: compared candidate judge models and proposed the two-judge design and its output format.
-
Bishnu Dev Changkakoti (Programme Director): set the fellowship's research agenda and directed the track.
Data and code
Not released. The questions are described here at the level of the track's paper draft.



