Label construction can determine measured detector performance on small auditory brainstem response benchmarks

Background: Two small public corpora, the PhysioNet earndb tone-pip set and the Mendeley click set, recur as evaluation data for objective auditory brainstem response (ABR) detection. Neither corpus includes expert annotations of response presence. Studies that cannot commission their own annotations therefore construct the label automatically, most commonly by declaring a response present when two repeated sub-averages correlate above a threshold—the clinical two-replication reproducibility criterion. Whether detector comparisons built on that automatic label would survive a switch to expert annotation has not been tested.

Methods: Audiologist annotations published by Liu et al. (2025) were mapped onto the recordings (one rater, 204 earndb conditions; six raters, 86 Mendeley conditions) and the mapping validated against the record inventories. On identical data, we constructed the reproducibility surrogate and four detectors spanning the capacity range (the instrument statistic wnsfmp, logistic regression, gradient boosting, a multilayer perceptron). Each learned detector was trained under each label and scored against each label (a 2 × 2 factorial) by leave-one-subject-out cross-validation; discrimination is reported pooled and as a frequency-adjusted AUROC; inference rests on exact subject-level permutation tests.

Results: The surrogate agreed with expert judgment at κ = 0.167 (earndb) and 0.248 (Mendeley), against κ = 0.691 among the six Mendeley audiologists. It is largely a function of signal-to-noise ratio: a single-replicate instrument statistic predicted it with AUROC 0.789 on sub-averages in which the expert saw no response, and a simulation with no response reproduced this. Training on the surrogate cost 0.139–0.304 pooled AUROC against expert judgment (perceptron: every informative listener, exact p = 0.031), a loss arising across listeners and frequencies rather than within them; scoring fixed predictions against either label did not reorder detectors. Changing both, as a study would, reordered them: wnsfmp fell from 0.827 to 0.662, while the perceptron rose from 0.713 to 0.898 (interaction −0.350, exact p = 0.031); about 37% of this is pooling across frequencies with different expert prevalence, and the within-frequency remainder was not resolved. Rescoring against individual Mendeley raters moved AUROC by 0.065–0.082, below the smallest difference eight listeners can resolve (0.096–0.136). The stimulus level alone reached AUROC 0.931 within 1 kHz, exceeding every detector there.

Conclusion: On these corpora, measured detector performance depends on how the label is constructed and who provided it, at magnitudes equal to or larger than the differences between detectors. The label acts chiefly through what detectors learn, and the proxy leaks the input. We claim neither label is correct; we show that the choice determines the answer and release the mappings and a CPU-reproducible pipeline