For quite a while now, there has been concern about how survivor bias and the over-reliance on $p$-values may lead to spurious false positives becoming conventional wisdom over time. Even worse there have been controversies in which famous claims have turned out not to be replicable. I will use this space to gather some of the discussion surrounding this issue, looking both at the structure of scientific publication intrinsically encourages false positives to survive, but also how healthy scientific debate ensures that false claims don't stand forever. Consider this an evolving set of documents, replete with placeholders and half-thoughts. I find this to be a fascinating intersection of my own mathematical interest in model selection and statistical decision making with the prevalent idea in modern political rhetoric that science is somehow "has it all wrong." --- A first paper worth looking at is [Investigating the replicability of the social and behaivoral sciences](https://www.nature.com/articles/s41586-025-10078-y), a massive effort including over 100 authors. Their opening paragraph captures the main thing we expect from scientific results. >*A central aim of science is to discover regularities in nature. If a claimed discovery is true, independent researchers should be able to conduct a similar investigation and reach similar conclusions. A replication attempt involves testing the same research question as a previous investigation with independent evidence, whether the evidence is a new data collection or existing secondary data that were not used in the previous investigation.* One fascinating insight was that whether or not a given result was considered successfully replicated is affected by how you define your measure of replication success. Plenty more to think through, but I just wanted to bookmark this here! ![[2026 Replicability Nature.pdf]]