Higgs Boson Confirmed at 5-sigma Standard Deviations at 125 GeV
experimental physics - Help interpreting "five sigma" standard? - Physics Stack Exchange
hypothesis testing - Origin of "5$\sigma$" threshold for accepting evidence in particle physics? - Cross Validated
Statistics behind the 5.9 sigma Higgs boson result
Videos
The Higgs-discovery experiment is a particle-counting experiment. Lots of particles are produced by collisions in the accelerator, and appear in its various detectors. Information about those particles is stored for later: when they appeared, the direction they were traveling, their kinetic energy, their charge, what other particles appeared elsewhere in the detector at the same time. Then you can reconstruct “events,” group them in different ways, and look at them in a histogram, like this one:
[Mea culpa: I remember this image, and others like it, from the Higgs discovery announcement, but I found it from an image search and I don’t have a proper source link.]
These are simultaneous detections of two photons (“diphotons”), grouped by the “equivalent mass” 
of the pair. There are tons and tons and tons of photons rattling around these collisions, and directional tracking for photons is not very good, so most of these “pairs” are just random coincidences, unrelated photons that happened to reach different parts of the detector at the same time.
Because each collision is independent of all the others, the filling of each little bin is subject to Poisson statistics: a bin with 
events in it has an intrinsic “one-sigma” statistical uncertainty of 
. You can see the error bars in the total-minus-fit plot in the bottom panel: on the left side, where $N\approx 6000$ events per interval in the top figure, the error bars are roughly 
events; on the right side, where there is less signal, the error bars are appropriately smaller.
The “one-sigma” confidence limit is 68%. Therefore, if those data were really independently generated by a Poissonian process whose average behavior were described by the fit line, you would expect the data points to be equally distributed above and below the fit, with about 68% of the error bars crossing the fit line. The other third-ish will miss the fit line, just from ordinary noise. In this plot we have thirty points, and about ten of them have error bars that don’t cross the fit line: totally reasonable. On average one point in twenty should be, randomly, two or more error bars away from the prediction (or, “two sigma” corresponds to a 95% confidence limit).
There are two remarkable bins in this histogram, centered on 125 GeV and 127 GeV, which are different from the background fit by (reading by eye) approximately 
and 
events. The “null hypothesis” is that these two differences, roughly 
and 
, are both statistical flukes, just like the low bin at 143 GeV is probably a statistical fluke.
You can see that this null hypothesis is strongly disfavored, relative to the hypothesis that “in some collisions, an object with mass near 125 GeV decays into two photons.”
This diphoton plot by itself doesn’t get you to a five-sigma discovery: that required data in multiple different Higgs decay channels, combined from both of the big CERN experiments, which required a great deal of statistical sophistication. An important part of the discovery was combining the data from all channels to determine the best estimate for the Higgs’s mass, charge, and spin. Another important result out of the discover was the relative intensities of the different decay modes. As another answer says, it helped a lot that we already had a prediction there might be a particle with this mass. But I think this data set shows the null hypothesis nicely: most of ATLAS’s photon pairs come from a well-defined continuum background of accidental coincidences, and the null hypothesis is that there’s nothing special about any of the photon pairs which happen to have an equivalent mass of 125 GeV.
I think this question may arise from a difference between somewhat rough layman's-terms presentations and the more careful statistics which goes on in the actual labs. But even after a given body of data has been analyzed to death, there is no formal way to capture in full the evidence underlying the way knowledge of physics grows. The evidence surrounding the Higgs mechanism, for example, would not be nearly as convincing if the Higgs mechanism itself were not an elegant combination of ideas which already find their place in a coherent whole.
The hypothesis that one is gathering evidence against is always the hypothesis that we are mistaken as to how a given body of data (such as a peak in a spectrum) came about. The mistake could be quite simple, as for example when in fact the underlying distribution is flat and the peak is an artifact of random noise. But usually one has to consider the possibility that the peak is there but is owing to something else than the mechanism under study. The hypothesis one is testing in the strict sense---the sense of ruling out at some level of confidence---is the set of all other ways we have thought of yet as to how the data could arise. In this set of ways we only need to consider ways that reflect known physics and known amounts of noise etc. in the apparatus.
I think what the community of physicists do is a bit like Sherlock Holmes: we try to think of plausible other ways the data could arise, and then give reasons as to why those other ways can be ruled out. The final step, where we proceed to the claim that the leading candidate explanation is what really happened, is not a step that can be quantified by any statistical measure. This is because it relies not only on a given data set, but also on a judgement about the quality of the theory under consideration.
History and origin
According to Robert D Cousins$^{1}$ and Tommaso Dorigo$^{2}$, the origin of the $5\sigma$ threshold origin lies in the early particle physics work of the 60s when numerous histograms of scattering experiments were investigated and searched for peaks/bumps that might indicate some newly discovered particle. The threshold is a rough rule to account for the multiple comparisons that are being made.
Both authors refer to a 1968 article from Rosenfeld$^3$, which dealt with the question whether or not there are far out mesons and baryons, for which several $4 \sigma$ effects where measured. The article answered the question negatively by arguing that the number of published claims corresponds to the statistically expected number of fluctuations. Along with several calculations supporting this argument the article promoted the use of the $5\sigma$ level:
Rosenfeld: "Before we go on to survey far-out mass spectra where bumps have been reported in $(K\pi\pi)_{3/2},(\pi \rho)^{--}$ we should first decide what threshold of significance to demand in 1968. I want to show you that although experimentalists should probably note $3\sigma$-effects, theoreticians and phenomenologists would do better to wait till the effect reaches $>4\sigma$."
and later in the paper (emphasis is mine)
Rosenfeld: "Then to repeat my warning at the beginning of this section; we are generating at least 100 000 potential bumps per year, and should expect several $4\sigma$ and hundreds of $3\sigma$ fluctuations. What are the implications? To the theoretician or phenomenologist the moral is simple; wait for $5\sigma$ effects."
Tommaso seems to be careful in stating that it started with the Rosenfeld article
Tommaso: "However, we should note that the article was written in 1968, but the strict criterion of five standard deviations for discovery claims was not adopted in the seventies and eighties. For instance, no such thing as a five-sigma criterion was used for the discovery of the W and Z bosons, which earned Rubbia and Van der Meer the Nobel Prize in physics in 1984."
But in the 80s the use of $5\sigma$ was spread out. For instance, the astronomer Steve Schneider$^4$ mentions in 1989 that it is something being taught (emphasize mine in the quote below):
Schneider: "Frequently, 'levels of confidence' of 95% or 99% are quoted for apparently discrepant data, but this amounts to only two or three statistical sigmas. I was taught not to believe anything less than five sigma, which if you think about it is an absurdly stringent requirement --- something like a 99.9999% confidence level. But of course, such a limit is used because the actual size of sigma is almost never known. There are just too many free variables in astronomy that we can't control or don't know about."
Yet, in the field of particle physics many publications where still based on $4\sigma$ discrepancies up till the late 90s. This only changed into $5\sigma$ at the beginnning of the 21th century. It is probably prescribed as a guidline for publications around 2003 (see the prologue in Franklin's book Shifting Standards$^5$)
Franklin: By 2003 the 5-standard-deviation criterion for "observation of" seems to have been in effect
...
A member of the BaBar collaboration recalls that about this time the 5-sigma criterion was issued as a guideline by the editors of the Physical Review Letters
Modern use
Currently, the $5\sigma$ threshold is a textbook standard. For instance, it occurs as a standard article on physics.org$^6$ or in some of Glen Cowan's works, such as the statistics section of the Review of Particle Physics from the particle data group$^7$ (albeit with several critical sidenotes)
Glen Cowan: Often in HEP, the level of significance where an effect is said to qualify as a discovery is $Z = 5$, i.e., a $5\sigma$ effect, corresponding to a p-value of $2.87 \times 10^{−7}$ . One’s actual degree of belief that a new process is present, however, will depend in general on other factors as well, such as the plausibility of the new signal hypothesis and the degree to which it can describe the data, one’s confidence in the model that led to the observed p-value, and possible corrections for multiple observations out of which one focuses on the smallest p-value obtained (the “look-elsewhere effect”).
The use of the $5\sigma$ level is now ascribed to 4 reasons:
History based on practice one found that $5\sigma$ is a good threshold. (exotic stuff seems to happen randomly, even between $3\sigma$ to $4\sigma$, like recently the 750 GeV diphoton excess)
The look elsewhere effect (or the multiple comparisons). Either because multiple hypotheses are tested, or because experiments are performed many times, people adjust for this (very roughly) by adjusting the bound to $5\sigma$. This relates to the history argument.
Systematic effects and uncertainty in $\sigma$ often the uncertainty of the experiment outcome is not well known. The $\sigma$ is derived, but the derivation includes weak assumptions such as the absence of systematic effects, or the possibility to ignore them. Increasing the threshold seems to be a way to sort of a protect against these events. (This is a bit strange though. The computed $\sigma$ has no relation to the size of systematic effects and the logic breaks down, an example is the "discovery" of superluminal neutrino's which was reported to be having a $6\sigma$ significance.)
Extraordinary claims require extraordinary evidence Scientific results are reported in a frequentist way, for instance using confidence intervals or p-values. But, they are often interpreted in a Bayesian way. The $5\sigma$ level is claimed to account for this.
Currently several criticisms have been written about the $5\sigma$ threshold by Louis Lyons${^{8,}}$$^9$, and also the earlier mentioned articles by Robert D Cousins$^{1}$ and Tommaso Dorigo$^{2}$ provide critique.
Other Fields
It is interesting to note that many other scientific fields do not have similar thresholds or do not, somehow, deal with the issue. I imagine this makes a bit sense in the case of experiments with humans where it is very costly (or impossible) to extend an experiment that gave a .05 or .01 significance.
The result of not taking these effects into account is that over half of the published results may be wrong or at least are not reproducible (This has been argued for the case of psychology by Monya Baker $^{10}$, and I believe there are many others that made similar arguments. I personaly think that the situation may be even worse in nutritional science). And now, people from other fields than physics are thinking about how they should deal with this issue (the case of medicine/pharmacology$^{11}$).
Cousins, R. D. (2017). The Jeffreys–Lindley paradox and discovery criteria in high energy physics. Synthese, 194(2), 395-432. arxiv link
Dorigo, T. (2013) Demystifying The Five-Sigma Criterion, from science20.com 2019-03-07
Rosenfeld, A. H. (1968). Are there any far-out mesons or baryons? web-source: escholarship
Burbidge, G., Roberts, M., Schneider, S., Sharp, N., & Tifft, W. (1990, November). Panel discussion: Redshift related problems. In NASA Conference Publication (Vol. 3098, p. 462). link to photocopy on harvard.edu
Franklin, A. (2013). Shifting standards: Experiments in particle physics in the twentieth century. University of Pittsburgh Press.
What does the 5 sigma mean? from physics.org 2019-03-07
Beringer, J., Arguin, J. F., Barnett, R. M., Copic, K., Dahl, O., Groom, D. E., ... & Yao, W. M. (2012). Review of particle physics. Physical Review D-Particles, Fields, Gravitation and Cosmology, 86(1), 010001. (section 36.2.2. Significance tests, page 394, link aps.org )
Lyons, L. (2013). Discovering the Significance of 5 sigma. arXiv preprint arXiv:1310.1284. arxiv link
Lyons, L. (2014). Statistical Issues in Searches for New Physics. arXiv preprint arxiv link
Baker, M. (2015). Over half of psychology studies fail reproducibility test. Nature News. from nature.com 2019-03-07
Horton, R. (2015). Offline: what is medicine's 5 sigma?. The Lancet, 385(9976), 1380. from thelancet.com 2019-03-07
In most applications of statistics there is that old chestnut about 'all models are wrong, some are useful'. This being the case, we would only expected a model to perform at a given level since we are describing some incredibly complicated process using some simple model.
Physics is very different, so intuition developed from statistical models isn't so appropriate. In Physics, in particular particle physics which deals directly with fundamental physical laws, the model really is supposed to be an exact description of reality. Any departure from what the model predicts must be completely explained by experimental noise, not a limitation of the model. This means that if the model is good and correct and the experimental apparatus understood the statistical significance should be very high, hence the high bar that is set.
The other reason is historical, the particle physics community has been burned in the past by 'discoveries' at lower significance levels being later retracted, hence they are generally more cautious now.