Bad style - HARKing and the replication crisis

Publication bias

Computing research validate research claims using statistical significance as the standard of evidence
Statistical evidence usually assumes 95% confidence (p <= 0.05)
analysis of 362 papers published found that 97% reject the null
papers that were incorrecct are not published
Publication bias: Papers supporting their hypotheses are accepted for publication at a much higher rate than those that do not.
HARKing (Hypothesizing After the Results are Known) or known as ‘outcome switching’
- Post-hoc reframing of experimental intentions to present a p-fished outcome as having been predicted from the start.
p-hacking: Manipulation of experimental and analysis methods to produce statistically significant results.
p-fishing: seeking statistically significant effects beyond the original hypothesis.

A survey of over 2000 psychology researchers, John et al. examined the prevalence of questionable experimental practices (forms of HARKing):

Failing to report all dependent measures, which opens the door for selective reporting of favourable findings – 63.4%;
Deciding to collect additional data after checking if the effects were significant – 55.9%;
Failing to report all of the study’s conditions – 27.7%;
Stopping data collection early once the significant effect is found – 15.6%;
Rounding off a p value (e.g., reporting p = .05 when the actual value is p = .054) – 22.0%;
Selectively reporting studies that worked – 45.8%;
Excluding data after looking at the impact of doing so – 38.2%;
Reporting an unexpected finding as having been predicted – 27.0%;
Reporting a lack of effect of demographic variables (e.g., gender) without checking – 3.0%;
Falsifying data – 0.6%

File drawer effect: Null findings tend to be unpublished and therefore hidden from the scientific community.

Solutions

Preregistration: Registries in which researchers preregister their intentions, hypotheses, and methods (including sample sizes and precise plans for the data analyses) for upcoming experiments

Preregistered Reports: Papers (Reports) are submitted for review prior to conducting the experiment. Registered reports include the study motivation, related work, hypotheses, and detailed method; everything that might be expected in a traditional paper except for the results and their interpretation.

• Redefine or abdandon statistical significance. • Create data repositories and support replication