{"id":23048,"date":"2019-11-15T18:11:01","date_gmt":"2019-11-15T12:11:01","guid":{"rendered":"https:\/\/www.enago.com\/academy\/?p=23048"},"modified":"2019-11-19T14:50:24","modified_gmt":"2019-11-19T08:50:24","slug":"chasing-the-p-value-statistical-significance-and-the-perils-of-p-hacking","status":"publish","type":"post","link":"https:\/\/www.enago.com\/academy\/chasing-the-p-value-statistical-significance-and-the-perils-of-p-hacking\/","title":{"rendered":"Chasing the P-Value: Statistical Significance and the Perils of P-Hacking"},"content":{"rendered":"

In the world of research, we learn about the importance of statistical significance. The p-value (the p stands for \u201cprobability\u201d) is crucial. Usually, a p-value of less than 0.05 is considered significant. The smaller the value, the more significant the results.You can also listen to this article as an audio recording.<\/p>\n\n

In general, journals publish articles that report one or more statistically significant effects. It puts tremendous pressure on researchers to produce significant results. At times, researchers may turn to malpractice called p-hacking to get the results they desire.<\/p>\n

Here, two researchers, Tom and Lisa, discuss the pressures to produce statistically significant results and how this can harm the integrity of science?<\/p>\n

Seeking Statistical Significance<\/h2>\n

Tom: \u201cI can\u2019t seem to get this method correct! I\u2019m sure I should get a significant result, but my p-value is around 0.8. What am I doing wrong?\u201d<\/p>\n

Lisa: \u201cHave you checked your protocol?\u201d<\/p>\n

Tom: \u201cYes, it seems ok. My supervisor wants to publish this in a high impact journal \u2013 it could help us to secure funding for our next project. However, the results we\u2019re getting aren\u2019t statistically significant. I\u2019m thinking of making changes to our method.\u201d<\/p>\n

Lisa: \u201cWhat sort of changes?\u201d<\/p>\n

Tom: \u201cI\u2019m going to collect more data. I might drop some of the outlying data points \u2013 I\u2019m sure they are wrong! Then I can try different ways of analyzing the data.\u201d<\/p>\n

The Problem of P-Hacking<\/h2>\n

Lisa: \u201cBut isn\u2019t that, p-hacking?\u201d<\/p>\n

Tom: \u201cWhat\u2019s p-hacking?\u201d<\/p>\n

Lisa: \u201cP-hacking is when a researcher, either knowingly or unknowingly, makes choices after seeing their data to help them get a significant result. This includes choices or tweaks like those you mentioned. For example, dropping outlying data points or changing the way you analyze data.\u201d<\/p>\n

Tom: \u201cI haven\u2019t heard of p-hacking before.\u201d<\/p>\n

Lisa: \u201cAs you know, there can be a lot of pressure to produce low p-value. Studies with statistically significant results are much more likely to be published. This can directly affect the funding a researcher receives, as well as their career prospects. It can be very tempting to turn to \u00a0research malpractices to get a low p-value.\u201d<\/p>\n

Types of P-Hacking<\/h2>\n

Tom: \u201cI think I need to know more about p-hacking so that I can avoid it. Can you give me some tips?\u201d<\/p>\n

Lisa: \u201cP-hacking often gives a p-value of 0.05, or less. It is because researchers often stop \u2018tweaking\u2019 once they have achieved a significant result. A cluster of values around p=0.05 suggests p-hacking. However, other types of p-hacking can be more difficult to spot.\u201d<\/p>\n

Tom: \u201cWhat are they?\u201d<\/p>\n

Lisa: \u201cThe first type is known as \u2018overhacking<\/a><\/strong>.\u2019 This is when a researcher continues to hack their data in an attempt to get a lower p-value. Instead of stopping once they reach a value below 0.05, they continue. They do this because a lower p-value suggests a more compelling result.\u201d<\/p>\n

Tom: \u201cWhat are the other types?\u201d<\/p>\n

Lisa: \u201cThe next type is selection bias<\/strong>. This is when a researcher has different p-values as a result of carrying out different analyses on the data, or analyzing different variables. However, even if they have several p-values under 0.05, they choose only the lowest for publication. This does not give an accurate picture of the data.<\/p>\n

\u201cThe third type is selective debugging<\/strong>. \u2018Bugs\u2019 can happen if a researcher chooses an unsuitable statistical test, or if there are problems with data coding. Researchers should always try to spot these errors. Selective debugging is when a researcher only corrects bugs when it helps to get a significant result. Once they have a significant result, they stop looking for bugs.\u201d<\/p>\n

Tom: \u201cSo the researchers are selecting for bugs that give a false positive<\/a> result?\u201d<\/p>\n

Lisa: \u201cYes, that\u2019s right.<\/p>\n

The Consequences of P-Hacking<\/h2>\n

Tom: \u201cWhat can happen if a researcher is caught, p-hacking?\u201d<\/p>\n

Lisa: \u201cP-hacking can have grave consequences. It undermines the value of the research. It could lead to journals retracting suspect articles, and the loss of future funding.\u00a0 As well as wasting valuable time and money, p-hacking could even reduce public confidence in science.\u201d<\/p>\n

Tom: \u201cThat does sound serious. How common is p-hacking?\u201d<\/p>\n

Lisa: \u201cSome studies have shown that p-hacking is widespread. P-hacking could also be a serious problem for meta-analyses. In these large-scale studies, researchers rely on earlier work for their analysis. If the earlier research has been p-hacked, the results will not be reproducible. A 2015 study<\/a> asked 100 research groups to replicate 100 published results. Of these, only 40 replicated well. In the other 60, the replication achieved much smaller effects than the initial result.\u201d<\/p>\n

Preventing P-Hacking<\/h2>\n

Tom: \u201cI understand why p-hacking is a problem. How can it be prevented?\u201d<\/p>\n

Lisa: \u201cThe best way to avoid p-hacking is to avoid making changes after you have seen the data. Of course, it can be difficult to resist! You could consider pre-registration. This is when you prepare a detailed research plan, including the statistical analysis<\/a> you plan to use. You then submit the plan to an online register, such as the Open Science Framework<\/a>. If you end up publishing your results, anyone can check your method against your plan. This makes it much harder to p-hack your data.\u201d<\/p>\n

Tom: \u201cThat sounds great. Is there anything else I can do?\u201d<\/p>\n

Lisa: \u201cJust pre-plan your work, and stick to it. Only make changes if you realize you have made a genuine error. You can also replicate your own work.\u201d<\/p>\n

Dealing with Non-Significant Results<\/h2>\n

Tom: \u201cI know that I can\u2019t change my method to get a significant result. Does this mean I won\u2019t be able to publish my work?\u201d<\/p>\n

Lisa: \u201cNot necessarily. As p-hacking becomes more widely recognized, journals may start to reduce their preference for significant results. Journals could also help by providing a platform for pre-registration. Researchers can consider other types of statistical analysis, as well as focusing on the quality of their research plan and data collection.\u201d<\/p>\n

Tom: \u201cThank you for your help!\u201d<\/p>\n

You can read more about statistical significance in this\u00a0article<\/a>. Have you encountered anyone p-hacking? Share your thoughts and experiences in the comments below.<\/p>\n

\n\t\t<\/a>\n\t\t<\/div>","protected":false},"excerpt":{"rendered":"

In the world of research, we learn about the importance of statistical significance. The p-value…<\/p>\n","protected":false},"author":8169,"featured_media":23050,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"om_disable_all_campaigns":false,"footnotes":""},"categories":[747,2],"tags":[1452],"ppma_author":[1908],"class_list":["post-23048","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-content-structure","category-academic-writing","tag-manuscript-drafting-tips"],"better_featured_image":{"id":23050,"alt_text":"","caption":"","description":"","media_type":"image","media_details":{"width":750,"height":430,"file":"2019\/06\/ChasingTheP-Value_Image.jpg","sizes":{"thumbnail":{"file":"ChasingTheP-Value_Image-170x150.jpg","width":170,"height":150,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-170x150.jpg"},"medium":{"file":"ChasingTheP-Value_Image-401x230.jpg","width":401,"height":230,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-401x230.jpg"},"large":{"file":"ChasingTheP-Value_Image-750x430.jpg","width":750,"height":430,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-750x430.jpg"},"better-amp-small":{"file":"ChasingTheP-Value_Image-100x100.jpg","width":100,"height":100,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-100x100.jpg"},"better-amp-normal":{"file":"ChasingTheP-Value_Image-260x200.jpg","width":260,"height":200,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-260x200.jpg"},"better-amp-large":{"file":"ChasingTheP-Value_Image-450x300.jpg","width":450,"height":300,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-450x300.jpg"},"publisher-tb1":{"file":"ChasingTheP-Value_Image-86x64.jpg","width":86,"height":64,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-86x64.jpg"},"publisher-sm":{"file":"ChasingTheP-Value_Image-210x136.jpg","width":210,"height":136,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-210x136.jpg"},"publisher-mg2":{"file":"ChasingTheP-Value_Image-279x220.jpg","width":279,"height":220,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-279x220.jpg"},"publisher-md":{"file":"ChasingTheP-Value_Image-357x210.jpg","width":357,"height":210,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-357x210.jpg"},"publisher-tall-sm":{"file":"ChasingTheP-Value_Image-180x217.jpg","width":180,"height":217,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-180x217.jpg"},"publisher-tall-lg":{"file":"ChasingTheP-Value_Image-267x322.jpg","width":267,"height":322,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-267x322.jpg"},"publisher-tall-big":{"file":"ChasingTheP-Value_Image-368x430.jpg","width":368,"height":430,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-368x430.jpg"},"Book Review":{"file":"ChasingTheP-Value_Image-320x430.jpg","width":320,"height":430,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image-320x430.jpg"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]}},"post":23048,"source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2019\/06\/ChasingTheP-Value_Image.jpg"},"acf":{"faq_main_heading":"","faq_heading_one":"","faq_heading_two":"","faq_heading_three":"","faq_heading_four":"","faq_heading_five":"","faq_heading_six":"","faq_description_one":"","faq_description_two":"","faq_description_three":"","faq_description_four":"","faq_description_five":"","faq_description_six":""},"views":1678,"single_webinar_page_date":null,"single_webinar_page_time":null,"session_agenda":null,"who_should_attend_this_session":null,"about_the_speaker_field":null,"co-webinar-sec":null,"co_webinar_sec_one":null,"speaker-name":null,"webinar-date":null,"webinar-time":null,"webinar-s-image":null,"custum_webinar_category":null,"authors":[{"term_id":1908,"user_id":8169,"is_guest":0,"slug":"eneditor","display_name":"Enago Academy","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/046a0ceeb5c38172654db93f9919593bc2e4e1391702eb8b7248865941ddbe18?s=96&d=identicon&r=g","author_category":"","user_url":"","last_name":"Academy","first_name":"Enago","job_title":"","description":"Enago Academy, the knowledge arm of Enago, offers comprehensive and up-to-date resources on academic research and scholarly publishing to all levels of scholarly professionals: students, researchers, editors, publishers, and academic societies. It is also a popular platform for networking, allowing researchers to learn, share, and discuss their experiences within their network and community. The team, which comprises subject matter experts, academicians, trainers, and technical project managers, are passionate about helping researchers at all levels establish a successful career, both within and outside academia."}],"_links":{"self":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/posts\/23048","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/users\/8169"}],"replies":[{"embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/comments?post=23048"}],"version-history":[{"count":0,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/posts\/23048\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/media\/23050"}],"wp:attachment":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/media?parent=23048"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/categories?post=23048"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/tags?post=23048"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/ppma_author?post=23048"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}