{"id":20478,"date":"2018-12-15T20:00:58","date_gmt":"2018-12-15T14:00:58","guid":{"rendered":"https:\/\/www.enago.com\/academy\/?p=20478"},"modified":"2022-04-29T22:47:44","modified_gmt":"2022-04-29T16:47:44","slug":"should-researchers-include-a-confidence-index-of-their-results","status":"publish","type":"post","link":"https:\/\/www.enago.com\/academy\/should-researchers-include-a-confidence-index-of-their-results\/","title":{"rendered":"Should Researchers Include a Confidence Index of Their Results?"},"content":{"rendered":"<p>The ongoing reproducibility crisis has drawn attention to the way experiments are reported. Alarmingly, a Nature survey revealed that 70 % of scientists were unable to reproduce another scientists\u2019 experiments and 50 % were unable to reproduce their own experiments. It is common scientific practice to back your results statistically with a p-value. However, the reproducibility crisis suggests that this is not enough. One way to <a href=\"https:\/\/www.nature.com\/articles\/d41586-018-07589-2\" target=\"_blank\" rel=\"noopener\">add clarity to data interpretation<\/a> may be to use a confidence index, that tells you the chances of your results being true.<\/p>\n<h2>What is a Confidence Index?<\/h2>\n<p>In scientific experiments, you test a treatment on a sample of a population. You ensure that you select a random sample (e.g. clinical trial participants) representative of a population. The confidence index will tell you how certain you can be that your treatment will have the same effect (as occurred in your experiments) on the entire population.<\/p>\n<p>It is a convention within scientists to <a href=\"https:\/\/www.amstat.org\/asa\/files\/pdfs\/P-ValueStatement.pdf\" target=\"_blank\" rel=\"noopener\">report their data<\/a> with a p-value, and although it is similar to a confidence index, it is not the same. The difference is that the p-value implies that your result is 95 % accurate, whereas a confidence index tells you the probability of your result being accurate.<\/p>\n<p>P-value calculations are dependent on:<\/p>\n<ul>\n<li>Sample size \u2013 larger sample sizes will generate more accurate results.<\/li>\n<li>Response frequency \u2013 the greater a particular response, the more accurate your results.<\/li>\n<li>Population size \u2013 this is only important when your population is small.<\/li>\n<\/ul>\n<p>A confidence index, which <a href=\"https:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4676938\/\" target=\"_blank\" rel=\"noopener\">could be based<\/a> on Bayesian probability, would include considerations such as:<\/p>\n<ul>\n<li>Random variables \u2013 unknown factors resulting from a lack of information.<\/li>\n<li>Prior probability \u2013 take account of available information.<\/li>\n<li>Hypothesis truth \u2013 probability that your hypothesis is true or false.<\/li>\n<\/ul>\n<p>A confidence index is crucial when making decisions about future experiments such as treatments for clinical trials. The confidence index should also give an idea of the limitations of the studies.<\/p>\n<h2>Justifying Results with P-values<\/h2>\n<p>Conventionally, scientists rely on p &lt; 0.05 to deem their results publishable. It tells other researchers that your results are valid. If your p-value is greater than 0.05, you probably will not publish the data. Some argue that this results in valuable research not being made available in the literature.<\/p>\n<p>Data is typically reported as follows: \u201cModerator analyses revealed that a higher temperature at bedtime was associated with lower sleep efficiency (SE) (<em>b<\/em>\u2009=\u2009\u2212\u200911.6\u00a0pp;\u00a0<em>p<\/em>\u2009=\u20090.020).\u201d<\/p>\n<p>We can see the p-value is less than 0.05, however does this mean this statement is true?<\/p>\n<p>In 2016, the American Statistical Association (ASA) <a href=\"https:\/\/www.enago.com\/academy\/the-reproducibility-project-cancer-biology-to-replicate-only-18-studies-now\/\" target=\"_blank\" rel=\"noopener\">published guidelines<\/a> for using a p-value and cautioned that a p-value by itself, \u201cdoes not provide a good measure of evidence regarding a model or hypothesis\u201d.<\/p>\n<p>This is because the p-value does not tell you whether:<\/p>\n<ol>\n<li>Your sample was representative of the population.<\/li>\n<li>The studied hypothesis is true or false.<\/li>\n<li>The data was produced by random chance.<\/li>\n<\/ol>\n<h2>Include the Confidence Index<\/h2>\n<p>Considering the above, the ASA cautions that the p-value should not be a \u201csubstitute for scientific reasoning\u201d and many other factors such as \u201cgood study design and conduct\u201d are important.<\/p>\n<p>Yes, any issues with the p-value should be<a href=\"https:\/\/www.nature.com\/news\/1-500-scientists-lift-the-lid-on-reproducibility-1.19970\" target=\"_blank\" rel=\"noopener\"> addressed by transparency<\/a> of experimental methods and analysis of the data in context with previous literature. After all, researchers are trained to assess the validity of the results of a study by considering all the relevant factors.<\/p>\n<p>However, Steven Goodman thinks research could be clarified if the results are quantified with a confidence index. This could minimize so-called p-hacking (manipulation of data in order to achieve p-values &lt; 0.05) and rather give actual values of probability. Do you think this clarity could help sort out the reproducibility crisis? Let us know your thoughts in the comments section below.<\/p>\n<div style=\"display:flex; gap:10px;justify-content:\" class=\"wps-pgfw-pdf-generate-icon__wrapper-frontend\">\n\t\t<a  href=\"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/posts\/20478?action=genpdf&amp;id=20478\" class=\"pgfw-single-pdf-download-button\" ><img data-src=\"https:\/\/www.enago.com\/academy\/wp-content\/plugins\/pdf-generator-for-wp\/admin\/src\/images\/PDF_Tray.svg\" title=\"Generate PDF\" style=\"width:auto; height:45px;\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" class=\"lazyload\"><\/a>\n\t\t<\/div>","protected":false},"excerpt":{"rendered":"<p>The ongoing reproducibility crisis has drawn attention to the way experiments are reported. Alarmingly, a&hellip;<\/p>\n","protected":false},"author":8169,"featured_media":20480,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"om_disable_all_campaigns":false,"footnotes":""},"categories":[1319,1336],"tags":[1495],"ppma_author":[1908],"class_list":["post-20478","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-beyond-publishing","category-phd-cafe","tag-scientific-transparency"],"better_featured_image":{"id":20480,"alt_text":"reproducibility crisis","caption":"","description":"Researchers have come up with a new way of finding a solution for the reproducibility crisis. The confidence index that is calculated on the research will give the scientists an idea about the results of the study being true.","media_type":"image","media_details":{"width":750,"height":430,"file":"2018\/12\/pexels-photo-1323592-1.jpeg","sizes":{"thumbnail":{"file":"pexels-photo-1323592-1-170x150.jpeg","width":170,"height":150,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-170x150.jpeg"},"medium":{"file":"pexels-photo-1323592-1-401x230.jpeg","width":401,"height":230,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-401x230.jpeg"},"large":{"file":"pexels-photo-1323592-1-750x430.jpeg","width":750,"height":430,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-750x430.jpeg"},"better-amp-small":{"file":"pexels-photo-1323592-1-100x100.jpeg","width":100,"height":100,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-100x100.jpeg"},"better-amp-normal":{"file":"pexels-photo-1323592-1-260x200.jpeg","width":260,"height":200,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-260x200.jpeg"},"better-amp-large":{"file":"pexels-photo-1323592-1-450x300.jpeg","width":450,"height":300,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-450x300.jpeg"},"publisher-tb1":{"file":"pexels-photo-1323592-1-86x64.jpeg","width":86,"height":64,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-86x64.jpeg"},"publisher-sm":{"file":"pexels-photo-1323592-1-210x136.jpeg","width":210,"height":136,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-210x136.jpeg"},"publisher-mg2":{"file":"pexels-photo-1323592-1-279x220.jpeg","width":279,"height":220,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-279x220.jpeg"},"publisher-md":{"file":"pexels-photo-1323592-1-357x210.jpeg","width":357,"height":210,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-357x210.jpeg"},"publisher-tall-sm":{"file":"pexels-photo-1323592-1-180x217.jpeg","width":180,"height":217,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-180x217.jpeg"},"publisher-tall-lg":{"file":"pexels-photo-1323592-1-267x322.jpeg","width":267,"height":322,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-267x322.jpeg"},"publisher-tall-big":{"file":"pexels-photo-1323592-1-368x430.jpeg","width":368,"height":430,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-368x430.jpeg"},"Book Review":{"file":"pexels-photo-1323592-1-320x430.jpeg","width":320,"height":430,"mime-type":"image\/jpeg","source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1-320x430.jpeg"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]}},"post":20478,"source_url":"https:\/\/www.enago.com\/academy\/wp-content\/uploads\/2018\/12\/pexels-photo-1323592-1.jpeg"},"acf":{"faq_main_heading":"","faq_heading_one":"","faq_heading_two":"","faq_heading_three":"","faq_heading_four":"","faq_heading_five":"","faq_heading_six":"","faq_description_one":"","faq_description_two":"","faq_description_three":"","faq_description_four":"","faq_description_five":"","faq_description_six":""},"views":778,"single_webinar_page_date":null,"single_webinar_page_time":null,"session_agenda":null,"who_should_attend_this_session":null,"about_the_speaker_field":null,"co-webinar-sec":null,"co_webinar_sec_one":null,"speaker-name":null,"webinar-date":null,"webinar-time":null,"webinar-s-image":null,"custum_webinar_category":null,"authors":[{"term_id":1908,"user_id":8169,"is_guest":0,"slug":"eneditor","display_name":"Enago Academy","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/046a0ceeb5c38172654db93f9919593bc2e4e1391702eb8b7248865941ddbe18?s=96&d=identicon&r=g","author_category":"","user_url":"","last_name":"Academy","first_name":"Enago","job_title":"","description":"Enago Academy, the knowledge arm of Enago, offers comprehensive and up-to-date resources on academic research and scholarly publishing to all levels of scholarly professionals: students, researchers, editors, publishers, and academic societies. It is also a popular platform for networking, allowing researchers to learn, share, and discuss their experiences within their network and community. The team, which comprises subject matter experts, academicians, trainers, and technical project managers, are passionate about helping researchers at all levels establish a successful career, both within and outside academia."}],"_links":{"self":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/posts\/20478","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/users\/8169"}],"replies":[{"embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/comments?post=20478"}],"version-history":[{"count":0,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/posts\/20478\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/media\/20480"}],"wp:attachment":[{"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/media?parent=20478"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/categories?post=20478"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/tags?post=20478"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.enago.com\/academy\/wp-json\/wp\/v2\/ppma_author?post=20478"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}