Yesterday, I attended a (zoom) seminar where multiple hypothesis testing was taken very seriously---you can watch it here if you're interested. I decided I needed to think more about this and, luckily for me, David Mackenzie had already thought very carefully about and blogged about it recently. He evaluates many different Stata commands including the one I heard about in the talk yesterday.
But the most eye-opening thing was this simple calculation that I admit (shamefully) I had never thought about before. Let's say you examine the impact of four different treatments, and you look at four different outcomes. Not crazy numbers, right? David writes,
Suppose that none of the treatments have any effect on any outcome (all null hypotheses are true), and that the outcomes are independent. Then if we just test the hypotheses one by one, then the probability that of one or more false rejections when using a critical value of 0.05 is 1-0.95^20 = 64%.
64%! That's quite a high probability, right?! Wouldn't it be crazy to then write an entire paper on that one result? The multiple hypothesis testing adjustments provide methods to adjust for the fact that we are testing multiple hypotheses.
Thank you to all of the authors of the Stata commands. You have made it easier for all of us to make the right adjustments, and of course, thank you to David for helping us think about these issues.
Wow, thank you for posting! I had been thinking about how to compute Anderson FDR q-values but didn't know there was pre-written STATA code for this. This may also be helpful to some: https://www.stat.berkeley.edu/~mgoldman/Section0402.pdf
ReplyDelete