Should We Use Figures for Statistical Inference? Evidence from RD

Alternative titles of this blog entry: 

Is the .05 P-Value Threshold too Lenient? 

or

When In Doubt, Simulate! 

But here goes: 

Whenever I teach RD design, I always urge students to make the pictures. In fact, I think the reason I love RD so much (and dislike IV so much) is precisely that we can see the magic right in front of our eyes. But every now and then, I see an RD figure that is .. less than impressive. And I find myself doubting the results. But should I?  

Kirabo Jackson wondered the same thing, and so he made a gif to show us simulated RD plots associated with various t-statistics. Brilliant! Yes, when the t-statistics are really big, the plots are beautiful, but even with a t-statistic of 3.8, the plot is "less than compelling". Clearly, our standards for beautiful RD plots are much more demanding than our standards for beautiful tables. So what should we do? Be more lenient with the RDers? More demanding of our tables? I'm not sure---read the comments for more insight on that. 

In the meantime, another big lesson: If you're ever not sure of something, simulate it! Read the entire thread. Kirabo includes the very simple simulation code he used to make his gif.