Why IVs Can Be Really Tricky---Even the Good Ones!

After several discussions with a colleague last week, it has come to my attention that I may be more critical of instrumental variables approaches than the typical applied micro economist. To be clear, I'm not talking about the use of IVs within RD or RCT designs. I'm talking about your standard IV paper. And I'm not even sure if the phrase "critical of" is the right one to use. After all, I use IVs in several of my papers. Maybe a better phrase would be "cautious about" or even "careful when using"...

In any case, there are some IVs that I tend to really like. One example: "judge-leniency" IVs. In a recent blog post, David Mackenzie explained the basic idea behind these IVs with an example from Kling (2006, AER). Imagine you want to know the impact of incarceration length on subsequent labor market outcomes. It's almost impossible to answer this question with standard OLS approaches because, as David writes, "people who get longer prison sentences might be different from those who get given shorter sentences, in ways that matter for future labor earnings." What to do? Exploit the fact that some judges are more lenient than others when sentencing. This means that people who, by pure luck, end up with a lenient judge will have a shorter sentence for reasons that have nothing to do with them. Pretty believable, right? At least, I buy it. And the excellent news is that this main approach can be used in many different scenarios with different types of "judges" (see the blog entry for examples). 

But it turns out that even with this really great IV, there are still problems, besides the most obvious one that you need access to lots of administrative data to be able to do this. First, you need to really know the institutional details about assignment. Are the judges really randomly assigned? The second is about the exclusion restriction: Even if the judges are randomly assigned, are we sure that the only way they affect outcomes is via your variable of interest? A third is about the monotonicity assumption, something we do not typically have to worry about in other IV contexts. Again, read the blog entry for more details. 

For now, I will leave you with this. IV approaches can often allow you to answer really important questions in very precise ways. I will certainly not tell you to omit the IV estimates from your paper. I want to see those numbers. It's just that I strongly urge you when writing up your results to be very careful about emphasizing where identification is coming from. As such, my preference is usually to focus on the reduced form estimates instead of the IV estimates. The reduced form is really where the magic is--IV estimates are just an interesting way to (potentially) interpret those reduced form estimates (but only under certain assumptions). 

For a discussion of the problems with an IV that I often use, see here.  For a more sympathetic view of IV approaches, I urge you to get in touch with my colleague, Jorge Agüero, who has developed some of his own really cool IVs.