The practice of using covariates is widespread in both experimental and correlational research, yet controversy remains on whether (and when) it is appropriate. Traditionally touted by statisticians as a useful method to soak up noise in a dependent variable and boost power, the practice recently has been recast in a negative light because of Type I error inflation (e.g., Simmons, Nelson, & Simonsohn, 2011). To make informed decisions about using covariates, researchers need to know more about the actual size of the associated costs and benefits.

In a series of simulations, we quantified the tradeoff between power and Type I error rates from flexible, post-hoc covariate use in experimental designs, and we found that including a single, strong covariate, even when unplanned, can substantially boost power while only slightly inflating Type I error (Wang, Sparks, Hess, Gonzales, & Ledgerwood, 2017, JESP). When more than one covariate is flexibly included or when the covariate is weakly correlated with the dependent variable, however, the tradeoff becomes more unfavorable. We provide researchers guidance on how to appropriately incorporate such exploratory analyses.