Robust standard errors are frequently used in statistics, often in an unthinking way. They are certainly valuable and useful in the context of linear regression; however, they do not serve a similar purpose in maximum likelihood estimation of non-linear models. The basic issue is that the kind of mis-specification that robust standard errors can address in linear regresion does not spoil consistency results for the parameter estimates. On the other hand, it does spoil those results in the context of maximum likelihood estimation of non-linear models. Thus at best you end up with a consistent estimator for the variance of a parameter estimate that is itself inconsistent, which isn’t really of interest except as a diagnostic tool for detecting bad models. I’ll explore this in more detail below.

In my time working in health science, I have been troubled by the number of times I have encountered statisticians and practitioners of statistics who are absolutely sure that either the variables or residuals in a linear regression must be approximately normally distributed, and the model is invalid otherwise.

This idea is completely false. In this post I want to explore why it is nonetheless so widely believed by professionals in the field.

What is linear regression?

It’s one of the first major topics you learn about when taking statistics class. In essence, it allows you to model the mean of a response variable \( y_i \) as depending on some explanatory variables \( x_i \), where the subscript \( i = 1…n \) labels different observations. More concretely,

There are models for just about anything you can think of. Fundamental physics, economic markets, epidemics, brain function, climate, click-through rates, animal behavior, election outcomes, to name a few. There’s a universe of possibilities for ways to model any given phenomenon, and only a few small hidden oases in an otherwise barren desert. What’s a model-builder to do?

What is imputation?

Imputation is ‘filling in’ missing values in a dataset. So, for example, say I have a dataset consisting of the name, sex, sex and height of some individuals as follows: