How much does the method matter? An empirical comparison of ways to quantify the influence of residential self-selection

Various techniques have been applied to address residential self-selection (RSS) bias in estimating the impact of the built environment on travel behavior. We investigate how much the diverse results obtained are due to the method used, focusing on the most-often applied approaches of statistical control modeling, propensity score-based techniques, and sample selection modeling. First, a companion paper identifies 47 plausible ways to estimate a key quantity of interest, namely, the proportion of the total apparent effect of the built environment on travel behavior that is due to the built environment itself, which we call the “built environment proportion”, or BEP. Using a single sample, the present paper provides empirical estimates of 28 of the 47 BEP formulas, and creates and applies a framework for comparing those BEPs. We find that BEPs based on modular effects or treatment effects involving probability weights, and those applied to two-equation models, are among the ten best (based on the goodness of fit of the underlying models and on various measures of quality of the BEP). For our dataset, the ten best BEP averages fall between 0.506 and 0.772, with an overall average of 0.617. Thus, our best estimate is that for this application, about 38% of the total apparent effect of the built environment is due to RSS. However, the range of these averages indicates that even when controlling for as many confounding factors as possible, and even when using the best methods for computing the BEP, we still find substantial variation in the answer to the question: how much of the built environment's apparent total influence on travel behavior is due to self-selection?