Estimator of Agreement with Covariate Adjustment

The parameter $$\kappa $$ κ is a general agreement structure used across many fields, such as medicine, machine learning and the pharmaceutical industry. A popular estimator for $$\kappa $$ κ is Cohen’s $$\kappa $$ κ ; however, this estimator does not account for multiple influential factors. The primary goal of this paper is to propose an estimator of agreement for a binary response using a logistic regression framework. We use logistic regression to estimate the probability of a positive evaluation while adjusting for factors. These predicted probabilities are then used to calculate expected agreement. It is shown that ignoring needed adjustment measures, as in Cohen’s $$\kappa $$ κ , leads to an inflated estimate of $$\kappa $$ κ and a situation similar to Simpson’s paradox. Simulation studies verified mathematical relationships and confirmed estimates are inflated when necessary covariates are left unadjusted. Our method was applied to an Alzheimer’s disease neuroimaging study. The proposed approach allows for inclusion of both categorical and continuous covariates, includes Cohen’s $$\kappa $$ κ as a special case, offers an alternative interpretation, and is easily implemented in standard statistical software.