Using hierarchical models to attribute sources of variation in consumer assessments of health care

The Consumer Assessments of Healthcare Providers and Systems (CAHPS) Medicare Advantage (MA-CAHPS) survey has provided extensive and uniform data for 8 years on the quality of Medicare health plans in the United States. The complex structure of the data makes hierarchical modelling an appropriate analytic tool. After describing the CAHPS survey and the analytic methods used in standard reports, we review research using two multilevel modelling strategies, each addressing a different aspect of the structure of the CAHPS data. The first fits a 2-level Fay-Herriott-type model to data aggregated by plan to estimate plan-level correlations among summary scores on different items. By forming separate measures for healthier and sicker members of each plan, we were able to determine which items measured distinct dimensions of quality depending on health status. The second analysis evaluated the relative contributions of geography and organizational units to the various quality measures, and the amount of variation over time in each. Geographical variation predominated for aspects of member experiences that are not typically under the direct control of plans, and the geographical effects were very stable over time. Each of the two analyses can be regarded as a simplification for particular objectives of a larger underlying model. Further methodological development is needed to better characterize variation in quality. (April 15, 2007)