The Course Experience Questionnaire as an Institutional Performance Indicator

Abstract

Data from the 1996 Course Experience Questionnaire (CEQ) were analysed using the Rasch measurement model. This analysis indicates that 17 of the 25 CEQ items fit a unitary scale that measures course quality as perceived by graduates. Graduates are located on the interval measurement scale produced in the Rasch analysis. The interval nature of the scale renders the graduates’ scores amenable to analyses that are not wisely employed using ordered raw CEQ scores. Analysis of variance indicates that variations in graduates’ responses are attributable to field of study and institutional factors. In order to compare universities, corrections are made for the course mix of each institution to produce expected institutional scores. These are compared with observed institutional scores to determine those universities that have performed above, at, or below expectation. (Individual institutions are not identified in this analysis).

Important issues relating to the educational and statistical significance of the findings have emerged. The data collected through the CEQ do not represent a simple random sample of all graduates. Instead, the data model is a hierarchical one, with individual graduates nested within courses, which are nested within institutions. This requires analysis using multilevel analytical tools. Conventional analyses substantially underestimate the standard errors of aggregated measures (such as institutional means) and therefore report institutional differences as significant when they are not. The implications of the measurement and analytical problems for policy decisions over the distribution of funding among institutions and among courses within institutions are discussed.