|[ Proceedings Contents ] [ WAIER Postgraduate Awards 2009 ]
Comparing single-level and multilevel regression models in analysing Rasch measures of numeracyJessica Elderfield
Abstract for a Master of Education thesis submitted to Edith Cowan University in May 2008. Jessica Elderfield received a WAIER Postgraduate Award in absentia at Forum 2009.
This thesis described a research study that investigated the empirical differences between two competing regression methods used to produce value-added performance indicator information for the monitoring of school effectiveness, and the practical consequences for schools when using this information for school improvement and accountability purposes. The two regression methods under review were single-level Means-on-Means regression and Multilevel Modelling. The study involved data from 24 government secondary schools with a total of 2862 students in 132 Year 8 classes in Western Australia. The dependent variable was a Rasch-created linear measure of Year 8 Numeracy from data of a Mathematics assessment, specially designed by the Department of Education. The main independent variable was a Rasch-created, linear measure of Year 7 Numeracy from student data of the Western Australian Literacy and Numeracy Assessment. Students' scored responses on items from both assessments were calibrated on a common Western Australian Monitoring Standards in Education linear scale which enabled: (1) the movement of a student's performance to be measured over time; and, (2) the application of subsequent statistical analyses from which valid inferences could be made. There were four other independent variables: (1) gender (male or female); (2) ethnic group (Aboriginal and Torres Strait Islander, or non-Aboriginal and Torres Strait Islander status); (3) language background (English or other than English); and the school-level variable; and (4) school socioeconomic status.
Application of the single-level Means-on-Means regression model to measures aggregated at the student, class and school-levels found that the amount of variance in students' numeracy measure that can be explained by the prior performance measure increased considerably at escalating levels of aggregation (individual, class, school). This highlighted the presence of aggregation bias under this regression method. Application of the Multilevel regression model to individual student measures found: (1) that once the effects of prior performance and other student background information were taken into account, the effects of the school reduced in their significance; and (2) that more of the variation in student performance was found at the student-level, followed by the class-level. In addition, a majority of schools were found to contain classes that were differentially effective in producing quality student outcomes. This is significant given that historically the Means-on-Means regression analyses of Western Australia's literacy and numeracy assessment data has only been applied at the school level. A comparison of the residuals (performance indicators) calculated under each regression model found that there were marked positional differences between the ranks of both classes and schools when analysed using the single-level Means-on-Means regression method as opposed to the Multilevel regression method. It was also found that, under the single-level Means-on-Means model as compared to the Multilevel model, some classes changed in their category of 'effectiveness in producing quality student outcomes': (1) two classes went from being labelled with 'as expected performance' to 'lower than expected performance'; (2) one class went from being labelled with 'higher than expected performance' to 'as expected performance'; (3) one class went from being labelled with 'lower than expected performance' to 'as expected performance'; and most notably, (4) seven classes went from being labelled with 'as expected performance' to 'higher than expected performance'.
The findings of this study suggest that residual based performance indicators calculated using the Multilevel model are more accurate and fair than those produced using the single-level Means-on-Means regression model, and would enable both schools and teachers to report on accountability and investigate a greater range of class and school effectiveness issues with more confidence. The implication of the findings highlight the need for further research into the benefits of using multilevel, longitudinal models to measure school performance in Australia. Furthermore, the instability of residual estimates as performance indicators has been shown in this study and it is recommended that the limitations of their use be heeded.
Keywords: Multilevel modelling, school effectiveness, performance indicators
|Author: Jessica Elderfield BPsych, MEd Email: email@example.com
Please cite as: Elderfield, J. (2009). Thesis abstract: Comparing single-level and multilevel regression models in analysing Rasch measures of numeracy. In Proceedings Western Australian Institute for Educational Research Forum 2009. http://www.waier.org.au/forums/2009/elderfield.html