|[ Proceedings Contents ] [ Forum 1997 Abstracts ] [ WAIER Home Page ]
Monitoring standards in the arts: A testing program using an outcomes frameworkBeverley J. Pascoe
Education Department of Western Australia
Since the inception of education standards in the form of Student Outcome Statements, the Education Department of Western Australia has conducted system level testing at Years 3, 7 and 10 in all learning areas apart from Languages Other than English. During 1996 the bold step was taken to test The Arts. This paper describes the testing program which was used to test The Arts using the Student Outcome Statements as a framework. It describes the processes used in the Monitoring Standards in Education project and the conceptualisation of a methodology to test The Arts, the strategies used in the testing and their relationship to the Student Outcome Statements, and the methods of marking the tests and analysing the data. A brief discussion of the difficulties encountered in sampling is also included, together with an overview of the final results of the testing.
During the initial stages of development extensive research was conducted into current Arts education philosophy on assessment, and consultation with teachers and other people with expertise in The Arts was carried out together with piloting and trialing in schools to establish appropriate methodologies for testing. The innovative strategies developed for the testing were based on similar philosophies of those being adapted in 1997 for the National Assessment of Educational Progress (NAEP) testing in the Arts in the United States of America. Extensive consultation with Dr Carol Myford, who is on the US National Assessment Governing Board, formed a part of the early development stage. The tests were developed by the Evaluation Branch of the Education Department of Western Australia in collaboration with Educational Testing Centre and, importantly, with input from practising arts educators.
Many of the assessment methods developed for the MSE testing measure, for the first time, the depth and complexity of student knowledge and skills in the learning area, and innovative standardised assessment procedures have been developed which allow students to work in ways that reflect good classroom practice. Where possible, 'hands on' strategies are used to monitor student performance and open-ended assessment items provide students with the opportunity to perform to the maximum of their abilities. The MSE assessments permit reliable evidence to be gathered on outcomes that are not readily addressed through machine-marked paper and pencil tests.
MSE does not identify schools or individual students. However, at the completion of testing and reporting, school versions of MSE tests are produced to provide schools with a reference point to compare their students' performance against the state as a whole, and as a consequence to implement their own programs to address any specific areas of need. This procedure increases the value of MSE to teachers and schools and MSE materials are widely used by schools as part of the school assessment procedures for their management information systems. Schools are made aware that MSE tests, although well researched and highly regarded, are limited by the constraints of system level testing and should be regarded as a 'snapshot' only of student performance, to be used to substantiate teachers' own records and judgements.
In Western Australia The Arts learning area comprises the five disciplines of dance, drama, media, music and visual arts, and statements are generic across these five areas. However, each is a discrete discipline with its own techniques and conventions. The statements describe student progress and achievement in four strands; 'Communicating Arts Ideas' and 'Using arts skills, techniques, technologies and processes' both of which relate to expression of The Arts; and 'Responding, reflecting on and evaluating the Arts' and 'Understanding the role of the Arts in society' which relate to appreciation of The Arts .
MSE reports the performance of representative samples of Years 3, 7 and 10 students and provides information about the achievements of subgroups of students: girls and boys; Aboriginal and Torres Strait Islanders and students from non-English speaking backgrounds and, as 1996 was the first year that collection of MSE data in the Arts was undertaken, this data will establish benchmarks for future testing in this learning area.
As audience and critics they analyse and reflect on the use of arts elements and develop criteria for critically appreciating and making informed judgements about their own arts works and that of others. They should also recognise the role and value of arts in their lives. Students need this type of participation and evaluation of various forms to develop their aesthetic understanding and practice.
These objectives were foremost in the minds of test developers and it was agreed unanimously that they could not be assessed using the multiple cho ice, machine scorable testing devices so often used in system-level testing.
To address all four Outcome Statement strands related to expression and appreciation of The Arts, students at Years 3, 7 and 10 were required to complete two test forms, the 'Analysis' and the 'Process'.
An attempt was made to use as much local content as possible, as well as variety in culture and genre, in stimulus materials presented. The stimulus material included videotaped excerpts for dance, drama and media, audiotapes for music and colour prints of paintings for visual arts.
Because of the subjective nature of The Arts, when assessing such notions as aesthetics and critical analysis it was important that students were able to express their own opinions of mood, feeling, likes and dislikes. For this reason, answers which extracted personal opinions were not assessed but were included in tests as prompts for discussion and justification of responses. This provided the opportunity to gain an insight into students' knowledge and skills.
In the case of the performing arts of dance, drama and music where students worked in groups, they were encouraged to brainstorm ideas related to the stimulus and to write their ideas on an individual basis before joining their groups. They were then made aware of the evaluation criteria and given specific instructions as to how to combine their ideas to plan their performance as a group within the guidelines, notating their plan using conventional notations and/or words and diagrams. After being given a specific time allowance for rehearsal, groups performed as teachers videotaped their performances. Afterwards, students completed a critique of their group's performance on an individual basis. Students were evaluated on their individual planning sheets, the group plan, the group's performance, and the individual critique of their group's performance.
In the case of media and visual arts where students worked individually, they brainstormed ideas and were then made aware of the evaluation criteria and given specific instructions as to how to plan their presentation within the guidelines. They then completed a plan before commencing their piece of work and, after completion, students completed a critique of their own work. Students were evaluated on their individual planning sheets, the piece of work presented, and the critique of their work.
Stimulus material ranged from the one word stimulus, 'Fire,' for dance, to short stories, poems, videotape excerpts and paintings.
In order to standardise marking of performances for the performing arts, teachers were trained in the use of assessment materials and were required to videotape student performances. The videotapes were then marked centrally by markers with expertise in music who had been trained in marking procedures.
Through the use of common items and common stimulus material, both the 'Analysis' and 'Process' tasks allowed for linking of items through Years 3, 7 and 10, thus providing valuable information on student progression through the Student Outcome Statement levels. Each student was required to complete both the 'Analysis' and the 'Process' task in order to capture both strands of the Student Outcome Statements and data was analysed to report on students' overall results from the combination of both tests. Marking keys, which described the categories for each item indicating various levels of understanding, were developed.
At the end of the day, markers had marked a series of common tests and felt confident they had reached a unified understanding of levels and a clear understanding of interpretation of the marking key. However, they exchanged telephone numbers in order to make contact to discuss any unusual or difficult examples which may not have emerged during marker training. Markers were requested to rotate Year 3, 7 and 10 tests when marking in order not to lose track of the development of levels through the year groups.
Marking of the Process tests in the performing arts involved viewing of students' performances on video in relation to individual plans, group plan, and student appraisals. Markers worked in pairs with the whole group marking the same video-taped performance. When the marking of each student group was completed, discussion took place among all markers, with each pair giving explanations to justify levels allocated. This procedure was repeated throughout the day, rotating through Year groups 3, 7 and 10 until markers were confident they had reached an understanding of 'levelness' of performance.
Only those students at the Year 10 level who were currently studying the relevant discipline at the time of sampling were tested. Numbers for sampling were restricted by the limited number of students undertaking arts options at secondary schools and the limited numbers of schools offering the complete range of arts disciplines, together with the fact that a decision had been made to test each school in one of the five arts disciplines only. The following data demonstrates the limitations to the Year 10 students available for the arts sample:
|Discipline||Tot al enrolled||Total population||% of population|
|No. of arts subjects|
offered in Year 10
|No. of schools|
The low proportion of male enrolments to female enrolments in arts programs was cause for some concern. The breakdown of arts enrolments by gender is shown in the following table.
|Females enrolled||Males enrolled|
The following figure demonstrates the method used by MSE to report results of the overall sample in relation to Student Outcome Statement levels.
Curriculum Corporation (1994). The Arts - a curriculum profile for Australian schools. Victoria: A.E. Keating (Printing) Pty Ltd.
Education Department of Western Australia (1996). Student Outcome Statements. The Arts (draft version). Perth: Education Department of Western Australia.
Honeyman, A. (1996). Monitoring standards in education: A handbook of the operations, policies and procedures of the Education Department of Western Australia (draft version). Perth: Education Department of Western Australia.
Pascoe, B. J. (1995). The influence of primary school music programmes on student choice of music studies in lower secondary schools. Perth: Edith Cowan University (unpublished thesis).
Rasch, G. (1980). Probabilistic models for intelligence and attainment tests (expanded edition). Chicago: University of Chicago Press (original work published in 1960).
|Author: Beverley J. Pascoe MEd is a teacher who is currently on secondment
to the Education Department of Western Australia Monitoring Standards
in Education project. She has carried out research in music education
in both primary and secondary schools in Western Australia and
is currently undertaking PhD research in the measurement of classroom
music knowledge and skills utilising an outcomes framework.
Please cite as: Pascoe, B. J. (1997). Monitoring standards in the arts: A testing program using an outcomes framework. Proceedings Western Australian Institute for Educational Research Forum 1997. http://www.waier.org.au/forums/1997/pascoe.html