[ Proceedings Contents ] [ Schedule ] [ Abstracts ] [ WAIER Home ]

Evaluating a university program

Clare McBeath
Faculty of Education, Curtin University
Renu Sharma
Moira Watson

Research and Development Division, West Coast College of TAFE
This paper examines an evaluation research project into Curtin University's courses in Vocational Education and Training teacher education by an independent research body. The reasons for employing an outside research body to review a university program are discussed.

Research methods include a review of the literature, including State and Federal government policy directives, interviews with TAFE and industry management and with Curtin lecturers and surveys of past and present students.

Stage 1 includes an appraisal of similar courses offered locally and nationally, collection and collation of information about the courses, a study of the course content, and a literature review of relevant evaluations including previous evaluations conducted specifically on these courses. Stage 2 consists of interviews with lecturers, identification of key stakeholders and the preparation, piloting and refinement of the survey instruments. In Stage 3 the survey will be distributed and data collated and analysed. The results and recommendations on ways and means of enhancing course effectiveness and saleability will comprise the final report in Stage 4. This paper reports on data collected in Stages 1 and 2.


Introduction

The Vocational Education and Training program under discussion is reviewed every five years. The review process is an accountability procedure to ensure that courses are regularly updated and their relevance evaluated against the needs of the students and changing practices in society and industry.

The courses were reviewed and extensively restructured three and a half years ago. Now the courses are in their fourth year and it is necessary to begin the review process again, to evaluate again in time for rewriting during 2002 and for offering the next upgrade in 2003.

Meanwhile the training scene itself has changed. There has been a general downgrading of desired professional qualifications in the training scene. University degrees tend to be undervalued by TAFE colleges, the State Departments responsible for training and by ANTA on the Federal scene. Very few government-employed trainers receive either time or financial assistance to undertake university studies any more. The exceptions to these constraints are coming from industry, but industry trainers are more difficult to reach and their numbers in the courses are still small.

Economic rationalism has brought about changes in the language of training. An overemphasis on training as little more than the assessment of observable competencies has encouraged a minimalisation of reflective practice and critical thinking. As much as we may deplore this, we still have to be aware of its existence and make sure that the courses acknowledge the strengths and weaknesses of such approaches.

Furthermore, industry reform and the devaluing of university professional qualifications have made it much harder to market university courses for trainers. Student numbers are falling across Australia. In these circumstances, the university certainly needed to review the courses and see what could be done to make them more attractive to the market for which they were designed.

Why evaluate courses

Virtually all the literature on curriculum development emphasises the importance of evaluation as part of the course development process. However it is probably the most neglected part of development. Often this is because funded development tends to use up the budget on needs analysis and on the production of course materials, and evaluation of the completed materials is dropped for economic reasons. Evaluation may take place during the writing and trialing of the materials, but rarely as a formal process once the materials have been used for a number of years.

Even in the university context, where the five year review and evaluation process is supported by policy and practice, more often than not, the process is not specifically funded and staff are rarely given release time to do it properly. Sometimes an outside reviewer is called in as part of the process and some of the more important decisions are left in his or her hands, but most of the work usually remains the responsibility of overworked staff.

Yet there are many reasons why courses should be regularly reviewed and updated. Basically, it is to make them more saleable in a highly competitive field, but there is also an intrinsic satisfaction in being able to say about one's own course materials that they are current, relevant, of high quality, well structured and well written.

Specifically, the purpose of our evaluation is to find out the worth or value of the courses in a changing environment. Curriculum teams need to find out how good their courses and materials are, and whether they continue to work in practice, or whether parts have to be changed or adapted.

Some evaluators claim that the requirements are to verify that the course objectives match the course outcomes. However we need to go far beyond this as educational programs may have many aims, objectives, and outcomes which are not explicitly expressed. The range of stakeholders or audiences will also have differing agendas.

Our perspectives therefore have to be wide and varied. The stakeholders in these courses include not only past, present and future students, but their employers, their potential employers, and both State and Federal Government departments, which have an interest in industry training and are currently influencing standards of professional development in the training field. The University is also a stakeholder, as its requirements and regulations have a significant impact on standards and quality, and the length, size and level of the courses. The lecturing staff who wrote and now teach and assess the individual units are also very powerful stakeholders. They are the ones who have been most closely monitoring the units and noticing where there might be discrepancies and changing factors influencing the course. The lecturers also are dedicated to their fields of expertise and often have a very wide perspective across its theory and practice in many different contexts.

The evaluation plan

It was felt that an outside, independent evaluator would give a new and different perspective on the value of the courses. An evaluator selected from the market served by the courses would give better insights into that market's interest and response. The Curtin team began looking around at training research groups in Perth, as well as seeking some funding to make the evaluation possible.

Even though the TAFE market has been shrinking and the industry trainer grou p growing and becoming identified as an important future client, the university had to turn to TAFE to find a research organisation equipped to handle the research. The Research and Development Division at West Coast College was invited to draw up a proposal. Curtin staff had worked with them before and were impressed by the standard of their work.

The research team conceptualised its role to include recommendations for the maintenance and improvement (defined as effectiveness and saleability) of the Curtin courses. They identified their objectives as follows

From these data they proposed to establish the value of the teacher education courses and their relevance to industry and government vocational education and training needs. They further proposed to provide short term and long term recommendations for improvement of the courses. The final report is planned to give clear insights into the strengths, weaknesses and future opportunities for the program.

Methodology and current position in terms of completion

The evaluation is being carried out in several stages as described below.

Stage 1Background research and comparison with other courses in terms of delivery mode, this has been completed. Comparison in terms of course content is still in progress.
Stage 2Consultation with the course coordinator and lecturers through interviews followed by an initial evaluation of interview responses, this has been completed. This stage also includes the preparation of survey instruments for students and stakeholders, these are currently under development.
Stage 3Will comprise utilisation of the survey instruments currently under development to survey stakeholders, current students and former students.
Stage 4Will begin to draw together conclusions through analysis of data (carried out on completion of data collection) and preparation and collation of results.
Stage 5Dissemination of conclusions through a final report.

The project is partially complete, with Stage 1 awaiting only some further comparison with other courses for full completion and Stage 2 complete, apart from finalising and clearing the survey instrument.

Preliminary findings

Comparison with other courses thus far has related to delivery mode. Further comparison addressing content will be made. The provision is unique in Western Australia with Associate, Undergraduate, and Graduate levels and also a Masters being offered externally. Only one other provider in WA offers Associate, Undergraduate, and Graduate levels for VET and these are on an intensive internal Saturday or Summer School basis. This provider also offers a Masters both internally and externally. Another organisation offers similar courses only from Graduate level through a mixture of internal and external units and Summer Schools. Nationally, one provider offers courses for VET both internally and externally at: Associate, Undergraduate, and Graduate levels.

Five lecturers teaching on the VET courses at Curtin University were interviewed. The interview schedule focused on three areas:- program specific, learner support and general evaluation. A number of common, (identified by most of the interviewees) and thus significant, themes were identified during analysis of the interviews. These highlight some of the strengths of the courses and also recognise a need for change in some areas.

The interviewees were unanimous in feeling that the courses maintain a good balance between theory and practical through the design of assignments which encourage the application of theoretical concepts to practical situations within the workplace. There is also an awareness of the need for ongoing modifications to keep pace with changes within the VET sector and in approaches to teaching and learning. This is particularly so with respect to two modules which are perceived to be less in tune with current needs and/or lacking in application of theory.

There was a consensus that the design of exercises and assignments is significant in building into the program both reflective and critical thinking skills. This is combined with a recognition that assignment schedules often coincide with the heaviest TAFE lecturer workload thus having a potential impact on the time available for reflection during completion.

Generally there was agreement that program credibility is enhanced through forging strong and supportive relationships with students and also developing positive links with employers.

Access and equity was also highlighted for the positive benefits of on-line delivery in terms of improving access and equality of opportunity for rural and remote students. Also for its potential in terms of re-formatting material for students with a visual impairment. However this is balanced by a consciousness of access problems if the internet is not easily available. Interviewees were also aware that attitudes to computers and preferred learning modes can also influence these considerations either positively or negatively with regard to individuals.

The major client groups are still perceived to be mostly TAFEs although there are also medical educators/health professionals and increasingly other government employees. There are also some from large corporations in private industry. Areas outside TAFE present opportunities for applying marketing strategies with a view to increasing numbers from these sectors. Employer support for students is often highly variable with TAFE offering no financial support although often being supportive in other ways. Support for students from industry is also very variable. Students who do not complete courses often cite time pressure, uneven workloads, conflicting job commitments and travelling as the reasons.

The next phase

Questionnaire development for surveying the three student groups (current, past successful and past withdrawn) is underway and should be complete in the near future, The questionnaires will be mailed as soon after completion and clearance through the university as relevant contact information is available.

Stakeholder/employer identification is also in progress - development of this survey instrument awaits completion as this is to some extent dependent on the sector make-up of the stakeholders/employers. Further comparison of the courses (in terms of content) with other similar courses will also be completed (information on content of some courses has not yet been received).

Responses to closed questions in questionnaires will be graphically represented and if number of responses are sufficient they will be subjected to statistical analysis. Once analysis is complete the final report will be compiled and submitted.

Conclusions to date

Investigation of other available courses has shown that the delivery mode employed is unique in Western Australia. The interviews have established that the lecturers are in agreement regarding the strengths of the program and are conscious of a need for regular updating of material. Work on this evaluation project has also provided the opportunity for collaboration between West Coast College of TAFE and Curtin University with the potential for greater cross sectoral links and understanding.

Authors: Dr Clare McBeath, Faculty of Education, Curtin University
Renu Sharma and Moira Watson, Research and Development Division, West C oast College of TAFE

Please cite as: McBeath, C., Sharma, R. and Watson, M. (2001). Evaluating a university program. Proceedings Western Australian Institute for Educational Research Forum 2000. http://www.waier.org.au/forums/2001/mcbeath.html


[ Proceedings Contents ] [ Schedule ] [ Abstracts ] [ WAIER Home ]
Created 19 Oct 2001. Last revised 21 May 2006. URL: http://www.waier.org.au/forums/2001/mcbeath.html
The Forum Proceedings are © Western Australian Institute for Educational Research. However
the copyright for each individual article remains with the authors of the article.
HTML: Roger Atkinson [rjatkinson@bigpond.com]