[ Proceedings Contents ] [ Schedule ] [ Abstracts ] [ WAIER Home ] Evaluating a university program Clare McBeath Faculty of Education, Curtin University Renu Sharma Moira Watson Research and Development Division, West Coast College of TAFE

This paper examines an evaluation research project into Curtin University's courses in Vocational Education and Training teacher education by an independent research body. The reasons for employing an outside research body to review a university program are discussed.

Research methods include a review of the literature, including State and Federal government policy directives, interviews with TAFE and industry management and with Curtin lecturers and surveys of past and present students.

Stage 1 includes an appraisal of similar courses offered locally and nationally, collection and collation of information about the courses, a study of the course content, and a literature review of relevant evaluations including previous evaluations conducted specifically on these courses. Stage 2 consists of interviews with lecturers, identification of key stakeholders and the preparation, piloting and refinement of the survey instruments. In Stage 3 the survey will be distributed and data collated and analysed. The results and recommendations on ways and means of enhancing course effectiveness and saleability will comprise the final report in Stage 4. This paper reports on data collected in Stages 1 and 2.

Introduction

The courses were reviewed and extensively restructured three and a half years ago. Now the courses are in their fourth year and it is necessary to begin the review process again, to evaluate again in time for rewriting during 2002 and for offering the next upgrade in 2003.

Meanwhile the training scene itself has changed. There has been a general downgrading of desired professional qualifications in the training scene. University degrees tend to be undervalued by TAFE colleges, the State Departments responsible for training and by ANTA on the Federal scene. Very few government-employed trainers receive either time or financial assistance to undertake university studies any more. The exceptions to these constraints are coming from industry, but industry trainers are more difficult to reach and their numbers in the courses are still small.

Economic rationalism has brought about changes in the language of training. An overemphasis on training as little more than the assessment of observable competencies has encouraged a minimalisation of reflective practice and critical thinking. As much as we may deplore this, we still have to be aware of its existence and make sure that the courses acknowledge the strengths and weaknesses of such approaches.

Furthermore, industry reform and the devaluing of university professional qualifications have made it much harder to market university courses for trainers. Student numbers are falling across Australia. In these circumstances, the university certainly needed to review the courses and see what could be done to make them more attractive to the market for which they were designed.

Why evaluate courses

Even in the university context, where the five year review and evaluation process is supported by policy and practice, more often than not, the process is not specifically funded and staff are rarely given release time to do it properly. Sometimes an outside reviewer is called in as part of the process and some of the more important decisions are left in his or her hands, but most of the work usually remains the responsibility of overworked staff.

Yet there are many reasons why courses should be regularly reviewed and updated. Basically, it is to make them more saleable in a highly competitive field, but there is also an intrinsic satisfaction in being able to say about one's own course materials that they are current, relevant, of high quality, well structured and well written.

Specifically, the purpose of our evaluation is to find out the worth or value of the courses in a changing environment. Curriculum teams need to find out how good their courses and materials are, and whether they continue to work in practice, or whether parts have to be changed or adapted.

Some evaluators claim that the requirements are to verify that the course objectives match the course outcomes. However we need to go far beyond this as educational programs may have many aims, objectives, and outcomes which are not explicitly expressed. The range of stakeholders or audiences will also have differing agendas.

Our perspectives therefore have to be wide and varied. The stakeholders in these courses include not only past, present and future students, but their employers, their potential employers, and both State and Federal Government departments, which have an interest in industry training and are currently influencing standards of professional development in the training field. The University is also a stakeholder, as its requirements and regulations have a significant impact on standards and quality, and the length, size and level of the courses. The lecturing staff who wrote and now teach and assess the individual units are also very powerful stakeholders. They are the ones who have been most closely monitoring the units and noticing where there might be discrepancies and changing factors influencing the course. The lecturers also are dedicated to their fields of expertise and often have a very wide perspective across its theory and practice in many different contexts.

The evaluation plan

Even though the TAFE market has been shrinking and the industry trainer group growing and becoming identified as an important future client, the university had to turn to TAFE to find a research organisation equipped to handle the research. The Research and Development Division at West Coast College was invited to draw up a proposal. Curtin staff had worked with them before and were impressed by the standard of their work.

The research team conceptualised its role to include recommendations for the maintenance and improvement (defined as effectiveness and saleability) of the Curtin courses. They identified their objectives as follows

• to overview the current offerings of the program
• to identify key factors attracting students to, and retaining students in, the program
• to identify the level of interest or disinterest among training managers in TAFE colleges, government departments and industry
• to identify future changes for strengthening the program
• to identify any untapped potential market

Methodology and current position in terms of completion

Stage 1	Background research and comparison with other courses in terms of delivery mode, this has been completed. Comparison in terms of course content is still in progress.
Stage 2	Consultation with the course coordinator and lecturers through interviews followed by an initial evaluation of interview responses, this has been completed. This stage also includes the preparation of survey instruments for students and stakeholders, these are currently under development.
Stage 3	Will comprise utilisation of the survey instruments currently under development to survey stakeholders, current students and former students.
Stage 4	Will begin to draw together conclusions through analysis of data (carried out on completion of data collection) and preparation and collation of results.
Stage 5	Dissemination of conclusions through a final report.

The project is partially complete, with Stage 1 awaiting only some further comparison with other courses for full completion and Stage 2 complete, apart from finalising and clearing the survey instrument.

Preliminary findings

Five lecturers teaching on the VET courses at Curtin University were interviewed. The interview schedule focused on three areas:- program specific, learner support and general evaluation. A number of common, (identified by most of the interviewees) and thus significant, themes were identified during analysis of the interviews. These highlight some of the strengths of the courses and also recognise a need for change in some areas.

The interviewees were unanimous in feeling that the courses maintain a good balance between theory and practical through the design of assignments which encourage the application of theoretical concepts to practical situations within the workplace. There is also an awareness of the need for ongoing modifications to keep pace with changes within the VET sector and in approaches to teaching and learning. This is particularly so with respect to two modules which are perceived to be less in tune with current needs and/or lacking in application of theory.

There was a consensus that the design of exercises and assignments is significant in building into the program both reflective and critical thinking skills. This is combined with a recognition that assignment schedules often coincide with the heaviest TAFE lecturer workload thus having a potential impact on the time available for reflection during completion.

Generally there was agreement that program credibility is enhanced through forging strong and supportive relationships with students and also developing positive links with employers.

Access and equity was also highlighted for the positive benefits of on-line delivery in terms of improving access and equality of opportunity for rural and remote students. Also for its potential in terms of re-formatting material for students with a visual impairment. However this is balanced by a consciousness of access problems if the internet is not easily available. Interviewees were also aware that attitudes to computers and preferred learning modes can also influence these considerations either positively or negatively with regard to individuals.

The major client groups are still perceived to be mostly TAFEs although there are also medical educators/health professionals and increasingly other government employees. There are also some from large corporations in private industry. Areas outside TAFE present opportunities for applying marketing strategies with a view to increasing numbers from these sectors. Employer support for students is often highly variable with TAFE offering no financial support although often being supportive in other ways. Support for students from industry is also very variable. Students who do not complete courses often cite time pressure, uneven workloads, conflicting job commitments and travelling as the reasons.

The next phase

Stakeholder/employer identification is also in progress - development of this survey instrument awaits completion as this is to some extent dependent on the sector make-up of the stakeholders/employers. Further comparison of the courses (in terms of content) with other similar courses will also be completed (information on content of some courses has not yet been received).

Responses to closed questions in questionnaires will be graphically represented and if number of responses are sufficient they will be subjected to statistical analysis. Once analysis is complete the final report will be compiled and submitted.

Conclusions to date

Authors: Dr Clare McBeath, Faculty of Education, Curtin University Renu Sharma and Moira Watson, Research and Development Division, West Coast College of TAFE Please cite as: McBeath, C., Sharma, R. and Watson, M. (2001). Evaluating a university program. Proceedings Western Australian Institute for Educational Research Forum 2000. http://www.waier.org.au/archives/forums/2001/mcbeath.html