[ Proceedings Contents ] [ Schedule ] [ Abstracts ] [ WAIER Home ]

Research into the effectiveness of online learning in higher education: Survey findings

Lou Siragusa
Curtin University of Technology		 WAIER logo
With the use of online learning environments firmly integrated into many higher education courses, the effectiveness of this mode of delivery has come under question and has been the subject of various studies in this area. This paper presents an ongoing research study seeking effective instructional design principles for the online learning environment in higher education. A survey was administered to students and their lecturers studying in various courses throughout the major universities in Western Australia who are using online learning environments. The survey utilised both quantitative and qualitative methods for collecting data including questionnaires and interviews. The student survey involved over 250 students who responded to an online questionnaire and 25 students who were interviewed. The survey focused on elements of online learning including content, structure, motivation, feedback, interaction and learning strategies. Among the findings, many students indicated that they were generally satisfied with using online learning support for the units they were studying, but are often disappointed with the way in which it is being used in some of the courses.

Introduction

Universities are under ever-increasing pressure to compete in an environment where globalisation of learning is becoming a reality through the delivery of their courses on the Internet. It is not uncommon for a student studying in higher education to have encountered an online learning environment in their first year of study. The literature would suggest that this pace is not slowing. New developments in online learning technologies is focusing on more "intelligent" content management systems that are, for example, able to change how the content is presented to individual students depending upon the learning style they are demonstrating (Martinez, 2001). With these continuing advances in online learning technologies, it is important to note that "... technology is not enough; ... it does not provide quick and easy solutions and ... educators need to build on their expertise and experience as teachers and apply them while working with others in the development of web-based teaching and learning" (Ryan, Scott, Freeman, & Patel, 2000, pp. 27-28).

The vast body of literature referring to instructional design models theories and models has been largely overlooked by designers of online learning environments. Although implementing online learning technologies is relatively easy, creating educationally effective and useful websites is still difficult (Pan, 1998). As online learning technologies continue to develop, it is more important to consider the use of effective instructional design principles for this environment (Chen, 1998). The way in which people read an HTML page on the computer screen is different from the way they read word processed documents on printed paper. Over recent years, authors have been documenting these differences and have acknowledged the need for further investigation into instructional design for Internet delivery. Greening (1998), for example, argued that "...generally, instructional designers either do not always appear to take advantage of the hypermedia technology, or do so without pedagogical foundation." Decisions made at the instructional design phase of course development can influence and encourage different learning strategies that can be used by students (Bull, Kimball, & Stansberry, 1998, pp. 40-41; McLoughlin & Oliver, 1998; Smith & Ragan, 1999, pp. 138, 233-4).

This paper will describe a study currently in progress aimed at identifying effective instructional design principles and learning strategies for web-based learning environments in higher education. This study involved the administration of a survey to students and their lecturers studying in a number of units throughout the five universities in Western Australia who are using web-based learning environments. This paper will report on the preliminary analysis of the student survey.

Research questions

This research asked the following two questions:
  1. What are the factors that make for effective instructional design in web-based learning in higher education? This involves examining students' and lecturers' perceptions of the factors that make for effective online learning environments. This also involves an investigation into how existing principles of instructional design can be applied to web-based learning.

  2. What are the learning strategies that influence students' successful learning in web-based learning environments? This involves examining how students are being instructed on how to use online learning and what learning strategies do lecturers encourage their students to use. This also involves an investigation into students' satisfaction towards the use of web-based learning.

Background

The focus of this study is to seek understanding of how instructional design principles can be applied to online learning environments. Instructional design is about promoting processes that lead to successful learning regardless of the delivery medium being used (Smith & Ragan, 1999, p. 18). In order to provide a structure for this study in which effective instructional design principles can be developed for online environments, the following three main discipline areas were examined: (1) Learning theories, learning philosophies and instructional design principles; (2) Research into student learning in higher education; and, (3) Online learning technologies in higher education.

Learning theories, learning philosophies and instructional design

The literature highlights a major shift from early behaviourism through to cognitive learning theories. Cognitivism is revealed as having the greatest influence on the development of instructional design theories and models (Smith & Ragan, 1999; Venezky & Osin, 1991, p. 76). The contemporary influence of constructivism has also significantly impacted upon instructional design (Smith & Ragan, 1999, p. 14). This has meant a shift from teacher-centred instruction to student-centred instruction. Earlier editions of instructional design models, such as those presented by Gagné, Briggs, & Wager (1992) and Dick, Carey, & Carey (2001) were based upon behaviourism. The later editions of these models incorporated findings from cognitive based studies acknowledging the importance of learner analysis, cognitive strategies, motivational strategies and information presentation strategies (Gagné et al., 1992). Unfortunately, it is evident that designers of online learning environments are largely over-looking this vast body of knowledge that has contributed to learning theories and in structional design models (McBeath & Siragusa, 2000).

Research into student learning in higher education

The way in which students learn in higher education has been the focus of study for many educators teaching in these institutions. Universities today are under increasing pressure to reduce public expenditure and to increase the numbers and diversity of their student population (Boud, Solomon, & Symes, 2001, p. 3). This presents challenges for university lecturers as they are now forced to accommodate students who demand flexible, targeted, accessible learning methods (Ryan et al., 2000, p. 12). The design of instruction for higher education courses needs to be sensitive to the students' needs regardless of the delivery medium to be utilised. Prior knowledge and experience have been found to affect how they process new knowledge being taught (Laurillard, 1993, p. 30).

Researchers have identified various approaches students have demonstrated towards a learning task including deep approach (obtain a deeper understanding of the content), surface approach (memorise facts and complete assessment tasks without a deep understanding of the content) and strategic approach (an alertness to marking schemes to obtain the highest possible grades) (Entwistle, 1987, p. 60). Student are capable of varying their approach to a learning task according to their interpretation of the demands of the learning situation (Laurillard, 1993, p. 32). Research has demonstrated, however, that students should be encouraged to adopt a deep approach towards the learning tasks they encounter for effective learning to take place (e.g., Biggs, 1987; Biggs, 1999; Entwistle, Hanley, & Hounsell, 1979; Entwistle, 1987; Laurillard, 1993). A number of models have been put forward to describe ways in which instruction can be designed to encourage students to adopt a deep approach to learning tasks such as the Presage, Process, Product (3P) model of student learning (Biggs, 1999, p. 18). The 3P model takes into account interrelated features of the learning experience which include: student factors (prior knowledge, ability, motivation), the teaching context (objectives, assessment, teaching), learning focused activities (appropriate deep approaches), and learning outcomes (facts, skills, transfer). Designers of instruction need to consider such models of student learning when designing for higher education courses regardless of the delivery medium.

Online learning technologies in higher education

The introduction of the use of the Internet for learning in higher education has been justified by its proponents for its potential to provide cost-effective flexible learning for a diverse student population (Ryan et al., 2000, p. 13). However, evidence has suggested that the expansion of universities to allow for a more diverse range of students using online technological solutions "... has led to a move from small group teaching to large group teaching, and from individual supervision to group supervision, with 'inevitable loss of quality of learning experience for students' " (Ryan et al., 2000, p. 14). Although online learning technologies may not have lived up to all of its promises, there have been situations where it has been put into practice to provide satisfactory solutions to particular problems. The World Bank-funded African Virtual University, for example, have utilised online learning technologies to provide an infrastructure for delivering education and training to less well-developed regions of Africa (Ryan et al., 2000, p. 17).

Online learning has undoubtedly altered the way courses are being delivered in many universities. The Internet is being used in a variety of ways from supporting traditional face-to-face delivered classes to the complete delivery of instruction over the web. However, we are only just beginning to explore the possibilities of online learning and coming to recognise its strengths and limitations (Ryan et al., 2000, p. 28). It is clear that that online learning technologies does not provide the complete cost-effective solutions once hoped for by earlier advocators of the use of this medium for learning. If students are expected to experience successful learning through the use of the Internet, as with the use of any other medium, the design of instruction must consider the learners' needs and how the learning will interact with this environment.

A model for online learning in higher education

The literature would suggest that there is a gap between the body of knowledge relating to learning theories, instructional design principles and research into student learning in higher education and the body of knowledge relating to online learning technologies. Although some authors are attempting to bridge this gap (e.g., Ryan et al., 2000), discussions relating to learning theories, for example, are mentioned at the beginning of the text and are rarely referred back too throughout the discussions relating to the development of online learning environments. This study attempts to tie this body of knowledge together with the survey findings in order to produce a model for describing instructional design principles for the online learning environment.

From the literature composed of discussions and research findings concerning the three main discipline areas acknowledged above, there are six distinct focus areas which have been identified as having direct influence on the design of effective online learning environments. These six focus areas have been categorised as: Structure, Content, Motivation, Feedback/Help, Interaction, and Learning Strategies. This has been illustrated in Figure 1.

Figure 1

Figure 1: A model showing focus areas relating to online
learning in higher education identified in the literature.

Although it is not possible to include all of the literature discussions within these focus areas in this paper, the following will provide a summary of some of the findings within these focus areas which have contributed to the development of this study. The findings within these focus areas have contributed to the formation of dimensions (and scales) for the survey instruments used in this study.

Structure

The way in which instructional information is structured and displayed to the learners can be interpreted in different ways by each learner in accordance with their own cognitive structure (Laurillard, 1993, p. 51). Most disciplines in higher education require students to remember large bodies of knowledge. The knowledge needs to be organised in such a way that it can be easily retrieved (Gagné et al., 1992, p. 83). Instructional theories, such as (Reigeluth, 1999a) elaboration theory, have attempted to prescribe how content should be structured. The structure of an online learning environment should also follows these principles to assist with student learning. Research has shown that students are capable of determining what features including the structure and appearance they like most and least about the website they use (Chandler & Maddux, 1998, p. 1059).

Content

The instructional designer's task is to determine the most appropriate way of delivering the unit's content to the students. Laurillard (1993) argued that effective teaching requires educators to know more than just the subject content; "They need to know the ways it can come to be understood, the ways it can be misunderstood, ... how individuals experience the subject" (p. 3). Instructional models, such as the systematic instructional design model developed by Dick et al. (2001), describe in detail how to conduct an instructional goal analysis and a subordinate skills analysis in order to determine which content is relevant for the instructional goal and in what sequence it should follow. In many studies, students have been asked to give their perceptions of the content's quality, completeness, relevance and sequence which they encounter on web-based learning environments (e.g. , Pujola, 1998, Appendix 1; Schlough & Bhuripanyo, 1998, p. 100; Swan, Bowman, Holmes, Sylvie, & Vargas, 1998-99, pp. 98-99). Students have also been asked to give their perceptions of the effectives of using the Internet for finding information relating to the unit they are studying (e.g., MacDonald & Mason, 1998, p. 41).

Motivation

The learners' will to sustain learning can only be achieved through intrinsic motivation (Bruner, 1966, p. 40). Ralph (1998, pp. 2-6) provided strategies for which educators can employ to assist their students develop intrinsic motivation including: develop positive relationships, attract learners' attention, enhance subject-matter relevance, build leaner confidence and promote learner satisfaction. The effectiveness of the lecturer has been attributed to the students' levels of motivation when using online learning environments (Agarwal & Day, 1998, p. 106; Foley & Schuck, 1998). Whether students are encouraged to participate in course activities and to ask questions, the type of help available to students, students being able to keep to a schedule, and so on, contribute to how students cope with using the Internet for learning (Mory, Gambill, & Browning, 1998). There are a number of factors that contribute to making a website pleasing to use such as the appearance of the website, the use of text and graphics, the amount of materials presented on each page, and so on (Summerville, 1998, pp. 431-7). How students persevere with technical problems and how these problems are resolved also contributes to students' levels of motivation (Everett, 1998; Mory et al., 1998).

Feedback/help

Numerous instructional design theories and models emphasise the importance of providing clear, timely and informative feedback to students about their performance and assisting them to proceed more effectively (Gagné et al., 1992, p. 196; Reigeluth, 1999b, p. 6). web-based learning environments are able to provided students with a mechanism to: submit their assignments; to receive prompt feedback; seek help with technical problems; and, to seek help with coursework matters (Thomas, Carswell, Price, & Petre, 1998). Evaluation feedback must be perceived by students to be fair and authentic for students to continue their desire to accomplish a learning goal (Ralph, 1998, pp. 5-6). Online learning content management systems (such as WebCT) have facilities for students and lecturers to be able to send and receive assignments and feedback. How effectively these facilities are used will affect the quality of the help and feedback that students receive.

Interaction

Learners need to be encouraged to contribute to their own learning through their interaction with not only the course materials, but also with their peers and their tutor (Bandura, 1977). Interaction between peers may include describing, explaining, questioning, discussing, defending, encouraging, supporting and assessing each other's work (Ralph, 1998, p. 145). The use of interaction between students and their tutor in web-based learning environments has often been described as the most important feature of a web-based learning environment (Wagner, 1998; Forsyth, 1996). How students and lecturers have used online interaction to assist with student learning has often been the subject of investigation in online learning environments (e.g., Jiang & Ting, 1998; Graham & Scarborough, 1999).

Learning strategies

According to Piaget and Inhelder (1969), students in higher education should have obtained the formal-operational stage of cognitive development where they are generally able to be more plan-full, strategic and efficient in their organisation and manipulation of the available information (Flavell, 1977, pp. 102-12). Assisting students to become more aware of their own learning strategies contributes to developing deep approaches (Entwistle, 1987). Through the process of thoughtful instructional design, effective learning strategies for the students to adopt can be developed (Smith & Ragan, 1999). Finding effective learning strategies for students to use has been the subject of investigation by educators working with online learning (Bull et al., 1998, pp. 40-41; Hawkes, Cambre, & Lewis, 1998; Shih, Ingebritsen, Pleasants, Flickinger, & Brown, 1998, p. 363). When students are observed using successful learning strategies, they should be encouraged to continue using them (Smith & Ragan, 1999). Students, for example, have been observed responding positively about taking a more active role in dealing with the course content and, therefore, were favourable towards the learning environment (Oliver & Omari, 1999).

Methodology

Quantitative and qualitative research paradigms

Researchers have frequently argued that good research practice involves the use of multiple methods to enhance the validity of the research findings (Mathison, 1988). Denzin (1970) described methodological triangulation as one of the six principle types of triangulation allowing for different research methods to be employed on the same object of study as a check on validity (Cohen & Manion, 1994). The quantitative and qualitative approaches used in this study relied upon the use of descriptive research in the form of questionnaires and semi-structured interviews used to describe and interpret what is or what exists (Cohen & Manion, 1994). Collecting students' perceptions of the effectiveness of their class website was an achievable and practical method of inquiry. Students within a learning environment are in a good position to evaluate instruction because of their experience with many other learning environments (Fraser, 1998).

Research design

The use of questionnaires (quantitative) and semi-structured interviews (qualitative) employed in this study allowed for the collection of data from large and varied groups of students and lecturers. After the data collection and analysis phase is complete, the results will contribute to the development of an instructional design model for online learning in higher education. Figure 2 illustrates the research design for this study.

Figure 2

Figure 2: Summary of the research design

The survey instruments

The survey examined as many online educational websites as possible through online lecturers and students who were willing to participate in the survey. By questioning as many students and lecturers as possible, a clearer picture of what they perceived to be effective instructional design principles for online learning could be made. The six main focus areas (structure, content, motivation, feedback, interaction and learning strategies) identified from the three main discipline areas acknowledged earlier in this paper have contributed to the design of the survey instrument questions for this study. The studies presented in the six focus areas identified numerous qualities of effective design of instruction which can be related to online learning environments in higher education. These qualities have been included in the design of the survey instruments.

The questions were designed to solicit students' perception (affective) of how well instructional design principles had been applied to their online learning environment. A separate questionnaire was prepared for the lecturers (although the findings of this questionnaire will not be reported in this paper). The majority of the questions in the questionnaire were written in a five-point Likert-scale style format. The questionnaires also contained a combination of factual questions and open ended questions. The questionnaires for the students was administered using an online questionnaire form, which submitted the results electronically to the researcher. After completing the questions, the respondents clicked on the Submit button allowing the responses to be emailed to the researcher. The responses were then automatically sorted into an Excel worksheet ready for analysis. The questions in the semi-structured interviews were similar to those given in the questionnaire; ho wever, the respondents were allowed to expand upon their answers if they wished. Table 1 displays the dimensions used for the questionnaire and interview schedule which were administered to the students. Table 2 provides a sample of some of the questions within each of the dimensions.

Table 1: Dimensions used for the students' questionnaire and interview schedule

DimensionDescription
Structure/
organization		This described how the information was displayed to students on the web.
ContentThis described the content of the learning materials that students worked through. Content included the subject/course content, assignments, activities, case studies, lecturer/tutorial/laboratory notes, reading materials, tests, etc.
MotivationThis described how the learning materials and the online learning environment were made appealing and interesting for students.
Feedback/helpThis described how students obtained coursework feedback and technical help.
InteractionThis described how students communicated with each other and with their instructor.
Learning strategiesThis described what learning strategies instructors encouraged students to use and what learning strategies students found effective.

Table 2: Sample questions items within each of the dimensions used
for the students' questionnaire and interview schedule

DimensionSample question items
Structure/
organization		The class web site was interesting and pleasant to use. (+)
I needed to constantly jump from page to page to find what I was looking for. (-)
ContentThe content was organised in an appropriate sequence. (+)
The content covered all essential information (both theory and practical). (+)
MotivationThe material that was presented on the class web site was interesting and meaningful to me. (+)
Online learning provided me with flexibility such as being able to study when it was convenient to me. (+)
Feedback/helpThere were clear instructions on how to submit assignments. (+)
I received prompt feedback from my lecturer/tutor about my assignments. (+)
InteractionI found the online discussions with my lecturer/tutor and other class members valuable. (+)
The interactions that I had with my lecturer/tutor affected the outcomes of my assignments. (+)
Learning strategiesI took more responsibility for my own learning working in a class with an online component. (+)
I was able to share with other students in this class what it was that I did not understand. (+)

Questionnaire piloting

As the student online questionnaire formed a significant part of the data collection for this study, it was necessary to pilot the questionnaire with a samples of students. Students from the Faculty of Education at Curtin University of Technology were selected for the initial pilot of the student questionnaire. The students who participated attended a face-to-face delivered class that was supported with an online component. Lecturers from the Faculty of Education at Curtin University were also asked to participate in the initial pilot. Based upon comments provided by the students and lecturers, the questionnaire was modified and made ready for administration to the general student population.

Administering the survey

During the second semester of 2001 and first semester of 2002, the major divisions from Curtin University of Technology, Edith Cowan University, Murdoch University, University of Notre Dame Australia and University of Western Australia were invited to participate in this study. The departments from within the university divisions contacted who agreed to participate in the survey are listed in Table 3.

Table 3: Departments within divisions who participated in the survey

UniversityDepartment/School/Faculty
Curtin University of TechnologyFaculty of Education
Science and Mathematics Education Centre
School of Physiotherapy
School of Media and Information
Edith Cowan University (Churchlands)Faculty of Business and Public Management
Murdoch UniversitySchool of Education
School of Environmental Science
University of Notre Dame AustraliaSchool of Business
University of Western AustraliaFaculty of Engineering, Computing and Mathematics
Department of Science

From these departments, 252 students completed the online questionnaire. From these 252 students, 25 students participated in a semi-structured interview.

Preliminary data analysis and results

As this is a study in progress, the following results are based upon preliminary analysis of the survey data. At the time of writing, a full analysis of both the quantitative and qualitative data was yet to be completed. The following will report on the overall data collected and will refrain from a detailed item-by-item analysis. At the preliminary stage of the analysis, however, interesting findings within the data collected were beginning to emerge as discussed in the following.

The preliminary analysis of the survey data focused on the main areas identified in the research questions.

Quantitative results

The online questionnaire data collected was analysed using the Lertap 5 (Nelson, 2002) item analysis software. With each dimension, Likert-type (1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, 5 = strongly agree) question items were selected for use as scales. Internal consistencies for each of the scales were established by calculating the coefficient alpha (Cronbach's alpha) as reported in Table 4.

Table 4: Reliability, mean ranges and scale means for student questionnaire
student n=252
ScaleReliability
(coefficient alpha)		No. of
scale items		Range of means
Lowest meanHighest mean
Structure/ organisation 0.7463.423.83
Content 0.7883.314.31
Motivation 0.7573.294.08
Feedback/ help 0.7983.113.90
Interaction 0.7353.383.62
Learning strategies 0.7372.813.69

Nelson (2001, p. 140) put forward Kaplan and Sacuzzo (1993) as having stated: "It has been suggested that reliability estimates in the range of .70 and .80 are good enough for most purposes in basic research" (p. 126). Therefore, the alpha values will be considered acceptable for the purposes of this study.

Table 4 also reports the range of means for each scale from lowest to highest. Figure 3 displays a sample of typical histograms showing student questionnaire responses from each scale. From the range of means and the sample of histograms, it can be seen that students appear to be mostly selecting the Agree (4) option in response to the affective questions. Although this may indicated that the majority of students are satisfied with the online learning environments they are using, there appears to be a reluctance to strongly agree with the affective statements presented in the questionnaire. As the majority of students appear not to be entirely satisfied with web-based learning, this may indicate areas of weakness within these environments needing further investigation.

Figure 3: Structure/Organisation and Content
Structure/OrganisationContent
Figure 3: Motivation and Feedback
MotivationFeedback
Figure 3: Interaction and Learning strategies
Interaction      Learning strategies      

Figure 3: Sample histograms of student questionnaire responses within each scale

Learning strategies

In the questionnaire, students were asked to identify what learning strategies they were encouraged to use on their online learning environment. Overall, students were able to identify some of the strategies they were using as illustrated in Figure 4.

Figure 4

Figure 4: Histograms of learning strategies used by students with online learning

Qualitative results

The questionnaire included a section which asked students for their permission to participate in an interview. From those who agreed to an interview and were contactable, 25 students were interviewed over the telephone. The interviews were conducted at the end of the semesters to ensure they had experienced an entire learning program with their online learning environment.

The qualitative data collected soundly confirmed the quantitative findings. The students appeared to be generally satisfied with using an online learning environment, even if it is just used to support face-to-face teaching. However, the students have identified areas of weakness with the way in which online learning environments were being used.

Although an interview schedule was prepared using the same dimensions as for the online questionnaire, students were encouraged to discuss any concerns they wish regarding the online learning environment they used for that semester. The way in which students responded to the interview questions varied. Some students were particularly keen to express problems they had encountered and some were enthusiastic to offer suggestions for improvements. For example, when students were asked questions such as, 'Where you happy with the way in which the material was presented on the web site?' a typical response was "Yes" to these types of questions. However, some students did make meaningfully critical comments regarding their satisfaction with the online learning environment. The following presents examples of comments made by students in response to the questions within each of the survey dimensions.

Structure/organisation

Students were asked for their opinions regarding how the material was presented on the website and if they had difficulties accessing it.
"Um, there was a mixture of stuff on there. Um, I think it could have been better organised."

"I'm quite experienced with using the web. I did spent a fair bit of time with peculiarities of the WebCT ... like pages that, um, hang for a time that you click to go to them and so you got to back out and go in a second time, sort of quite bizarre programming."

Some students were able to make adjustments to the way in which the material was presented on the website by reformatting it to suit their requirements.
"... I went into the WebCT and copied it out, and plonked it into my work, and then I start doing my assignments from there, by reformatting it. I'm very skilled with computing side of it, so I don't think I have any problems what so ever."

Content

Students were asked to comment on their level of satisfaction with the subject content that was on the unit's web page. Students were generally quite satisfied with the content that was provided. However, comments were made about the completeness of the unit's content, particularly when it is used to support face-to-face classes.
"Yeah, subject content was fine, ... we just had lecturers and went to manual tutorial classes and none of the notes from that lecture were put on the web or anything. You had to actually go to the lecture. If you missed the lecture [you had] to go to the lecture and copy the notes, and that was frustrating ... as a mature age student, because if you missed one lecture then I could not do the tute properly for that week because you need the notes because there was no text book."
Other students needed to have access to content materials that would support assessment activities which were also online.
"I think basically to just give out ... more information on the unit and more practical questions ... I wish they had you know more than just five to practice."

Motivation

Students were asked for their perception of how they were motivated to study with the online learning environment. They were asked, 'Did the information that was presented stimulate your interest for this unit?'
"...Um, but I [was] sort of clicking through these links to see why they have been put there, and um, I found some stuff that was not just very interesting but I found it very relevant to a lot of the questions that we have been discussing in the workshops."
Students were also asked if their online lecturer showed concern for the students' progress and if the lecturer's interest stimulated their interest for the unit.
"Yes, in a way because anytime when I had problems then I would email her and then she would come back to me with the reply."
Prompt replies to email messages was shown to be encouraging for students' progress throughout the unit.

Feedback

Students were asked if there was adequate help available with using the online learning environment. The students who were interviewed did not appear to need extra help. However, they perceived that some other students may have benefited from more online help.
"So, I don't think they had anything like, um, you know frequently ask questions with solutions to common problems people might have... I don't think there was much there, but um, I don't know how much [help] would be needed... I think it would probably would be helpful. A good thing to have a lot more on help online because ... I have a picture that quite a lot of students were actually using it [WebCT] less than they might because of lack of familiarity. So, if they found some nice pages there that explained a lot of things in a straight forward terms, then that could have been valuable."
Students were asked to comment on the promptness and effectiveness of the feedback they received from their online lecturer. Overall, students appeared to be quite happy with feedback regarding assignments. Some students were using an online assignment submission application and commented on its effectiveness.
"... submitting assignments online, you don't actually get an email back ... saying 'yeah we have got your assignment', ... you are not sure whether the lecturer did get the assignment, you don't actually get a response, you need to get a direct response ..., and that was a little bit, um, disappointing because I wasn't sure whether they had received my assignment..."
One student commented on how working in groups and receiving feedback from other students regarding their assignments was helpful. This also helped to increase motivation to succeed in the unit.
"... when my fellow students had some problem and I thought, 'oh, I'm not the only one' and you know we say 'OK, what do we do about it', and there was a learning lab system so we decided to go there."

Interaction

Students were asked to comment about their perception of the effectiveness of the use of interaction on their online learning environments. Interaction, whether through email, bulletin boards or chat rooms, appeared to be beneficial to many students.
"I preferred email because, um, I like checking my friends assignments and they check mine. You know what I mean. A single checking is not plagiarism or anything like that."
Students were asked if they wished there were more online discussions. Some were happy not to participate in online discussions, although they may have sent out the occasional email. Other students wanted more online discussions.
"Um, yeah. I would want that because the more you discuss maybe you might find you know better solutions and you get other peoples opinion I think that would be very good."

Learning strategies

Students where asked a number of questions to find out what sort of learning strategy they employed with their online learning experience. In many cases, students struggled to identify what learning strategies worked for them. Students were often set on a linear path to follow throughout the online unit, particularly where the online learning environment was used to automatically present a series of exercises for students to complete.
"Um, we were doing the problems over the website..., yes but I mean they had the answers on there and you lost two marks, two out of ten for every time you did it incorrectly and you had practise questions..."
Many students were encouraged to work in groups, particularly if they were to work on case studies and problem solving activities.
"We worked in groups, ... we were meant to meet each other but really we didn't as a group, we really only emailed each other, divided up the course work and then put it together, we integrated it all. So I mean it didn't work, so I don't know, in a way its negative, because it allows you to have direct communication so you're not going to have group meetings, you are not going to see each other in person, but in a way for the students its so much better because there is no point in meeting each other, it is much better to divide the contents up into sections, divide it up and do it."
Students often searched the Internet to assist them with the completion of assignments and presentations.
"Assignment and the work, assignments mainly. You had to go online and get information from other sites, analyse other sites..."
Some students commented on how they believed using the Internet for learning allowed them to take more responsibility for their own learning.
"... because some lectures are very repetitive and ... this helped me sort of take control of my own learning and have it at my own pace."

Likes and dislikes about online learning

At the end of the interview students were asked what they liked and disliked the most about the website they were using. The following is just a small sample of comments made by four students.
"It was good because we had the flexibility to do it any time we wanted..."

"Um, I think that the sort of instance feedback and required interaction with the lecturers and stuff and the flexibility are the two things. I think ... being able to talk to the lecturers like people, if you know what I mean [laughs]. Strange, but ... sharing ideas and things, um, I found out [to be] really good, and also just the flexibility of time and placement stuff that is so much better than other ways of doing things."

"Um, but the only probable disadvantage was the way it kept crashing and the deadlines and stuff sometimes could not be met..."

"Um, nothing to do with the online component when you lose the connection with the ISP, or the computer decides to freeze on you, things like that, technical difficulties beyond and above. But you know that was one thing, but ... quite a lot of the stuff [was] on CD as well, so that was ... really good backup."

The comments made by students during the interviews suggested that they were reasonably satisfied with the online learning environments they were using. Although some of these comments suggests certain problems with the online learning environment they were using, some comments also highlight certain strengths with web-based learning.

Discussion

As was identified in the quantitative analysis, students mostly agreed (4) with the positive statements about the effectiveness of the online learning environment they were using. Although they initially appeared to be generally satisfied with the website they were using, there were few 'strongly agree' (5) responses to the affective statements, indicating possible weaknesses in the effectiveness of the online learning environments. This was confirmed with the qualitative analysis of the interviews. The interviews revealed that students were reasonably satisfied with the online learning environment they used, but there were aspects of these websites which students perceived as needing improvement.

Some of the weaknesses identified in the interviews related to online learning content management systems (e.g., WebCT, Blackboard, etc.) which were used to deliver the learning materials on the Internet. This is not to imply that the weaknesses only rest with the content management systems themselves, but possibly there are weaknesses with how they are being utilised. Students using non-commercial online content management systems also found weaknesses in these systems; in the systems themselves and how they were being used.

The questionnaire provided open-ended questions as to how each of the components (dimensions) of the online learning environments could be improved. Students provided responses to these open-ended questions which will require further analysis. To thoroughly identify the areas of weakness, an item-by-item analysis of the questionnaire responses is also needed.

Some of the students who participated in the interviews have demonstrated what the literature refers to as "deep approach" to their learning. They display a desire to want to fully utilise the online learning environment and the Internet in general. Some of these students have gone so far as to attempt to change the structure and sequence in which the learning materials are presented on these websites. It has also become clear throughout this preliminary analysis of the survey data that many students are "web-aware" and have certain expectations as to how the Internet should be used as online learning environments.

Conclusion

As this is a study in progress, full analysis of the survey data is yet to be completed. The preliminary analysis reported here, however, does clearly indicate particular weaknesses perceived by students about how online learning environments are being employed. Whether the online learning environments are used to supported face-to-face delivered classes, or are being used to completely deliver instruction to students in remote locations, there are elements which are used in both cases. Many students are accustom to viewing many websites on the Internet. They appear to be making comparisons with what they see in non-educational Websites with the online learning environments they are being exposed to. Hence, they are becoming increasingly more critical of the web pages they access as well as other aspects of the Internet including online communication facilities which are being utilised.

The vast amount of literature relating to learning theories, instructional design principles and online learning technologies should be an initial source of reference in the design of online learning environments. While we struggle to keep up with the online learning technologies, it is important to remember that, in higher education, we are delivering learning experiences to students. These online learning experiences can be made more effective with the employment of sound instructional design principles. Further analysis and development of this study will attempt to align time-tested learning theories and instructional design principles with the design of effective online learning environments.

As detailed analysis of both the quantitative and qualitative data continues, further recommendations will inevitable emerge. It is expected that these recommendations will develop into a model for the instructional design of online learning environments in higher education.

References

Agarwal, R., & Day, E. (1998). The Impact of the Internet on Economic Education. The Journal of Economic Education, 29(2), 99-110.

Bandura, A. (1977). Social learning theory. Englewood Cliffs, New Jersey: Prentice Hall.

Biggs, J. B. (1987). Student approaches to learning and studying. Melbourne: Australian Council for Educational Research.

Biggs, J. B. (1999). Teaching for quality learning at university. Buckingham: Society for Reserch into Higher Education & Open University Press.

Boud, D., Solomon, N., & Symes, C. (2001). New practices for new times. In D. Bound & N. Solomon (Eds.), Work-based learning: A new higher education? Buckingham: The Society for Research into Higher Education & Open University Press.

Bruner, J. S. (1966). Toward a theory of instruction. Cambridge, MA: Harvard University Press.

Bull, K. S., Kimball, S. L., & Stansberry, S. (1998). Instructional design in computer mediated learning. Paper presented at Coming Together: Preparing for Rural Special Education in the 21st Century. Conference Proceedings of the American Council on Rural Special Education (18th, Charleston, SC, March 25-28, 1998). U.S., Oklahoma, Charleston, SC.

Chandler, B., & Maddux, C. D. (1998). Chandler, B. & Maddux, C. D. Paper presented at SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington DC, March 10-14, 1998). Proceedings. Nevada, Washington DC.

Chen, L. L. (1998). Web-based distance instruction: Design and implications of a cybercourse model. In Proceedings of Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT). Sponsored by the Research and Theory Division (20th, St. Louis, MO, February 18-22, 1998), U.S., Michigan.

Cohen, L., & Manion, L. (1994). Research Methods in Education (4th ed.). London: Routledge.

Denzin, N. K. (1970). The Research Act in Sociology: A Theoretical Introduction to Sociological Method. London: The Butterworth Group.

Dick, W., Carey, L., & Carey, J. O. (2001). The Systematic Design of Instruction (5th ed.). New York: Longman.

Entwistle, N. J. (1987). Changing perspective in education: Understanding classroom learning. London: Hodder and Stoughton Educational.

Entwistle, N. J., Hanley, M., & Hounsell, D. J. (1979). Approaches to learning and levels of understanding. British Journal of Educational Research, 5, 99-114.

Everett, D. R. (1998). Taking Instruction Online: The Art of Delivery. Paper presented at SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceedings. Kentucky, Washington, DC.

Flavell, J. H. (1977). Cognitive Development. Englewood Cliffs, New Jersey: Prentice-Hall, Inc.

Foley, G. & Schuck, S. (1998). Web-based conferencing: Pedagogical asset or constraint? Australian Journal of Educational Technology, 14(2), 122-140. http://www.ascilite.org.au/ajet/ajet14/foley.html

Forsyth, I. (1996). Teaching and learning materials and the Internet. London: Kogan Page.

Fraser, B. J. (1998). Classroom environment instruments: Development, validity and applications. Learning Environments Research, 1, 7-33.

Gagné, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of Instructional Design (4th ed.). Fort Worth: Holt, Rinehart & Winston, Inc. Harcourt Brace College Publishers.

Graham, M. & Scarborough, H. (1999). Computer mediated communication and collaborative learning in an undergraduate distance education environment. Australian Journal of Educational Technology, 15(1), 20-46. http://www.ascilite.org.au/ajet/ajet15/graham.html

Greening, A. (1998). WWW support of student learning: A case study. Australian Journal of Educational Technology, 14(1), 49-59. http://www.ascilite.org.au/ajet/ajet14/greening.html

Hawkes, M., Cambre, M., & Lewis, M. (1998). The Ohio SchoolNet Telecommunity Evaluation. Paper presented at the North Central Regional Educational Lab., Oak Brook, Illinois. Oak Brook, IL.

Jiang, M., & Ting, E. (1998). Course design, instruction, and students' online behaviors: A study of instructional variables and students' perceptions of online learning. Paper presented at the Annual Meeting of the American Educational Research Association (San Diego, CA, April 13-17, 1998). U.S., New-York, San Diego, CA.

Kaplan, R. M., & Sacuzzo, D. P. (1993). Psychological testing: Principles, applications, and issues. Pacific Grove, California: Brooks/Cole.

Laurillard, D. (1993). Rethinking university teaching: A frame work for the effective use of educational technology. London: Routledge.

MacDonald, J., & Mason, R. (1998). Information skills and resource-based learning in an Open University course. The Journal of Open and Distance Learning, 13(1), 38-42.

Martinez, M. (2001). Designing learning objects to personalize learning. [viewed 11 Jul 2002] http://reusability.org/read/chapters/martinez.doc

Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13-17.

McBeath, C. and Siragusa, L. (2000). The place of instructional design in higher education in the computer age. In Flexible Learning for a Flexible Society, Proceedings of ASET-HERDSA 2000 Conference. Toowoomba, Qld, 2-5 July. ASET and HERDSA. http://www.ascilite.org.au/aset-archives/confs/aset-herdsa2000/procs/mcbeath.html

McLoughlin, C. & Oliver, R. (1998). Meeting the needs of gifted and talented students through technology supported distance teaching. Australian Journal of Educational Technology, 14(1), 35-48. http://www.ascilite.org.au/ajet/ajet14/mcloughlin.html

Mory, E. H., Gambill, L. E., & Browning, J. B. (1998). Instruction on the web: The online student's perspective. Paper presented at SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceedings. US; North-Carolina, Washington, DC.

Nelson, L. R. (2001). Item analysis for tests and surveys using Lertap 5 [Software manual]. Perth, Western Australia.

Nelson, L. R. (2002, 26th June 2002). Lertap 5! [viewed 4 Aug 2002] http://www.lertap.curtin.edu.au/index.htm

Oliver, R. & Omari, A. (1999). Using online technologies to support problem based learning: Learners' responses and perceptions. Australian Journal of Educational Technology, 15(1), 58-79. http://www.ascilite.org.au/ajet/ajet15/oliver.html

Pan, A. C. (1998). Optimize the web for better instruction. Paper presented at SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceedings." U.S., New-Jersey, Washington, DC.

Piaget, J. & Inhelder, B. (1969). The psychology of the child. New York: Basic Books.

Pujola, J. T. (1998). Ewebuation. Edinburgh Working Papers in Applied Linguistics, 9, 104-115.

Ralph, E. G. (1998). Motivating teaching in higher education: A manual for faculty development. Still water, Oklahoma: New Forums Press Inc.

Reigeluth, C. M. (1999a). The elaboration theory: Guidance for scope and sequence decisions. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instruction theory (Vol. II, pp. 425-453). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Reigeluth, C. M. (1999b). What is instructional design theory and how is it changing? In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instruction theory (Vol. II, pp. 5-29). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Ryan, S., Scott, B., Freeman, H. & Patel, D. (2000). The virtual university: The Internet and resource-based learning. London: Kogan Page.

Schlough, S. & Bhuripanyo, S. (1998). The development and evaluation of the Internet delivery of the course "Task Analysis". Paper presented at SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceedings, U.S., Wisconsin.

Shih, C. C., Ingebritsen, T., Pleasants, J., Flickinger, K. & Brown, G. (1998). Learning strategies and other factors influencing achievement via web courses. Paper presented at Distance Learning '98. Proceedings of the Annual Conference on Distance Teaching & Learning (14th, Madison, WI, August 5-7, 1998). U.S., Iowa, Madison, WI.

Smith, P. L. & Ragan, T. J. (1999). Instructional Design (2nd ed.). New York: John Wiley & Sons, Inc.

Summerville, J. B. (1998). The role of awareness of cognitive style in hypermedia. Proceedings of Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT) Sponsored by the Research and Theory Division (20th, St. Louis, MO, February 18-22, 1998). U.S., Kansas, St Louis, MO.

Swan, K., Bowman, J., Holmes, A., Sylvie, S. & Vargas, J. (1998-99). "Reading" the web: Making sense on the information superhighway. Journal of Educational Technology Systems, 27(2), 95-104.

Thomas, P., Carswell, L., Price, B. & Petre, M. (1998). A holistic approach to supporting distance learning using the Internet: Transformation, not translation. British Journal of Educational Technology, 29(2), 149-161.

Venezky, R., & Osin, L. (1991). The Intelligent Design of Computer-Assisted Instruction. New York: Longman Publishing Group.

Wagner, E. D. (1998). Interaction Strategies for Online Training Designs. Paper presented at Distance Learning '98. Proceedings of the Annual Conference on Distance Teaching & Learning (14th, Madison, WI, August 5-7, 1998). US, California, Madison, WI.

Please cite as: Siragusa, L. (2002). Research into the effectiveness of online learning in higher education: Survey findings. Proceedings Western Australian Institute for Educational Research Forum 2002. http://www.waier.org.au/forums/2002/siragusa.html

[ Proceedings Contents ] [ Schedule ] [ Abstracts ] [ WAIER Home ]
Created 30 Aug 2002. Last revised 21 May 2006. URL: http://www.waier.org.au/forums/2002/siragusa.html
The Forum Proceedings are © Western Australian Institute for Educational Research. However
the copyright for each individual article remains with the authors of the article.
HTML: Roger Atkinson [rjatkinson@bigpond.com]