[ Abstract for this presentation ] [ Proceedings Contents ] [ Schedule ] [ Abstracts ]

Multi-site action research case studies: Practical and theoretical benefits and challenges

Mary Delfin Pereira and Roger Vallance
University of Notre Dame Australia
WAIER logo
A curriculum initiative project was implemented in four schools in Singapore over a span of five to six weeks during 2004. In conducting research in the diverse schools, there was also an opportunity to study the interactions between action research and multi-site case studies. Though action research and case studies are frequently used in education to research curriculum initiatives, their interactions are seldom explored. Moreover, the practical benefits and challenges of multi-site case studies in an action research are little discussed.

The project employed a number of different schools: girls only, boys only and co-educational schools; different levels of performance in a graded situation; multiple teachers and classes within each site; and control and experimental conditions for the curriculum implementation. Thus, by examining the particular benefits and challenges presented by this project, it is hoped that this paper will contribute to a better understanding of case study action research through describing:


Background

Before describing how the research materialised and features of the research itself, it may be pertinent to examine the researcher's teaching experiences. In a curriculum initiative in which the researcher designed the curriculum, the researcher's experience working in schools in Singapore as well as in the US. proved useful in a number of areas.
  1. In the designing of the curriculum, the researcher could draw on her experiences teaching an integrated English and Literature program in Singapore and as a Language Arts teacher in the US. The knowledge gained during her Masters Education course in the US. also aided in the curriculum design.

  2. Her prior experience as a teacher helped in building a rapport with the teachers and students with whom that she came into contact during the course of the research.

  3. Her knowledge of generic practices common across schools enabled her to adapt and adjust according to individual school's needs and requirements.

  4. Her teaching experience also aided in the analysis of the qualitative and quantitative data collected. In addition, the experience enabled her to draw conclusions from the findings that may have relevance to other schools as well.
Before the research was undertaken, a pilot survey was conducted in January, 2004 to discover the number of secondary schools in Singapore that have an integrated English and Literature Program. The survey was also used to ascertain how the English Departments in the schools viewed an integrated English and Literature Program and to discover if any schools would be interested in participating in the research. Six schools articulated initial interest and four schools eventually participated in the research. Among the four schools, two were co-educational schools, another was a boys' school and the last, a girls' school. The Secondary One students of these schools were selected to participate in the research. One of the schools wanted all its Secondary One Express and Normal (Academic) classes to participate in the research. (Express students and Normal (Academic) students take four and five years respectively to complete their secondary education.) The three other schools were willing to have some classes participate as control classes. Table 1 displays the four schools and information about participant classes and teachers in the schools.

Table 1: Participating schools and classes and teachers in each school

SchoolsTypeExperimentalControl
ClassesTeachers ClassesTeachers
1Co-educational8*600
2Boys3131
3Girls3322
4Co-educational2333
* Originally there were 9 classes, but one, a Normal (Academic) class, had to be excluded from analysis due to discrepancies in some of the data collected from the class.

One of the schools, the boys' school, already has an integrated English and Literature Program. However, there are distinct differences between the school's program and the proposed program. The existing program in the school is a Literature-Based English Program (LBEP) whereas the proposed program is a Literature-Driven English Program (LDEP). In the LBEP, most of the teaching time is spent on literary analysis. Limited time is spent on teaching language skills. When these skills are taught, the literature text is not used as a medium of instruction. The LDEP aims to correct this imbalance by connecting the explicit teaching of language skills to the study of the literature text. The literature text becomes the vehicle through which students are taught to be better writers and readers. While in a LBEP it is hoped that students will gain language skills through interaction with the literature text, in the LDEP nothing is left to chance. The students are guided to see the literature text as a model of well-crafted language. It is hoped they will, through the process of theory application, be able to transfer knowledge of the craft into their own use of the language.

The four week LDEP was designed and implemented in the four schools in July 2004. The program was designed to run for four weeks because schools could only allow a limited period of time for the research to be conducted in their schools. In the end, due in part to holidays and school events, the program ran for between five and six weeks in all the four schools. The intent of the research was to discover if the program can be successful in a naturalistic environment with many mediating variables within schools and among schools. The mediating variables included:

Multi-site action research case study

A research that involves a curriculum initiative is multi-faceted in many respects. There are many variables that could influence the outcome and as such, it would be very difficult to reach any conclusion regarding whether the curriculum has been effective or not. That was the main reason the research was designed to include at least four schools with varying student bodies and cultures to test the experimental curriculum. After all, "a finding emerging from the study of several very heterogeneous sites would be more robust" (Shofield 2000, p.80). As such, conclusions reached from the findings derived from the four schools would be more persuasive than if the experimental curriculum was tested on the students of one school. Figure 1 displays an overview of the research.

Figure 1

Figure 1: Overview of the Research on the LDEP

Thus, multi-site case study was the chosen option for the research on testing the effectiveness on the LDEP in imparting language skills.

Theoretical benefits

Firstly, a case study was decided upon as the preferred method of conducting the research on the LDEP. "(C)ase studies will often be the preferred method of research because they may be epistemologically in harmony with the reader's experience and thus to that person a natural basis for generalization" (Stake 2000, p.19). Therefore, since the intended audience would be schools, a project undertaken in individual schools would enable other schools' administration and teachers to make comparisons and determine if the outcomes are relevant to them.

Instead of concentrating on a single site, it was decided to conduct the research in multi-sites because '(t)he same case study may involve more than one unit of analysis. This occurs when, within a single case, attention also is given to a subunit or subunits.' (Yin 1994, p.41, emphasis in the original). Thus, each school and each class within each school becomes a case study with sub-units. The conclusions drawn from the findings from each school are studied in relation to the school as well as in comparison to the other schools. In this way, what Yin (1994) considers as a possible problem in embedded case study design can be avoided. As he puts it, a major problem with '(a)n embedded design... occurs when the case study focuses only on the subunit level and fails to return to the larger unit of analysis' (Yin 1994, p.44). While effectiveness of the LDEP is analysed in relation to each mediating variable, in the end, the researcher returns to the main research question, namely is the LDEP effective in imparting language skills in spite of the mediating variables.

By studying the multi-sites as individual case studies as well as a larger single case study, sub-unit analysis as well as cross-comparisons can be made. In doing so, either a 'literal replication' or a 'theoretical replication' (Yin 1994, p. 46) can be made. A literal replication occurs when the intended replication of measured outcomes is achieved. In a theoretical replication, the intended replication does not produce comparable measurable outcomes. However, these different outcomes generate explanations to account for the lack of comparability in terms of the variables in the design. These explanations could lead to the creation of hypotheses.

If a literal replication is made, in that the curriculum in effective in the various sub-units, then it may be possible to come up with a 'theoretical framework...(which) later becomes the vehicle for generalizing to new cases' (Yin 1994, p.46). However, on the other hand, if the conclusions arrived at are different, then there will be a theoretical replication. In the case of theoretical replication, the researcher may surmise that the curriculum may be effective in some schools or classes but not in others due to some pertinent mediating variables exerting an influence over the findings.

Therefore, an action research into a curriculum initiative in multi-site cases provides a number of advantages which also argue for the validity of the research findings and conclusions drawn from them. These advantages include:

  1. Testing the curriculum in a 'real' situation. Schools vary, students differ and so do classes. It would be difficult to reach any conclusion about a curriculum that is tested on a single school. The school culture or organisational structure may have an influence on whether the curriculum succeeds or not. The school may have high-achievers and thus, the motivated students could be a reason why the curriculum succeeds. There would be many plausible reasons for a curriculum succeeding in one school and just as many possible reasons why it may not in another. Therefore, depending on a single site or school would make the findings applicable only to that school and perhaps to schools with very similar characteristics. However, multi-site cases afford the researcher with the opportunity to test the curriculum in situations that include more variables.

  2. Multi-site cases will lead to a greater coverage or sample of potential variables. In the case of the research on the LDEP, there are many variables with regard to ability, gender, school and class cultures, and teaching styles. These variables exist within as well as across the schools studied.

  3. The greater the number of mediating variables it is more likely that the findings will be more robust. A curriculum that is tested in all four schools is the only constant amidst a myriad of variables. If it succeeds, it would be easier to draw a conclusion that the inherent merits in the curriculum were the most likely reasons for the improvement observed in the students' performances.

  4. Any conclusion formed from an analysis of similar findings collected from multi-site case studies may lead to a 'naturalistic generalization' (italics is the author's), derived at by recognising the similarities of objects and issues in and out of context and by sensing the natural covariations of happenings. To generalise this way is to be both intuitive and empirical' (Stake 2000, p.22). Thus, the conclusions arrived at from such analyses could be extrapolated to other schools with similar contexts.

  5. The 'naturalistic generalization' could result in a wider potential interest and audience. Other schools in Singapore, and perhaps, in other countries as well, have a wider choice of schools with which to compare theirs. As such, the likelihood of finding schools with similar circumstances in multi-site cases is high.

Theoretical challenges

Multi-site case studies may present many advantages, but they also come with challenges as well. The challenges faced by the research during the course of multi-site case studies include:
  1. Reconciling the many differences and conflicts in pertinent variables. For instance, when findings were analysed from the perspective of gender, the girls from the two co-educational schools and a girls' school were grouped together. However, there are differences between the culture of a co-educational school and a single-sex school. Therefore, in the analysis what was done was to analysis the performance of all the girls as one sub-unit of gender. Then, a second sub-unit was created to differentiate the performances of the girls from the co-educational schools from the performance of the girls in the girls' school.

  2. Deciding on the meaningfulness of the variables was a second challenge. A variable on its own is of no importance unless it has the potential to influence the finding. Initially, the outcomes of the performances in the writing assessment of the classes taught by the same teachers in the control and experimental grou ps were examined. However, such a comparison would be unnecessary and irrelevant. In the first place, the control group did not register any improvement. It would be more pertinent to compare the outcomes of the control and experimental classes taught by the same teachers in one of the schools. Then, the teaching style would be the control variable in both the experimental and control classes taught by the same teacher. The different curricular used in the experimental and control classes would become the mediating variable.

  3. The third challenge is how the discrepancy in outcomes between schools or classes can be explained. Since there are many variables, it would be difficult to decide on which variable or variables may have influenced the difference in the outcomes. For example, Schools 1, 2 and 4 registered similar improvements but the improvement achieved by the students in School 3 was significantly lower. Since the three other schools would also have many mediating variables, what was different about the variables found in School 3 that had an impact on the overall performance of the students? Would School 3 being an all-girls' school become an influencing factor since the other three schools were co-educational and boys' schools? Or were there other factors involved?

  4. The final challenge rested in the premise that if there were discrepancies to which answers could not be easily sought, there can be no replication. Then, the results would be pertinent only to the individual cases and would have no significance for any other schools. Fortunately, in the research on the LDEP, there were more similarities than discrepancies and an attempt could be made to explain whenever discrepancies did crop up. Within classes with discrepancies there were similarities as well as differences between these classes and others.

Practical benefits

Apart from the theoretical benefits and challenges, multi-site case studies offer practical benefits as well. These practical benefits include:
  1. Diversity of variables which could lead to a greater understanding of how effective the curriculum can be. A curriculum that is effective in more schools with more classes would have more practical benefits than one that is found to be effective in only one school or a few classes. The possibility of coming up with a literal replication or a theoretical replication is higher. In all the four schools, there was improvement in the performances of the experimental classes, leading to a literal replication from which a theoretical framework could be created (Yin 1994). At the same time, there were some discrepancies in the outcomes of the writing assessment of one school and a couple of classes in particular. The degree of improvement was not as high as for the other schools or classes. A closer examination of the qualitative data led to a theoretical replication (Yin 1994) whereby possible explanations were derived from the differences found in the way the program was implemented and in similar attitudes of the students in the classes. As an aside, it must be noted here that the use of mixed methods led to a richer analysis. There was the quantitative data derived from the pre-test and post-test results and the content analysis of the surveys that students completed. The qualitative data included data from interviews, observations, teachers' log book entries and field notes. The use of quantitative and qualitative data led to "the multiple sources of evidence (which) essentially provide(d) multiple measures of the same phenomenon." (Yin 1994, p.92).

  2. In multi-site case studies, there are also multiple conditions which reflect the real situation in the educational arena. By trying out the curriculum in these different conditions, without manipulating or controlling any of the variables, any finding in relation to the effectiveness of the curriculum would be more reliable. In the research on the LDEP, it was a 'take us as you find us' situation. The researcher did not attempt to impose any conditions but adapted and adjusted according to the needs and requirements of the schools. For example, in one school the periods were one hour long whereas in the other schools the periods were either thirty or thirty-five minutes long. The lesson outline and lesson plans were adjusted to reflect these differences in timetabling.

  3. Multi-site case studies also present the researcher with a large amount of data to work on. There are two advantages to this. The first advantage is that should there be any problem with any particular data, there are other sources on which the researcher can rely. In the LDEP research, data from the surveys and the pre-test and post-test results of one class was excluded when it was found that there is a high possibility that the data might be unreliable. However, there was data from sixteen other classes that could be analysed. The second benefit is the rich source of data allowed for theoretical replication in the case of discrepancies in outcomes. The classes with discrepancies came from different schools but from observations, conversations with teachers and interviews with teachers and students, similarities between the classes emerged. It was then, possible to form a theory about the discrepancies.
These practical advantages made the argument for reliable hypotheses arrived at from the findings more plausible.

Practical challenges

In addition to practical advantages, some of the practical challenges included:
  1. The limited time and resources available to the researcher in the conduct of the research. The researcher was working alone in schools that could afford only a limited time of a few weeks to complete the program. It was a strain on the researcher and perhaps, if the researcher had devoted her attention to one case study, some of the limitations of the research could be eliminated. One limitation was the short period of time the researcher could spend with the teachers from each school. As a result she left it to the teachers to get back to her if they had any difficult in the implementation of the LDEP. Unfortunately, many of them did not, and the researcher became aware of the problems only during school visits and interviews at the end of the program.

  2. There was also the problem of whether there would be replication since research was being conducted in multiple sites with very different characteristics.

  3. The large amount of quantitative and qualitative data meant a lot of time and energy would be needed to analyse it.

Overcoming the practical challenges

Some steps were taken by the researcher to overcome the challenges.

Firstly, the researcher made herself available to the teachers and schools whenever they needed her. Through the use of email and mobile phones, the teachers were able to get in touch with the researcher at any time. In cases of emergencies, the researcher was able to go to the schools at the shortest possible time. In one instance, when a teacher had doubts about how to teach a particular lesson, she contacted the researcher at home at night and the researcher obliged.

Though the workload was great, the prospect of meeting the challenge of analysing the large amount of data proved to be exciting. The time and resources spent on discovering what the findings meant was worthwhile.

Conclusion

There are theoretical and practical challenges to a multi-site action research. On the other hand, the theoretical and practical benefits of such a research are enormous. The theoretical advantages aid in validating the research and the practical benefits makes the findings more reliable. For these reasons, a multi-site action research undertaken to explore a curriculum initiative was found to be fruitful and rewarding.

References

Shofield, J. W. (2000). Increasing the generalizability of qualitative research. In R. Gomm, M. Hammersley & P. Foster (Eds.), Case Study Method: Key Issues, Key Texts. London: Sage Publications.

Stake, R. E. (2000). The case study method in social inquiry. In R. Gomm, M. Hammersley & P. Foster (Eds.), Case Study Method: Key Issues, Key Texts. London: Sage Publications.

Yin, R. K. (1994). Case Study Research: Design and Methods (2nd ed. Vol. 5). Thousand Oaks, California: Sage Publications, Inc.

Authors: Mary Delfin Pereira and Roger Vallance, University of Notre Dame Australia, Fremantle Campus, 19 Mouat St, Fremantle, Western Australia 6959. Email: delfinpereira@lycos.com, rvallance@nd.edu.au

Please cite as: Pereira, M. D. and Vallance, R. J. (2005). Multi-site action research case studies: Practical and theoretical benefits and challenges. Proceedings Western Australian Institute for Educational Research Forum 2005. http://www.waier.org.au/forums/2005/pereira.html


[ Abstract for this presentation ] [ Proceedings Contents ] [ Schedule ] [ Abstracts ]
Created 8 Aug 2005. Last revised 8 Aug 2005. URL: http://www.waier.org.au/forums/2005/pereira.html
The Forum Proceedings are © Western Australian Institute for Educational Research. However
the copyright for each individual article remains with the authors of the article.
HTML: Roger Atkinson and Clare McBeath