A systematic observation schedule was developed in order to examine boys’ and girls’ attention in class and the extent and type of academic interactions with teachers and other pupils. Specifically, a five minute section of a video tape showing a seven year old in a primary school was observed in order to find out whether boys are less attentive and more disruptive. A review of the existing research in this tradition revealed that systematic observation was the best method to address the study’s research question. It has been used to study teaching since the 1930s when researchers began to explore teacher-student interactions and other classroom behaviors (Evertson & Green, 1986). By the 1960s, systematic observation had become the dominant method in the field of classroom research.
Systematic observation can be defined as “ a research tool that entails the direct observation of the behaviour and the recording of that behavior in terms of categories that have been devised prior to the start of data collection” (Bryman, 2008, p. 254). The basic purpose of systematic observation is to provide a brief description of the educational system (Croll, 1986). The observational data can be used in order to analyse teacher behavior, investigate pupil-teacher interaction patterns, and measure the effectiveness of different approaches to teaching (Croll, 1986; Ober, Bentley & Miller, 1971). According to Croll (1986), those within and outside the educational system can gain valuable information through the results derived from systematic observation.
Some fundamental aspects of systematic observation are worth taking into account before conducting any research. First of all, a clear focus is necessary before the data collection. The research problem needs to be explicitly determined so that the observer knows exactly who or what is to be observed (Bryman, 2008; Croll, 1986). In addition, a key point to systematic observation is the design of the coding scheme. The categories of the behaviour that are to be observed and the ways that behaviour should be assigned to those categories must be explicitly defined and established in advance (Bryman, 2008). Another characteristic of systematic observation, pointed out by Croll (1986) is related to the quantitative nature of the observational data. The analysis is based on statistical techniques which produce precise and accurate results that can be related to other data. Finally, the role of the observer is to strictly follow the observational procedure, ensuring that the same events will be coded in the same way by different observers (Croll, 1986).
At this point, it is important to present some examples of research that has used systematic observation techniques. Undoubtedly, two of the best-known schedules of systematic observation used in educational research are Flanders Interaction Analysis Categories (FIAC) and Observation Research and Classroom Learning Evaluation (ORACLE). FIAC was developed by Ned Flanders (1970) in order to assess the different levels of control exercised by the teacher over classroom activities (Hammersley, 1986). Since then, it is being used by the majority of researchers in a great variety of settings, either in its original version or modified to meet the needs of each study. Similarly, the ORACLE study was developed to investigate the different patterns of teachers’ and pupils’ behaviour and the supposed effects of these patterns on pupils’ progress (Graham & Hughes, 1995). The research involved two separate observation systems: a Pupil Record and a Teacher Record (Galton, Simon, & Croll, 1980).
An important study using systematic observation was conducted by Blatchford and his colleagues (2005), to examine the impact of class size on pupils’ behaviour. The observational part of this research retrieved data from children in both large and small classes (age 10 – 11 years). The schedule comprised the following categories: time spent in different work settings, time spent in different school subject areas, description of how children behaved when in three social modes (with their teachers, with other children, and when not interacting). A systematic observation schedule was also adopted by Humphreys and Smith (1987) in order to investigate rough and tumble play among children (7, 9, and 11 years) in the school playground. By the same token, Blatchford et al. (2006) used systematic observation techniquess to measure the extent to which pupils (8-10 years) were influenced by the SPRinG program (social pedagogic research into grouping). A conclusion based on the observational data was that group work can improve peer interactions if pupils have the appropriate training in the skills of group working. In brief, based on the research evidence as well as the literature review, systematic observation seems to prevail in the field of educational research.
Method
Procedure
A systematic observation method was used to examine whether boys are less attentive and more disruptive. A five minute section of a video tape of a seven year old in a primary classroom was observed. The aim was to provide a description of the “ target” child’s behaviour in the classroom during the observation interval. Teachers and other pupils were observed only when they interacted with the target child. Based on specific criteria, a set of categories was determined prior to the start of data collection in order to code the observed behaviour. The five minutes section of the video was divided into 30 continuous ten-second time intervals. The job of the observer was to record on the coding scheme the behaviour that was occurring during each interval (see Appendix 1).
Observation Categories
The schedule comprised categories that provided a description of how the target child behaved when in three social contexts: interaction with the teacher, pupil contacts, and target child not interacting. Each of these social contexts was divided into sub-categories based on the activities that the target child was involved during the observation. Explicit definitions of these categories follow.
Pupil X (Target) – Teacher Interaction
The current category referred to the child’s contribution when interacting with the teacher about academic matters and included the following subcategories:
1. Attend: The child’s attention was directed to the teacher by listening to her and interacting only with bodily movements without responding or initiating.
2. Respond: The child interacted by replying to the teacher’s questions or requests.
3. Initiate: The child made the first move to begin an interaction with the teacher (i. e. asked a question).
4. On-task: Interaction between the teacher and the child which served an educational purpose.
5. Off-task: The child’s behaviour was inappropriate or unrelated to the teacher’s requests during the interaction (e. g. not answering a question).
Pupil X (Target) – Pupil Interaction
The category comprised the target child’s contacts with other pupils but not the teacher.
1. On-task: All interactions with other pupils that involved activities relevant to the classroom tasks.
2. Off-task: The target was involved with other pupils in non-work related activities (i. e. fooling around).
No Interaction
The category included all the child’s individual behaviours when not interacting with the teacher or other pupils.
1. On-task: The child was involved in acceptable classroom behaviours individually (i. e. writing in the book).
2. Off-task: The child was not engaged in the classroom activities. Failure to attend to or work on the assigned task, breaking the classroom rules (i. e. getting out of seat, talking out, disturbing others, rocking on his chair) or staring blankly away from the task (i. e. daydreaming).
Comments
The present category was used when no other activity code was suitable for the observed behaviour.
The design of the coding scheme was based on the basic principles of systematic observation with reference to the current study. The categories were devised to provide information relevant to the specific research question. For instance, it was important to investigate the work and non-work related behaviour in order to conclude if the target child was disruptive or not. Thus, the on-task and off-task categories were created. Each category was exhaustive and explicitly defined, covering all the possible behaviours that may occur in the classroom setting. In addition, each behaviour was coded just by ticking a box making the design of the schedule easy to operate.
Results
The observational data were analysed using frequencies of occurrence for each pre-specified category as presented in Table 1. Due to the fact that the observation was based only on five minutes of interaction, no further statistical analysis was conducted.
The overall trend shows a prevalence of the “ no interaction” category with a percentage of 54. 55% during the five minutes of observation. The most frequent behaviour observed during the no interaction context was the off-task behaviour when the child was not engaged with the classroom activities. The off-task behaviour occurred 15 times (62. 5%) indicating that the child was involved in inappropriate or unrelated to the classroom activities most of the time (i. e. he was playing with his hands, rocking on his chair). An occurrence of 37. 5% was reported for the on-task behaviour when the child was not interacting with the teacher or other pupils.
Less frequent but significant was the child’s interaction with the teacher. The results illustrated that the child most of the time was attending and listening to the teacher (56. 25%). Five times the pupil asked a question and two times he responded to a teacher’s question. At the particular category mode neither on-task nor off-task behaviour were reported. Finally, no significant trends were observed when the pupil was interacting with other pupils in the classroom. The target child was only three times involved in an off-task interaction with another pupil, whereas on-task behaviour occurred only once.
After analysing the observational data an inter-observer agreement test was carried out in order to determine the schedule’s levels of reliability (Robson, 2002). A kappa of . 59 was found indicating a fair agreement between the two observers (Appendix 2).
Table 1
Frequency of occurrence for the observed behaviour
Observed Category
Frequency of occurrence
Percentage
%
Pupil X – Teacher Interaction
Attend
9
56. 25
Respond
2
12. 5
Initiate
5
31. 25
On-Task
0
0
Off-Task
0
0
Total
16
36. 36
Pupil X – Pupil Interaction
On-Task
1
25
Off-Task
3
75
Total
4
9. 09
No Interaction
On-Task
9
37. 5
Off-Task
15
62. 5
Total
24
54. 55
Note. X represents the pupil being observed.
Discussion
Based on the observational data analysis it can be concluded that the seven years old child was more disruptive and less attentive as he had spent the majority of time engaged in off-task activities. However, the results cannot be generalized as the observation was based only on five minutes of interaction and further analysis is needed to be able to investigate the research question.
Despite the well-organized design of the observation schedule, some important limitations should be mentioned. Perhaps the most obvious difficulty in terms of research design was to establish the coding scheme. Due to the observer’s lack of training the research implied some important drawbacks. To begin with, the schedule failed to meet some of the basic assumptions described previously for the use of this technique. To illustrate, the categories used to describe the child’s behaviour were not mutually exclusive and most of the times it was difficult to distinguish between two different variables as they were overlapped. For instance, it was difficult to identify whether the target child was attending to the teacher or involved in an on-task interaction with her. Moreover, a number of behaviours that occurred during the five minutes period were not included in the coding system. Thus, they were recorded as comments, breaking the rules of systematic observation. An example of such behaviour was the teacher ignoring the target child and not responding to his requests. A final concern is related to the issue of reliability that arised from the use of systematic observation. As stated earlier, the Cohen’s Kappa was . 59 indicating a fair inter-observer reliability level. Higher levels of agreement could be achieved if the observers were more trained and had more time to design the observation schedule.
Taking into account the above limitations, some important directions for future research can be concluded. Firstly, the overall result showed that more categories were needed for the purposes of the present study. If more operational definitions were used and exclusive categories were included, more valid conclusions could be made. For instance, further explanation of the work and non-work related behaviour was needed in order to avoid the overlap of the categories. Furthermore, a good point for future research could be to identify the direction of the interaction prior to the data collection. The interactions could be classified as teacher to – target child, target child – to teacher, target child – to pupils, and pupils to – target child. This could provide a better insight into the interactions and the relationships that occur in the classroom.
A general review of the studies based on a systematic observation schedule reveals many aspects that are inherently questionable. There has been considerable debate within the systematic observation tradition, concerning the validity and reliability of the results produced (Hammersley, 1993). A general criticism is related to the pre-determined nature of systematic observation categories. The researchers adapting this point of view (Walker & Adelman as cited in McIntyre & Macleod, 1986) claim that a framework irrelevant to the setting being observed may be imposed when using systematic observation schedules as many systems are not suitable for all kinds of classroom contexts. Similarly, Delamont and Hamilton (1986) propose that systematic observation techniques fail to understand the perspectives in which the classroom interaction occurs and specifically the intentions of the teachers and pupils involved. In addition to these concerns, systematic observation has been proved weak in providing any evidence on the mental activities of the participants due to the fact that direct behaviour is being observed. Therefore, the observer is not allowed to discover why people do things. A counter argument is presented by McIntyre and Macleod (1986) who support that the observer is able to understand the shared meanings within a classroom as he is a member of the same culture, and thus the categorization of the events occurring can be based on these shared cultural values. Apart from that, the main strength of systematic observation is based on the objective description of children’s and teachers’ behaviour in regular classroom situations (Martin, 1977).
It has also been asserted that the prespecified coding systems are unable to go beyond the established categories. The potential of systematic observation to uncover any global concepts that lie behind the small parts of the observed behaviour is limited. Consequently, the natural patterns of classroom interaction are disregarded by the use of arbitrary time sampling (Delamont and Hamilton, 1986). Another methodological concern of systematic observation is related to the interference of the technique. For instance, reactive effects may occur if the teacher or the students change their behaviour due to the presence of the observer. Moreover, systematic observation has come under criticism in terms of its tendency to focus predominantly on one particular system, that of Flanders (1970). According to this perception, the procedures used by Flanders and consequently by all other systematic observation schedules are ideologically committed (Delamont, 1976). However, McIntyre and Macleod (1986) argue for the value of systematic observation by highlighting that FIAC is only one system among many. It has to be remembered that, in the field of social research most of the methods used are ideologically based as the researchers usually focus their attention on particular aspects while neglecting others (McIntyre & Macleod, 1986).
Finally, certain criticisms have been implied in connection with the issue of generalizability (Stenhouse, 1975). As Blatchford (2005) observes, due to the time consuming nature of data collection, the analysis is usually based on total frequencies of behaviours. Consequently, many of the conclusions about what happens in classrooms are based on quantification, thus generalization could be misleading as numbers cannot reflect social reality. Evidence against this assumption is provided by Croll (1986) who underlines the practical advantages of systematic observation in that it allows data to be collected on a substantial scale, producing cumulative and replicable results. As he notes: “ Systematic observation techniques can be used across a large number of classrooms and a long period of time by a large number of observers all engaged on a common purpose” (Croll, 1986, p. 6). Ober et al., (1971) gave weight to this view, by emphasizing the distinct and analytic qualities of the method which allow the observer to provide an accurate description of classroom activities.
Obviously, there are some undeniable limitations to systematic observation. However, the reasons for conducting a study using this method prevail. It is certainly accurate and effective as a methodological tool when direct behavour is the focus of the analysis. Moreover, the combination of systematic observation with methods that provide evidence on the meanings of the observed behaviour can contribute to the field of research (McIntyre & Macleod, 1986). Considering all the above, there is no doubt that many areas of the educational research can be benefited from the use of systematic observation.
