The Consistency and Stability of School Effect in Colombian High School Education

Consistency

This property is studied through the correlation among the residuals of the adjusted models (Murillo & Hernández-Castilla, 2011) with information gotten at the same time (Thomas et al., 1997) and different variables of criteria (Kyriakides & Creemers, 2008). This property is related to the dimensionality of school effect (Kyriakides & Creemers, 2008). Studies on this property have been conducted in primary and preparatory levels using cognitive and socio-affective measures.

In elementary education, the most important study is by Mandeville and Anderson (1987) who found for 2,083 students of 423 schools in the USA, correlations between mathematics and language were between 0.63 to 0.70; the correlations among degrees were between 0.08 to 0.16. The results suggest that the school effect is consistent but remains at the same level. Another important study was done by Mortimore, Sammons, Stoll, Lewis, and Ecob (1988) who reported correlations between mathematics and reading at 0.41. The conclusion was that good results in one topic were related to the good results in others. Later, Sammons, Nuttall, and Cuttance (1993) did a re-analysis of this information and reported higher correlations (r=0.61 between mathematics and reading). In general, the results showed the correlations for the subjects is between 0.70 to 0.75 and for different criterial measures of 0.00 and 0.05 (Bosker & Scheerens, 1989).

In secondary education, Willms and Raudenbush (1989) in their longitudinal studies reported for Type A school effect a correlation between 0.46 to 0.73 and for Type B the effects were between 0.19 to 0.71 which according to them proves consistency among subjects. Similarly, Bosker and Scheerens (1989) found that the correlation among subjects was between 0.45 to 0.75 and between 0.35 to 0.70 among different criterial variables. These results suggest that if the school is successful in one subject, then it may be successful in the others (Willms & Raudenbush, 1989). The magnitude of the correlations, lower than primary, could be effect for teacher and better measures in this level.

Thomas et al. (1997) found correlations of 0.25 (English literature and French) and 0.75 (total scores and English) in longitudinal studies done from 1990 until 1992 using multilevel models. The authors concluded that there are substantial differences among subjects for the schools and they posited that these results are due to the teachers. Besides, Cuttance (1987), Thomas and Mortimore (1996); Thomas and Goldstein (1994) and Fitz-Gibbon, Tymms, and Hazlewood (1990) found moderate correlations between English and Arithmetic with values between 0.46 to 0.48. These results contradict the ones obtained by Goldstein and Rasbash (1993) who found a correlation of 0.29 between Mathematics and English. Hence, they concluded that the ranking of schools depends on the adjusted variables and the curriculum measures more than the consistency among subjects.

In relation to the correlations for different product measures, Rutter et al.'s research (1979) reported positive correlations with values between 0.23 to 0.65 among subjects and between academic and social success and r=0.80 for class attendance and delinquency. Similarly, Mortimore et al. (1988) reported correlations for writing and class attendance, and Mathematics and self-concept. Despite these positive results, these researchers claim that cognitive and affective measures are independent. Opdenakker and Van Damme (2000) and Thomas (2001) ensure that the school effect is higher for cognitive compared to affective results. This may be because the latter has the least attention in the curriculum and their measure is imprecise (Reynolds et al., 2011).

In contrast, Iberoamerican research has reported consistency among subjects. Murillo (2007a) found moderate consistency with values nearest to 0.5 for Mathematics and language and correlations between cognitive and affective products measure (r= 0.26 and r=0.40 for school satisfaction at Mathematics and language respectively). For the same subjects, in Mexico, Zorrilla (2008) reported high consistency (r=0.86 in 2002 and r= 0.87 in 2003) and in Argentina, Cervini (2010) presented high correlations (r=0.78 in primary r= 0.80).

According to Murillo (2005a), the research in general reported moderate resistance with correlations close to 0.5. For this reason, it is important to include more subjects and different results variables. In relation to the products measure by Reynolds et al. (2011) acceptable consistency between subjects was found, with higher magnitude in primary than secondary schools. However, results are not conclusive about the relationship between cognitive and affective measures because there is little research on it.

Stability

This property is related to permanence by the time (Murillo, 2005a; Reynolds et al., 2011). This property refers to the characteristics of the school and teachers. To investigate stability, the correlation within residuals of models in two different periods of time (Scheerens & Bosker, 1997; Willms & Raudenbush, 1989) is studied. Due to this, in their analysis, researchers used longitudinal models and repeated measures. With regard to this property, authors such as Kyriakides and Creemers (2008) argued that it is necessary to study the theoretical approaches to find out about the source of the stability, for example, if the change is natural or programmed; whether it depends on school characteristics, teachers, or internal or state politics. In this sense, the authors ensure that the school effectiveness should be studied as a dynamic system because there is interaction between different factors such as students’ background, school characteristics and the educational system. In consequence, the schools can identify their weaknesses and take the necessary action to improve and increase their school effect (Kyriakides and Creemers, 2010).

A pair of studies in elementary education was conducted by Mandeville and Anderson (1987) and by Mandeville (1988). The first study showed correlations between 0.52 to 0.33 for different cohorts. This instability may be explained by the early age of students and their development. As for consistency, Bosker and Scheerens' (1989) study found correlations between 0.35 to 0.65 for different cohorts (years) and correlations between 0.10 to 0.65.

In secondary education, Bosker et al. (1988, as cited in Bosker & Scheerens, 1989) reported correlations between 0.40 to 0.80 for secondary levels. However, they claimed that it was over estimated because there is dependence inside the information as the measure was similar for students in the different levels and the criterial variable is accumulative along the time. Thomas et al. (1997) reported index of correlations to subjects between 0.38 to 0.92. The subjects with lowest stability were French and the highest was history. In general, they found higher stability to the total scores.

In USA, Willms and Raudenbush (1989) reported for the A school effects correlations of 0.87 to the total scores for 1980 and 1984; r=0.13 for Mathematics and r= 0.73 for English. In addition, for B school effect the correlation index was equal to 0.70 in the total score; r= -0.013 for Mathematics and r= 0.79 for English. Considering that these results reflected the school interaction over time, the authors concluded that the instability of the school effect was due to the subjects because of the change of teachers and the implementation of major or short-term education reforms. Bosker and Scheerens (1989) showed the results for cohorts between 0.70 to 0.95 and for levels between 0.25 to 0.90. They argued that the study stability within levels and signatures is an important topic (Luyten, 2004) because it can show the specificity and effectiveness of teaching and when these are high it may be due to the accumulation of the effects every year (Goldstein, 1997). If these are low, it could be due to the changes that the schools undergo in order to adjust to external circumstances.

In England, Mangan, Pugh, and Gray (2005) studied 541 secondary schools for a period of ten years (1992-2001); the variable was the score in GCSE. They used time series and did trajectories on the time frame. The results showed little increase; between 0.06 to 0.0124, where the rate of improvement was between 1.2 to 5.9. According to the geographic distribution, the schools displayed different patterns of improvement which were unstable. 63% of the schools posted a decrease in three or more years in the number of highest scoring students. The previous performance had a positive relationship with the current performance, but this effect reduces quickly with dynamic models for one previous year showed a value of 0.52 and for two previous years the value was 0.21. In relation to the prediction, the gender of the students and the management of the school were included as variables which resulted in 95% of the schools posting important differences for three years. The authors concluded that stability only occurs for a short time, maximum for three years and because of that, it was difficult to make predictions.

In Iberomerica, there is not much information about stability. Hence, it is necessary to do more research with different levels and background variables and criterial variables (Murillo, 2005a). In addition, the effectiveness and change of teachers should also be studied (Kyriakides & Tsangaridou, 2008).

Research questions

The questions that guide the study are as follows:

a) Does school effect change within subjects for the Colombian educational system?

b) Is school effect perdurable over time in the Colombian educational system?

Bearing in mind this purpose, the research investigated the estimation of consistency in high school for mathematics, language, natural and social science over seven years in the 2000 decade as well as the estimation of the stability of school effect for high school during the same time.

Participants

The sample is found in Table 1. The information shows that the percentage of girls is higher than boys for all years except 2000. To test for stability, the same schools were studied for the seven years; 6,844 schools and 49,210 students.

Variables, test and questionnaire

The research included two variables: results and adjustment similar to other similar studies (Cervini, 2010; Murillo, 2005b, 2007a, 2008; OREALC/UNESCO, 2010; Zorrilla, 2008).

The product variable was scored using a standarized test for mathematics, language, natural science (physics, chemistry and biology) and social sciences (geography, history and philosophy). These results are scaled with average 50 and standard deviation 10; minimum 0 and maximum 100 (ICFES, 2011c).

The adjusted variables are for the students and schools. The variables for students were:

• Gender (GENERO): 0= boys; 1= girls.
• Mother tongue (LENGUA): 1= native language, 0= Spanish.
• Sociocultural and socioeconomic indexes (ZNSC1, ZNE).

The index was synthetic in accordance with the methodology of Becerra-Avella (2010). These are standarized variables with average 0 and standard deviation 1; these included parents’ occupation and education and monthly income.

The variables for the school were:

• Socio-economic level (NSE_ESCUELA), standarized with average 0 and standard deviation
• Tuition (Pension): ordinal variable with seven categories.

The information about the variables was obtained through a standardized test and a 90-item questionnaire on students, parents, and schools’ background. Until 2007, the standardized test contained 35 items covering each subject and this number was later reduced to 24 items (ICFES, 2005). The psychometric quality, done with Rasch model showed a good andadjustment. The test had an equating process until 2000 (ICFES, mail in 2013). When the students registered for the test, they were given the questionnaire to fill up.

Data analysis

The analysis was done with MLWin with different phases. The first was the preparation of information which included depuration and preparing for the program. According to this process, 10% and 20% was eliminated; the main criteria was missing values and schools with less than ten students. The second was the estimation of every model without adjusted variables. In addition, the model was adjusted for maximum probability. Finally, the deviance was obtained to know the adjustment of models.

The consistency was calculated for mathematics, language, social and natural sciences with multilevel models of three levels: student, school and state with MLWin 2.3. Once the models were adjusted, the standardized residuals and their correlations were obtained with SPSS 18. The models adjusted in MLWin are as follows:

Level 1: $y_{ij}={\beta }_{0ijk}cons+{\beta }_{1}GENER{O}_{ijk}+{\beta }_{2}LENGU{A}_{ijk}+{\beta }_{3}ZNSC{1}_{ijk}+{\beta }_{4}ZN{E}_{ijk}+{\beta }_{5}NSEESCUEL{A}_{ji}+{\epsilon }_{ij}$

Level 2: ${\beta }_{0jk}={\beta }_{00k}+{\mu }_{ojk};{\beta }_{1jk}={\beta }_{10k}+{\mu }_{1jk};{\beta }_{2jk}={\beta }_{20k}+{\mu }_{2jk};{\beta }_{3jk}={\beta }_{30k}+{\mu }_{3jk};{\beta }_{4jk}={\beta }_{40k}+{\mu }_{4jk};{\beta }_{5jk}={\beta }_{50k}+{\mu }_{5jk}$

Level 3: ${\beta }_{00k}={\gamma }_{000}+{\nu }_{00k};{\beta }_{10k}={\gamma }_{100}+{\nu }_{10k};{\beta }_{20k}={\gamma }_{200}+{\nu }_{20k};{\beta }_{30k}={\gamma }_{300}+{\nu }_{30k};{\beta }_{40k}={\gamma }_{400}+{\nu }_{40k};{\beta }_{50k}={\gamma }_{500}+{\nu }_{50k}$

Where:

$y_{ijk}$ is the result variable and i = 1 ,..., ni is every student, j = 1,..., nj every of the school,

k= 1...., nk every of the state; $e_{ijk},{\mu }_{ijk},V_{ok}$ are the errors of every level.

${\beta }_{njk}$ = it is the variables included in the models.

${\beta }_{0jk}$ = it is the average score for the student 0 of the school j and state k, it is compound by ${\beta }_{00k}$ it is the average of the intercept of the school, and ${\mu }_{0jk}$ the random error.

${\beta }_{1jk}$ = it is the average of the slope, it shows the variability between school j and state k, it is compound by ${\beta }_{10k}$ is equal the average of the slope by the schools, and ${\mu }_{1jk}$ the random error.

${\beta }_{00k}$ = it is the average score by the intercept for the three level, it is compound by ${\gamma }_{000}$ is the general average and ${\nu }_{00k}$ random error.

${\beta }_{10k}$ = it is the average slope of the three levels and it is compound by ${\gamma }_{100}$ the average of intercept and ${\nu }_{10k}$ the random error.

For the model introduces the variables in the fixed part after that, there is variation in the average and slopes.

The stability was calculated for cohorts with longitudinal multilevel model, we used MLwiN 2.3. The model is the following:

$y_{ij}={\beta }_{0ij}cons+{\beta }_{1ij}pensio{n}_{1j}$

${\beta }_{0ij}={\beta }_0+{\mu }_{0j}+e_{0ij}$

${\beta }_{1ij}={\beta }_1+{\mu }_{1j}+e_{1ij}$

$\left[\begin{array}{c}{\mu }_{0j}\\ {\mu }_{1j}\end{array}\right]$ N(0, ):= $\left[\begin{array}{c}{\sigma }_{u0}^2\\ {\sigma }_{u01}^2{\sigma }_{u1}^2\end{array}\right]$

$\left[\begin{array}{c}{e}_{0ij}\\ e_{1ij}\end{array}\right]$ N(0,):= $\left[\begin{array}{c}{\sigma }_{e0}^2\\ {\sigma }_{e01}^2{\sigma }_{e1}^2\end{array}\right]$

Where:

$Y_{ij}$ = it is the score of the test by the time i of the school j.

${\beta }_{0j}$= it is the average score according to the time for the school

${\beta }_{1j}$ = it is the slope of the variable tuition.

${\sigma }_{e1}^2$ = it is the variability of first models, time.

${\sigma }_{u1}^2$ = it is the variability of second level by school.

Consistency

The results show that the school effectiveness is equal for all the subjects and have implications for the organization. In high school, the consistency is moderate similar to Cervini’s (2010) research in Argentina and Zorrilla's (2008) in Mexico. However, this study's findings contradict the results obtained by Cuttance (1987), Thomas and Mortimore (1996) and Thomas and Golsdtein (1994) who reported moderate correlations among subjects and with the results of Golsdtein and Rasbash (1993) who showed low correlations close to 0.29. These results allow for an assumption that consistency is similar to the other Latin American countries. Hence, it is possible to affirm that these results are due to the similar characteristics of the school. It is important to mention that only mathematics posted a significant change in school effectiveness.

The consistency in this research is high with values more than 0.7, except for mathematics which is moderate and low for 2000 in particular, with language. These results are similar to those obtained by Bosker and Scheerens (1989), Cervini (2010), Zorrilla (2008) and Reynolds et al. (2011).

In summary, the Colombian school effect, either positive or negative, is similar across the board for every subject as consistency was established among subjects except for matemathics because the schools that have good results in this subject are not good at the other subjects (Murillo, 2007a).

Stability

The results for stability showed that there is change due to time for every subject. Consequently, it is possible to assume that the Colombian school is not stable, as the school effect is not stable over time. In addition, these results depend on the socio-economic status of the school. These findings are similar to Mangan, Pugh, and Gary (2005) who affirmed that there are school changes which lead to instability over time. In the current research, it was found that along the ten-year period, ascendant and descendant tendencies were noticeable. This behavior is evidence of instability.

With regard to the results for stability, it is difficult to compare the results of this study with those of emblematic studies such as Thomas et al. (1997) who showed that stability depended on the subject or the study of Thomas, Peng, and Gray (2007) who found that the 55% of schools are effective over time, with many unexpected changes. For this reason, only in a few schools, is it possible to find improvement.

The results for stability and consistency\show that although there is similarity in the teaching-learning process among subjects, this changes from one year to the next. The variability could be the answer to the arrangement of the external circumstances (Goldstein, 1997). Despite changes in institutional or governmental policies, they influence all the subjects in the same way. These results show the need to consider the dynamic models (Kyriakides & Creemers, 2010). Like other studies in Latin America, the results of this research show:

… the important repercussion in the index of effectiveness of the schools... if the school effect is not consistent, we would analyze the effectiveness in a different measure of the results; if they are not stable, it will be necessary to have the index measure for some years consecutive… (Murillo, 2007a, p. 83)

Quality and equity education and social equity

The research about school effect informs us about the effectiveness of school. This shows the results of educational processes that are ahead in the schools and reflect the politics of state or country. These results are the principal tools for informing about the quality of education.

The quality of education comprises five dimensions: relevance, pertinence, equity, effectiveness and efficiency (Blanco, 2008). Relevance and pertinence refer to the aim and sense of education. The other dimensions have to do with guaranteeing access and permanence inside the educational system like equity and the achievements of educational goals. They also are related to the way money is invested for education. The studies about consistency and stability are useful for providing information about these dimensions.

The results of this research show that in Colombia, schools are not very successful in producing good results in different subjects and guaranteeing their stability. This will impact on the quality of education. If Colombia wants to become developed in the way that the UNESCO (2012) suggests, the country should work towards decreasing the differences in the quality of the education because these have a considerable effect on opportunities for employment and on national development. In conclusion, the country must be seriously concerned about putting in place fairer, participatory and equitable societies that allow for a harmonious and prosperous coexistence among citizens.