Exploring Data Use Among Teachers in the Republic of Ireland
Abstract
Systematic data use among educators in Ireland has become a significant feature of recent policy documents from the Department of Education and Skills (DES, 2011a; DES, 2013). A comprehensive examination of teachers' data use, their attitudes towards data and their confidence in their skills to use data has yet to be conducted in the Republic of Ireland. This study sought to examine current data use practices among Irish primary teachers, to explore teachers' confidence in relation to their data literacy skills and to examine the extent to which they valued data. Primary school teachers, (n=217) across 18 schools, completed three scales to measure data use, data confidence and data value; Teacher Data Use Scale (TDUS), the Teacher Confidence with Data Scale (TCDS) and the Teacher Data Value Scale (TDVS) respectively. Results indicated a significant predictive relationship between data confidence, data value and data use. High levels of data confidence were found to lead to greater data use. Disparity emerged between confidence levels across data literacy skills with teachers reporting most confidence in relation to key data related terminology and least confidence in relation to basic statistics. Findings are discussed in the context of current and future educational realities in the Republic of Ireland.
Keywords: Data-ınformed decision making, school ımprovement, data literacy
Introduction
The Irish social and economic landscape has been marked by considerable change in recent years. Our economic plight has refocused political attention on education; with educational change being marked out as a potential route to recovery. Irish educators have been tasked with improving their schools and the Literacy and Numeracy for Learning and Life document, (DES, 2011a), along with the School Self- Evaluation Guidelines for Primary Schools (DES, 2013) have been presented as the blueprints for change. Both of these documents specify that whole school data analysis should play a functional role in future school development practices. They propose the data enquiry cycle as a mechanism for change. However, it is unclear how Irish schools will set about this change, it is unclear whether they are ready for this change, and it is unclear if they want to change. It is essential that these three ambiguities are explored from both a theoretical and concrete research base.
Data-Informed Practice in Education
The notion of using data in schools is by no means a novel initiative, with data-informed practices in education being commonplace for more than a decade in the U.S.A., Australia and in England. However, it would appear that the effectuality of these approaches has not always been evident. Romero and Ventura's (2010) review of the literature on the art of data-mining, suggests that while there is a consistent trend towards increased use of data-based practices in schools internationally, there are inconsistencies with regard to how the data is used and with regard to the success of these initiatives. Two of the most regularly cited obstacles to making progress with data are teacher attitudes towards data and teachers' data-literacy, including their ability to use data to inform planning.
Johnson (2004) contends that while "few of us are statisticians at heart" there is an essential requirement to enable teachers "to make meaning out of raw data" (p. 6). Data literacy is described as "the ability to examine multiple measures and multiple levels of data, to consider the research and to draw sound inferences" (Love, 2004, p. 22). A series of decoding skills must be acquired prior to becoming data literate. Earl and Katz, (2006) suggest that educators need to be able to differentiate between sound and unsound data and to have a strong sense of basic statistical concepts and measurements. Researchers also indicate that teachers need to understand the limitations of certain data and what interpretations are considered valid (Firestone & Gonzalez, 2007). However, the evidence suggests that “educators are woefully under-prepared to engage in data-based decision making (Earl & Katz, 2006, p. 4). Supovitz and Klein (2003) conducted explorative research in the area of data use in schools in the U.S.A. and found that only 19% of school leaders felt that they had the technical skills to manipulate the data in order to use it to answer the questions that they wanted to ask (p. 38). Given the substantial head-start that American school leaders have had over Irish principals with regards to coming to terms with data, one could reliably presume that the level of data expertise among our school leaders and within our schools is even lower.
A general "mistrust" of data has been noted among educators (Earl and Katz, 2006). This mistrust was evidenced in Ingram, Louis, and Schroeder's (2004) findings that teachers tend to disregard data in favour of their own “personal metric” for evaluating their instructional effectiveness. It was highlighted that teachers “base their decisions on experience, intuition and anecdotal information (professional judgment)” instead of systematically collected information (p. 128). Young and Kim (2007) detected the same doubting disposition towards test data, also suggesting that teachers view assessment results as separate from rather than integral to their teaching. Heritage and Chen (2005) propose that educators are more likely to believe in the value of the data if they have the skills to use them, further highlighting the critical importance of developing teachers' ability to understand, manipulate and use data. Saunders (2000), however, who warns against a blind trust in data which could be dangerous and have detrimental consequences, highlights the importance of educators' ability to put data in perspective.
A comprehensive examination of teacher's data use, their attitudes towards data and their confidence in their skills to use data has yet to be conducted. How can Irish primary school teachers be encouraged and supported to use data and how well prepared are teachers to use data effectively and safely? As Ireland attempts to embark on a data-informed approach to school improvement it is essential that these questions are answered in order to inform appropriate support mechanisms and professional development for teachers. This study seeks to examine Irish primary teachers' current data use practices and to explore teachers' confidence in relation to their data literacy skills, along with the extent to which they value data.
Methodology
Sample
A sample of 18 primary schools stratified by socioeconomic status (DEIS (DEIS: Delivering equality of opportunity in schools. DEIS schools are located in socio-economically disadvantaged areas and receive additional teaching and financial resources in order to reduce the educational disparity which accompanies socio-economic discrepancies) and non-DEIS) and location (urban and rural) took part in this study. All teachers within these schools were invited to participate. A total of 300 questionnaires were distributed by the researcher to all consenting teachers within these schools. Identities were anonymous and confidentiality of responses was assured. A number of questionnaires were not returned while others were completed incorrectly. A total of 217 questionnaires were deemed appropriate for analysis. Of these, 176 teachers indicated the type of teaching that they were involved in; mainstream teaching at senior levels (n= 98), mainstream teaching at junior levels (n= 39) and learning support teaching (n= 39).
Materials
Three scales were designed by the researcher in consultation with the literature to measure teachers' data use, data confidence and data value. Existing measures were deemed inappropriate for the educational context in the Republic of Ireland. All three instruments were pilot tested (n=18) prior to use in this study.
Data Use
The Teacher Data Use Scale (TDUS) consisted of 17 statements pertaining to teachers' use of data in schools (e.g. Item 4: I use data to inform my teaching and planning). Teachers were asked to indicate to what extent each statement was reflective of their own practice on a 4-point scale ranging from 1 (to a great extent) to 4 (not at all). Two subscales were derived from the TDUS, namely Individual Data Use (10 items) and Collaborative Data Use (7 items). Reliability analysis indicates strong internal consistency for the TDUS (Cronbach's Alpha = 0.88) and both subscales (IDU = 0.83; CDU =0.86).
Data Confidence
The Teacher Confidence with Data Scale (TCDS), consisted of 18 statements which described various data literacy skills (e.g. Item 10: I know what sort of questions to ask about data; Item 12: I am able to consider measurement error when comparing results across pupils/time). Teachers were asked to indicate their level of confidence in their competence in relation to each skill on a 4-point scale ranging from 1 (to a great extent) to 4 (not at all). Reliability analysis indicates strong internal consistency for the TCDS (Cronbach's Alpha = 0.94) Four subscales were derived from the TCDS, namely Confidence with Terminology (percentile, standard score, sten, raw score), Confidence with Basic Statistic (normal distribution, reliability & validity, measurement error etc), Confidence with Analysis (analysis of school-based data, class-based data etc.) and Confidence with Reporting (discussing with colleagues, parents, written report, reporting to external educational partners) with Cronbach's Alpha ratings of; 0.92, 0.87, 0.84 and 0.80 respectively.
Data Value
The Teacher Data Value Scale (TDVS) included 8 statements, relating to the value of data for teachers (e.g. Item 2: Standardised Test Data offers very limited information). Teachers were asked to indicate the extent to which they agreed or disagreed with each statement on a 5-point scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). Cronbach's Alpha for the TDVS was 0.61. While it is generally recommended that Cronbach's Alpha should be greater than 0.7, it is common to find quite low values for scales with less than ten items. In such cases an examination of the mean inter-item correlation for the scale is appropriate. Briggs and Cheek (1986) recommend an optimal inter-item correlation of 0.2 to 0.4. The inter-item correlation for the TDVS was 0.2.
Procedure
Copies of the three scales were sent in hard-copy to each of the 18 schools for distribution to all consenting teachers. Teachers were asked to complete scales in their own time. The scales took approximately 20 minutes to complete. Principals were requested to gather all scales and return them (via post) to the researcher within three weeks. Completed scales were received from all 18 schools within the allocated time period.
Results
Preliminary Analysis
Preliminary analyses were conducted to ensure no violation of assumptions. All scales were normally distributed. Descriptive statistics for each scale were analysed (see Table 1 below).
Comparison between Individual Data Use and Collaborative Data Use
A paired-samples t-test was used to examine whether there were any significant differences among teachers in terms of how they used data (i.e. individually or collaboratively). Analysis revealed that teachers were significantly more likely to use data individually (M=1.93, SD=0.68) that to use data collaboratively (M=2.50, SD=0.52, t(173)=12.09, p=0.0005). The eta squared statistic (0.10) indicated large effect size.
ANOVA was used to examine differences between teacher types (mainstream senior, mainstream junior and support) on each of the three variables; data use, data confidence and data value. Findings indicated that there was no significant difference between groups in terms of data use F (2,149) = .63, p= >0.05 and data value F (2, 167) = 1.10, p= >0.05. Analysis indicated significance at the P = <0.05 alpha level between teacher types in relation to data confidence F (2, 155) =1.30, p= 0.01. Post-hoc comparisons using the Tukey HSD test indicated that the mean data confidence score for support teachers (M=66.25, SD=10.12) was significantly different from junior level mainstream teachers (M=59.18, SD=10.47, p=0.01). Results indicate that support teachers have significantly greater data confidence than junior level mainstream teachers.
Relationship between Data Use, Data Confidence and Data Value
Pearson product-moment correlation coefficient was applied to measure the relationship between the variables under study. There was a medium positive correlation between data use and data confidence with high levels of data confidence associated with high levels or data use. Analysis indicated a medium positive correlation between data use and data value and a medium positive correlation between data confidence and data value.
Variance in Data Use across Data Confidence Groups
SPSS was used to analyse valid responses for the Teacher Confidence with Data Scale (TCDS, n=192) in order to identify suitable cut-off points to subdivide the variable data confidence into three relatively equal groups. Scores of the TCDS ranged from 16-61 on a 72 point scale. Groups were identified as follows; High Confidence >=71, (n=68), Medium Confidence 60-70, (n=64), Low Confidence <=59, (n=6859). A one-way between groups ANOVA was conducted to examine the impact of data confidence on data use, as measured using the Teacher Data Use Scale (TDUS). Results revealed a statistically significant difference at the p<0.01 level in data use scores across the three groups F(2, 115)=18.54, p=0.0005 with a large effect size (eta squared =0.15). Post-hoc comparisons using the Tukey HSD test indicated that the High Confidence group (M=53.04, SD =7.53) used data significantly more that the Medium Confidence group (M=49.31, SD=7.81, p=0.05) and the Low Confidence Group (M=43.65., SD=8.31, p=0.0005). The Medium Confidence group used data significantly more that the Low Confidence group p=0.001 (see Table 2).
Variance in Data Value across Data Confidence Groups
An analysis of variance was conducted to identify whether differences existed between groups (high, medium and low confidence) in terms of data value. Results yielded statistically significant differences in data use scores for the three groups F(2,181)=14.22, p=0.0005, with a strong effect size (eta squared at 0.13). Post-hoc analysis using Tukey HSD revealed that those with high data confidence (M=28.85, SD=3.40) valued data significantly more than those with medium confidence (M=27.42, SD=2.78, p=0.03) and those with low data confidence (M=25.83, SD=3.15, p=0.0005). Those with medium data confidence valued data significantly more than those with low levels of data confidence p=0.01 (see Table 3).
Influence of Data Confidence and Data Value on Data Use
A standard multiple regression was conducted to examine whether data confidence and data value were predictive of data use and to identify which variable was most predictive. Preliminary analyses were also conducted to ensure no violation of the assumptions of normality, linearity, multicollinearity and homoscedasticity. The results supported the appropriateness of multiple regression analysis. The results demonstrated that 43% of the variance in data use was explained by the model as a whole (i.e. data confidence and data value). The ANOVA results further indicated that the model reached a statistical significance, F (2, 155) =18.25, p < 0.0005. The standardised beta values were then inspected in order to determine which of the independent variables, included in the model, contributed significantly to the prediction of data use. The strength of the beta value for each independent variable in the model indicates whether the variable makes a significant unique contribution to explaining the dependent variable when the variance explained by all other variables in the model is controlled for (Pallant, 2007). The beta values for data confidence (.320, p < 0.0005) and data value (.187, p = 0.02) were significant; that is, data confidence and data value made statistically significant contributions to the prediction of the participants’ data use. Further examination of standardised coefficients revealed that, of these two variables, data confidence makes the largest unique contribution (8% approx) although data value also makes a statistically significant contribution.
Analysis of Teacher Data Confidence across Data Literacy Skills: Confidence with Terminology , Confidence with Reporting, Confidence with Analysis and Confidence with Basic Statistics
A Repeated Measures ANOVA was used to determine whether there were any significant differences between confidence levels across the four data literacy skills; Confidence with Terminology, Confidence with Basic Statistics, Confidence with Analysis and Confidence with Reporting. The Bonferroni adjustment was applied. Post-hoc analyses were conducted as appropriate. Results indicated significant differences in confidence across skills, Wilks’ Lambda = .42, F(3,189)=86.86, p=0.0005. Figure 1 illustrates the confidence levels for each of the four data literacy skills. Confidence with terminology (M=4.69, SD=0.66) was significantly higher than confidence with reporting (M=3.80, SD=1.64, p=0.0005), confidence with analysis (M=3.99, SD=0.73, p=0.0005) and confidence with basic statistics (M=3.70, SD=0.78, p=0.0005). Confidence with basic statistics was significantly lower than all other skills at the p=<0.01 alpha level.
Conclusion
This study has yielded a number of significant findings which can help inform policy and planning in relation to data use in Irish primary schools. A significant relationship emerged between data use, data confidence and data value. In depth analyses revealed that teachers with high levels of data confidence were significantly more likely to use data than teachers with medium or low levels of data confidence. Furthermore, in line with Heritage and Chen's (2005) position that teachers are more likely to value data if they have the skills to use data, results indicated that teachers with high levels of data confidence valued data significantly more than those with low levels of data confidence. Analyses indicated a predictive relationship between data use, data confidence and data value, with data confidence and data value accounting for 40% of the variance in data use. Data confidence emerged as the most significant contributor. These findings suggest that if the intention of the Department of Education and Skills is for teachers to use more data, then it is essential that professional development is provided to build teachers' confidence and competence with regard to data literacy skills.
In relation to teachers' use of data currently, it emerged that teachers are significantly more likely to use data individually as opposed to collaboratively. As data use is most effective when examined and used collaboratively (Earl & Katz, 2006) it would be important for future research to examine this pattern of practice further in order to establish how best to support and encourage systematic collaborative data use. Findings suggested that levels of data use were not influenced by teachers' positions in school i.e. junior mainstream, senior mainstream and support teaching, however it did emerge that support teachers were significantly more confident than junior mainstream teachers when it came to data literacy. As standardised assessments are generally not administered in the junior levels, it is likely that teachers at these levels are less familiar with many of the concepts pertaining to data literacy. However, in order to facilitate whole-school, collaborative, data-informed decision making and avoid potential power imbalances, it is essential that all teachers have the confidence and skills to analyse and use data. In accordance, it would be important that any training initiatives in data use are targeted at the whole- school level.
A disconcerting disparity emerged in relation to teacher confidence across data literacy skills. In line with international research (Earl & Katz, 2006; Supovitz and Klein, 2003), results suggest that Irish primary teachers lack confidence with regards to their competence in basic statics. Their reported confidence levels for basic statistics were significantly lower than reported levels for all other skills. Somewhat illogically, given the inextricable relationship between standardised scores and basic statistical knowledge such as normal distribution, teachers reported extremely high levels of confidence in relation to their ability to explain key terminology (i.e. percentile etc). In cognisance of Saunders' (2000) caveat and the age old adage that a little knowledge is a dangerous thing, it is imperative that if teachers are being encouraged to use data to inform their decisions that they have the skills to differentiate between sound and unsound data and to have a strong sense of basic statistical concepts and measurements.
Acknowledgements
The author(s) declare that there is no conflict of interest.
References
Chen, E., Heritage, M., & Lee, J. (2005). Identifying and monitoring students' learning needs with technology. Journal of Education for Students Placed at Risk, 2, 309-332. DOI:
Department of Education and Skills (2011). Literacy and numeracy for learning and life: The national strategy to improve literacy and numeracy among children and young people 2011-2020. Dublin: DES
Department of Education and Skills (2013). School Self-Evaluation Guidelines for Primary Schools. Dublin: DES
Earl, L., & Katz, S. (2006). Leading in a data rich world: Harnessing data for school improvement. Thousand Oaks, CA: Corwin
Firestone, W. A., & Gonzalez, R. A. (2007). Culture and processes affecting data use in school districts. In P. A. Moss (Ed.), Evidence and decision making: The 106th yearbook of the National Society for the Study of Education, Part I (pp. 132-154). Malden, MA: Blackwell. DOI:
Ingram, D., Louis, K. S., & Schroeder, R. G. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106, 1258-1287. DOI:
Johnson, R. S. (2002). Using data to close the achievement gap: How to measure equity in our schools. Thousand Oaks, CA.: Corwin Press.
Love, N. (2004). Taking data to new depths. Journal of Staff Development, 4, 22-26. OECD (2009). Pisa 2009. Paris: OECD
Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems Man and Cybernetics Part C. Applications and Reviews, 40(6), 601-618. DOI:
Saunders, L. (2000). Understanding schools' use of 'value-added' data: The psychology and sociology of numbers. Research Papers in Education, 2, 241-258. DOI:
Supovitz, J., & Klein, V. (2003). Mapping a Course for Improved Student Learning: How Innovative Schools Systematically Use Student Performance Data to Guide Improvement. Philadelphia: Consortium for Policy Research in Education. DOI:
Young, V., & Kim, D. (2007). Using assessments for instructional improvement: A literature review. Education Policy Analysis Archives, 19, 1-40. DOI:
Copyright information
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.