Vol. 30 No. 2 (2022)
Articles

Does the Choice of Observation Instrument Matter?

PDF
  • Jennifer Lewis,
  • Asli Ozgun-Koca,
  • Lenuel C Hernandez,
  • Christopher Nazelli,
  • Kate Rollert French
Kate Rollert French
Wayne State University

Published 2022-03-04

Keywords

  • teaching observation

Copyright (c) 2022 Jennifer Lewis, Asli Ozgun-Koca, Lenuel C Hernandez, Christopher Nazelli, Kate Rollert French

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Abstract

Does the choice of observation instrument make a difference in the feedback and ratings that teachers receive? This study explores how lessons are rated differentially across various observation instruments. To investigate this question, ten randomly selected mathematics lessons were rated using six different observation instruments. Overall scores varied little across instruments. Our analyses indicate that differences in scores can be attributed to what we call instrumental occlusion, instrumental emphasis, and element density. This article concludes with implications for the selection and use of observation instruments in school settings.  

PDF

References

  1. Ball, D. L., Thames, M. H., & Phelps, G. (2008). Content knowledge for teaching: What makes it special? Journal of Teacher Education, 59(5), 389-407. https://doi.org/10.1177/0022487108324554
  2. Berry, III, R. Q., Rimm-Kaufman, S. E., Ottmar, E. M., Walkowiak, T. A., & Merritt, E. (2010). The Mathematics Scan (M-Scan): A measure of mathematics instructional quality. Unpublished measure, University of Virginia. Retrieved from http://www.socialdevelopmentlab.org/wp-content/uploads/2012/05/M-Scan_measure_Final.pdf
  3. Blazar, D., Braslow, D., Charalambous, C. Y., & Hill, H. C. (2017). Attending to general and mathematics-specific dimensions of teaching: Exploring factors across two observation instruments. Educational Assessment, 22(2), 71-94. https://doi.org/10.1080/10627197.2017.1309274
  4. Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593-2632. https://doi.org/10.1257/aer.104.9.2593
  5. Danielson, C. (2013). The Framework for Teaching evaluation instrument. The Danielson Group.
  6. Darling-Hammond, L., Wei, R. C., & Johnson, C. M. (2009). Teacher preparation and teacher learning: A changing policy landscape. In G. Sykes, B. Schneider, & D. N. Plank (Eds.), Handbook of Education Policy Research (pp. 613-636). Routledge. https://doi.org/10.4324/9780203880968.ch48
  7. Desimone, L., & Le Floch, K. C. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1–22. https://doi.org/10.3102/01623737026001001
  8. Eco, U., & Weaver, W. (1994). How to travel with a salmon & other essays. Harcourt Brace.
  9. Erickson, F. (2006). Definition and analysis of data from videotape: some research procedures and their rationales. In J. L. Green, G. Camilli & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 177-191). Routledge. https://doi.org/10.4324/9780203874769-15
  10. Fenstermacher, G. D., & Richardson, V. (2005). On making determinations of quality in teaching. Teachers College Record, 107(1), 186-213.
  11. Goe, L., Bell, C., & Little, O. (2008). Approaches to evaluating teacher effectiveness: A research synthesis. National Comprehensive Center for Teacher Quality.
  12. Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., et al. (2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition & Instruction, 26(4), 430-511. https://doi.org/10.1080/07370000802177235
  13. Hill, H.C., & Grossman, P. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371-384. https://doi.org/10.17763/haer.83.2.d11511403715u376
  14. Hill, H. C., Sleep, L., Lewis, J. M., & Ball, D. L. (2007). Assessing teachers’ mathematical knowledge: What knowledge matters and what evidence counts? In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 111–155). Information Age.
  15. Hull, J. (2013). Trends in teacher evaluation: How states are measuring teacher performance. Center for Public Education. Retrieved from http://www. centerforpubliceducation.org.
  16. Kersten, T. A., & Israel, M. S. (2005). Teacher evaluation: Principals’ insights and suggestions for improvement. Planning and Changing, 36(1/2), 47-67.
  17. Learning Mathematics for Teaching Project (2011). Measuring the mathematical quality of instruction. Journal of Mathematics Teacher Education, 14(7), 25-47.
  18. Lewis, J. M., Reid, D., Bell, C., Jones, N. D., & Qi, Y. (2020). The mantle of agency: Principals’ use of teacher evaluation policy. Leadership and Policy in Schools. https://doi.org/10.1080/15700763.2020.1770802
  19. Lord, B. (1994). Teachers’ professional development: Critical colleagueship and the role of professional communities. In N. Cobb (ed.), The future of education: Perspectives on national standards in America (pp. 175-204). College Entrance Examination Board.
  20. Marzano, R. J., Toth, M. (2013). Teacher evaluation that makes a difference: A new model for teacher growth and student achievement. ProQuest Ebook Central.
  21. Maulana, R., Helms-Lorenz, M., & Van de Grift, W. (2017). Validating a model of effective teaching behaviour of pre-service teachers. Teachers and Teaching, 23(4), 471-493. https://doi.org/10.1080/13540602.2016.1211102
  22. Millman, J. (1981). Handbook of teacher evaluation. Beverly Hills, CA. Sage Publications.
  23. National Council of Teachers of Mathematics. (1991). Professional standards for teaching mathematics. National Council of Teachers of Mathematics.
  24. No Child Left Behind Act of 2001, Pub. L. 107-110, 20 U.S.C. § 6319 (2002).
  25. Popham, W. J. (2013). On serving two masters: Formative and summative teacher evaluation. Principal Leadership, 13(7), 18-22.
  26. Sawada, D. & Piburn, M. (2000). Reformed teaching observation protocol (RTOP). Arizona Collaborative for Excellence in the Preparation of Teachers.
  27. Schoenfeld, A. (2013). Classroom observations in theory and practice. ZDM Mathematics Education, 45(4), 607-621. https://doi.org/10.1007/s11858-012-0483-1
  28. Van der Lans, R. M., van de Grift, W. J. C. M., van Veen, K., & Fokkens-Bruinsma, M. (2016). Once is not enough: Establishing reliability criteria for feedback and evaluative decisions based on classroom observation. Studies in Educational Evaluation, 50, 88-95. https://doi:10.1016/j.stueduc.2016.08.001