Teaching, learning and assessment and the grade for overall effectiveness - a case of mistaken identity

This post highlights the flaws of using the aspect grade for teaching, learning and assessment as a limiting factor for the grade awarded for overall effectiveness, and with it the implications for the reliability and validity of inspection outcomes for general further education colleges.
      The OfSTED Handbook for the Inspection of Further Education and Skills http://www.ofsted.gov.uk/resources/handbook-for-inspection-of-further-education-and-skills-september-2012  makes it clear that the aspect grade for  quality of teaching, learning and assessment is a limiting  factor for  the overall grade for effectiveness, for example, to be awarded an outstanding grade for overall effectiveness, the grade for the quality of teaching, learning and assessment must also be outstanding.

     Using a similar approach as undertaken by Waldegrave and Simons (2014) to analyse the relationship between grades awarded for school inspections, the following table summarises the relationship between the different inspection grades awarded during 125 general further education college inspections and which took place between January 2013 and June 2014.

Aspect grade	Overall grade for effectiveness
	1	2	3	4*
Agreement with the grade for outcomes for learners	100%	74%	93%	50%
Agreement with the grade for the quality of teaching, learning and assessment	100%	100%	100%	75%
Agreement with grade for the effectiveness of  leadership and management	100%	86%	90%	75%

*only 4 colleges were awarded a grade 4, which has an impact on the associated correlations.

     It can be clearly seen from the data that the teaching, learning and assessment aspect grade corresponds most strongly with the overall grade for effectiveness, which is not surprising given the guidance in the handbook.   Out of 125 GFE college inspections undertaken in the specified 18 month period, there was only 1 occasion when there was a difference between the two grades, and on this occasion the overall grade for effectiveness was lower than the grade for teaching, learning and assessment.  . 
     However, the direct relationship between the grade for overall effectiveness and the quality of teaching, learning and assessment is not without its’ problems.  In the further education sector, unlike schools, individual lesson grades are still being used by OfSTED inspectors to summarise judgments about the quality of teaching, learning and assessment within a lesson.    Both Matt O’ Leary and Rob Coe identify the serious challenges with the use of observations in the grading of teaching and learning.   Waldegrave and Simons (2014) cite Coe’s synthesis of a number of research studies, which raises serious questions about the validity and reliability of lesson observation grades.  When considering value added progress made by students and a lesson observation grade (validity) , Coe states that in the best case there will be only 49% agreement between the two grades and in the worst case there will be only 37% agreement.  As for the reliability of grades Coe’s synthesis suggests that in the best case there will be 61% agreement and in the worst care only 45% agreement between two observers.
     As such, it would seem that using the teaching, learning and assessment grade as the driver for the grade for overall effectiveness is not consistent with the current best available evidence, and indicates that systems of accountability within the further educations sector have yet to be fully informed by the evidence-based practice movement.  Accordingly, in order to bring about more effective and more meaningful accountability processes the following are worthy of consideration.

• the direct relationship between the teaching, learning and assessment aspect grade and the overall grade for effectiveness should be replaced by a holistic judgment.
• the design of 'inspection regimes' to be subject to open and transparent college effectiveness/college improvement 'state of the art' reviews to ensure processes are likely to generate valid and reliable judgments.
• as part of the process of self-assessment colleges reduce, if not eradicate,  their over-reliance on lesson observation grade profiles in making judgments about teaching, learning and assessment.

References

Coe, R, Lesson Observation: It’s harder than you think, TeachFirst TDT Meeting, 13th January 2014

O’Leary M (Forthcoming in 2015) ‘Measurement as an obstacle to improvement: moving beyond the limitations of graded lesson observations’, chapter in Gregson, M. & Hillier, Y. (eds) Reflective Teaching in Further, Adult and Vocational Education, London: Bloomsbury.

O’Leary, M. (2014). Lesson observation in England’s Further Education colleges: why isn’t it working and what needs to change? Paper presented at the Research in Post-Compulsory Education Inaugural International Conference: 11th –13th July 2014, Harris Manchester College, Oxford.

Waldegrave, H., and Simons, J. (2014). Watching the Watchmen: The future of school inspections in England, Policy Exchange, London