Why the delusional voodoo of grading lesson observations is not going stop any time soon.

Despite the impassioned words of headteachers, such as Tom Sherrington and John Tomsett on the reliability and validity of graded lesson observations, they are still being used in around 50 per cent of schools.   Drawing upon the pioneering work of Pawson and Tilley (1997) I will use two core ideas/concepts from realist evaluation i.e ‘programme theory,’ and ‘mechanisms’  to help explain, why despite evidence that lesson observations should only be used for low-stakes development purposes, some head-teachers and senior leadership teams still wish to continue with the practice of grading lessons.

What do we mean by a ‘programme theory’?

Any intervention - in this case graded lesson observations - introduced or used by head-teachers and senior leaders - should be informed by some form of programme theory i.e. a statement along the lines of .   if we do X then this will deliver change Y.   In other words, a programme theory is simply an ‘if… then....." statement.  So in the case of head-teachers and senior leadership teams who wish to use to continue to graded lesson observations within their school, they may be being guided by a programme theory - which goes along the following lines : ‘If we use graded lesson observations this will then deliver an increase in the quality of teaching and learning within the school.’

Given the diverse nature of headteachers and senior leadership teams it is unlikely that there is only one programme theory being ‘used' in schools who are continuing to use graded lesson observation.  Other programme theories which could be being used include:

‘If we use graded lesson observations this will lead to greater control over teachers and how well they perform.’

‘If we use graded lesson observations this will then lead to fewer poorly performing teachers within the school.’

"If we use graded lesson observations then we will be able to measure the quality of teaching and learning within the school.'

Of course, this is not a comprehensive list of programme theories and I'm sure you will be able to think of programme theories which include either OfSTED or governing bodies.

In the context of ‘realist evaluation’ what do we mean by a ‘programme mechanism?’

The important thing to remember is that interventions do not in themselves work, they work because people make them work. So an intervention provided an opportunity/constraint X and this prompted me to do Z.    As Wong et al (2013) explain - interventions work by changing the decision-making of the subjects of an intervention. (Now I know describing teachers as 'subjects' of intervention may be troubling for some readers, but stay with me).  With this in mind, the question becomes: how does the use of graded lesson observations influence the thinking and decision-making of teachers?

What programme mechanisms might be considered to be active when considering whether to either to implement or continue to use graded lesson observations?

Drawing upon the work of Pawson and Tilley (1997) on the use of CCTV systems in car parks to reduce car crime, (and which is scarily useful), these are some of the mechanisms head-teachers and senior leadership teams maybe seeking to influence through the use of graded lesson observations.

The 'gotcha mechanism’ graded lesson observations could reduce the number of 'inadequate /requires improvement' lessons as it make more likely that persistently 'poor' teachers are identified and competency and capability procedures are used to to ensure the 'poor' teacher leaves the school.

The 'you’ve been framed mechanism' graded lesson observations could reduce the number of inadequate lessons by encouraging teachers to raise their game on a day to day basis, so as to avoid being graded 'requires improvement/inadequate’.

The 'effective deployment mechanism' graded lesson observation may facilitate the effective deployment of professional development to support those teachers who have been identified as requiring support in order to improve teaching and learning

The 'publicity mechanism' graded lesson observations may indicate to teaching staff that the SLT takes seriously the improvement of teaching and learning. The teachers may then decide to 'up their game' in order to avoid being judged 'requiring improvement/inadequate'

The 'memory jogging mechanism' notices in the staffroom about graded lesson observation protocols may remind teachers of their 'vulnerability' and prompt them to take greater care their lessons meet the required standard

And I'm sure there are others you can think of such as:

The 'get out before they catch me mechanism' graded lesson observations may encourage some teachers who think they are weak practitioners  decide to leave before they are deemed inadequate.

The ‘the compliance mechanism’ graded lesson observations may demonstrate who is ‘in-charge’ within the school and encourage teachers to comply with the desired model of teaching, learning and assessment.

What are the implications of the above analysis the use of graded lesson observation in schools?

First, if we want to understand why headteachers and senior leaders may be extremely reluctant to give up on an established intervention, then we need to take into account both the programme theories they hold and causal mechanisms they are seeking to influence. If the reliability and validity of graded lesson observations is irrelevant to either the programme theory or causal mechanism, then the ‘evidence’ is unlikely to have much impact.

Second, proponents of non-graded observation may need to adopt a slightly different take in disagreeing with colleagues. In the first instance we need to understand colleagues ‘programme theories’ and views of ‘causal mechanisms.’ Relying on evidence, may not be sufficient to bring about changes in those ‘programme theories’ or ‘causal mechanisms’. Daniel Dennett in his 2013 book Intuition Pumps And Other Tools for Thinking suggests that at times of disagreement we should use Rapoport’s Rules:

1. You should attempt to re-express your target’s position so clearly, vividly, and fairly that your target says, “Thanks, I wish I’d thought of putting it that way.”

2. You should list any points of agreements (especially is they are not matters of general and widespread agreement).

3. You should mention anything you have learned from your target.

4. Only then are you permitted to say so much as a word of rebuttal or criticism.

And finally

Any intervention – be it graded or ungraded lesson observations - must be seen within the context within which it is applied, so in other words there is no such thing as a context free intervention. Indeed, context is an essential component of ‘realistic evaluation'. So in the case of schools, much will depend upon; the relationships between staff; the values, beliefs and culture of the school; the impact upon the school of external shocks – be it funding or an Ofsted inspection; or the stance that senior leadership take on the use of external evidence, as to how an intervention such as graded lesson observations play out. Remember interventions – be it graded or ungraded lesson observations - do not in themselves work, they work because teachers and headteachers make them work.