Thursday 14 December 2017

School research lead and simple sabotage

As we approach the end of the calendar year it seems only sensible to have an end of year list.   So in the spirit of 'bah humbug' I thought I'd provide a checklist of those actions where you may have tried to sabotage yourself as school research lead or where others may have tried to undermine you.  This checklist is based on Simple Sabotage by Galford, Frisch and Greene - which in turn is based on guide to sabotage produced by the US Office of Strategic Services in 1944


A checklist of acts of sabotage


Action
You
Others
Insist on doing everything by ‘channels.’ Never permit short-cuts to be taken to expedite decisions


Make speeches. Talk as frequently as possible and at great length.  Illustrate your points by long anecdotes and accounts of personal experience. Never hesitate to make a few comments that .we are doing it for the pupils’


Where possible refer all items to ‘committees’ for further discussion and consideration.  Attempt to make the committees as large as possible – never less than five.


Bring up irrelevant issues as frequently as possible


Haggle over precise wordings of communications, minutes and other documents


Refer back to matters already decided upon at the last meeting and attempt to re-open the question or the advisability of the discussion


Advocate ‘caution’ Be reasonable and urge your colleagues to be ‘reasonable’ and avoid haste which might lead to embarrassment later on


Be worried about the propriety of any decision – raise the question of whether such action as is contemplated lies within the remit of the group or whether it might conflict with the policy of some higher eschelon


CC everyone into to all your emails and discussions



So what are you to do if there are ticks in either column.  Well if it's you sabotaging yourself, the answer is simple STOP doing it - and do something else instead which increases your chance of success, for example, have working groups of four or less.  If it is others trying to sabotage  - call the behaviour out - if a decision has been made and subsequently someone tries to re-open the discussion - just say - 'Thank you for your comments, a decision has already been made and we are moving on' - and then move on

Friday 8 December 2017

The School Research Lead : How to stop doing what doesn't work

Tuesday 5 December  saw a strange alignment between the real-world and Twitter.  That night saw a Coalition for Evidence-Based Education discuss the notion of strategic abandonment (Thank you @DrCarolineCreaby).  Whilst later in the evening#DebatED  discussed 'whether an interest in education research is more about identifying what doesn't work as suggesting what will'.  This was then following up on Thursday with a #UKEdResChat focussing on 'how do we define 'what works' in educational research? Should we also be focussing on what doesn't work?"

So with that in mind it seems sensible to examine a process for disengaging from strategies and interventions which appear not to be working.   (McGrath, 2011) has identified a disciplined process for getting out of projects and which includes these steps
  1. Decide in advance on periodic checkpoints for determining whether to continue or not
  2. Evaluate the project’s upside against the current estimated costs of continuing.  If it no longer appears that the project will deliver the returns anticipated at the outset, it may be time to stop 
  3. Compare the project with other candidate projects that need resources.  If this one looks less attractive than they do, it may be time to stop? 
  4. Assess whether the project teams may be falling prey to escalations pressures (all we be ok as long as we make the project bigger) 
  5. Involve an objective, informed outsider in the decisions about whether to continue, instead of leaving it up to the project team members
  6. If the decision is made to stop, spell out the reasons clearly
  7. Think though how capabilities and assets developed during the course of the projects might be recouped
  8. Identify all who will be affected by the project’s terminations; draw up a plan to address disappointments or 'damage' they might suffer
  9. Use a symbolic event – a wake, a play, a memorial – to give people closure
  10.  Make sure that that the people involved get a new, equally interesting opportunity p83

 Given what we know about educational research and interventions, it is impossible to avoid things that do not work.  As such the choice is simple - continue with practices and interventions that do not work or release the resources for use in some area where they might.  However, in doing so, it is important to maximise what can be learnt from failure - and which may lead to success next time.

Reference

McGrath, R. (2011). Failing by Design. Harvard Business Review (April, 2011), 77-83.


Thursday 30 November 2017

New : Teachers and performance related pay - what's the evidence?

Last week Schools Week published a piece by Lee Miller – Deputy Chief Executive of the Thinking Schools Academy Trust (TSAT) titled Performance related pay will solve teacher retention crisis which describes a new teacher pay structure for the TSAT – which includes
  • From September 2018 a minimum starting salary of £25,000 (£2,000 above the national average)
  • All progression at the upper pay levels is based solely on performance (excellent teaching)
  • Teachers at the top of the scale – who exceed performance targets receive a 3% non-consolidated bonus
Having read the article – my first response was to ask: What research evidence is there to support the pay structure which has been put forward?

Pietro Marenco (@PMarencoHR and scienceatwork.com) very kindly tweeted an extract from (Weibel et al., 2009) who undertook an meta-analysis of the effect of pay for performance on performance and who state:

Our meta-analysis clearly demonstrates that the task type moderates the effect of pay for performance on performance. Pay for performance has a strong, positive effect on performance in the case of noninteresting tasks. Pay for performance, however, tends to have a negative effect on performance in the case of interesting tasks. The vignette study reveals (a) why pay for performance sometimes undermines performance and (b) how pay for per- formance produces hidden costs, which also need to be accounted for. 

Pay for performance causes a cognitive shift, that is, it strengthens extrinsic motivation for behavior (causes a price effect) and at the same time weakens intrinsic motivation for behavior (causes a crowding-out effect). Depending on the strength of these two opposing effects, pay for performance either hurts or promotes personal efforts: The more intrinsic motivation was there at the beginning, the more of it can be destroyed. 

Hidden costs arise even if the price effect is stronger than the crowding-effect. The loss of intrinsically motivated behavior has always to be compensated by external rewards.  P18

Which tends to suggest that in the context of teaching – a complex task if ever there was one (Shulman, 2004) - performance related may have a negative effect on performance and may ‘crowd-out’ intrinsic motivation (and associated discretionary effort.

If we turn now to research evidence which relates specifically to the education sector, we can look at the (EEF, 2017) summary on performance pay which states:

The results of rigorous evaluations, such as those with experimental trials or with well-controlled groups, suggest that the average impact of performance pay schemes has been just above zero. Some approaches appear to show more promise, such as bonuses or enhanced pay to attract teachers to challenging schools, or loss aversion, where the award has to be paid back if student results fall below a certain level. 

And which goes onto to state:

The evidence is not conclusive. Although there has been extensive research into performance pay, much of this is either from correlational studies linking national pay levels with general national attainment or from naturally occurring experiments. More recent randomized trials have had mixed results. Overall, it is hard to make causal claims about the efficacy of performance pay on the basis of the existing evidence

In addition, a systematic review by (Bajorek and Bevan, 2015)  of performance-related-pay in the UK public sector who found.

…. some evidence that PRP schemes can be effective across the three domains of the public sector for which there was evidence available (health, education and the civil service), but findings within and between the sectors are mixed, with scheme effectiveness often dependent on scheme design and organisational context 

Of particular interest to those who manage the teaching workforce is that there may well be - gender and age differences in the response to PRP

(Leigh, 2012)) and (Jones, 2013) report evidence to indicate that male teachers respond more positively to and support PRP schemes than their female counterparts, and Jones (2013) also highlighted that women are more likely to reduce their hours under PRP than men. Evidence also suggests that teachers with more experience display negative reactions to PRP in comparison to early career teachers (Jones, 2013; Leigh, 2013), although it is  PRP in the UK public sector  unclear whether this results from hostility to changes in the system, or previous negative experiences to PRP.(p91)

Finally, we could examine the work of (Lynch et al., 2016) on teacher retention who found

Factors that are significantly associated with intent to stay in the profession could be labelled as ‘protective factors’, worthy of attention among school leaders and policymakers. 
Unsurprisingly, we found that by far the strongest predictor is ‘job satisfaction’. Among the other factors, the strongest predictors are: 
being proud to work at the school 
having adequate resources 
being well supported and valued by school management 
having an effective governing body 
appropriate pay for level of responsibility. P17

(Since I posted this blog @Jonathan_Haslam of the Institute of Effective Education has pointed me in the direction of this IEE Best Evidence in Brief on performance pay)

In summary

On the one hand, it would appear that the research suggests that:
  • Performance related pay is not suited to complex tasks such as teaching 
  • Performance related pay may reduce intrinsic motivation
  • The impact of performance related on pupils’ results is just above zero
  • Male teachers tend to respond more positively to PRP than their female counterparts
  • The introduction of PRP may lead to female teachers reducing the number of hours taught
  • More experienced teachers are more likely to display negative reactions to PRP compared to early career teachers 
  • Job satisfaction is by far the biggest predictor of teachers’ intention to stay in the profession
On the other hand
  • There is a problem of recruitment and retention
  • The research evidence could be viewed as being inconclusive
  • Pay does appear to influence teacher retention rates
  • The senior team of a MAT clearly believe that they have the expertise to make it work
  • There would appear to be stakeholder support from the relevant teaching union
So where does this leave us?

Evidence-based practice involves the conscientious, explicit and judicious use of the best available evidence – research, organisational data, practitioner expertise and stakeholder  in order to make decisions which will hopefully lead to a favourable outcome.

Given the nature of the evidence – the decision as to whether to proceed with PRP within a MAT is a judgment call.  That said, in making that judgement one hopes that the decision-makers have undertaken an appropriate process of due diligence and have asked the following questions.
  • Is the issue an important problem for which a remedy is sought and that can be locally implemented?
  • How substantial are the desirable anticipated effects?
  • How substantial are the undesirable anticipated effects?
  • How robust and secure are the different sources - research, practitioner expertise, stakeholder views and school data - of evidence?   
  • Does the balance of the desirable and undesirable effects favour the proposal or a realistic alternative?
  • How large are the resource requirements – attention, time, money, professional learning?
  • What impact does the decision have on educational equity?  Will it help close gaps in attainment?
  • Are there important ethical issues which need to be taken into account? 
  • Are there key stakeholders – teachers, parents, trustees, who would not accept the distribution of the benefits, harms and costs?
  • Would the intervention adversely affect the autonomy of teacher, department, school or MAT?
  • Are there important barriers that are likely to limit the feasibility of implementing the intervention (option) or require consideration when implementing it?
  • Is the intervention or strategy sustainable?
And finally

How strong is the recommendation ranging from a strong recommendation – where benefits clearly outweigh costs or vice versa, with consistent supporting evidence from research evidence, practitioner expertise, school data and stakeholders – without major limitations.  Or is it a weak recommendation with uncertainty in estimates of benefits and costs Some supporting evidence from research evidence practitioner expertise – though with major limitations.

References

BAJOREK, Z. M. & BEVAN, S. M. 2015. Performance-related-pay in the UK public sector: A review of the recent evidence on effectiveness and value for money. Journal of Organizational Effectiveness: People and Performance, 2, 94-109.
EEF 2017. Teaching and Learning Toolkit : Performance Pay. London: Education Endowment Foundation.
JONES, M. D. 2013. Teacher behavior under performance pay incentives. Economics of Education Review, 37, 148-164.
LEIGH, A. 2012. The economics and politics of teacher merit pay. CESifo Economic Studies, 59, 1-33.
LYNCH, S., WORTH, J., BAMFORD, S. & WESPIESER, K. 2016. Engaging Teachers: NFER Analysis of Teacher Retention. Slough: NFER.
SHULMAN, L. 2004. The Wisdom of Practice-Collected Essays of Lee Shulman: Vol 1, ??????, Jossey-Bass.
WEIBEL, A., ROST, K. & OSTERLOH, M. 2009. Pay for performance in the public sector—Benefits and (hidden) costs. Journal of Public Administration Research and Theory, 20, 387-412.