Updated - When the evidence is not all that it seems and the case of the 0.84 effect size for leading teaching learning and development

As we approach the end of term, we enter the season of INSET days and a tsunami of PowerPoint presentations by consultants and other experts.  In these presentations we often see the same research referenced and used to justify a particular position or course of action, though without reference to the research's limitations  Indeed, one of the most frequently recycled set of findings is Robinson et al's (2009) research on the impact of different dimensions of leadership on student outcomes, and as illustrated in the following figure (Robinson et al 2009, p39)

Robinson (2011) states this figure illustrates that : The most powerful way that school leaders can make a difference to the learning of their students is by promoting and participating in the professional learning and development of their teachers.  The average impact (0.84) of these leadership practices on student outcomes was twice that of any other leadership dimension (p104)

With headteachers in high achieving schools headteachers and principals more likely to :

• participate actively in teacher learning and development than their colleagues in low-achieveing schools;
• promote and participate in staff discussions on teaching and learning;
• to be seen as a source of instructional advice.

Limitations

Unfortunately, there are a number of limitations to these findings and which are not widely reported.  So here goes:

• The effect size for the 'leading teacher learning and development' dimension was generated from only six of the twelve studies used to calculate the effect sizes elsewhere in the study
• In these six studies the vast majority of the evidence came from primary/elementary schools (Robinson et al 2009)
• None of the studies used in the calculation of the effect size came from the United Kingdom, with the majority of schools being based in the United States
• It would appear that the most recent study used in the calculation of the 0.84 effect size came from 1994 
• The findings do not explain why the dimensions is associated with higher achievement or describe the skills and knowledge needed to be an effective leaders of professional learning (Robinson, 2011)

Robinson (2011) states that when asked - school leaders talk about the symbolic importance of leaders involvement in professional learning. Yet Robinson argues that there are other reasons; first, leaders who collaborate and work with their staff are better placed to engage in professional discussions with teachers; second, headteachers' endorsements of external support/consultants are likely to be more credible as teachers know that their leaders are experiencing what they are endorsing.  However, Robinson surmises that perhaps the most important reason for effect is that the involvement of leaders in teacher professional learning make them more aware of the problems generated by the learning and what is necessary for success, which allows the headteacher to ensure the resources are in place for success.

So what are the implications of this analysis?

At first blush, there would appear to be several implications about what actions school leaders should take to improve student outcomes

• It would appear that we don't know what works in secondary schools, particularly in UK secondary schools.
• Given the importance that Robinson that places are leaders making resources available to support professional learning, it maybe that in secondary schools that resourcing strategically is the dimension which has the most impact.
• Indeed, it may be that within elementary/primary schools strategic resourcing is the key factor but it's impact has manifested itself in the 'leading teacher learning and development'.  So possibly there is a misattribution in the original analysis.

However, despite these reservations Robinson et al's research can still provide some guidance for school leaders.

1. If you are a primary school leader - leading and participating in teacher learning and development - may be your best bet in improving pupil outcomes .  
2. If you are a secondary school leader - you may wish to focus on the resourcing of teaching learning and development and well as trying to create a school culture which explicitly supports such activity
3. If you are secondary school leader - the nuts and bolts of leading teacher learning and development will rest with other colleagues - be it Deputy Heads and Heads of Department - and it will be important to appoint individuals into such positions who have the capacity and capability to undertake the leading of teaching learning and development.

Note - since I originally wrote this post I have been in contact with Professor Robinson, who has provided me with some clarification.  As Professor Robinson states:

These findings are NOT about what heads should be doing. They are about the impact  of different types of leadership practice on student outcomes and I state that, while  half of the studies used teacher ratings of principals, the other half were rating  school-wide leadership. There were not enough studies to do separate  meta-analyses. So they should be interpreted as indicative of the impact of type of leadership practice not of the impact of principalship. (email correspondence, 17 August, 2016)

And finally

If you are at a session where a consultant or external expert uses the above figure in one of their PowerPoint presentations, please remember to ask them the following: 

• How many studies were included in Robinson et al's research ?(27).  
• How many studies were used to calculate the effect sizes ? (12).  
• How many studies were used in calculating the 0.84 effect size? (6) 
• How many studies included data from secondary schools? (the minority as not possible to be specific)
• How many studies were conducted in the UK? (none)
• How many studies were conducted in the 21st century? (none)

References

Robinson, V.M., Hohepa, M. and Lloyd, C., 2009. School leadership and student outcomes: Identifying what works and why ; Best Evidence Synthesis, New Zealand Ministry of Education

Robinson, V., 2011. Student-centered leadership (Vol. 15). John Wiley & Sons.