The school research lead : What works and the problem of side-effects

An essential element of the school research lead's role is to help incorporate evidence – research, school data, stakeholder views and practitioner expertise into the decision-making process.  Now to assist us with this task (Drucker, 2001) provides us with two useful ‘rules of thumb’ to be applied when incorporating the evidence into the decision-making process, which are:

• Act if on balance the benefits greatly outweigh cost and risk.
• Act or do not act, but do not ‘hedge’ or compromise.  (p196)

However, identifying the risks and costs of whether to adopt a new intervention or strategy may not be as straightforward as you might hope.   (Zhao, 2017) argues that educational research has largely ignored the potential harms arising from research into 'what works.'  As such, it may not be that easy for you as an evidence-based school research lead to identify the negative side effects associated with an intervention, which in turn makes it extremely difficult calculate whether the on balance the benefits of an intervention greatly outweigh the risks and costs.  So to aid with you with this task of weighing up costs and benefits, the rest of this post will consider the following:

• What do we mean by a side-effect
• Why do we need to study side effects
• What are the apparent reasons for the lack of concern for risks of interventions in education
• What are the implications for your role as a school research lead?

What do we mean by a side-effect?

Zhao uses the Cambridge Online Dictionary to define a side effect as an unwanted or unexpected results or condition that comes along with the desired effects of something.

Why do we need to study side-effects?

Well in the context of evidence-based school leadership and management the study of side-effects is extremely important.  For example, poorly selected, designed and implemented educational interventions may have significant detrimental impacts on the life changes of young people, or teacher’s reputations may be unnecessarily tarnished with failure.

Second, implicit within evidence-based practice is the notion of informed decision-making, with stakeholders in the decision being actively engaged in the decision-making processes, with their values, rights and preferences being taken into account.  This will be extremely difficult, if not impossible to do, unless stakeholders and decision-makers are fully aware of the side effects of an intervention.

Third, if we are going to have productive and useful discussions about ‘what works, for whom, to what extent, for how long, and in what context’ then it is necessary to research evidence which allows us to gain a fair minded view of the potential and costs of the intervention.  Zhao argues that research not including sufficient discussion of the side effects of an intervention leads to unhelpful discussions as to the pros and cons of an intervention, as opponents to the intervention may adopt extreme positions given the lack of consideration of side effects by the advocates of the intervention

What are the apparent reasons for the lack of concern for risks of interventions in education?

Zhao argues that there are probably many reasons as to the lack of concern about the risks associated with interventions.  First, given that education is perceived as a good thing, this leads to individuals potentially ignoring negative unintended consequences.  Second, the negative unintended consequences may take some time to emerge.  Third, if a narrow definition of success is adopted, then it becomes much more difficult to observe unintended consequences, as they may be deemed to be out of scope.  Finally, there may be a range of pressures – be it political, commercial or economic – which may lead to wilful blindness towards the side effects.

What are the implications for your role as a school research lead?

So if we accept (Zhao, 2017) argument about insufficient attention being paid by educational to the side-effects of interventions, what are the implications for your role as the school research lead in supporting evidence-based school leadership and management.

First, as (Kahneman et al., 2011) argues when making decisions we are prone to overconfidence, planning fallacies and optimistic biases.  As such, given the relative lack of research evidence on the side effects of intervention, you are going to have make sure there are attempts to identify potential side effects and negative unintended consequences before any decision is made.

Second, when appraising research evidence about an intervention a useful screen as to the trustworthiness of the evidence is whether there are statements and discussions about the side effects of the intervention.  If there is little or no discussion about side-effects then this should act as a warning sign that the research evidence is providing an incomplete report on what works.

Third, when receiving advice from external consultants or hearing the views of internal advocates of an intervention, it is extremely important to ask them about any anticipated side-effects of the intervention.  Again, depending on how well this questions is answered will determine the trustworthiness of the advice.

Fourth, if having decided to implement an intervention it is important to be vigilant for any side effects or unanticipated consequences.  However, it’s not enough just to focus on the subjects of the ‘intervention’ it is also important to consider the implications for ‘bystanders’ or other directly involved in the implementation of the intervention.  An intervention may provide benefits for its’ intended beneficiaries but the costs are being borne by others.

Fifth, to help identify the unintended consequences of an implementation then there is value in using the premortem technique which will help you imagine circumstances when your planned for intervention has failed.  By using the premortem technique this could reduce your chances of falling prey to both a range of cognitive biases and groupthink 

Sixth, just because an intervention has negative unintended consequences does not automatically mean that the intervention should be abandoned.  It may be that there are some low-cost mitigating strategies that can be adopted, which means you can still benefit from the intervention, but now at an acceptable level of unintended consequences.

And finally

If the anticipated results of an intervention are too good to be to true, then they probably are.  

References

DRUCKER, P. F. 2001. The Essential Drucker, Butterworth-Heinemann Oxford.

KAHNEMAN, D., LOVALLO, D. & SIBONY, O. 2011. Before you make that big decision. Harvard business review, 89, 50-60.

ZHAO, Y. 2017. What works may hurt: Side effects in education. Journal of Educational Change, 18, 1-19. which can be found at  http://zhaolearning.com/2017/02/17/what-works-can-hurt-side-effects-in-education/

What's the role of teacher intuition in evidence-based/informed practice?

At Thursday's Chartered College of Teaching Inaugural Conference the issue of the relationship between teacher intuition/instinct and evidence-based practice was briefly discussed.  In this post, I will consider  the role of professional intuition in evidence-based decision-making, and under what circumstances that intuition can be trusted.  To help us get to grips with this issue I will be turning to the work of (Kahneman and Klein, 2009) and (Kahneman and Klein, 2010) who represent two quite different schools of thought on decision-making  – naturalistic decision-making and heuristics and biases –  but nevertheless have reached a number of areas of agreement.  As such the rest of this section will:

• define what is meant by skilled intuition;
• describe the circumstances which may contribute to the quality of an intuitive judgment; 
• determine whether the circumstances which support the development of skilled intuition exist in a school environment;
• consider the implications of for school leaders of fractionation – having genuine expertise in some but not all aspects of a role.
• consider the implications for evidence-based school leaders

What is skilled intuition?

Both Kahneman and Klein agree on Simon’s (1992) definition of skilled intuition as 

“The situation has provided a cue: This cue has given the expert access to information stored in memory, and the information provides the answer. Intuition is nothing more and nothing less than recognition” (p. 155). 

Kahneman and Klein note that this definition is extremely helpful in that it helps clarify what is meant by intuition, as it makes clear that intuition is not some mystical process but is just a process of pattern recognition

Under what circumstances is skilled intuition developed?

The determination of whether an intuitive judgment can be trusted is a function of both the environment is which the judgment is made – does the environment provide valid cues as to the issue or issues faced – and do decision-makers have the opportunity to learn the various cues.  Kahneman and Klein argue that medicine and firefighting take place in fairly ‘high validity’ environments, in that there are relatively stable relationships between the observed cues and the results of a range of decisions.  However, there are other environments such as financial and political forecasting which take place in what Kahneman and Klein describe as zero-validity environments. 

Do schools provide an appropriate environment to develop skill intuition?

Schools are likely to be a both a highly valid and uncertain environment.  By this I mean that a teacher may develop the skill to identify the most likely strategies to work with particular students, but that does not mean that it will work every time.  Head of departments and headteachers may have the skills and knowledge of what to do in trying to ‘turnaround’ an under-performing department or school, but that does not mean that this will lead to success on all occasions.   In these situations, the uncertainty is a function of the wide range of developments that can happen in any one class or school at any one time.

What are the problems with fractionation?

Kahneman and Klein state one of the challenges for decision-makers is that the fractionation of skill may lead to over-confidence.  School leaders more often than not have a number of different tasks, not all of which will they be as equally skilled. A school leader may have risen through the ranks of the school by being an expert teacher.  However, this may lead to them in their new role having to make judgment in areas in which they have no real knowledge and skills.  However, just because a school-leader has developed skills, for example in intuitively judging the academic success of a particular year group, that does not mean that those skills will transfer across to other areas such as staffing or financial decisions.  However, there is a real risk that the individual’ successes in other aspects of their role will lead to overconfidence – by both themselves and others – in their decision-making abilities, leading subsequently to poor decision-making.

What are the implications of this discussion for developing your skills as an evidence-based school leader?

• On a day to day basis in the classroom - intuition and pattern recognition is going to be an important part of the teacher's decision-making process
• However, when making less time-pressured and more deliberative decisions, professional intuition should not be trusted on its’ own.  Intuition should be seen as the starting point, and evidence which disconfirms your ‘gut’ feeling should be actively sought.
• Make sure that a systematic search strategy for multiple sources of evidence is developed early in the decision-making process
• Beware of over-confidence in your decision-making abilities, just because you have been able to make ‘good’ intuitive judgments in one part of your role, that does not mean you will be able to make such judgments in an area in which you are less skilled.
• Ensure conditions are created where the evidence informing the decision can be genuinely and openly challenged, and where constructive dissent is rewarded rather than discouraged or punished.
• Avoid jumping to conclusions and actively seek to defer judgment until all the evidence has been made available and appraised.

References

KAHNEMAN, D. & KLEIN, G. 2009. Conditions for intuitive expertise: a failure to disagree. American Psychologist, 64, 515.

KAHNEMAN, D. & KLEIN, G. 2010. Strategic decisions: When can you trust your gut. McKinsey Quarterly, 13, 1-10.

The school research lead and the need for research literacy - the case of leadership and correlation to organisational climate

A major challenge for the school research lead is whether they are able to help colleagues develop the capacity and capability to interpret research findings.  In particular, can school research help colleagues to correctly interpret the statistical findings which are often found in research reports.  So this post, I will look at the specific challenges of using the correlation coefficient(r ) when seeking to interpreting Daniel Goleman’s  work on leadership and emotional intelligence (Goleman, 2000).  To help us do this I will:

•  Summarise Goleman’s work on leadership that gets results.
• Briefly explain what is meant by the term correlation coefficient and the implications for interpreting Goleman's work
• Consider some broader points about the challenge of using correlation in educational leadership research and the implications for practice

Daniel Goleman and leadership that gets results

Goleman argues that leaders often mistakenly assume that their leadership is a function of their personality, rather than is something can be chosen to be meet the needs of a specific circumstance.  Goleman claims that successful leaders have strengths in six emotional intelligence competences* i.e.: self- awareness, self-regulation, motivation, empathy, and social skill.   Moreover, these components of emotional intelligence can be combined in different ways and which reflect six basic styles of leadership i.e. coercive, authoriative, affiliative, democratic, pace-setting and coaching, which are summarised in Table 1 Moreover, the very best leaders are not wedded to one particular approach and can change approaches depending upon the demands of the situation.

Table 1 Six approaches to leadership 

Leadership style	Brief explanation
Coercive	‘Do what I say’
Authoriative	‘Come with me’
Affiliative	‘People first’
Democratic	‘Let’s decide together’
Pace-setting	‘Let’s make this happen, and now.’
Coaching	‘People development

*note - I'm not going to get drawn into discussion about the evidence 'for and against' emotional intelligence - other than to suggest you have a look the work of Professor Rob Briner

The correlation coefficient and leadership styles

Based on research interviews with nearly 4000 executives out of a database of 20,000 executives worldwide,  Goleman goes onto to demonstrate the impact that different leadership styles  have on the organisational climate (working atmosphere).    Using the correlation coefficient (r ), Goleman seeks to to quantify both the strength and direction of a relationship between leadership style and organisational climate .     Now, the correlation coefficient (r) can range from +1 – where there is perfectly positive correlation to -1, where there is a perfectly negative correlation.  On the other hand, a value of 0 would suggest that there is no relationship between the two variables i.e. leadership style and organisational climate.

Goleman’s research indicates that  the relationship between leadership style and organisational climate reflects a range of intermediate values between -1 and +1.  Table 2 shows the relationship between different styles of leadership and aspects of organisational climate.  So looking at the table we can see that the coercive (-0.26) and pace-setting (- 0.25) styles have a negative overall correlation with organisational climate.  Whereas, the -authoritative (+0.54), affiliative (0.46), democratic (+0.43) and coaching (+0.42) – styles have a positive correlation correlation with organisational climate.  However, Goleman notes that all styles can have their uses, and no one style should be relied upon nor excluded, when seeking to tackle the ranges of circumstances which are faced by leaders.

Table 2 Leadership style and organisational climate (Goleman, 2000 p19)

Interpreting the correlation coefficient

(Cumming and Calin-Jageman, 2017) argue that when interpreting an r value – let’s say 0.54 – and which hints at some relationship between two variables X (leadership style)and Y (organisational climate), a number of different things could be going on.

• ·        There may be a causal link between leadership style  and organisational climate changes in leadership style may lead to changes in organisational climate, or changes in organisational climate are leading to changes in organisational style.
• ·         Something else may be going one, with other variables impacting on either leadership style  or organisational culture or both.
• ·        There are no causal links, and what we are seeing is what Cumming and Calin Jageman describe as ‘seeing a face in the clouds’. 

As such Cumming and Calin Jageman note that all correlations do is to give use some form of guidance as to what to investigate further.  The r values may suggest what and where to look, however, it needs to be made clear that correlation does not imply causation.  In other words, just because X appears to be linked with changes in Y, that does not mean that changes in X are causing changes in Y

So far we have made no reference to how should interpret the size of different correlation coefficient or r values – be they -1, 0, 0.5. 0.7 or +1.  For example in Table 3, (Hinkle et al., 2003) provided the following guidance for interpreting correlation.

Table 3 – Rule of Thumb for Interpreting the Size of a Correlation Coefficient

Size of Correlation	Interpretation
0.90 to 1.00 (- 0.90 to –-1.0)	Very high positive (negative) correlation
0.70 to 0.90 (- 0.70 to - 0.90)	High positive (negative) correlation
0.50 to 0.70 ( - 0.50 to -0.70)	Moderate positive (negative) correlation
0.30 to 0.50 ( - 0.30 to -0.50)	Low positive (negative) correlation
0.00 to 0 0.30 (    0.00 to- 0.30)	Little if any correlation

If we were to use this table as a guide, this would suggest that in general, with the exception of the authoritative style, there is little/low levels of correlation between leadership styles and organisational climate.  Nevetheless, (Cumming and Calin-Jageman, 2017) argues that r values need to be interpreted in context, as correlation is used in such a wide-ranging number of ways and settings, that reference values for r provide little help in interpreting the data.  Indeed, Cummings and Calin Jageman argue that an r value of 0.3 may suggest there is a relationship between two variables, even though when graphed the data looks like a shot-gun blast.

What about the correlation determinant r squared.

However, if we are looking for the strength of association between the two variables, we may wish to use the coefficient of determination r squared – which gives you the proportion of shared variance.  So if we have two variables (A – the independent variable and B the dependent variable) and we have a correlation coefficient + 0.5 then our coefficient of determination will be 0.25. 

So what does a coefficient of determination of 0.25 mean, well Professor Steve Higgins (@profstig) of the University of Durham, has provided me some useful guidance.  Well at best

• A might cause B 25% of the time OR
• B might cause A 25% of the time OR
• C causes them both (and is not present all of the time) OR
• it is a pure coincidence OR
• it is some of the above in some unknown combination

In other words,  there is some relationship explaining about a quarter of the overlap, but we don’t know anything about the cause

We can now calculate Goleman’s correlation coefficient into the coefficient determinant and interpret what it means for the relationship between leadership style and organisational climate

Table 4 Leadership style, correlation coefficient and correlation coefficient determinant

Leadership style	Correlation Coefficient  r	Coefficient determinant r  squared
Coercive	- 0.26	+ 0.07
Authoriative	+ 0.54	+ 0.29
Affiliative	+0.46	+ 0.21
Democratic	+0.43	+ 0.18
Pace-setting	-0.25	+ 0.06
Coaching	+ 0.42	+0.18

So what does this suggest, well for both the coercive and pace-setting styles relatively small amounts of variances in leadership style and organisational climate appear to be shared (-0.07 and 0.06 respectively) As for the relationship between an authoritative style and organisational climate the data suggests that at best only 29% of the total variance between the two variables is shared.

Other issues to take into account when interpreting the correlation coefficient

Again, I would like to thank Professor Steve Higgins for bringing these issues to my attention.  First, the correlation coefficient should be calculated from a random sample of the population.  In the case of Goleman’s work, this would appear to be the case with 4,000 executives randomly selected from a database of 20,000 executives worldwide.  That said, those 20,000 executives were not in themselves a random sample of executives.  Second, Goleman does not report the error associated with these correlations, as such, we do not know how precise they are, as no confidence limits have been reported.

Implications

It would seem to me that there are a number of implications in using ‘correlation’ when making recommendations about leadership and management.

• Correlation does not mean there is a causal link between two variables – in other just because you have a positive correlation that does not mean the changes in leadership style have caused the change in organisational climate.  Indeed, it may be the change in organisational climate may cause the change in leadership style.
• It’s important to have some rules of thumb for interpreting the size of the correlation coefficient.  In this example the largest correlation coefficient (authoritative 0.54) is around the borderline between positive and low correlation.  In other words, just because you have is for your study the largest correlation coefficient does not mean that the size of correlation coefficient is large.
• If you are looking to make judgments about the strength of the relationship between two variables, then you need to calculate the coefficient of determination.  Again, in this example, we have two values (coercive 0.06 and pace-setting 0.07) which are close to zero, which suggests there very little of the variance in leadership style and organisational climate seem to be shared.
• Before making recommendations about leadership and management which uses correlations, it might be useful to look at a basic statistical textbook.

References

CUMMING, G. & CALIN-JAGEMAN, R. 2017. Introduction to the New Statistics: Estimation, Open Science, and Beyond, Abingdon, Routledge.

GOLEMAN, D. 2000. Leadership that gets results. Harvard business review, 78, 4-17.

HINKLE, D. E., WIERSMA, W. & JURS, S. G. 2003. Applied statistics for the behavioral sciences.