Is it any good : Merit is not the only fruit

If you are a school research lead, practioner inquirer or someone interested in what works in schools, this post is for you.   In this post,  I will use Stufflebeam and Coryn's (2014) extended definition of evaluation to help examine some of the values associated with in-school evaluation.  I will then go onto consider some of the tasks associated with evaluations and consider the operational implications for the school research lead.

So what do we mean by and extended definition of evaluation?

A useful place to start is Stufflebeam and Coryn's  (2014) extended definition of evaluation, which they describe as:

… the systematic process of delineating, obtaining, reporting, and applying descriptive and judgmental information about some subject’s merit, worth, probity, feasibility, safety, significance, and or equity. (p14)

Stufflebeam and Coryn acknowledge they might have included other values and when conducting an evaluation, as such discussions will need to take place as to whether other values, relevant to the context, should also be include.   That said, it is likely that most evaluations will include some, if not all, of the seven included values.

So what are the extended values?

In a recent post, I discussed the distinction between 'merit' and 'worth', so will only briefly revisit them now.

• Merit - the intrinsic quality or excellence of the service/programme/project/innovation without reference to costs.

• Worth - the quality of the item, taking into account both context and costs.

More detailed descriptions of the other five extended values are detailed below.

• Probity - has the activity/programme under review been conducted honestly and with due regard to ethical considerations such as honesty, integrity and ethical behaviour.  As such evaluators, should check a programme's uncompromising adherence to the highest moral standards and err on the side of too much consideration of probity.

• Feasibility – does the service/programme consume more resources than available or are there political considerations which make the activity undeliverable.  As such an evaluator's decision may justify the non-continuation of a programme

• Safety – are those engaging with the service/programme subject vulnerable to harm, for example,  physical or psychological, and is applicable to evaluations in all fields

• Significance – what is the potential of the service/programme and its importance within a given context.   Sometimes programmes are only relevant in the short-term, or only have local interest.  On the other hand, some programmes have a relevance far beyond the evaluation setting.  As such, a key questions for the evaluator is whether the project/service scaleable and will it work in other different settings.

• Equity – this can include: provision to all; access for all; equal participation; impact on different groups.  Kellaghan cited y Stufflebeam and Coryn argues that there are seven indicators of the existence of equity:
1. A societys public educational services will be provided for all people
2. People from all segments of the society will have equal access to the services
3. There will be close to equal participation by all groups in the use of the services
4. Levels of attainments  - for example, years in the education systems - will be substantially the same for different groups
5. Levels of proficiency in achieving all of the education system's objectives will be equivalent for different groups
6. Levels of aspiration for life pursuits will be similar across societal groups
7. The education systems will make similar impacts on improving life accomplishments of all segments of the population (especially ethnic, gender, socio-economic groups) that the education serves.  (Stufflebeam and Coryn, p14)

So how do we operationalise evaluation?

Stufflebeam and Coryn characterise the work of evaluators under four main headings.

• Delineating – determining key questions, audiences, values, criteria, budget, information sources and where appropriate budget

• Obtaining – obtaining, aggregating and analysis relevant information

• Reporting – providing the sponsor, other audiences and stakeholders with feedback about the outcomes of the evaluation

• Applying – how can the evaluator assist the evaluation sponsor apply the findings of the evaluation

The final feature of Stufflebeam and Coryn’s definition of evaluation to be considered concerns the nature of the information included in the evaluation

• Descriptive information – this should provide a range of factual statements that ‘objectively’ describe the programme/service.  This could include to whom is the service offered; when was if offered; how many people engaged with the provision; how resourced – human, physical and financial; the cost of the provision

• Judgmental information  - this includes getting the views of what those involved in the service/innovation/provision think of the ‘quality’ of the service.  This should involve judging the provision against a set of values and criteria

So what are the implications for the school research lead?

For me there are several implications for the school research lead and senior leaders within a school.

• If the role of the school research lead involves helping colleagues to try work out what works, this should be seen as a necessary but not sufficient condition for evaluations. 
• The evaluative questions - need to include : what works, for whom, to what extent, in what context etc.
• Evaluation will require the application of values to the work of colleagues, which will require the evaluator to be particularly skilled in managing the internal politics within a school.
• If a school is conducting a range of practitioner inquiries, it's important to try and ensure that all types pupils are 'covered' by the inquiries.
• Be clear whether the evaluation is about bring improvement or providing an overall judgement of the programme/evaluand.
• Informal evaluations may be useful in generating discussion, but are unlikely to provide sufficient rigorous evidence to justify scaling up informally evaluated programmes within a school.  However, they may be useful within a formative evaluative context.
• Formal and detailed evaluations are necessary where the outcome of the evaluation is likely to involve a critical operational decision within the school.

Some final words

Although this post has focussed primarily on school evaluation and has put the 'school research lead' in the foreground of the evaluative process, being a skilled evaluator is a responsibility for every teacher within a school.   Indeed, applying the values of the extended definition to current practices may provide a very stimulating 'provocation' which results in changes in practice.

References

Stufflebeam, D.L. & Coryn, C. (2014). Evaluation Theory, Models & Applications (second edition), Jossey Bass, San Franciscon.

SLT and Quick Fixes : Why assumptions may make an ass out of you and me!

If you are interested in making better decisions, then this post is for you. In particular, if you are a middle or senior leader within a school this post will help you to look at the challenges in undertaking high-quality problem solving processes. Recent research by Robinson, Meyer, Le Fevre and Cinema (2014) suggests that many leaders go about solving problems with little or no explicit testing of both underpinning assumptions or causal reasoning. Rather leaders assumed the the validity of their causal assumptions and offered or sought ideas about how to fix the problem-at hand. The rest of this post will review the study's theoretical framework, methods and data sources, findings and significance of the study. The final section of this post will consider some of the practical implications for leaders - at all levels within a school - in their attempts to engage in high quality problem solving processes.

So what was the purpose of the study?



Leaders are operating in an increasingly turbulent world, with a need to generate solutions to complex problems. However, many such problems are wicked in that they are difficult if not impossible to solve - with no ideal solution, few if any measures of success and with  more than more than cause  - and require the participation of others in problem-solving process. As such, the purpose of Robinson et al's research was to .. examine the quality of leaders’ real-world problem- solving as they initiate conversations with those whose cooperation is essential for resolving problems of equity and excellence (p2)



Robinson et al argue that the quality of leaders’ problem-solving is important because leader's decisions have ethical and educational consequence for others. For example, decisions may lead to a waste of resources which may impact upon the availability of resources for some pupils. Alternatively, decisions may be taken about staff, which are based on an incomplete problem-solving process.



So what was the study's theoretical framework?



Robinson et al draw on theories of interpersonal effectiveness (Argyris, 1991; Argyris & Schon, 1974) and theories of negotiation, so as to identify behavioral indicators of when leaders tested and checked, instead of assuming the validity of their assumptions, with it being argued that such behaviour are likely to increase the quality of inferential reasoning. Information is sought about the other party's perspective through a process of summarising and checking and the use of genuinely open questions.
  The research focused on whether the leaders tested three types of assumptions: first, assumptions about the presence of a problem, and this in critical in engaging others in the problem-solving process; second, assumptions about the cause of the problem and associated causal reasoning; third, assumptions about the solution to the problem.

So what methods and data sources were used?

Twenty-seven aspiring and experienced educational leaders, who were enrolled on a graduate course in educational leadership, participated in the study. . They were asked to identify an important issue which was important to them and having gained the consent of the other party involved, record a conversation that sought to address the issue . The leaders then transcribed their own conversations and annotate the transcript with any unarticulated thoughts and observations, which had occurred during the conversation. The researchers then annotated the scripts to identify each main assumption and any attempt to check the validity of such assumptions. These analyses were then subject to further analysis both, quantitative and qualitative and subsequent validation.

So what are the study's findings?

Leaders were much more likely to assume rather than test the validity of the main assumptions they made about their selected problem. As Robinson et al state : Leaders’ problem-solving typically involved gaining agreement about the existence of a problem and moving straight to a discussion of how it could be fixed, with little if any inquiry into its causes. (p7)

Robinson et al identify two reasons why this might be the case. Leaders tended not to reveal their underpinning assumptions, particularly if these assumptions involved criticism of the other party, which suggests a link between interpersonal skills and the quality of problem-solving. Second, given current conceptions of 'heroic leadership' maybe the leaders thought it was their responsibility to find a solution and fix the problems. Some of the unexpressed thoughts of the leaders suggested that they felt under pressure to find solutions of the other party within the discussion.

So what is the scholarly significance of the study?

Robinson et al argue identify three reasons why the the study is of scholarly significance.

1. Despite the current 'fashion' for evidence-based inquiry, the study would suggest that there appears to be little evidence that such inquires are taking place, with conversations focussing on solution development rather than the testing of underpinning assumptions
2. Although leaders often thought that the other party's behaviour or actions had contributed to the problem at hand, this was very rarely raised directly in the conversations. This suggests a close relationship between the inter-personal skills and the problem-solving skills of the leader. In other words, an unwillingness by leaders to raise issues with the other party - be it for reasons for a lack of skill or a concern for maintaining relationships - will have an impact on the quality of the problem-solving process.
3. Questions remain about the extent it is possible to manage some of the cognitive processes involved in such conversations, particularly vis a vis System 1 reasoning processes. On the other hand, research is being undertaken to see whether it is possible to support the development both cognitive processes and behaviours in order to improve the problem-solving process.

So what are the practical implications of the study?

Now bearing in mind the relatively small sample of school leaders involved in the research, it is important that only provisional implications are drawn from the student.  That said, for me, there would appear to be three implications of the study for leadership practice at all levels within a school.

1. Leaders many need to change their stance, from heroic problem-solver to that of the facilitator of the conditions that allow the development of genuine partnerships for problem-solving. The role of leader may well be about developing the skills to help others unpack their assumptions and causal reasoning rather than 'selling' the leader's own answer to a problem.
2. Given the time commitment required for high quality problem solving processes, leaders will need to learn the skill as to when deep and genuine problem-solving processes are required. As such, the leader's focus will need to be in the first instance in a deep understanding of the nature of the problem, in particular checking out the assumptions about the problem, before working how to proceed. 
3. And to misquote Daniel Kahneman -it may well be a case of Thinking Once, Thinking Twice and Acting Fast

References

Argyris, C. (1991). Teaching smart people how to learn. Harvard Business Review, 69(3), 99- 109.

Argyris, C., & Schon, D. (1974). Theory in practice: Increasing professional effectiveness.
San Francisco: Jossey-Bass.

Robinson, V. , Meyer, F. , Le Fevre, D. and Sinnema, C. (2014) Leaders' Problem-Solving Capabilities : Exploring the 'Quick-Fix' Mentailty Paper submitted to the AERA annual meeting 2015

Are you a genuine inquirer?

If you have anything to do with inquiry within schools, this post is for you. In particular, if part of your role is to have 'professional' conversations and dialogues with colleagues, this post will illustrate both the challenges in having genuine and open conversations and the prevalence of pseudo inquiry.  In doing so, I will draw upon the work of Le Fevre, Robinson and Sinnema (2014) who recently undertook a study of how 13 school leaders engaged in inquiry, and whose main finding is that leaders engage in predominantly pseudo inquiry, with little evidence of genuine and open-minded inquiry.    Given the importance of this finding and its implications for school-based inquiry and leadership and management, Le Fevre et al's work requires further examination.  As such, the rest of this post will :

• distinguish between and inter-personal and intrapersonal inquiry
• examine Genuine Inquiry as Open-mindedness;
• explore some of the barriers to Genuine Inquiry;
• summarise the study's research methods and findings;
• consider the implications for both schools and school research leads.

So what is the difference between inter-personal and interpersonal inquiry?

First, it is important to clarify that we are examining inter-personal rather than intra-personal inquiry. Le Fevre et al argue that intra-personal inquiry involves, for example,  personal reading or private reflection.  Whereas interpersonal inquiry involved some form of personal interaction - normally face to face

So what do we mean by open-mindedness?

Le Fevre et al argue that open-mindedness as associated with being critically open to alternative possibilities, a recognition that they indeed may be wrong about the issue at hand be willing to change their position,  With this open-mindedness incorporating a recognition that their beliefs may be incorrect or misinformed, and that new evidence becoming available may require a shift in a position or view.

So what do we mean by 'pseudo-inquiry'?

Le Fevre state that pseudo-inqury takes place where there is not a commitment to open-mindedness and ...  has the surface characteristics of inquiry, it is not driven by a desire to learn,  Genuine inquiry requires conversations to be motivated (either consciously or unconsciously) by a desire to learn and to be drive by a stance of open-mindedness (p884)

Pseudo inquiry can often be indicated by the types of questions which are asked.  For example, if the questions posed are designed to communicate your own point of view, in a subtle and often implied manner.  - Don't you agree that it would be a good idea to ..... Alternatively, a number of simple straightforward questions are asked in order to create an air of consensus, and which makes more difficult to subsequently disagree.  Finally, there is the form of pseudo-inquiry where the answer to the questions are already known to the inquirer.   

So what are some of the barriers to Genuine Inquiry?

Cognitive Biases

Le Fevre et al argue thats some of the main barriers to genuine inquiry are cognitive biases.  In a previous post I identify a number of cognitive biases which can get in the way of genuine inquiry and those include:

• Confirmation bias - the tendency to selectively search or interpret information in a way that confirms your perceptions or hypotheses.
• Conjunction fallacy - the tendency to assume that specific conditions are more probable than a single general one.
• Endowment effect - the tendency that people often give more value to on an object they already have than they would be pay to acquire it.
• Fundamental attribution error - the tendency to over-emphasize personal factors and underestimate situational factors when explaining other people's behaviour
• Halo effect - the tendency for a person's positive or negative traits to extend from one are of their personality to others' perceptions of them. 
• Hindsight bias - a memory distortion phenomenon due to the benefit of feedback about the outcome of an event, people's recalled judgment of the likelihood of that event are typically closer to the actual outcome that their original judgments were.
• In-group bias - the tendency for people to give preferential treatment to others they perceive to be members of their own group.(Wilke and Mata, 2012).

Perceptions of risk

Le Fevre et al argue that perceptions of risk can have a significant impact on the nature of inquiry, leading inquirers to avoid that will lead to negative emotional reactions.  Le Fevre et al go onto argue There are two sorts of risks; one, is of up-setting the relationship, and the second is of being challenged (p886).  As such, when individuals are motivated by the need to confirm rather than challenge their beliefs and underpinning assumptions, they are far less likely to take relationship risks.

Limited skills in Genuine Inquiry

A third barrier to effective Genuine Inquiry is related to the interpersonal skills of the inquirer.  Le Fevre et al reference research which indicates that educational leaders - headteacher etc - are far better at putting forward their own point of view than listening and learning about the perspectives of others.  Le Fevre et al argue that this lack of skill reflects both the cognitive and behavioural elements of skills and that school leaders are perfectly capable of asking open and inquiring questions, which seek to get to the bottom of an issue relating to a college.  Unfortunately, the thoughts that occupy their minds get in the way of them doing so.  Le Fevre et al suggest that leaders don't check others' understandings of an issue for fear or disagreement or if a parent raises an objection this leads to a feeling of being stuck and which gets in the way of further inquiry; they don't check understanding of the other person's perspective because they have forgotten what has been said as they are more interested in what they are going so ay next.

So what research did Le Fevre et al undertake?

The study involved 13 educational leaders on a university graduate programme in educational leadership, all of which volunteered to take part in the research.  The characteristics of the groups are as follows:

• 9 females/4 makes
• 8 worked in primary schools and 5 worked in secondary schools
• 4 had less than 2 years senior management experience, 6 had 3-5 years and 3 had 6-10 years
• The majority were between 31 and 50 years of age

The participants were recorded having a conversation - of approximately 8 minutes - with a female actor playing the role of a teacher, with the conversations being based on an identical scenario.  Immediately after the conversation the educational leader was asked to complete a self-assessment of their interpersonal skills.  One week later the educational leaders were each issued with a transcript of the conversation and were asked to annotate the transcript with any thoughts and feelings they had during the conversation which they had not expressed in the conversation.  The educational leaders were subsequently asked to write up to half a page detailing what they were seeking to achieve during the conversation.

Data analysis and findings

The transcripts of all 13 conversations were repeatedly listened and led to three categories of transpctin being identified

• No inquiry
• Pseudo-inquity 
• Pseudo-inquiry and limited genuine inquiry

A fourth category of genuine inquiry had been expected, but no conversations fitted into that category.  A second step involved identifyin patterns within each of the categories to help establish the reasons for the absence of genuine inquiry, with the whole transcript being used as the unit of analysis.  Indeed, the main finding of the study

... is that leaders engaged in minimal genuine inquiry.  One conversation involved no inquiry of any sort, two conversations involved inquiry but it was all pseudo and the 10 conversations were predominantly pseudo inquiry.  While much inquiry, as signalled by such linguistic feature as questioning, was common-place, our analysis of the unspoken thoughts and feelings revealed considerable closed rather than open-mindedness.  Self-assessment data indicated that the leaders themselves wre aware of their limited capability in genuine inquiry (p890)

Le Fevre et al go onto state : While interpersonal inquiry, in the context of professional learning and problem-solving is highly values, our research suggests that genuine inquiry in situations of anticipated difference and disagreement, at least is relatively rare. When conversations are risky and likely to rouse negative emotions, genuine inquiry is overtaken by the desire to wine, and to do so while avoiding risk and negative emotion. (p895)

So what are the implications of the study?

Le Fevre et al argue that the study has the following implications:

• it confirms previous research in both business and educational settings
• given that research in other fields suggests that participants are likely to perform better in a scenario situation than in real life, the study suggests the genuine-inquiry 'failings' are greater than this research suggests.  As such, further research in real-life settings is required
• to develop leaders skills as genuine inquirers may require significant disruption to the their current practice and ways of work.  In the first instance, this may best be achieved by some form of 'private' intrapersonal inquiry.  Having challenged their own assumptions in private, this may lead to a willingness to do so in public interpersonal inquiry.

So what are the particular implications for the school research lead?

For me, three implications spring to mind:

1. The need to accept that in all likelihood everyone of us involved in educational inquiry have a long way to go in developing our practice.
2. The current emphasis on lesson study and joint practice development, may indeed by putting the 'cart before the horse' and colleagues may first require a structured individual programme of support to attempt to unpack their underlying beliefs, values and biases before they are in a position to engage in genuine inquiry with others.
3. Building school cultures which are genuinely open and not pseudo - will take time, effort and real commitment.  School leaders are going to have to show the need to demonstrate that developed their own genuine and open-minded ways of thinking and behaving, if genuine inquiry is to have real roots within a school.

References

Le Fevre, D. M., Robinson, V. M., & Sinnema, C. E. (2014). Genuine Inquiry Widely Espoused Yet Rarely Enacted. Educational Management Administration & Leadership, 1741143214543204.

Wilke A. and Mata R. (2012) Cognitive Bias. In: V.S. Ramachandran (ed.) The Encyclopedia of Human Behavior, vol. 1, pp. 531-535. Academic Press