SLT and Quick Fixes : Why assumptions may make an ass out of you and me!

If you are interested in making better decisions, then this post is for you. In particular, if you are a middle or senior leader within a school this post will help you to look at the challenges in undertaking high-quality problem solving processes. Recent research by Robinson, Meyer, Le Fevre and Cinema (2014) suggests that many leaders go about solving problems with little or no explicit testing of both underpinning assumptions or causal reasoning. Rather leaders assumed the the validity of their causal assumptions and offered or sought ideas about how to fix the problem-at hand. The rest of this post will review the study's theoretical framework, methods and data sources, findings and significance of the study. The final section of this post will consider some of the practical implications for leaders - at all levels within a school - in their attempts to engage in high quality problem solving processes.

So what was the purpose of the study?



Leaders are operating in an increasingly turbulent world, with a need to generate solutions to complex problems. However, many such problems are wicked in that they are difficult if not impossible to solve - with no ideal solution, few if any measures of success and with  more than more than cause  - and require the participation of others in problem-solving process. As such, the purpose of Robinson et al's research was to .. examine the quality of leaders’ real-world problem- solving as they initiate conversations with those whose cooperation is essential for resolving problems of equity and excellence (p2)



Robinson et al argue that the quality of leaders’ problem-solving is important because leader's decisions have ethical and educational consequence for others. For example, decisions may lead to a waste of resources which may impact upon the availability of resources for some pupils. Alternatively, decisions may be taken about staff, which are based on an incomplete problem-solving process.



So what was the study's theoretical framework?



Robinson et al draw on theories of interpersonal effectiveness (Argyris, 1991; Argyris & Schon, 1974) and theories of negotiation, so as to identify behavioral indicators of when leaders tested and checked, instead of assuming the validity of their assumptions, with it being argued that such behaviour are likely to increase the quality of inferential reasoning. Information is sought about the other party's perspective through a process of summarising and checking and the use of genuinely open questions.
  The research focused on whether the leaders tested three types of assumptions: first, assumptions about the presence of a problem, and this in critical in engaging others in the problem-solving process; second, assumptions about the cause of the problem and associated causal reasoning; third, assumptions about the solution to the problem.

So what methods and data sources were used?

Twenty-seven aspiring and experienced educational leaders, who were enrolled on a graduate course in educational leadership, participated in the study. . They were asked to identify an important issue which was important to them and having gained the consent of the other party involved, record a conversation that sought to address the issue . The leaders then transcribed their own conversations and annotate the transcript with any unarticulated thoughts and observations, which had occurred during the conversation. The researchers then annotated the scripts to identify each main assumption and any attempt to check the validity of such assumptions. These analyses were then subject to further analysis both, quantitative and qualitative and subsequent validation.

So what are the study's findings?

Leaders were much more likely to assume rather than test the validity of the main assumptions they made about their selected problem. As Robinson et al state : Leaders’ problem-solving typically involved gaining agreement about the existence of a problem and moving straight to a discussion of how it could be fixed, with little if any inquiry into its causes. (p7)

Robinson et al identify two reasons why this might be the case. Leaders tended not to reveal their underpinning assumptions, particularly if these assumptions involved criticism of the other party, which suggests a link between interpersonal skills and the quality of problem-solving. Second, given current conceptions of 'heroic leadership' maybe the leaders thought it was their responsibility to find a solution and fix the problems. Some of the unexpressed thoughts of the leaders suggested that they felt under pressure to find solutions of the other party within the discussion.

So what is the scholarly significance of the study?

Robinson et al argue identify three reasons why the the study is of scholarly significance.

1. Despite the current 'fashion' for evidence-based inquiry, the study would suggest that there appears to be little evidence that such inquires are taking place, with conversations focussing on solution development rather than the testing of underpinning assumptions
2. Although leaders often thought that the other party's behaviour or actions had contributed to the problem at hand, this was very rarely raised directly in the conversations. This suggests a close relationship between the inter-personal skills and the problem-solving skills of the leader. In other words, an unwillingness by leaders to raise issues with the other party - be it for reasons for a lack of skill or a concern for maintaining relationships - will have an impact on the quality of the problem-solving process.
3. Questions remain about the extent it is possible to manage some of the cognitive processes involved in such conversations, particularly vis a vis System 1 reasoning processes. On the other hand, research is being undertaken to see whether it is possible to support the development both cognitive processes and behaviours in order to improve the problem-solving process.

So what are the practical implications of the study?

Now bearing in mind the relatively small sample of school leaders involved in the research, it is important that only provisional implications are drawn from the student.  That said, for me, there would appear to be three implications of the study for leadership practice at all levels within a school.

1. Leaders many need to change their stance, from heroic problem-solver to that of the facilitator of the conditions that allow the development of genuine partnerships for problem-solving. The role of leader may well be about developing the skills to help others unpack their assumptions and causal reasoning rather than 'selling' the leader's own answer to a problem.
2. Given the time commitment required for high quality problem solving processes, leaders will need to learn the skill as to when deep and genuine problem-solving processes are required. As such, the leader's focus will need to be in the first instance in a deep understanding of the nature of the problem, in particular checking out the assumptions about the problem, before working how to proceed. 
3. And to misquote Daniel Kahneman -it may well be a case of Thinking Once, Thinking Twice and Acting Fast

References

Argyris, C. (1991). Teaching smart people how to learn. Harvard Business Review, 69(3), 99- 109.

Argyris, C., & Schon, D. (1974). Theory in practice: Increasing professional effectiveness.
San Francisco: Jossey-Bass.

Robinson, V. , Meyer, F. , Le Fevre, D. and Sinnema, C. (2014) Leaders' Problem-Solving Capabilities : Exploring the 'Quick-Fix' Mentailty Paper submitted to the AERA annual meeting 2015

Are you a genuine inquirer?

If you have anything to do with inquiry within schools, this post is for you. In particular, if part of your role is to have 'professional' conversations and dialogues with colleagues, this post will illustrate both the challenges in having genuine and open conversations and the prevalence of pseudo inquiry.  In doing so, I will draw upon the work of Le Fevre, Robinson and Sinnema (2014) who recently undertook a study of how 13 school leaders engaged in inquiry, and whose main finding is that leaders engage in predominantly pseudo inquiry, with little evidence of genuine and open-minded inquiry.    Given the importance of this finding and its implications for school-based inquiry and leadership and management, Le Fevre et al's work requires further examination.  As such, the rest of this post will :

• distinguish between and inter-personal and intrapersonal inquiry
• examine Genuine Inquiry as Open-mindedness;
• explore some of the barriers to Genuine Inquiry;
• summarise the study's research methods and findings;
• consider the implications for both schools and school research leads.

So what is the difference between inter-personal and interpersonal inquiry?

First, it is important to clarify that we are examining inter-personal rather than intra-personal inquiry. Le Fevre et al argue that intra-personal inquiry involves, for example,  personal reading or private reflection.  Whereas interpersonal inquiry involved some form of personal interaction - normally face to face

So what do we mean by open-mindedness?

Le Fevre et al argue that open-mindedness as associated with being critically open to alternative possibilities, a recognition that they indeed may be wrong about the issue at hand be willing to change their position,  With this open-mindedness incorporating a recognition that their beliefs may be incorrect or misinformed, and that new evidence becoming available may require a shift in a position or view.

So what do we mean by 'pseudo-inquiry'?

Le Fevre state that pseudo-inqury takes place where there is not a commitment to open-mindedness and ...  has the surface characteristics of inquiry, it is not driven by a desire to learn,  Genuine inquiry requires conversations to be motivated (either consciously or unconsciously) by a desire to learn and to be drive by a stance of open-mindedness (p884)

Pseudo inquiry can often be indicated by the types of questions which are asked.  For example, if the questions posed are designed to communicate your own point of view, in a subtle and often implied manner.  - Don't you agree that it would be a good idea to ..... Alternatively, a number of simple straightforward questions are asked in order to create an air of consensus, and which makes more difficult to subsequently disagree.  Finally, there is the form of pseudo-inquiry where the answer to the questions are already known to the inquirer.   

So what are some of the barriers to Genuine Inquiry?

Cognitive Biases

Le Fevre et al argue thats some of the main barriers to genuine inquiry are cognitive biases.  In a previous post I identify a number of cognitive biases which can get in the way of genuine inquiry and those include:

• Confirmation bias - the tendency to selectively search or interpret information in a way that confirms your perceptions or hypotheses.
• Conjunction fallacy - the tendency to assume that specific conditions are more probable than a single general one.
• Endowment effect - the tendency that people often give more value to on an object they already have than they would be pay to acquire it.
• Fundamental attribution error - the tendency to over-emphasize personal factors and underestimate situational factors when explaining other people's behaviour
• Halo effect - the tendency for a person's positive or negative traits to extend from one are of their personality to others' perceptions of them. 
• Hindsight bias - a memory distortion phenomenon due to the benefit of feedback about the outcome of an event, people's recalled judgment of the likelihood of that event are typically closer to the actual outcome that their original judgments were.
• In-group bias - the tendency for people to give preferential treatment to others they perceive to be members of their own group.(Wilke and Mata, 2012).

Perceptions of risk

Le Fevre et al argue that perceptions of risk can have a significant impact on the nature of inquiry, leading inquirers to avoid that will lead to negative emotional reactions.  Le Fevre et al go onto argue There are two sorts of risks; one, is of up-setting the relationship, and the second is of being challenged (p886).  As such, when individuals are motivated by the need to confirm rather than challenge their beliefs and underpinning assumptions, they are far less likely to take relationship risks.

Limited skills in Genuine Inquiry

A third barrier to effective Genuine Inquiry is related to the interpersonal skills of the inquirer.  Le Fevre et al reference research which indicates that educational leaders - headteacher etc - are far better at putting forward their own point of view than listening and learning about the perspectives of others.  Le Fevre et al argue that this lack of skill reflects both the cognitive and behavioural elements of skills and that school leaders are perfectly capable of asking open and inquiring questions, which seek to get to the bottom of an issue relating to a college.  Unfortunately, the thoughts that occupy their minds get in the way of them doing so.  Le Fevre et al suggest that leaders don't check others' understandings of an issue for fear or disagreement or if a parent raises an objection this leads to a feeling of being stuck and which gets in the way of further inquiry; they don't check understanding of the other person's perspective because they have forgotten what has been said as they are more interested in what they are going so ay next.

So what research did Le Fevre et al undertake?

The study involved 13 educational leaders on a university graduate programme in educational leadership, all of which volunteered to take part in the research.  The characteristics of the groups are as follows:

• 9 females/4 makes
• 8 worked in primary schools and 5 worked in secondary schools
• 4 had less than 2 years senior management experience, 6 had 3-5 years and 3 had 6-10 years
• The majority were between 31 and 50 years of age

The participants were recorded having a conversation - of approximately 8 minutes - with a female actor playing the role of a teacher, with the conversations being based on an identical scenario.  Immediately after the conversation the educational leader was asked to complete a self-assessment of their interpersonal skills.  One week later the educational leaders were each issued with a transcript of the conversation and were asked to annotate the transcript with any thoughts and feelings they had during the conversation which they had not expressed in the conversation.  The educational leaders were subsequently asked to write up to half a page detailing what they were seeking to achieve during the conversation.

Data analysis and findings

The transcripts of all 13 conversations were repeatedly listened and led to three categories of transpctin being identified

• No inquiry
• Pseudo-inquity 
• Pseudo-inquiry and limited genuine inquiry

A fourth category of genuine inquiry had been expected, but no conversations fitted into that category.  A second step involved identifyin patterns within each of the categories to help establish the reasons for the absence of genuine inquiry, with the whole transcript being used as the unit of analysis.  Indeed, the main finding of the study

... is that leaders engaged in minimal genuine inquiry.  One conversation involved no inquiry of any sort, two conversations involved inquiry but it was all pseudo and the 10 conversations were predominantly pseudo inquiry.  While much inquiry, as signalled by such linguistic feature as questioning, was common-place, our analysis of the unspoken thoughts and feelings revealed considerable closed rather than open-mindedness.  Self-assessment data indicated that the leaders themselves wre aware of their limited capability in genuine inquiry (p890)

Le Fevre et al go onto state : While interpersonal inquiry, in the context of professional learning and problem-solving is highly values, our research suggests that genuine inquiry in situations of anticipated difference and disagreement, at least is relatively rare. When conversations are risky and likely to rouse negative emotions, genuine inquiry is overtaken by the desire to wine, and to do so while avoiding risk and negative emotion. (p895)

So what are the implications of the study?

Le Fevre et al argue that the study has the following implications:

• it confirms previous research in both business and educational settings
• given that research in other fields suggests that participants are likely to perform better in a scenario situation than in real life, the study suggests the genuine-inquiry 'failings' are greater than this research suggests.  As such, further research in real-life settings is required
• to develop leaders skills as genuine inquirers may require significant disruption to the their current practice and ways of work.  In the first instance, this may best be achieved by some form of 'private' intrapersonal inquiry.  Having challenged their own assumptions in private, this may lead to a willingness to do so in public interpersonal inquiry.

So what are the particular implications for the school research lead?

For me, three implications spring to mind:

1. The need to accept that in all likelihood everyone of us involved in educational inquiry have a long way to go in developing our practice.
2. The current emphasis on lesson study and joint practice development, may indeed by putting the 'cart before the horse' and colleagues may first require a structured individual programme of support to attempt to unpack their underlying beliefs, values and biases before they are in a position to engage in genuine inquiry with others.
3. Building school cultures which are genuinely open and not pseudo - will take time, effort and real commitment.  School leaders are going to have to show the need to demonstrate that developed their own genuine and open-minded ways of thinking and behaving, if genuine inquiry is to have real roots within a school.

References

Le Fevre, D. M., Robinson, V. M., & Sinnema, C. E. (2014). Genuine Inquiry Widely Espoused Yet Rarely Enacted. Educational Management Administration & Leadership, 1741143214543204.

Wilke A. and Mata R. (2012) Cognitive Bias. In: V.S. Ramachandran (ed.) The Encyclopedia of Human Behavior, vol. 1, pp. 531-535. Academic Press

The School Research Lead and Working Out What Works : Distinguishing between merit and worth.

If you work in a school and have anything to do with evaluating teaching, learning and school effectiveness, this post is for you.  This post, the first of two, is aimed at anyone who wants to have a better understanding of how to figure out 'what-works' in schools.  In doing so, I will be using Stufflebeam and Coryn's 2014 book - Evaluation Theory, Models and Applications (second edition) to provide an introduction to educational evaluation.

So what is evaluation?

Stufflebeam and Coryn begin by using a basic definition of evaluation developed by the Joint Committee* in 1994 which describes evaluation as ... the systematic assessment of of the worth or merit of an object**.   Stufflebeam and Coryn argue that given the evaluation's root term is value, then evaluations inevitably involve some form of value judgement.  The Joint Committee's definition attempts to address the values issue by distinguishing between merit and worth, differences which are substantive when engaging in evaluation.

So how do we distinguish between merit and worth?

Stufflebeam and Coryn highlight the different characteristics of merit and worth in Table 1 (2014, p9)

Merit	Worth
May be assessed on any object of interest	Is assessed only on objects that have demonstrated an acceptable level of quality
Pertains to the intrinsic value of the object	Pertains to the extrinsic value of the object?
Pertain to quality, that is, an object’s level of excellence	Pertains to an objects’ quality and values or importance within a given context
Is assessed using the question.  Does the object do well what it is intended to do?	Is assessed using the questions, Is the object of high quality and also something the target group needs.
Is tied to the accepted standard of quality for the type of object being evaluated	Is tied to accepted standards of quality and to data from a pertinent needs assessments
Concerns the object’s rating on standards of quality against competitive objects of the same type	Entails judgment of the object’s quality and importance and value to a particular consumer group
May by assessed through the comparison of an object with standards or competitive objects	Assessments of worth may be comparative or non-comparative

In general, when we discuss an object's merit we are discussing whether that object is doing what it is supposed to do.  For example, feedback advances pupil learning whereas, repeating a year appears to have a negative impact on pupil learning. Merit, in other words, can be seen as intrinsic merit in the absence of costs.      Worth, on the other hand, does not just restrict itself to the intrinsic value of an object, it also includes  reference to the need for an object and its associated costs.  For example, phonics appear to be low cost intervention which advances pupil learning by four months, whereas the use of teaching assistants is a high cost intervention, which has limited impact on pupil learning. So a programme or innovation may be judged to meeting its own internal objectives, however, in doing so it might be both too expensive and not meeting a broader set of needs.

So what concepts are related to needs and needs assessment?

Stufflebeam and Coryn go on to state that the characteristics of worth suggest that it is impossible to evaluate an object's worth unless consideration is taken into account as to whether that object is a meeting specific needs.  And in considering the meeting of needs, we need to distinguish between the identification of a need and whether that need is being met through some form of intervention.  These differences area  highlighted Table 2.

Table 2 Concepts Related to Needs and Needs Assessment  (Adapted from Stufflebeam and Coryn, 2014 p11)

Concept	Definition	Example
Defensible purpose	A desired end that has been legitimated	Pupils’ development of mathematical skills
Need	Something that is necessary or useful for fulfilling a defensible purposes	Competent, effective teaching in mathematics
Outcome need	An achievement or outcome required to meet a defensible purpose	Pupils’ demonstration of proficiency in mathematics
Intervention need	A certain service, competent service provider, or other helping agent	Competent teachers in KS4 courses in mathematics
Needs assessment	A systematic investigation of the extent to which treatment/and or outcome needs are being met	Examination of pupils results in GCSE Mathematics and evaluation of the involved teachers

So what are the implications for the School Research Lead

The preceding analysis of the nature of evaluation has a several implications for the School Research Lead.

1. Much of the discussion around the role of the School Research Lead is about assisting colleagues in finding out 'working out what works' and as such focuses on the merit rather than the worth of the object of evaluation. 
2. Working out what works also needs to consider questions such as - works for who and under conditions - and are needs being met?
3. Some interventions may be of merit, but on the other hand, may not be worth it in terms of the effort expended in implementation.  

Some final words one evaluation - Part One

Values are an inescapable component of evaluations.  Accordingly, evaluations need to be undertaken in a systematic manner, with a defensible rationale being provided for the values being used to evaluate a subject;  rigorously collecting information and subjecting that information to open and transparent analysis; and communicating those findings to colleagues, senior leaders and other stakeholders.   This will require evaluators to have two broad sets of skills; first, the ability to undertake or assist in the technical evaluation of an object; second, having high-level inter-personal skills to ensure all the relevant participants are engaged before, during and after the evaluation. (Stufflebeam and Coryn, 2014)

Next week's post with look at the extended definition of evaluation and the implications for school research leads.

References and further reading

Stufflebeam, D.L. & Coryn, C. (2014). Evaluation Theory, Models & Applications (second edition), Jossey Bass, San Franciscon.

*The Joint Committee is a standing committee that was established in the  1975.  Its members have been drawn from 15 professional societies is the United States and Canada that are concerned with improving evaluation in education.

**A future post will examine Stufflebeam and Coryn' extended definition of evaluation the systematic assessment of an object's worth, merit, probity, feasibility, safety, significance and or/equity (p11) and the role of values in evaluation.

New : A spiral of disciplined inquiry

If you are a teacher, head of department or headteacher and interested in disciplined inquiry, this post is for you.  In this post, I will examine Timperley, Kaser and Halbert's (2014) Spiral of Inquiry, and consider the implications for teachers and school research leads undertaking practitioner inquiry. Kaser and Halbert (2009, 2013)) and Timperley (2011) have previously written about 'spirals of inquiry'.  However, they argue that the 2014 version of the model is different from previous iterations

• By placing learners at the centre of inquiry
• Recognising that inquiry is both an individual, joint and collective endeavour - which can take place not just within a school but between schools
• The model builds on emerging knowledge from the learning sciences (Dumont, 2010) 
• It's about changing the experiences of learners through new learning and new action.

So what is a spiral of inquiry?

Timperley et al describe the spiral of inquiry process as a multi-tiered and multi-connected process, which is framed by two central questions :What’s going on for our learners?  How do we know? (p7) and is illustrated in the following figure

Phase 1 Scanning
During the scanning phase the focus is on what's going on for your learners.  It's about being open and seeking new sources of evidence.   When focussing on what's going on for your learners, the focus is not just on the academic outcomes of learning but includes:

• asking learners where they are going with  their learning; 
• the social nature of learning; 
• the role of emotions in learning; identifying and acknowledging individual differences in learners; stretching all learners to make a contribution; 
• are learners able to identify what quality works like - and what they need to do to improve; 
• and, can learners make connections across subject areas but also within and outside the school.  finding out what's going on just for learners, but with parents, families, the community and other stakeholders.

Phase 2 Focussing
The phase of the spiral inquiry involves asking the question : Where are we going to concentrate our energies so that we can deliver the experience and outcomes for our learners (p10).  In doing so, this will involve:

• using information to identify a focussed and concentrated area for learning and action; 
• seeking more information to gain a deeper more accurate understanding of the situation; takes advantages of existing strengths and capabilities as well as identifying areas for development and further capacity building; 
• identifies a common theme which participants can 'own' and engage with.

Phase 3 Developing a hunch
In this phase the focus is on How are YOU contributing to this situation? (p13).  Hunches are often based on intuition and are not grounded in evidence.  In this phases, we must have the confidence to bring these hunches to the fore, and explore the underpinning assumptions and evidence.   However, to explore deeply held views, will require 'emotionally safe learning environments for teachers, so these hunches can be explored without fear.   As such developing hunches is concerned with with:

• getting values and beliefs about your own teaching practice out into the open; 
• identifying those practices that are within your domain and you can do something about;
• checking our assumptions, our evidence, and our initial interpretations for validity before taking the next step.

Phase 4 New Learning
This phases involves asking a number of inter-related questions : what do you need to learn; how do we go about learning it; and where can learn more about what to do.  This inevitably involves engaging with the latest research literature on learning and what makes a difference for our learners.  However, it is recognised, that works somewhere else may need to be adapted if not transformed, if it is to work in a different setting.  For this part of the spiral to work it requires;
learning to be connected to the needs of our learners;

• linked to the issues emerging from the other phases of the spiral; 
• having a clear understanding of why new approaches may be more beneficial for learners than previous approaches;
• 'learning' being supported over the long-run with both time and resources.

Phase 5 Taking action
This phase consists of taking informed action which will make enough of a difference.  However, this action involved learning in depth about new and innovative ways of doing things and which is informed by a profound awareness of why these new practice are more worthwhile than previous actions.  It is also about evaluating the impact of the new practices on learners, whilst at the same time creating conditions of openness and trust amongst teachers.

Phase 6 Checking
Finally the phase involves asking 'Have you made enough of a difference?' Checking whether you have made enough of a difference in an integral part of evidence-informed inquiry.  It is also about others having accessing to information about whether the innovation has made a difference to learners.  And last but not least, it's about what comes next.

Some initial observations
The spiral of inquiry is an incredibly useful model to help school research leads support colleagues engage in teacher inquiry.  It provides a well-structured and thoughtful process, which captures some of the messiness of the inquiry process.  The model also provides a way of thinking collaborative activity with colleagues both within and between schools. Nevertheless, as with all such models, there are always matters for further consideration, which may have been omitted or require further discussion.    As such, three issues require further consideration: the expertise and skill levels of the teacher participants; connections with evidence-based medicine; and, the evaluation and checking of impact

Expertise and skill levels of the teacher participants
Timperley et al state that : The spiral of inquiry provides a mental model this is systemic but not simplistic (p23), and I would agree.  However, the various phases of the spiral will require the development of sophisticated skills of inquiry, and which would be incorrect to assume that many, if not all, teachers have the appropriate skill level and expertise.  Skills required include: the ability to understand the various mental models being used to generate the assumption which underpin the development of well-formed hunches; the ability to interrogate the 'academic literature' for ideas, processes and interventions which may be usefully deployed to meet the challenge at hand; or finally, the skills necessary to gain genuine insight of the concern of learners, parents and the

Connections with evidence-based medicine
For me, a core weakness of much of what is written about evidence-informed teacher inquiry derives from a lack of awareness of some of the processes associated with evidence-based practice, and it could be said, that education already has plenty of spirals, but often this represents an unnecessary re-invention of the wheel.   An example of this is Timperley et al's claim that placing learners, and other stakeholder such as parents, at the centre of inquiry is an inherently novel feature of the spiral of inquiry. Unfortunately, evidence-based medicine has since its very inception placed significant emphasis on the needs, values and concerns of patients. For example, Straus et al al (2011) describe evidence-based medicine as requiring ... the integration of the best research evidence with our clinical expertise and our patient's unique values and circumstances.

Checking and have we made enough of a difference  
The questions which underpins the checking phase of the spiral, is whether we have 'made enough of a difference?  In particular, Timperley et al argue that only through careful checking can we decide whether we have made enough of a difference.  Checking whether you have made a difference may seem a relatively straightforward task, however,  it is not without its challenges and complexities.  A useful place to start is with Stufflebeam and Coryn's (2014) definition of evaluation, which is as follows: evaluation is the systematic assessment of an object's merit. worth, probity, feasibility, safety, significance, and or/equity. (p12) If we take merit and worth as a starting point, merit being th the intrinsic quality or excellence without reference to costs  and worth-  the quality of the item, taking into account both context and costs.  So the question becomes, not are we making enough of a difference, but rather are we making enough of a difference given both the context and costs.  And, this development of the notion of 'difference' can be continued with each of the criteria - such as significance and feasibility.

To conclude:
The spiral of inquiry is an important contribution to the developing discipline of evidence-based teacher inquiry.  It is an approach which has much potential, particularly given the focus on what is going on for learners.  However, for the potential of the model to be realised,  it needs to be implemented  by practitioners with the necessary skills, knowledge and expertise to make the most of the sophistication of the approach.

References

Dumont, H, Istance, D and Benavides, F (2010) The Nature of Learning: Using Research to Inspire Practice, OECD Publications, Paris.

Straus, S.E., Glasziou, P., Richardson, W. S. & Haynes, B.R. (2011)  Evidence Based Medicine : How to practice and teach it, (4th edition), Churchill Livingston.

Halbert, J and Kaser, L (2013) Spirals of Inquiry for Equity and Quality, BCPVPA Press, Vancouver. Further information available at www.bcpvpa. bc.ca/node/108.

Kaser, L and Halbert, J (2009) Leadership Mindsets: Innovation and Learning in the Transformation of Schools, Routledge, London.

Stufflebeam, D.L. & Coryn, C. (2014). Evaluation Theory, Models & Applications (second edition), Jossey Bass, San Franciscon.

Timperley, H (2011) Realizing the Power of Professional Learning, Open University Press, London.

Timperley, H, Halbert, J and Kaser, J. (2014) A framework for transforming learning in schools: Innovation and the spiral of inquiry, Centre for Strategic Education, Victoria