The School Research and the So What question

As we approach the end of the academic year, school research leads will not doubt  be pulling together a programme of work for the coming year.  Inevitably,  colleagues will be putting forward a range of research questions which they hope to have included in the school’s research/inquiry agenda.  However, as harsh as it may seem the most appropriate initial response to these requests is to ask as  (Booth, Colob, Williams, J., & Fitzgerald, 2016) suggest: So What? Does this question really matter? What would happen if we didn’t answer your question?  Would it make a difference to the life chances of pupils or the well-being of colleagues?

In order to help your colleagues answer the - So What – question, we will borrow a three step process developed by (Booth et al., 2016) in helping researchers develop questions that matter.

1. Name the topic:  I am trying to learn/find out about ………..
2. Ask an indirect question about the topic in order to identify what you do not know about the topic
3. Answer So What? by motivating your question by asking a second indirect questions that explains why you asked your first indirect questions. 

Let’s have a look at a few worked examples

• I am interested in working on marking strategies because I want to find out the most time effective way of carrying out marking because I want to reduce unnecessary work load on teaching staff

• I am interested in children’s play because I want to find out the best way of incorporating play into my teaching as I want to increase children’s independence as learners

• I am interested in understanding the nature of parental engagement with their children’s learning so I can  enlist parents help in increase their children’s effectiveness as readers.

• I am interested in looking into incremental coaching as I want to find a way to develop teachers’ pedagogical skill so as to improve pupil outcomes.

• I am interested in understanding the role of school research champions because I want to find out how best to access academic research because I want teaching and learning to be informed by best currently available research knowledge.

Now just because a colleague cannot immediately answer these three questions, does not mean that his or her topic of interest is without merit.   However, what it does suggest that the idea or question is not yet robust enough to be allocated resources in its support.  Moreover, further refinement of the response to - So What – may lead to a situation where senior leaders care sufficiently about the question – as to make sure it becomes a future priority. 

Reference

Booth, W. C., Colob, G., G, Williams, J. M., J., B., & Fitzgerald, W. T. (2016). The Craft of Research (Fourth Edition). Chicago: The University Of Chicago Press.

The school research lead, bullsh.t and what to do about it

One of the many challenges facing school research leads and other teachers who use social media – be it Twitter, Facebook and/or blogs – is to try and make sure they do not fall victim to bulls..t.   So in this post we will use James Ball’s recently published book Post-Truth: How bullshit conquered the world to help us both what identify bulls..t is and the strategies to adopt so we are less likely to fall victim to it.  However, it should be noted that bullsh.t is not confined to social media and may often be heard, dare I say it, on a regular basis in school staff rooms and senior leadership team meetings

What do we mean by bullsh.t?

Ball draws upon the work of Harry Frankfurt in order to distinguish between lies, untruths and bullsh.t and summarises Frankfurt’s argument as: to tell a lie, you need to care about some form of absolute truth or falsehood, and increasingly public life is run by people who don’t care much either way – they care about their narrative.  (Ball, 2017) p6

Ball goes onto cite Frankfurt who concludes

Someone who lies and someone who tells the truth are playing on opposite sides, so to speak, in the same game. Each responds to the facts as he understands them, although the response of the one is guided by the authority of the truth, while the response of the other defies that authority, and refuses to meet its demands.

The bullshitter ignores these demands altogether.  He does not reject the authority of the truth, as the liar does, and oppose himself to it.  He pays no attention to it all, by virtue of this, bullshit is a greater enemy of the truth than lies are.

Ball argues that a bullshitter will say what works to get the result they want, regardless of whether it is true or not.  Ball goes onto say that in his view that this serves as an accurate description of many political campaigns.   No doubt there are some Tweeters and other social media commentators who would say that given the performative nature of external school accountability regimes this is also an accurate description of much educational leadership in schools and multi-academy trusts.

So how are you to protect yourself from bullsh.t?

Ball suggests a number of strategies – aimed at social media – which you could use to prevent yourself falling victim to bullshit.   However, I argue that these strategies can also be extended to the ‘real’ world and be used in other settings.

Burst your bubble

Ball argues observes that many of us live in an online bubble – where we follow and communicate with people who have the same views as ourselves.   Maybe we need to make sure we follow and meaningfully engage with people who have different views than ourselves.  With the aim of this engagement being to understand the views of others, rather than seeking to convince other as to veracity of our own arguments.   Indeed, in the physical world, the could involve secondary teachers engaging in dialogue and conversation with colleagues in either or both the primary or further education sector.  Alternatively, it may be as simple as just spending some time with a colleague who specialises in a subject to completely different to your own, say a science teacher with an art teacher.

Engage system two

Ball makes reference to System One – instinctive reaction and System Two – more deliberative thinking.  With System One thinking Ball argues that we are far more likely to quickly RT and share Tweets and other social media which are consistent with how we see things.  However, this risks spreading bullsh.t – as we have not subject these Tweets or blogs to appropriate critical scrutiny.  On the other hand, Ball argues, that if we subject some of this material to more critical scrutiny and asking - is there evidence to back up the claims being made; where has the evidence come from; do the authors recognise the limitations of the claims being made – then it means we are far more likely not so share bullshit.

Learn some stats

Ball argues that if you have a very basic grasp of statistics then you are far less likely to fall prone to bullsh.t.  In the context of your work as a school research lead it would be useful to get to grips of the notion of effect sizes and their limitations.   A useful introductory text can be found in the work of (Ellis, 2010) It would probably also be worth dipping into an introductory texts on statistics such as (Cumming & Calin-Jageman, 2017) – which is particularly useful at looking at p-values and confidence limits  and which is supported by a range of videos which can be found on YouTube.

Treat narratives you believe in – just as sceptically as those you don’t

Just because a narrative you believe in is being promoted in social media does not mean that it is right.  We may see tweets or posts which sees coaching as the magic bullet of school improvement or tweet which supports the use of Lesson Study or that the use of research evidence is the key to improving practices in the classrooms.  On such occasions, we may need just to step back and say – is there an alternative view or views on this?  If so, what is it?  Are there any elements of these alternative views which are robust.

Try not to succumb to conspiratorial thinking

Ball argues that in a world of bullsh.t there is a tendency for there to be a rise in conspiratorial thinking – and that others are ‘out to get us’.  Where you are tempted to thinking conspiratorially – it is always worth remembering Hanlon’s Razor – which can be described in a number of ways. For example “Never attribute to malice that which is adequately explained by stupidity" or "Don't assume bad intentions over neglect and misunderstanding."

And some final words

School research leads have a particular responsibility for rooting out ‘bullsh.t’ in schools.  On the other hand, school research leads need to make sure they too are not proponents of other people’s bullsh.t and need to adopt an appropriately self-critical stance of their own work.  This may not be easy, but that does not make it any less vital.

References

Ball, J. (2017). Post-Truth: How bullshit conquered the world. London: Biteback Publishing

Cumming, G., & Calin-Jageman, R. (2017). Introduction to the New Statistics: Estimation, Open Science, and Beyond. Abingdon: Routledge.

Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results: Cambridge University Press.

The school research lead, jobs to be done and what you can learn from the humble milk-shake

In a recent Education Endowment Foundation blog (Yeomans, 2017) states:

New research in education makes for interesting reading. But ultimately, commissioning and conducting it is only worthwhile if it is useful to teachers and can inspire helpful changes in their everyday practice – either leading to new practices being adopted, tweaks to what they currently do, or a reduction in the time spent on ineffective approaches.

So in this post, we will use the work (Christensen, Dillon, Hall, & Duncan, 2016) and their notion of ‘job-theory’ to try and understand the challenges of translating research outcomes into changes in classroom practice.  In doing so, we will consider how the humble ‘milk-shake’ maybe an essential element in trying to understand how to bring about more evidence-informed classrooms.

Job Theory and Milk Shakes

(Christensen et al., 2016) argue that job theory is based on the notion that if we use a product or service, we are essentially ‘hiring’ something to get a job done.  In this context, (Christensen et al., 2016) view a ‘job to be done’ as something that helps an individual make progress in his to her working life.  If the product does the job well, then in similar circumstances we are likely to use that product again.  On the other, hand if the product does a poor job we are far less likely to use it again, and will look for something else to help us make progress.   

(Christensen et al., 2016) go onto explain the notion job-theory of  by examining why individuals may purchase a ‘milk-shake’ on his or her commute to work.    Or put another way, I wonder what job arises in people’s lives that causes them to come to this restaurant to ‘hire’ a milk-shake?  Using this perspective market research was undertaken in a fast-food restaurant watching: What time people bought milk-shakes? How were they dressed? Who were they with? Did they buy food as well, or did they just drive off with it.

The research found that a more than expected number of milk shakes were being sold to individuals who came into the fast-food restaurant before 9.00 am, did so alone, only purchased a milk-shake, and then drove off with it.    These individuals were then asked what ‘job’ they were trying to do for themselves that causes them to stop and buy a milk shake.  Initially, the customers found it difficult to answer the question, but eventually the researchers were able to come up with an answer.  Customers were buying milkshakes to ‘Help me stay awake and occupied while I make my morning commute more fun.’

Of course, buying a milk-shake to make your morning commute more interesting isn’t the only reason why people buy milk-shakes in the morning or at other times of day.  However, it does illustrate the what causes them to undertake a specific course of action in a particular circumstance in order to make progress in some aspect of his or her life.

So as (Christensen et at, 2016) note - a job would appear to have two key elements: progress and circumstance.  Progress represents movement towards achieving some aim, objective or aspiration, and a job is nearly always being a process rather an event to enable progress.  On the other hand, circumstance represents the specific context in which the need for progress arises.  As such circumstance takes into account a range of issues, for example, in what context does the job take place, who needs the job to be done, when does the job need to done. 

In addition, a job is not only about progress in a given circumstance, but also emotional and social dimensions – be it what people think is important, what they believe, how they relate to others and how they want to be seen by others.

Job Theory and Research Use

So how can we use job theory to think about how we can close the research-practice gap. Well one way is for commissioners/publishers and producers of research to pose a number of questions about the potential impact and transferability of the research

How will the research help teachers make progress in their working lives?  This progress could take multiple forms, for example, improved pedagogical knowledge, pedagogical content knowledge or subject expertise,

Who will use the research? At what stage of his or her career? In what type of school, with what type of pupils? When will they use it? How often are they likely to use it?

What’s currently getting in the way of teachers making the progress they wish to make.  Will the research help them get over these obstacles?

How are teachers making do with sub-optimal solutions by ‘cobbling’ together something that just about works?   Or are they not doing anything at all?

How would teachers define what ‘quality’ means for a better solution, and what tradeoffs are they willing to make (adapted from Christensen et al., 2016)

Alternatively, you may be a school research lead screening and reviewing research evidence which may or may not be used in your school.  This set of questions will help you identify why colleagues might ‘hire’ the research so that they can make progress in his or her working life.  If you can’t identify the ‘job to be done’ or the circumstances in which that ‘job’ will take place, then it’s highly unlikely that your colleagues will ‘hire’ the research. 

And some final words

Closing the research-practice gap in schools is never going to be easy.  It is tempting to spend time thinking about how to improve the communication of the research or how we might make the research better – or in the case of our milk-shake – how do we let the world know we’ve made a better milk-shake.  However, unless that research help teachers get a job done – no matter how interesting and flavoursome the research – it’s unlikely to be used.

References

Christensen, C. M., Dillon, K., Hall, T., & Duncan, D. S. (2016). Competing Against Luck: The Story of Innovation and Customer Choice.

Yeomans, E. (2017). What happens next? Generating evidence is a start, helping teachers use it is the real challenge

Retrieved from https://educationendowmentfoundation.org.uk/news/what-happens-next-generating-evidence-is-a-start-helping-teachers-use-it-is