Helping School Research Leads Ask Better Questions Part Three - The CIMO Format

In my last two posts I have looked at how the PICO format, which has been borrowed from evidence-based medicine, can help school research leads help colleagues ask well formulated and answerable questions.  In this post I will explore a similar approach,  
developed by Denyer & Tranfield (2009) for use in a social science or organisational context.  Denyer and Tranfield argue that well-crafted review questions need to take into both the organisational context and the relationship between an intervention and an outcome.  Adapting the work of (Pawson, 2006),  Denyer and Tranfield have developed a structured and contextual approach to developing an answerable question (CIMO) and which provides a better focus on both the context and the mechanism(s) by which change is brought about.

What is CIMO?

CIMO is an acronym for the the components of a well-formulated question for use in a social science or organisational context.

C — Context. Which individuals, relationships, institutional settings, or wider systems are being studied?

I — Intervention. The effects of what event, action, or activity are being studied?

M — Mechanisms. What are the mechanisms that explain the relationship between interventions and outcomes? Under what circumstances are these mechanisms activated or not activated?

O — Outcomes. What are the effects of the intervention? How will the outcomes be measured? What are the intended and unintended effects?

Denyer and Tranfield provide a worked example of a question framed with these components: 

“Under what conditions (C) does leadership style (I) influence the performance of project teams (O), and what mechanisms operate in the influence of leadership style (I) on project team performance (O)?” (Denyer and Tranfied, 2009 p 682)

Educational Examples

Using these elements it is now possible to frame answerable questions, several examples of which can be found below.

Under what circumstances does a further education college middle manager’s leadership style influence the academic performance of students, and what are the mechanisms of middle management leadership style which affect student performance (adapted from (Denyer & Tranfield, 2009)

Under what conditions does re-taking GCSE English provide an effective mechanism for developing 16 year old full-time further education students’ English skills, where those students previously achieved a grade D?  What are the processes associated with re-sitting GCSE English which affect English skills 

Is the use of flipped learning an effective mechanism for engaging  full-time 16 year old level one further education college students effective in reducing the risk of non-attendance, where there has previously been a history of non-attendance in school.  
What are the mechanisms of flipped learning which affect student attendance.

Under what circumstance are graded lesson observations effective in improving lecturers teaching where those teachers have previously been judged to be inadequate or requiring improvement, and what are the mechanisms of graded lesson observation which affect teacher performance.

So what are benefits of using CIMOs and phrasing questions in such a manner?. 

A number of benefits spring immediately to mind:

1. CIMO provides a framework for formulating problems in a structured manner, and the very process of developing the question promotes understanding of the issue at hand.
2. By formulating questions in this manner is that subsequently provides the basis for undertaking a systematic review and in particular provides guidance as to what literature to review and the data to be considered.
3. CIMOs provide  a basis for allowing researchers/bloggers and tweeters to attempt to agree the question to which they are trying to contribute.

In future posts I will consider how school research leads can help colleagues critically appraise the relevant literature and produce critically appraisal of various topics.

Denyer, D., & Tranfield, D. 2009. Producing a systematic review. In D. Buchanan, & A. Bryman (Eds.), The SAGE handbook of organizational research methods: 671-689. London: SAGE Publications Ltd.

Pawson, R. 2006. Evidence-based policy : A realist perspective. London: Sage Publications.

School Research Leads and Asking Better Questions - Part Two

In my previous post I adapted the work of Strauss et al (2011)  to help school-based research leads go about the task of devising well formulated questions.  In this post I intend to look further at the differing types of questions that can be asked.   But first a quick recap of the PICO format.   PICO is an acronym for the components of a well-formulated question and which are as follows:

P — Pupil or Problem. How would you describe the group of pupils or problem?
I — Intervention. What are you planning to do with your pupils?
C — Comparison. What is the alternative to the intervention (e.g.different intervention)?
O — Outcomes. What are the effects of the intervention?

PIOC and Different Question Types

Having created a well-formulated question, it is worth reflecting on the type of question-which can be created and which has been adapted from (Stillwell, Melnyk, & Williamson, 2010).

Pedagogical intervention - To determine which pedagogical intervention leads to the best outcome for pupils/outcomes

• For students requiring students requiring additional learning supported how does the provision of 1 to 1 support compared with group support affect retention rates in the first term?

Etiology - To determine greatest success/risk factors

• Are level 3 BTEC Extended Diploma students who have grade C or above in GCSE Mathematics compared with those students who do not, more likely to successfully complete their two year programme of study?

Diagnosis - To determine which test is more accurate in diagnosing learning  needs

• For post 16  students requiring support in the development of English skills,  are GCSE grades a better indicator of needs compared to specific on-line screening tool (eg BKSB)?

Prognosis or prediction To determine the course over time and likely complications of a particular pedagogical intervention

• Do weekly tutorials for students with poor records of attendance improve timely completion of coursework within three months of the initiation of the weekly tutorials?

Meaning To understand the meaning of an experience for a particular group of  students

• How do further educations students with grade D or below in GCSE English perceive re-siting GCSE English during the first year of post-16 education?

How to ask the most appropriate question?

Straus et al 2011 have suggested a series of filters which could be used to identify the most appropriate question to ask in a particular situation.  I have adapted the suggested filters so they can be easily transferred to an educational setting.

• Which question, if answered, will be most useful for our pupils/students/learners' well being - academic or personal?
• Which question will be most useful for subject leaders, heads of department in gaining a better understanding of the issues at hand?
• Which question will be most useful in helping to improve the department, school or college?
• Which question is most likely to re-occur and will need to be revisited in the future?
• Which question is most interesting to you as an evidence-based practitioner and contribute most to your personal professional development?

I'm sure that some colleagues are saying that they do not have sufficient time to formulate questions in such a structured manner.  On the other hand, as Strauss et al (2010) so clearly articulate - again amended for evidence-based educational practitioners - there are a number of clear benefits of such an approach:

1. Focussing our scarce professional development time on the needs of our learners.
2. Making it easier to communicate with colleagues.
3. As our knowledge grows we are role-modelling to our colleagues what it looks like to be a research informed practitioners

I hope you agree with me that by asking better questions we can expand our skills as evidence-based practitioners.  In future posts  I will continue to explore the challenge of asking better questions and will be drawing upon a number of differing perspectives.

References
Straus, S.E., Glasziou, P., Richardson, W. S. & Haynes, B.R. (2011)  Evidence Based Medicine : How to practice and teach it, (4th edition), Churchill Livingston.
Stillwell, S. B., Melnyk, B. M., & Williamson, K. M. (2010). Asking the Clinical Question : A key step in evidence based practice. American Journal of Nursing, 210(3).

The school research lead and asking better questions - part one

At the recent ResearchED Research Leads one-day conference one of the tasks that emerged for the role of the school-research lead is to help colleagues ask well-formulated and answerable questions. Indeed, there was some discussion as to whether this required input from a higher education institution.  Fortunately for both current and prospective research leads there is a wealth of material produced by the evidence-based medicine movement (Straus, S, Glasziou, P., Richardson, W.S.and Haynes, R.B (2011) Evidence-Based Medicine : How to practice and teaching it, 4th edition) which can help with the development of well-formulated and answerable questions, which I will now adapt for the use of school research leads.

Background and Foreground Questions

Straus et al state that in the first instance it is necessary to make the distinction between background and foreground questions, with background questions being made up of two parts:
* A question root (who, what, how, when, how ) with a verb
* An issue or matter of interest

So examples of educational background questions could be:

* How does homework improve student achievement?
* What are the benefits of e-learning?
* When is the best-time to give students diagnostic tests?
* Who is best placed to undertake performance reviews and appraisals?
* Where can you find examples of effective 'flipped' learning

Whereas a foreground question asks far more specific questions about a particular action, intervention or innovation, for example, does 24/7 access to iPads compared to the use of Chromebooks improve the timely completion of homework tasks. 

The PICO Format

Invariably foreground questions can be constructed using the PICO format developed for evidence-based medicine and which contains 4 basic elements which I have adapted for use in schools.

P —  Pupil or Problem. How would you describe the group of pupils or problem?
I  —  Intervention. What are you planning to do with your pupils?
C —  Comparison. What is the alternative to the intervention/action/innovations
O —  Outcomes. What are the effects of the intervention/action/intervention?

Two examples of foreground questions could be :

1. For pupils requiring additional learning support (P) how does the provision of 1 to 1 support (I) compared with group support (C) affect achievement rates.
2. For pupils aged 16 who failed to achieve at least at a grade C in GCSE English (P) and subsequently retake GCSE English (I) at the end of the academic year, how well do they achieve (O) compared to students who have been prepared and entered for iGCSE English (C)

Benefits for school-research leads
So what are the benefits for school research leads of using the PICO format, well for me three benefits come to mind.

1. PICO provides a mechanism for developing answerable questions, which should make it easier to pinpoint the relevant evidence necessary to answer the question. 
2. PICO provides a structure by which the staff who are being supported by research leads are able to ask better questions for themselves.
3. PICO should reduce the time-pressure on research leads as it will lead to far more empowered colleagues.

Given these provisional benefits of using the the PICO format,  in future posts I will look at extending the use of the PICO format to help ask a far greater range of well-formulated questions.

Maturity and the case of graded lesson observations - can the FE sector handle the truth!

Last Friday saw the TES publish an article on graded lesson observations within the further education sector where Lorna Fitzjohn, OfSTED's national director for learning and skills stated : “The big question that we’ve had from some leaders in the sector is: is the sector mature enough at this point in time not to have us (OfSTED) grading lesson observations?”  In responding to this statement it seems sensible to ask the following questions:

1. What does the research evidence imply about the validity and reliability of graded lesson observations?
2. What is the best evidence-based advice on the use of graded lesson of observations?
3. What do the answers to the first two questions imply for the maturity of the leadership and management of the further education sector?

What does the research evidence imply about the validity reliability of lesson observations?

      O’Leary (2014) undertook the largest ever study into lesson observation in the English education system and investigated the impact of lesson observations on lecturers working in the FE sector. Lecturers' views when asked about graded lesson observations were summarised as :

Over four fifths (85.2%) disagreed that graded observations were the most effective method of assessing staff competence and performance. A similarly high level of disagreement was recorded in response to whether they were regarded as a reliable indicator of staff performance. However, the highest level of disagreement (over 88%) of all the questions in this section was the response to whether graded observations were considered the fairest way of assessing the competence and performance of staff. 

     Not only are there 'qualitiative' doubts about the reliability and validity of lesson observations, there are also 'quantitative' based objections to their use, particularly when un-trained observers are used. Strong (2011) et al found that the correlation coefficient for untrained observers agreeing on a lesson observation grade was 0.24 which runs counter to sector leaders advice to  both Principals and Governors to 'trust your instincts.'
     Furthermore, Waldegrave and Simons (2014) cite Coe’s synthesis of a number of research studies, which raises serious questions about the validity and reliability of lesson observation grades. When considering value added progress made by students and a lesson observation grade (validity) , Coe states that in the best case there will be only 49% agreement between the two grades and in the worst case there will be only 37% agreement. As for the reliability of grades Coe’s synthesis suggests that in the best case there will be 61% agreement and in the worst care only 45% agreement between two observers.

      The above would suggest graded lesson observations provide an extremely shaky foundation on which to make judgements about the quality of teaching and learning within the further education sector.   It is now appropriate to turn to the current best evidence-based advice on the use of graded lesson observations.

So what is the best evidence-based advice on the use of lesson observations?

     Earlier this year at the Durham University Leadership Conference Rob Coe stated that the evidence suggests

Judgements from lesson observation may be used for low-stakes interpretations (eg to advise on areas for improvement) if at least two observers independently observe a total of at least six lessons, provided those observers have been trained and quality assured by a rigorous process (2-3 days training & exam). High-stakes inference (eg Ofsted grading, competence) should not be based on lesson observation alone, no matter how it is done

      In other words, the current practice of a graded lesson observation system based on one or two observations per year - which is the case in the vast majority of colleges - is not going to provide sufficient evidence for low stakes improvement never mind high stake lecturer evaluation.  That does not mean lesson observations should not take place but as Coe suggests they need to take place alongside reviews of other evidence : such as student feedback, peer-review and student achievement.

  

So what does the above mean for the leadership and management of the further education sector?

     First, there would appear to be a clear research-user gap, with sector leaders giving out advice which is inconsistent with the best available evidence. Second, if leaders and managers within the sector have to rely upon graded lesson observations to attempt to demonstrate and track improvement of teaching and learning, then the sector needs to begin to develop a wider and more sophisticated range of processes and measures to judge the quality of teaching and learning. Third, the reliance on graded lesson observation suggests a prevailing leadership and management culture which is founded on an assumption that to improve performance requires the removal of poor performers.  For me, Mark Tucker (2013) sums up the way forward

There is a role for teacher evaluation in a sound teacher quality management system, but it is a modest role. The drivers are clear: create a first rate pool from which to select teachers by making teaching a very attractive professional career choice, provide future teachers the kind and quality of education and training we provide our high status professionals, provide teachers a workplace that looks a lot more like a professional practice than the old-style Ford factory, reward our teachers for engaging in the disciplined improvement of their practice for their entire professional careers, and provide the support and trust they will then deserve every step of the way.

And if we take that as the definition of a mature system, then maybe FE has a long way to go.

References
Coe, R, Lesson Observation: It’s harder than you think, TeachFirst TDT Meeting, 13th January 2014
O’Leary, M. (2014). Lesson observation in England’s Further Education colleges: why isn’t it working and what needs to change? Paper presented at the Research in Post-Compulsory Education Inaugural International Conference: 11th –13th July 2014, Harris Manchester College, Oxford.
Strong, M., Gargani, J. and Hacifazliog, O. (2011). Do We Know a Successful Teacher When We See One? Experiments in the Identification of Effective Teachers, Journal of Teacher Education 62(4) 367–382.
Tucker, M. (2013). True or false: Teacher evaluations improve accountability, raise achievement. Top Performers (Education  Week blog), July 18, 2013 http://www.ncee.org/2013/07/true-or-false-teacher-evaluations-improve-accountability-raise-achievement/
Waldegrave, H., and Simons, J. (2014). Watching the Watchmen: The future of school inspections in England, Policy Exchange, London.

 

Research Leads Conference and Evidence-Based Practice - How to avoid re-inventing the wheel

On Saturday I had the privilege of attending ResearchED's Research Leads one-day conference.  It was an incredible day full of intellectual challenge mixed with the opportunity to meet some wonderful colleagues.  However, what was deeply ironic given that the event was being held in the Franklin Wilkins building at Kings' College London, and which is named after two of the discoverers of the 'double-helix', was that we were not standing on the 'shoulders of giants'.  Many of the issues and topics which we wrestled with have already, to a very large extent, been engaged with by practitioners in the fields of evidence-based medicine and more generically evidence-based practice.
     So how do we go about standing on the shoulders of giants, particularly with reference to the role of a research-lead in a school and how research-leads could go about performing their task.  Both Alex Quigley and Carl Hendrick did an admirable job in trying to map out the territory for the research-lead and some of the roles and tasks that need to be performed.  For me a fantastic starting point for building upon this discussion is the work of Barends, Rousseau, & Briner's (2014) and their pamphlet on the basic principles of evidence-based management.  Barends* et al state:

From the point of view of a school-research lead this definition has a number of implications.

1. Being a research-lead involves more than just 'academic' research, it involves drawing upon a wider range of evidence - including both individual school data and the views of stakeholders, such as staff, pupils, parents and the wider community.
2. This approach can be largely independent of work with higher education institutions.  It is not necessary to partner with a HEI to be an evidence-based practitioner or evidence-based school. It might help, specifically when supporting staff develop as evidence-based practitioner but it's not absolutely necessary.
3. At the heart of this process is being able to translate school issues or challenges into a well formulated and answerable problem and evidence-based medicine has a number of tools which can help with this process.

      So what does this mean for the research lead's role in a school? Well a highly relevant place to start is to look at those leadership capabilities which appear to have the biggest impact on student and in this case teacher learning.    Given that we a seeking to develop processes which are 'research' led there is no better place to start than the work of Viviane Robinson on Student-Centred Leadership, which is based on both a best-evidence synthesis of the impact of leadership on student achievement, and which identifies three key leadership capabilities: applying relevant knowledge; solving complex problems and building relational trust.  Below I have begun to  tentatively examine what this might mean for research leads, so here goes:

• Applying relevant knowledge - this about using knowledge about effective research in order to help colleagues become better practitioners, for example, helping to ask well-formulated and answerable questions.
• Solving complex problems - research is a tricky and slippery topic and it's about discerning the specific research challenges faced by a school and then crafting research processes in order to address those challenges.
• Building relational trust - which is an absolute pre-requisite for such work.  Research leads need to gain the trust of colleagues, otherwise it will be virtually impossible to develop a research agenda in schools.  And in undertaking this role, this require a specific set of skills which will allow both the research lead and practitioner to engage in effective evidence-based practice.

In order to help with the development of the role of the research-leads the next few posts in this series will draw upon material from both the evidence-based medicine and the broader field of evidence-based practice to look at the following:

• How to get better at asking well-formulated questions.
• How to write a short-paper on a Critically Appraised Topic
• Some of the current challenges faced by evidence-based practitioners, particularly in the field of medicine.

*This definition is partly adapted from the Sicily statement of evidence-based practice: Dawes, M., Summerskill, W., Glasziou, P., Cartabellotta, A., Martin, J., Hopayian, K., Porzsolt, F., Burls, A., Osborne, J. (2005). Sicily statement on evidence-based practice. BMC
Medical Education, Vol. 5 (1).
References
Barends, E., Rousseau, D. M., & Briner, R. B. 2014. Evidence-Based Management : The Basic Principles. Centre for Evidence Based Management (Ed.). Amsterdam.
Robinson, V. (2011) Student-Centered Leadership, Jossey-Bass, San Francisco

Dangerous Half Truths and Total Nonsense - Evidence Based Practice and two avoidable mistakes

A few weeks ago I wrote about how difficult it is to learn from others using casual benchmarking and the implications this had for leadership teams who followed the Skills Commissioners advice to visit other colleges.  In this post I continue to draw upon the work of Jeffrey Pfeffer and Robert I Sutton and their 2006 book Hard Facts Dangerous Half-Truths and Total Nonsense : Profiting from evidence-based management  and some of the practices which are confused with evidence based management. In addition to casual benchmarking Pfeffer and Sutton identify two other practices - ,  doing what seems to have worked in the past and following deeply held yet unexamined ideologies -  which can lead to both to poor decisions and detrimental outcomes for college stakeholders i.e. staff, students, parents, employers and the community.   These two practices are now explored in more detail.

Doing what seems to have worked in the past
     As colleagues move one from role to another, be it within a school or college the temptation is use that experience and apply it to a new setting.  Pfeffer and Sutton argue that problems arise when the new situation is different from the past and when what we have learned worked in the past, may have actually have been wrong.  There is a huge temptation to import management practices and ideas from one setting to another without sufficient thought about context, either old or new.   Pfeffer and Sutton identify three simple questions that can assist in avoiding negative outcomes arising from inappropriately repeating past strategies and innovations.

• Is the practice that I am trying to import into a new setting, say lesson planning, peer tutoring, e-learning, pastoral arrangements or information and guidance arrangements directly linked with previous success.   Was the success which is now being attempted to be replicated despite of the innovation which was adopted rather than because of it?
• Is the new - department, school, college or other setting - so similar to past situations that what worked in the past will work in the new setting?  For example, getting the balance right between delegation and control may be different between an outstanding school/college and one that is on a journey of improvement.
• Why was past practice  say graded lesson observations - effective, what was the mechanism or theory which explains the impact of your previous actions.  If you don't know what was the theory or mechanism that underpinned past success then it will be difficult to know what will work this time.  (Adapted from Pfeffer and Sutton p9)

Following deeply held yet unexamined ideologies 
     The third basis for flawed decisions-making is deeply held beliefs or biases which may cause school and college leaders to adopt particular practices, even though there maybe little or no evidence to support the introduction of the proposed practice or innovation. Pfeffer and Sutton suggest a final three questions to address this issue.

• Is my preference for a particularly leadership style or practice because it fits in with my belief about people -  for example, do I support the introduction of PRP for teachers because of my belief of the motivational benefits of higher salaries and payment for results?
• Do I require the same level of evidence or supporting data, if the issue at hand is something that I am already convinced what the answer is?
• Am I  allowing my preferences to reduce my willingness to obtain and examine data and evidence which may be relevant to the decisions.  Do I tend to read the educational bloggers/tweeters who agree with me or do I search out views that are in contrast to my own.  Am I willing to look at research which challenges my preconceptions for example Burgess et al (2013) and the small but positive effect of league tables.(Adapted from Pfeffer and Sutton, p 12)

To conclude :

     If an evidence based approach is taken seriously it could potentially change the way in which every school/college leader thinks about educational leadership and management, and subsequently change the way the behave as school/college leader.  As Pfeffer and Sutton state:
     First and foremost, (evidence based management) it is way of seeing the world and thinking about the craft of management.  Evidence-based management proceeds from the premise that using better, deeper logic and employing facts to the extent possible permits leaders to do their jobs better. Evidence based management is based on the belief that facing the hard facts about what works and what doesn't, understanding dangerous half-truths that constitute as much conventional wisdom about management, and rejecting the total nonsense that too often passes for sound advice will help organizations perform better (p13)

References
Burgess, S., Wilson, D., and Worth, J. (2013) A natural experiment in school accountability the impact of school performance information on pupil progress, Journal of Public Economics, vol 106, pp 57-67

Pfeffer, J and Sutton, R., (2006) Hard Facts Dangerous Half-Truths and Total Nonsense : Profiting from evidence-based management. Harvard Business Review Press.

What Works? Evidence for decision-makers - it's not quite that simple.

This week saw the publication of the What Works Network's What Works? Evidence for Decision-Makers report which in the context of education identified a number of practices, for example, peer tutoring and small-group tutoring which appear to work.  On the other hand, the report also identified a number of practices which appeared not to work, for example, giving pupils financial incentives to pass GCSE or pupils repeating a year.  However, whenever such  reports come out which identify supposedly what works there can be tendency to confuse 'evidence' with evidence-based practice.  In particular, there is a danger that published research evidence is used to reduce the exercise of professional judgment by practitioners.  Indeed, one of the great myths associated with evidence-based practice is that it is the same as research determined or driven practice.  As such, it is potentially useful for educationalists interested in evidence-based education to gain a far greater understanding of what is meant by evidence-based practice.
      Barends, Rousseau, & Briner's (2014) recent pamphlet on the basic principles of evidence-based management  define evidence based practice as the making of decisions through the conscientious, explicit and judicious use of the best available evidence from multiple sources by:

• Asking: translating a practical issue or problem into an answerable question
• Acquiring: systematically searching for and retrieving the evidence
• Appraising: critically judging the trustworthiness and relevance of the evidence
• Aggregating: weighing and pulling together the evidence
• Applying: incorporating the evidence into the decision-making process
• Assessing: evaluating the outcome of the decision taken

to increase the likelihood of a favourable outcome (p2)*.

In undertaking this task information and evidence is sought from four sources

1. Scientific evidence Findings from published scientific research.
2. Organisational evidence Data, facts and figures gathered from the organisation.
3. Experiential evidence The professional experience and judgment of practitioners.
4. Stakeholder evidence The values and concerns of people who may be affected by the decision.

In other words, evidence based practice involves multiple sources of evidence and the exercise of sound judgement.
     Furthermore, drawing upon Dewey's practical epistemology Biesta argues that the role of research is to provide us with insight as to what worked in the past, rather than what intervention will work in the future.  As such, all that evidence can do is provide with us a framework for more intelligent problem-solving.  In other words, evidence cannot give you the answer on how to proceed in any particular situation, rather it can enhance the processes associated with deliberative problem-solving and decision-making.
     So what are the key messages which emerge from this discussion.  For me at least there would be appear to be four key points

• Research evidence is not the only fruit - in other words when engaging in evidence based-practice research evidence is just one of multiple sources of evidence.
• Even where there is good research evidence, that does not replace the role of judgment in making decisions about how to proceed.
• All that research evidence can do is tell you what worked in the past, it can't tell you what will work in your setting or in the same setting at some time in the future.
• Research evidence provides a starting point for intelligent problem-solving

References
Barends, E., Rousseau, D. M., & Briner, R. B. 2014. Evidence-Based Management : The Basic Principles. Centre for Evidence Based Management (Ed.). Amsterdam.
Biesta, G. 2007 Why 'What Works' Won't Work : Evidence-based practice and the democratic deficit in educational research, Educational Theory, 57 (1) 21-22