This post seeks to examine some of the issues associated with using national benchmark data as a means of evaluating the performance of general further education colleges and consider whether Phil Rosenzweig's book The Halo Effect .... and the eight other business delusions that deceive managers can help our thinking about college performance.
As in previous posts I have used Statistical Process Control techniques to produce a funnel plot of general further education colleges' 16-18 long course success rates for the 2012-13 academic year.
Using ±3SD from the mean as the best estimate of expected variation, we can see that there are a large number of colleges, whose performance could be deemed to illustrate 'special-cause variation'.
However, there are limits with funnel plots, as with approximately 50% of colleges being outside the control limits it would suggest 'over-dispersion'. This level of dispersion indicates that there are a range of factors at work, which need to be taken into account when seeking to understand variations in college performance.
Furthermore, it this level of over-dispersion also suggests that we need to avoid some overly-simplistic explanations of differences in performance between colleges. Phil Rosenzweig in his book The Halo Effect .... and the eight other business delusions that deceive managers identifies a range of errors of logic or flawed thinking which distort our understanding of company (college) performance. The most common delusion is the Halo Effect, when a company's sales and profits are up (a college's retention, achievement and success rates) people (inspectors or significant other stakeholders) may conclude that this is the results of brilliant leadership and strategy or a strong and coherent corporate (college) culture. When performance falters, (success rates or position in leagues tables fall) they conclude it is the result of weak leadership and management (the Principal), and the company was arrogant or complacent (the college was coasting). Whereas the reality may be that little may has changed and that the college/company performance creates a HALO that shapes the way judgements are made about outcomes for learners, teaching learning and assessment, and leadership and management.
Rosenzweig identifies eight other delusions, which I would are argue are often seen in the FE sector and education at large. In the next section I have amended each of the delusions with the use of an educational example.
- The delusion of correlation and causality : two things may be correlated, that does not mean that one causes the other, for example, does a coherent college culture lead to an outstanding college, or is a coherent college culture the product of success.
- The delusion of the single explanation : there may be multiple explanations that lead to improvement, a clear vision, a focus on teaching and learning strategies, better leadership - and many of these factors are highly correlated, so focussing on one particular explanation may lead other possible or multi-causal explanations being ignored.
- The delusion of connecting the winning dots : if we look at successful colleges to try and see what they have in common, that does not mean the commonality explains success
- The delusion of rigorous research : Is the data available for making judgements about performance is as good as we think it is, for example, lesson grade profiles which we know are probably neither reliable or valid.
- The delusion of lasting success : Many once successful colleges will decline over time. What worked in the past, may not work now, and certainly may not work for the six years between inspections for outstanding colleges
- The delusion of absolute performance : College performance is relative, not absolute. A college can improve and still fall behind other colleges at the same time.
- The delusion of the wrong end of the stick : Some college's may have pursued a highly focused strategy eg sixth form colleges but that does not necessarily lead to success (see previous post)
- The delusion of organisational physics : College performance is a complex process, with feedback loops etc and changes often lead to unexpected consequences. Changes in success rates may be the product of changes in the nature of external assessment regimes rather than any specific organisational change.
Given the increasing pervasiveness of business management conventions within education, it would be surprising if the FE sector had escaped delusions that have deceive businesses and their managers.
Future posts will look at more detail particular aspects of management conventions and seek to compare them with the research literature to see if we can generate non-delusional management practice which is firmly based upon the research evidence.