Funnel plots and DIY statistical analysis - Part 2 - Sixth Form Colleges and Inspection Outcomes

Using the same approach and resources (produced by the Public Health Observatories) as described in a previous post I have produced a funnel plot for England's SFCs using the 2012-13, 16-18  long course success rates as the measure.

A common benchmark which is often used is the average level of performance, which in this case is a 16-18 long course success rate of 86.6% .  We can then estimate what is the expected level of variation around the benchmark, which in this example have been set at 2SD and 3SD from the mean, roughly 95% and 99.8 confidence limits.   Initial analysis of the data would suggest there are over 40 possible outliers,  performing at levels above or below what could be expected from 'normal' variations in performance.

In recent months there has been considerable discussion about ensuring Ofsted inspection judgements are informed by a increased level of statistical sophistication, for example, the Policy Exchange's recent report on the future of school inspections.  As such, it seemed reasonable to compare recent SFC inspection judgements with the funnel plot in order to provide a statistical perspective on the inspection grades.

Approximately a quarter of all SFCs have been inspected during the current academic year, and which gives us a range of inspection grades for outcomes for learners and which are summarised below

• 2 colleges' success rates were above the benchmark + 3SD limit (2 good grades for outcomes for learners)
• 9 colleges' success rates were within ± 3SD of the benchmark (7 good and 2 requires improvement grades)
• 12 colleges' success rates were below the benchmark - 3SD limit (3 good, 8 requires improvement, and 1 inadequate)

Given the partial nature of the data available and the recognition that inspection judgements are informed by more than piece of statistical data, it is difficult to draw any specific conclusions.  On the other hand, it should be possible to pose a range of questions which are worthy of further investigation.

• Allowing for the specialist nature of SFCs, more work is required to understand the over-dispersion of performance, with approximately 45% of colleges' success rates being above or below the funnel?
• Given this over-dispersion, how useful are national averages in benchmarking performance?
• Are too many good grades being awarded for outcomes for learners?
• Are there more inadequate SFCs than has previously been thought?
• What would be the implications for inspection grade if robust and statistically sound approaches to data analysis were used.

Finally, all of the above should be seen as highly provisional and as a small contribution to increasing the use of evidence based educational management.