The School Research Lead and Working Out What Works : Distinguishing between merit and worth.

If you work in a school and have anything to do with evaluating teaching, learning and school effectiveness, this post is for you.  This post, the first of two, is aimed at anyone who wants to have a better understanding of how to figure out 'what-works' in schools.  In doing so, I will be using Stufflebeam and Coryn's 2014 book - Evaluation Theory, Models and Applications (second edition) to provide an introduction to educational evaluation.

So what is evaluation?

Stufflebeam and Coryn begin by using a basic definition of evaluation developed by the Joint Committee* in 1994 which describes evaluation as ... the systematic assessment of of the worth or merit of an object**.   Stufflebeam and Coryn argue that given the evaluation's root term is value, then evaluations inevitably involve some form of value judgement.  The Joint Committee's definition attempts to address the values issue by distinguishing between merit and worth, differences which are substantive when engaging in evaluation.

So how do we distinguish between merit and worth?

Stufflebeam and Coryn highlight the different characteristics of merit and worth in Table 1 (2014, p9)

Merit	Worth
May be assessed on any object of interest	Is assessed only on objects that have demonstrated an acceptable level of quality
Pertains to the intrinsic value of the object	Pertains to the extrinsic value of the object?
Pertain to quality, that is, an object’s level of excellence	Pertains to an objects’ quality and values or importance within a given context
Is assessed using the question.  Does the object do well what it is intended to do?	Is assessed using the questions, Is the object of high quality and also something the target group needs.
Is tied to the accepted standard of quality for the type of object being evaluated	Is tied to accepted standards of quality and to data from a pertinent needs assessments
Concerns the object’s rating on standards of quality against competitive objects of the same type	Entails judgment of the object’s quality and importance and value to a particular consumer group
May by assessed through the comparison of an object with standards or competitive objects	Assessments of worth may be comparative or non-comparative

In general, when we discuss an object's merit we are discussing whether that object is doing what it is supposed to do.  For example, feedback advances pupil learning whereas, repeating a year appears to have a negative impact on pupil learning. Merit, in other words, can be seen as intrinsic merit in the absence of costs.      Worth, on the other hand, does not just restrict itself to the intrinsic value of an object, it also includes  reference to the need for an object and its associated costs.  For example, phonics appear to be low cost intervention which advances pupil learning by four months, whereas the use of teaching assistants is a high cost intervention, which has limited impact on pupil learning. So a programme or innovation may be judged to meeting its own internal objectives, however, in doing so it might be both too expensive and not meeting a broader set of needs.

So what concepts are related to needs and needs assessment?

Stufflebeam and Coryn go on to state that the characteristics of worth suggest that it is impossible to evaluate an object's worth unless consideration is taken into account as to whether that object is a meeting specific needs.  And in considering the meeting of needs, we need to distinguish between the identification of a need and whether that need is being met through some form of intervention.  These differences area  highlighted Table 2.

Table 2 Concepts Related to Needs and Needs Assessment  (Adapted from Stufflebeam and Coryn, 2014 p11)

Concept	Definition	Example
Defensible purpose	A desired end that has been legitimated	Pupils’ development of mathematical skills
Need	Something that is necessary or useful for fulfilling a defensible purposes	Competent, effective teaching in mathematics
Outcome need	An achievement or outcome required to meet a defensible purpose	Pupils’ demonstration of proficiency in mathematics
Intervention need	A certain service, competent service provider, or other helping agent	Competent teachers in KS4 courses in mathematics
Needs assessment	A systematic investigation of the extent to which treatment/and or outcome needs are being met	Examination of pupils results in GCSE Mathematics and evaluation of the involved teachers

So what are the implications for the School Research Lead

The preceding analysis of the nature of evaluation has a several implications for the School Research Lead.

1. Much of the discussion around the role of the School Research Lead is about assisting colleagues in finding out 'working out what works' and as such focuses on the merit rather than the worth of the object of evaluation. 
2. Working out what works also needs to consider questions such as - works for who and under conditions - and are needs being met?
3. Some interventions may be of merit, but on the other hand, may not be worth it in terms of the effort expended in implementation.  

Some final words one evaluation - Part One

Values are an inescapable component of evaluations.  Accordingly, evaluations need to be undertaken in a systematic manner, with a defensible rationale being provided for the values being used to evaluate a subject;  rigorously collecting information and subjecting that information to open and transparent analysis; and communicating those findings to colleagues, senior leaders and other stakeholders.   This will require evaluators to have two broad sets of skills; first, the ability to undertake or assist in the technical evaluation of an object; second, having high-level inter-personal skills to ensure all the relevant participants are engaged before, during and after the evaluation. (Stufflebeam and Coryn, 2014)

Next week's post with look at the extended definition of evaluation and the implications for school research leads.

References and further reading

Stufflebeam, D.L. & Coryn, C. (2014). Evaluation Theory, Models & Applications (second edition), Jossey Bass, San Franciscon.

*The Joint Committee is a standing committee that was established in the  1975.  Its members have been drawn from 15 professional societies is the United States and Canada that are concerned with improving evaluation in education.

**A future post will examine Stufflebeam and Coryn' extended definition of evaluation the systematic assessment of an object's worth, merit, probity, feasibility, safety, significance and or/equity (p11) and the role of values in evaluation.