Monday, December 22, 2014

Bang for buck in REF 2014

The Research Excellence Framework 2014 evaluates research in terms of outputs (e.g., journal publications), impacts (e.g., how research translates to real-world impact) and research environment (e.g., facilities, funding, people).  Universities try to maximise the proportion of 4-star and 3-star ratings they get which indicates world and internationally leading research.  To the wider world, the only things that count are the end products of research: outputs and (research's indirect cousin) impacts.  So what is the point of assessing research environment?  One possibility is that it can be used to ascertain which departments are squandering resources and which departments are doing fantastic research on a shoestring, and using that information to guide funding decisions.  For example, if a department is managing to produce a high proportion of 3-4* outputs on a 2* environment, then quality-related (QR) research money should be thrown at them because one can only imagine what they might achieve with better facilities.  On the other hand, if a department has a 4* environment but its modal outputs tend to be in the 2-3* range, then serious thought should be given to holding back some of their QR money because they are clearly wasting their excellent research environment facilities on relatively mediocre research.  How can we examine - and reward - greater bang for buck?

We attempt to quantify "bang for buck" by looking at the quality of research produced by departments (outputs and impacts) relative to the quality of their research environment. Are people in 4* research environments consistently producing 4* research outputs? Are people in lower rated research environments producing poorer quality outputs?

To examine this, we calculate each department's grade point average (GPA) based on their combined outputs (weighted at 65% in REF2014) and impact (weighted at 20%) and compare it to the GPA based on their environment alone.  In short this is the score for outputs and impacts divided by the score for research environment.  This gives us a "Bang for Buck" ratio, where a score of 1 indicates a university/unit is producing research in line with the quality of its environment. A score of greater than 1 indicates higher quality outputs relative to the research environment, while a score lower than 1 indicates outputs of a lower quality relative to the environment.  As my interest is in psychology, I've done the analysis for Unit of Assessment 4 (Psychology, Psychiatry and Neuroscience) - if time permits I’ll add the analysis for universities overall later.  I’m focussing on top research departments, which I've defined as those that received above average scores for their 3* and 4* outputs/impacts combined (the average score in UoA 4 is ~65%).  Table 1 below shows the rankings according to the Bang for Buck ratio for the top research departments. 

Table 1. Bang-for-buck scores for the top 46 institutions in UoA 4. Overall GPA is included for context, but not used in the calculation. 
Rank Institution name Overall GPA Environment GPA Impact GPA Outputs GPA b4b
1 Nottingham Trent University 2.894 2.50 3.73 2.73 1.19
2 University of Stirling 3.269 2.88 4.00 3.14 1.16
3 University of East London 2.931 2.63 4.00 2.67 1.14
4 Goldsmiths' College 2.981 2.75 3.60 2.84 1.10
5 University of Aberdeen 3.221 3.00 3.00 3.34 1.09
6 University of Portsmouth 2.799 2.63 3.53 2.61 1.08
7 University of Dundee 3.182 3.00 3.80 3.03 1.07
8 University of Warwick 3.290 3.13 3.47 3.27 1.06
9 University of East Anglia 3.099 3.00 3.60 2.97 1.04
10 University of Hull 2.805 2.75 3.27 2.68 1.02
11 University of Essex 3.302 3.25 3.53 3.24 1.02
12 University of Plymouth 3.040 3.00 3.20 3.00 1.02
13 Roehampton University 2.785 2.75 3.40 2.60 1.02
14 Bangor University 3.278 3.25 3.73 3.15 1.01
15 Lancaster University 3.014 3.00 3.27 2.94 1.01
16 Queen's University Belfast 2.987 3.00 3.73 2.76 1.00
17 University of Sussex 3.355 3.38 3.84 3.20 0.99
18 City University London 2.728 2.75 2.80 2.70 0.99
19 University of Leicester 3.062 3.13 3.00 3.07 0.98
20 Swansea University 3.176 3.25 4.00 2.91 0.97
21 University of Surrey 2.858 3.00 2.73 2.86 0.94
22 Royal Holloway, University of London 3.439 3.63 3.73 3.31 0.94
23 University of Kent 2.963 3.13 3.40 2.79 0.94
24 University of Liverpool 3.138 3.38 3.80 2.88 0.92
25 University of Southampton 3.212 3.50 3.80 2.96 0.90
26 University of Glasgow 3.205 3.50 3.20 3.14 0.90
27 Imperial College London 3.405 3.75 3.84 3.19 0.89
28 University of Oxford 3.623 4.00 3.86 3.47 0.89
29 University of Reading 3.043 3.38 3.68 2.77 0.88
30 University of York 3.460 3.88 3.47 3.36 0.87
31 University of Exeter 3.223 3.63 3.20 3.14 0.87
32 Cardiff University 3.522 4.00 3.80 3.33 0.86
33 University of Cambridge 3.508 4.00 3.64 3.35 0.86
34 University of Durham 3.038 3.50 3.27 2.86 0.84
35 Birkbeck College 3.462 4.00 4.00 3.17 0.84
36 University of Bristol 3.191 3.75 3.80 2.87 0.82
37 University of Birmingham 3.400 4.00 3.68 3.18 0.82
38 University of Nottingham 3.188 3.75 3.33 3.01 0.82
39 University of Leeds 3.066 3.63 3.36 2.85 0.82
40 Newcastle University 3.366 4.00 3.87 3.07 0.81
41 University of Edinburgh 3.349 4.00 3.82 3.06 0.81
42 King's College London 3.329 4.00 3.78 3.04 0.80
43 University College London 3.307 4.00 3.71 3.02 0.80
44 University of Sheffield 3.196 3.88 3.60 2.92 0.79
45 University of St Andrews 3.281 4.00 3.40 3.08 0.79
46 University of Manchester 3.232 4.00 3.60 2.94 0.77


The first thing that strikes me is the relatively small number of UoA 4 departments that have a bang for buck (b4b) ratio above 1. Only 16 of the top 46 departments are producing better outputs/impacts than their research environments would suggest.  The second thing I notice is that Oxford (.89) and Cambridge (.86), almost always considered to be in the top 3 research departments, are well down the table in terms of b4b.  Now, Oxford and Cambridge departments could hardly be considered underachieving, given their impressive outputs and impacts, but the b4b scores suggest they could be doing still more given the resources at their disposal. 

Of the 35 institutions in Table 1 that don't receive top marks for their environments, 25% of outputs and 58% of impacts are rated as 4*, indicating that huge swathes of world class research are done outside of traditional research powerhouses and elite institutions.  For me, the real stars in this analysis are Aberdeen, Dundee and Warwick - all three have very impressive GPAs for outputs and impact, but are still scoring very highly on the bang-for-buck measure.  On average, 35% of the outputs and 49% of the impacts from these three institutions are at 4* level, despite not getting top scores for their research environments (averaging only 8% at 4*).  

Does size matter?

Scanning the table, it also seems that larger departments tend to do less well in terms of bang-for-buck.  Table 2 shows the 10 largest departments (by number of staff submitted to UoA 4) along with their b4b scores. 

Table 2.  Bang-for-buck scores of the top 10 departments by size
Rank Institution name FTE Staff Submitted b4b2
1 University College London 286.6 0.80
2 King's College London 238.9 0.80
3 University of Edinburgh 117.3 0.81
4 University of Oxford 98.3 0.89
5 University of Cambridge 76.0 0.86
6 Cardiff University 69.3 0.86
7 University of Bristol 68.8 0.82
8 University of Manchester 67.7 0.77
9 University of Nottingham 53.2 0.82
10 Newcastle University 51.6 0.81

The mean b4b score of the top 10 largest departments is .825, meaning that the quality of research outputs and impact are considerably below what one might expect given their research environments. There is a general negative relationship between the size of department and the b4b score (r ~= -0.3) Are the departments in Table 2 too large to function optimally in terms of research outputs?  Is there an ideal department size in terms of producing quality research consistent with the research environment?  Table 3 shows the b4b scores for groups of departments ordered by size (e.g., the median number of staff for the top 10 largest departments is 72.64, the next 10 largest is 37.3 and so on).  

Table 3. Groups of departments ranked by size with median number of active researchers and b4b scores
Rank by Overall GPA Median FTE b4b
1-10 72.64 0.825
11-20 37.3 0.908
21-30 29.275 0.908
31-40 21.75 1.033
41-50 14.85 1.038

According to this data, the sweet spot for department size is somewhere between 15 and 22 active researchers. Beyond that, people are not making optimal use of their research environments. It's not clear why this would be the case - it may be because resource management becomes extra problematic over a given size or because a culture of resource wastage becomes normalised in larger departments.  There may be many reasons for such a relationship.

What have we learned?

Overall, I think this analysis suggests a couple of very important points. It is clear that there is huge value in not focusing research funding in a small number of elite institutions. Whatever the flaws of the REF, the overall research profiles clearly indicate that there is high quality research taking place across the UK - there are 46 departments with >65% research considered in world-leading or internationally-leading.  Furthermore,  most of the world-leading research in the UK is taking place in departments that lack 4* research environments.  What the bang for buck calculations show is that many departments are punching well-above their weight in terms of research produced and are demonstrating extremely good value for money. In addition, as departments increase in size there is a noticeable decrease in bang-for-buck scores. 

In terms of allocating research funding (QR money), there are at least two ways to utilise this information. One way is to allocate extra money to those departments with high bang for buck ratios, because giving them the opportunity to improve their research environments will likely lead to matched improvements in research outputs and impacts.  Another way is to disregard environment completely when allocating QR money, and focus only on outputs and impacts, which rewards departments purely by the research they actually produce, rather than factoring in the environment they already have in place.