REFs, RAEs and h’s….what about the students?

A student perspective on the role of research assessments, influence indexes and impacts factor measurements.

by Chris A.J. Brown


As a geographer studying the UK labour market, measurements are clearly an important part of my research – levels of employment, self-employment, unemployment, jobs. Comparing these measurements against each other and across space is another key part in examining change and possible repercussions. However, academia in the UK as a whole has been developing a growing obsession with different type of measurement…of itself. Universities and staff are bracing, or indeed energising themselves, for the next round of measurements that asses the quality of research being produced by higher education institutions (HEI). The Research Excellence Framework (REF) due to be completed in 2014 is the new incarnation of the Research Assessment Exercise (RAE) that was last completed in 2008. There is much discussion about the relevance, purpose and benefit of such quality assessment but it is part of a wider inception of measurements beginning to encroach upon the academic world, not least that of impact and influence. The London School of Economics has its own research project dedicated to the impact of academic research.  However, these assessments and measurements of research and academics is set at a somewhat bureaucratic level and focus and has left me asking the question: what is the purpose of these assessments and measurements from a student’s point of view?

A recent discussion seminar within our department focussed on a 2010 paper by Andrew Bodman which broadly looked at the ‘influentialness’ of geographers within academia using the ‘h’ index (I will encourage you to look up its definition yourself rather than attempt that here!).  The discussion among the group largely focussed around the purpose of such quantification– that of the measurement and impact of academic research and the challenges that come with it.  The forthcoming REF will determine which universities within the UK are producing quality research which in turn will be used to determine resource allocation and funding. Indirectly such measurement will also create another form of ‘league table’ with regards to which institutions are producing the ‘most’ or ‘best’ high quality research and to a certain extent the individuals associated therein. The Bodman paper was however specifically relating to the role of individuals and their level of influence on wider academic research, predominantly in relation to citations and references that come from publications.

As a postgraduate researcher I was left with a couple of questions. Given the likelihood and growing  impetus within academia to introduce measurement such as the REF or even individuals academic influence based on the ‘h’ index, at what stage should postgraduates or even undergraduates concern themselves with this measuring? Further to this, why is the level of influence or impact that academic research is having upon students not being considered? As is often the case, it is the definition that confuses me. In the case of Bodman’s paper it is the definition of influence, or lack of,  that struck me in this respect- ambiguous at best. To me measuring an individual’s impact or influence based on citations among peers, colleagues or even policy is a very lateral practice.  If such measurement and assessments are to become commonplace among HEI’s, then surely it is somewhat ignorant not to consider the influence or impact this research is having on current and future cohorts of undergraduate students, a number of whom will remain for postgraduate research and academic positions. After all a large proportion of the HEI population are students.  As undergraduates we are predominantly directed towards the key and leading thinkers in certain fields of research. Effectively these people or their work may subsequently influence dissertation topics and further to that postgraduate or PhD research. Ultimately leading to the production of papers included in future REFs and impact factor ‘h’ indexes. This in my mind is a measurement of influentialness that is being overlooked.

A somewhat intriguing analogy used within the Bodman paper was the idea that some of these measurements and indexing can be considered in a manner of a forest – some of the trees are taller and stand out from the crowd but are not necessarily stronger or broader than others. Furthermore, the collective whole – or forest – must play a certain amount in contextualising and supporting directly or indirectly the growth and development of others, but receive little recognition if measured by simply by height.  To put more ‘real worldy’, there are many reason why measuring the influence of individual academics and their work is flawed. Apparent success, influence and impact measured in this way may be down to improvements in technology allowing for better journal and citation access, field of study, working on a ‘hot topic’, or purely publishing in the right journal at the right time. And to risk taking the HEI forest analogy even further, surely the role of tree saplings [under/post-graduates] has just as pertinent a role.

While these measurements are clearly important at a certain level for resource management, funding, support, academic rigour and even individual confirmation (or gratification), for the students who make up the bulk of the HEI population it remains to be a somewhat abstract practice that is easier to overlook than concerns yourself with – although the figures may be used to try and influence a students decision on where to study. Should students be concerned or engaging with this? Clearly the purpose of such measurements and assessments is a more structural and institution purpose. Perhaps a student  ignorance of sorts is a good thing and allows us to focus solely on our own research and measurements of our own data.  But I do feel there is perhaps a missing link in this quantification of academic research – student academics.

P.S. The irony of the Bodman paper is that it has itself received a very good ‘impact factor’, which I assume is largely due to the relevant topic it was covering. I also realise therefore the irony of writing this piece as a personal blog, as undoubtedly I will be checking the figures that measure how many people read it! Have I caught the bug!?!


Bodman, AR. (2010). Measuring the influentialness of economic geographers during the ‘great half century’: an approach using the h index. Journal of Economic Geography. 10: 141-156.


PhD, year 1 – am I doing this right?

Reflections on my first year as a PhD student.

by Chris A.J. Brown


I began writing this blog post a while back when I reached the 6 months stage of my PhD. I felt the need to ‘get writing’ as I had come across the first instance of a widely cited period of uncertainty and possible jadedness of the PhD process- defining your research. What was my PhD actually about?; Am I doing this correctly?; How can where I am now turn into an 80,000 word piece of expertly written and unique piece of academic literature? In all honesty, even 12 months in, some of these feelings still present themselves but they have now been accepted as par for the PhD course and are in fact just part of shaping and defining the research. However, it made me think about a different part of the PhD. Quite often the individual mechanics of working on a PhD are lost in the rhetoric of impact, quality and the production of papers. Although these are key points, a PhD is just as much about training as it is about being a bona fide researcher. For me, one reason I chose to come back to University to work on a PhD was to gain further training, skills and knowledge in a particular field of research and to have a greater input when returning to a job.  The following revisits some of what I wrote around 6 months ago when I found the grit, definition and focus of my research left wanting and where it has left me now – about to move into 2nd year.

PhD students are often bombarded with training and guidance on what to expect of a PhD, to quote a description given to me years ago, is “the closest a man can get to bearing a child” – it is ultimately a very individual and at times isolating experience. Now this is a bit over-theatrical and implies an over-erring, pessimistic view but it does in some respect hold truth. Fortunately, the experience I have had so far has been positive and challenging but it has also made me realise there is an aspect to a PhD that is often not considered and perhaps represents this individual and ‘lonely’ phase.

Much of this guidance and training which is provided for prospective and current PhD students is well targeted and of genuine use in preparing for the journey ahead.  But six months into my PhD while trawling through bounds of literature, writing and casting a critical eye over my field of research, it became quite hard to know whether I was moving in the right direction. Supervisors have a key role to play here and fortunately mine have provided the perfect balance of guidance and space to make it my own.  The ‘standard’ training that is given to PhD students often comes prior to starting with further support throughout but is often more focussed on the main and practical aspects such as thinking about concepts and theory; posing strong research questions; starting the field, desk or lab work; beginning the write up; or preparing for the viva. During this phase of research it made me think about what I had experienced with regards to the actual mechanics and cognitive process of being a PhD student, and researcher in training.

For many, you arrive at the start of a PhD from two channels (although not exclusively). One is to begin with a very clear idea of what your research is about and it may be a broad continuation of either an undergraduate and/or postgraduate dissertation or interest. The second channel may be that you begin with a wholly new topic to which you only have a very broad idea as to what your research will be about. In my case, I was the second channel. This presented challenges. Although I was privy to many of the concepts of my PhD – UK labour market and unemployment – I began my PhD having recently been working for a well established research organisation in London. I was therefore somewhat disengaged from the rigour of the academic world and the processes required for writing towards a PhD. I had the basic broad knowledge of the research area but had not engaged in much of the large theories or concepts that lay behind such a topic – perhaps an obvious difference between the ‘academic bubble’ and the ‘real world’.

The challenge is that not only do you have to get to grips with your chosen field of study but, in my case, it was another period of getting to know a new University, staff, and colleagues. For some this is an easy process but for others it may not be so simple trying to balance these interlinked components of academia.  ‘Toing and froing’ with research ideas, designing research questions and seeking a gap to fill in broad spectrums of literature is an all encompassing task. Once you move into the twilight phase of first year, the PhD research should be shaping up nicely but you are presented with new issues surrounding how you will add to existing knowledge; will this be a unique and forward thinking piece of work that will be well considered by specialists in your field?; What academic papers may evolve from this piece of research?

Now, the first point to raise here is “yes, but all PhDs are different” – I am specifically coming from a social science perspective. What I have said above may not appply to everyone but the main points should broadly apply to all. As part of my PhD funding I am fortunate enough to be entitled and invited to attend and take part in a number of doctoral training events that help to network and bring together all the like minded inter-disciplinary PhD-ers in Scotland. These events are an invaluable part of the the PhD process. Not least because they provide some informal training and guidance on many aspects of studying towards an PhD but there should be emphasis on one word: training. The key thing to remember through the 1st year of the PhD is that you are still not the expert but you are in a process of training and specialising. The highs and lows that come with beginning a PhD are to be expected. The research is your project to which you need to take owenership and focussing on that will keep you enthused and determined.


I hope to add similar blogs about the PhD process and training as I go, partly for reasons mentioned about the individual mechanics of doing a PhD are often not considered but also as a means to write non-academically – another part of the training.
I would also ideally like for other PhD-ers and academics to comment and add to the discussion. And if anybody is interested, to add their own blogs or discussion I would be happy to help accommodate or help.  There exists many outlets to discuss such PhD topics such as ‘#phdchat’ or ‘#acwri’ on Twitter, and so I only hope to add to this useful support and advice network.