The importance of being REF-able

The project has a short article published on the excellent LSE Impact blog, on the importance of being REF-able, as part of the Accelerated Academy series. In it, we discuss the interaction between research evaluation frameworks and academics’ individual career goals, their sense of scholarly identity, and their disciplinary norms and practices. In the week or so since it was published, it’s been retweeted and shared many times, and I’ve been asked for links and hard copies via ResearchGate. Perhaps it has touched on something close to the heart of those working in academia? The article can be found here.

Writing that counts or writing that is counted?

At the beginning of December, I went to Leiden in the Netherlands to present a paper on behalf of the Academics Writing project. The conference was the Accelerated Academy, which brings together scholars from across Europe, mainly from STS (Science and Technology Studies) but also from other disciplines, to share research about the acceleration of higher education and the increased use of metrics for research evaluation. Clips of some sessions from the Prague conference in 2015 can be found here.

Our paper focused on the tensions that have emerged from the project around the REF. The UK’s national research evaluation exercise pushes academics in competing directions in their writing. The REF and institutional policies around it drive academics to write more and better journal articles in each REF period, and this has become an overarching goal for academics’ writing practices: to publish enough and to target the right journals is to keep their job, to get promoted and to be considered successful. But not all academics really want to publish this sort of journal article, or at least not at the expense of other genres. Historians value monographs, but these take a long time to write and because four research outputs can be submitted to the REF, four has become something of a magic number. Four monographs in a single REF period would be more or less impossible, but departments expect at least four publications, so historians are pushed towards writing journal articles. For Marketing academics, four translates as four-star; their ultimate target for publishing. They are encouraged to publish in four-star journals, which they see as a.) unrealistically difficult and b.) pushing them out of their discipline. If it’s too hard to get into a four-star marketing journal, one can always aim for a four-star management journal.

Finally, although since 2014 the REF rewards writing for impact beyond academia, writing for non-academic audiences was widely perceived by our respondents to be of secondary importance to writing for peer-reviewed, high-impact journals. Many were interested in writing for the media, writing blog posts, tweeting and the like, but didn’t feel that their institutions valued these genres. This view is unsurprising given the relentless counting and evaluating by universities of writing that is aimed primarily at other academics.

The project goes to summer school

I gave a lecture last week at an international doctoral summer school in Vienna, run by Klagenfurt University. The theme of the summer school was quality in scholarly practice in social sciences and humanities, and my lecture discussed findings from the Academics’ Writing project on how academics’ writing practices are influenced by measures of research quality such as the REF.

I shared extracts from interviews the project team has done with academics who talked about the pressures of having to publish certain numbers of papers in specified venues, often based on journal impact factors, and about feeling forced to publish in journals outside their discipline.

I also ran a workshop on getting published, and Klagenfurt University were generous enough to let me stick around for the whole week, listen to the other invited speakers and get to know the students, who came from Austria, Germany, France, Turkey, Italy, Bulgaria, Lithuania, Uganda, Nigeria, Brazil, Costa Rica and the USA. They were a passionate, articulate and multilingual group and I scribbled down many of their nuggets of wisdom, which sometimes bordered on the poetic. Here’s a selection:

“I write at night, like a witch”

“I feel oppressed by writing in English”

“You need to read texts against the grain”

“There are ghosts in the archive”

“The impact agenda makes bad autoethnographers of us”

I could pick any one of these and write about it all day, but I’ll leave you to ponder your own interpretation of these statements and how they might apply to you.

Holding our disciplinary ground

I had the pleasure of going to Ann Arbor last week to present at the International Writing Across the Curriculum conference on behalf of the Academics Writing project. Ann Arbor is home to the University of Michigan. It’s a lovely college town where, it seems, people have a passion for gardening and also apparently don’t lock their doors at night.

ann arbor garden

Our presentation focused on the ways in which academics’ disciplinary writing practices have been affected by changes at managerial level, particularly the REF. For some disciplines, the picture is rather bleak. For example, in Marketing, our participants were forced to aim their scholarly writing at specific target journals prescribed by the Association of Business Schools’ benign-sounding Academic Journal Guide, which ranks business and management journals according to their supposed quality.

Only 5 of the journals ranked highest (4-star) by the ABS guide in 2015 were in Marketing, making it a very small target to aim for. There is also a perception that most of these journals are US-based, and publish mainly quantitative work.

Here’s a taste of what our participants in working in Marketing departments in England said about this:

“Now it’s not just four star journals. It’s four star journals in marketing. Now the four star journals in marketing […] are US based four star journals. I think there were only 16. I can’t remember what the figure was but it was such an unbelievably low figure, of UK academics getting published last REF into US four star journals.” Charles, lecturer in Marketing

“Now I target management journals, which is one way of hitting a four star.” Diane, professor in Marketing

michigan league

There is much to comment on here about the primary purpose of publishing research, the effect of attempting to measure quality in this manner, the level of autonomy academics enjoy (or not) regarding what and where to publish, cultural differences, disciplinary boundaries, and much more. The other disciplines in our study are experiencing their own sets of pressures related to disciplinary values and practices, which you can find out more about by checking out the slides from the talk.

 

Many of the other presentations at the iWAC conference were pedagogically-oriented, focusing on improving students’ writing in the disciplines, but this still provided an opportunity to consider what it is that we are preparing our students for; what disciplinary writing actually looks like in the context of contemporary higher education.

Who do you work for anyway?

The project team had the pleasure of being invited to give a talk this week at a careers event at Manchester University, aimed at humanities researchers. The event included a panel discussion on the theme of What do you actually do?  The panel’s insights were not only very interesting and amusing, but also resonated with the findings of the Academics’ Writing project. For example, there was unanimous agreement that an academic’s job was roughly divided into three main areas: teaching, research and admin, which is consistent with what our research participants have said. However, the panel members added that there was also some “other stuff” too. Asked what this “other stuff” consisted of, they revealed that it included attending committee meetings, doing ethics reviews and the like. Much of this type of work we had categorised as “admin”, but some universities call it “service”.

As part of our analysis, we coded the transcripts of our interviews with participants, attaching a descriptive code to genres of writing that people talked about. 64 different genres emerged, but they did not all fit into the tripartite system of teaching-research-admin. Conducting peer reviews of books or articles, for example, is not exactly admin. It’s not for the benefit of the department or institution, nor the academic doing it, who receives no pay, credit or time for such work (although arguably, academics at the beginning of their career may do reviews to gain experience or help establish their reputation). Such work is central to academia as a way of upholding standards and creating disciplinary knowledge, but it is also time consuming, yet it is not accounted for in the workload allocation models academics are expected to adhere to. Similarly, responding to surveys and participating in disciplinary or professional fora do not fall neatly into research, teaching or admin, yet academics often see these activities as crucial for staying up to date with their field and contributing to their discipline.

This raises the question not just of what do academics actually do, but of whom they actually work for. Clearly, they are employed by their institution, but much of what they do could be argued to have benefits for others beyond or within the university. In the age of the REF,  is it one’s department that one primarily strives to benefit? This might actually work against any wider sense of disciplinary identity, since comparable departments must effectively compete with each other. What about the fee-paying students? Are they our “customers” who we must serve? Or is some of an academic’s work actually for themselves? After all, our h-index is ours alone, and when academics move institutions, they take their work with them. Or do they? Some argue that academics moving institutions prior to the REF be disallowed from submitting work done in the preceding year or two for the REF in their new institution.

Several panel members in Manchester, and our own participants, talked about their research writing as though it were something personal and rather indulgent. “I’m going to be selfish and use this time for myself” was the way one person described getting research writing done. Another piece of advice from a panel member was, “If you get a second that belongs to yourself, work on a paper”. Research writing was the thing that got squeezed by other work, and the thing most likely to get done in personal time; at home, on holiday and in the evening, partly because it was seen as something that belonged to the individual.

Cultures of counting: Metrics through a critical lens

Tuesday 24th May sees the Academics’ Writing project’s fourth and final workshop on the role of metrics in academic life. This time, we have invited two expert speakers to talk about what responsible metrics might look like in the context of both REF and TEF. The speakers are Professor Paul Ashwin of Lancaster University and Professor James Wilsdon of the University of Sheffield.

Metrics evoke a mixed reaction across the higher education community. A commitment to using data to inform decisions makes some enthusiastic about the prospect of granular, real-time analysis of our activities. Yet we only have to look at the blunt use of metrics such as
journal impact factors, h-indices and grant income targets, to be reminded 
of the pitfalls. Some of the most precious qualities of academic culture
 resist simple quantification, and individual indicators often struggle to do justice to the richness and plurality of our work.

Across both research and teaching, metrics are receiving greater emphasis from policymakers and managers. The November 2015 HE green paper outlines a new regulatory architecture, including the replacement of HEFCE with a new Office for Students, and the introduction of a Teaching Excellence Framework (TEF). Metrics are portrayed as crucial to the TEF, albeit with some scope for expert judgement alongside, and there are now fierce arguments raging across the sector about whether we need a TEF at all, and if so, how it should be designed, and what mix of quantitative indicators it should employ.

Metrics hold real power: they are constitutive of values, identities and livelihoods. How to exercise that power to more positive ends was the focus of The Metric Tide, a recent UK review of the role of metrics in research management and assessment. The Metric Tide sets out a framework for responsible metrics, and makes a series of recommendations for researchers, university managers, funders, policymakers and publishers.

In this seminar, James Wilsdon, who chaired The Metric Tide, will outline its main findings, and reflect on ongoing efforts to influence debates about UK research policy and funding, including over the design of the next Research Excellence Framework (REF), which is currently the focus of a further review by Lord Stern.

Paul Ashwin, in the context of the proposed TEF, will examine the challenges of developing measures of teaching quality that do not simply reflect institutional prestige.

Both speakers will consider what a culture of ‘responsible metrics’ might look like for research and teaching, and the opportunities and obstacles to achieving this.

James Wilsdon is professor of research policy at the University of Sheffield, and was chair of the UK’s Independent Review of the Role of Metrics in Research Assessment and Management. He is now chairing a European Commission expert group on the future of altmetrics.

Paul Ashwin is professor of higher education at the Lancaster University and a co-investigator in the ESRC/HEFCE funded Centre for Global Higher Education.

This talk is part of the Designing the Academic Self series, sponsored by the Academics Writing project, which is part of the Literacy research Centre at Lancaster and the Northwest Doctoral Training Center and is open to early career researchers and doctoral students from Lancaster, Manchester and Liverpool universities.  To book your place, visit Eventbrite. 

 

What can and can’t metrics tell us?

Tuesday 26th April sees the third of four interactive workshops on the role of metrics in academic life, run by the Academics Writing  project, alongside  Masud Khokhar and Tanya Williamson of Lancaster University library, called What can and can’t metrics tell us?

Higher education institutions in the UK have to justify how they spend their research funding and demonstrate that this spending resulted in high quality research outcomes. This is achieved mainly through the REF (research excellence framework) and bibliometric measures associated with it. The research activity of individual academics, departments and institutions is measured in various ways and the resulting data used for a range of purposes. But are we asking the right questions? And are we using the data in an appropriate way?

This interactive workshop is part of the Designing the Academic Self series, and explores the ways in which quantitative indicators of research output are used, by both institutions and individuals. We begin by exploring what exactly are we trying to measure, and consider how well are the metrics we generate can do this. We will talk about case studies of how metrics are currently being used in different UK institutions, and will consider the potential positive and negative effects of these on academics.

To book a place, visit Eventbrite