The project has a short article published on the excellent LSE Impact blog, on the importance of being REF-able, as part of the Accelerated Academy series. In it, we discuss the interaction between research evaluation frameworks and academics’ individual career goals, their sense of scholarly identity, and their disciplinary norms and practices. In the week or so since it was published, it’s been retweeted and shared many times, and I’ve been asked for links and hard copies via ResearchGate. Perhaps it has touched on something close to the heart of those working in academia? The article can be found here.
REF
Writing that counts or writing that is counted?
At the beginning of December, I went to Leiden in the Netherlands to present a paper on behalf of the Academics Writing project. The conference was the Accelerated Academy, which brings together scholars from across Europe, mainly from STS (Science and Technology Studies) but also from other disciplines, to share research about the acceleration of higher education and the increased use of metrics for research evaluation. Clips of some sessions from the Prague conference in 2015 can be found here.
Our paper focused on the tensions that have emerged from the project around the REF. The UK’s national research evaluation exercise pushes academics in competing directions in their writing. The REF and institutional policies around it drive academics to write more and better journal articles in each REF period, and this has become an overarching goal for academics’ writing practices: to publish enough and to target the right journals is to keep their job, to get promoted and to be considered successful. But not all academics really want to publish this sort of journal article, or at least not at the expense of other genres. Historians value monographs, but these take a long time to write and because four research outputs can be submitted to the REF, four has become something of a magic number. Four monographs in a single REF period would be more or less impossible, but departments expect at least four publications, so historians are pushed towards writing journal articles. For Marketing academics, four translates as four-star; their ultimate target for publishing. They are encouraged to publish in four-star journals, which they see as a.) unrealistically difficult and b.) pushing them out of their discipline. If it’s too hard to get into a four-star marketing journal, one can always aim for a four-star management journal.
Finally, although since 2014 the REF rewards writing for impact beyond academia, writing for non-academic audiences was widely perceived by our respondents to be of secondary importance to writing for peer-reviewed, high-impact journals. Many were interested in writing for the media, writing blog posts, tweeting and the like, but didn’t feel that their institutions valued these genres. This view is unsurprising given the relentless counting and evaluating by universities of writing that is aimed primarily at other academics.
Holding our disciplinary ground
I had the pleasure of going to Ann Arbor last week to present at the International Writing Across the Curriculum conference on behalf of the Academics Writing project. Ann Arbor is home to the University of Michigan. It’s a lovely college town where, it seems, people have a passion for gardening and also apparently don’t lock their doors at night.
Our presentation focused on the ways in which academics’ disciplinary writing practices have been affected by changes at managerial level, particularly the REF. For some disciplines, the picture is rather bleak. For example, in Marketing, our participants were forced to aim their scholarly writing at specific target journals prescribed by the Association of Business Schools’ benign-sounding Academic Journal Guide, which ranks business and management journals according to their supposed quality.
Only 5 of the journals ranked highest (4-star) by the ABS guide in 2015 were in Marketing, making it a very small target to aim for. There is also a perception that most of these journals are US-based, and publish mainly quantitative work.
Here’s a taste of what our participants in working in Marketing departments in England said about this:
“Now it’s not just four star journals. It’s four star journals in marketing. Now the four star journals in marketing […] are US based four star journals. I think there were only 16. I can’t remember what the figure was but it was such an unbelievably low figure, of UK academics getting published last REF into US four star journals.” Charles, lecturer in Marketing
“Now I target management journals, which is one way of hitting a four star.” Diane, professor in Marketing
There is much to comment on here about the primary purpose of publishing research, the effect of attempting to measure quality in this manner, the level of autonomy academics enjoy (or not) regarding what and where to publish, cultural differences, disciplinary boundaries, and much more. The other disciplines in our study are experiencing their own sets of pressures related to disciplinary values and practices, which you can find out more about by checking out the slides from the talk.
Many of the other presentations at the iWAC conference were pedagogically-oriented, focusing on improving students’ writing in the disciplines, but this still provided an opportunity to consider what it is that we are preparing our students for; what disciplinary writing actually looks like in the context of contemporary higher education.
Cultures of counting: Metrics through a critical lens
Tuesday 24th May sees the Academics’ Writing project’s fourth and final workshop on the role of metrics in academic life. This time, we have invited two expert speakers to talk about what responsible metrics might look like in the context of both REF and TEF. The speakers are Professor Paul Ashwin of Lancaster University and Professor James Wilsdon of the University of Sheffield.
Metrics evoke a mixed reaction across the higher education community. A commitment to using data to inform decisions makes some enthusiastic about the prospect of granular, real-time analysis of our activities. Yet we only have to look at the blunt use of metrics such as journal impact factors, h-indices and grant income targets, to be reminded of the pitfalls. Some of the most precious qualities of academic culture resist simple quantification, and individual indicators often struggle to do justice to the richness and plurality of our work.
Across both research and teaching, metrics are receiving greater emphasis from policymakers and managers. The November 2015 HE green paper outlines a new regulatory architecture, including the replacement of HEFCE with a new Office for Students, and the introduction of a Teaching Excellence Framework (TEF). Metrics are portrayed as crucial to the TEF, albeit with some scope for expert judgement alongside, and there are now fierce arguments raging across the sector about whether we need a TEF at all, and if so, how it should be designed, and what mix of quantitative indicators it should employ.
Metrics hold real power: they are constitutive of values, identities and livelihoods. How to exercise that power to more positive ends was the focus of The Metric Tide, a recent UK review of the role of metrics in research management and assessment. The Metric Tide sets out a framework for responsible metrics, and makes a series of recommendations for researchers, university managers, funders, policymakers and publishers.
In this seminar, James Wilsdon, who chaired The Metric Tide, will outline its main findings, and reflect on ongoing efforts to influence debates about UK research policy and funding, including over the design of the next Research Excellence Framework (REF), which is currently the focus of a further review by Lord Stern.
Paul Ashwin, in the context of the proposed TEF, will examine the challenges of developing measures of teaching quality that do not simply reflect institutional prestige.
Both speakers will consider what a culture of ‘responsible metrics’ might look like for research and teaching, and the opportunities and obstacles to achieving this.
James Wilsdon is professor of research policy at the University of Sheffield, and was chair of the UK’s Independent Review of the Role of Metrics in Research Assessment and Management. He is now chairing a European Commission expert group on the future of altmetrics.
Paul Ashwin is professor of higher education at the Lancaster University and a co-investigator in the ESRC/HEFCE funded Centre for Global Higher Education.
This talk is part of the Designing the Academic Self series, sponsored by the Academics Writing project, which is part of the Literacy research Centre at Lancaster and the Northwest Doctoral Training Center and is open to early career researchers and doctoral students from Lancaster, Manchester and Liverpool universities. To book your place, visit Eventbrite.
What can and can’t metrics tell us?
Tuesday 26th April sees the third of four interactive workshops on the role of metrics in academic life, run by the Academics Writing project, alongside Masud Khokhar and Tanya Williamson of Lancaster University library, called What can and can’t metrics tell us?
Higher education institutions in the UK have to justify how they spend their research funding and demonstrate that this spending resulted in high quality research outcomes. This is achieved mainly through the REF (research excellence framework) and bibliometric measures associated with it. The research activity of individual academics, departments and institutions is measured in various ways and the resulting data used for a range of purposes. But are we asking the right questions? And are we using the data in an appropriate way?
This interactive workshop is part of the Designing the Academic Self series, and explores the ways in which quantitative indicators of research output are used, by both institutions and individuals. We begin by exploring what exactly are we trying to measure, and consider how well are the metrics we generate can do this. We will talk about case studies of how metrics are currently being used in different UK institutions, and will consider the potential positive and negative effects of these on academics.
To book a place, visit Eventbrite
How are metrics affecting academics’ writing practices?
Academics are increasingly subject to a range of ‘metrics’, attempts to produce a quantitative measure of the reach and impact of research (and, increasingly, teaching). Research metrics include measures at the level of the individual, such as the h-index (a measure of an individuals’ research output and the extent to which it is cited), at the level of individual papers such as numbers of downloads or numbers of citations, and at the level of journals such as the ISI ‘journal impact factor’, worked out on the basis of citation rates across a journal’s outputs. Newer metrics – ‘altmetrics’ – try, in addition, to capture the wider range of ways in which research publications can have an impact in the virtual world, producing measures based on numbers of views, downloads, saves, shares and recommendations online. All these metrics are becoming increasingly important in recruitment to academic positions, promotion, and of course assessment of academic departments via research assessment exercises such as the REF.
The existence and influence of such metrics has a range of effects on academics’ writing practices. They impact, for instance, on the selection of journals to write for, genres to write in and topics to focus on. Academics are increasingly expected to engage in self-promotional virtual practices including maintaining professional webpages and academic social networking, and these activities can have a direct effect on the newer altmetric quantitative indicators.
The second of four interactive workshops takes place this week, exploring the ways in which quantitative indicators informed by digital technologies are influencing academics’ writing practices.We will report on preliminary findings from the Academics’ Writing project and attendees will have the opportunity to discuss their own experiences of using and being measured via metrics.
Workshop title: Designing the academic self: How are metrics affecting academics’ writing practices at UK universities?
Date: Tuesday 16 Feb 2016, 1.00 – 3.00 pm
Venue: Lancaster University, Charles Carter A15
This series of workshops is run with support from the Northwest Doctoral Training Centre, and is free and open to doctoral students, staff and researchers from Lancaster, Liverpool, and Manchester Universities. Please register your attendance via Eventbrite. For dates and themes of the whole series of workshops, click here.
You must be logged in to post a comment.