Dr John Towse is a senior lecturer in the Psychology department. John’s work spans a number of research topics but includes: working memory, executive functions, mathematical cognition, cybercognition, human dimensions of cyber security and bibliometrics.
I had the pleasure of meeting with John recently to discuss the important role that the the PROSPR Group is playing in the psychology department and beyond!
Q: What is the PROSPR Group?
A: PROSPR stands for promoting open science practices and the idea is that we’re a semi formal group that meet to try and help each other in developing open science ideas, open science practices and looking at opportunities to do good work in this space. It’s a relatively new thing that’s of growing interest in psychology generally. We describe ourselves in more detail on our website
Q: How was it formed?
A: It was formed in a number of ways but our staff members were interested, independently in a number of open science issues and we gradually began to realise that each of us were doing different things but they gelled together. In addition we also had some early career researchers and postgraduates that had begun to hear about open science and had gone on some workshops that were available or they’d read up on these issues. So again, they were expressing interest in what this all meant. So we started talking to each other and then we started realising that we could make it all a bit more regular. My colleague Dermot Lynott in the Department took the initiative of setting up and leading some meetings. We discussed contemporary papers both about questionable contemporary research practices and new ideas and then we eventually came up with a name for the group and our mailing list and our web presence and so it’s developed from there.
Q: What activities are the group engaged in and how do they link with research practices?
A: We’re actually engaged in quite a few different activities. One aspect of what we do is raising visibility of open science in psychology and so as a group of people who are excited and positive about some of the issues and opportunities for improved practices, innovative ideas and use of technology – we’ve tried to spread the word. We’ve done that in a number of ways. Some of us have gone to other universities and groups to talk about open science issues and indeed to talk about the PROSPR group.
For example, Neil McLatchie has recently taken up an invitation to visit another institution to give a research seminar and explain what we did and how we did it and encourage them to set up a similar group, which we’re glad to say, they have. Others of us have gone to talk about particular issues and ideas that we’ve been working on and investing in to try to promote those. That’s sometimes been done through the university.
Separately, David Ellis has been advocating for good practices as a JISC data champion. These are just some examples of what all sorts of PROSPR members have been doing. Postgrads have discussed their enthusiasm for being involved in many-lab replications of key findings and so on.
We’ve also been getting involved in some of the university wide initiative such as Data Conversations. We’ve happy to do that to come and learn and to talk about the work that we’ve been doing.
Q: How do the groups activities relate to the substantive work of psychology researchers?
A: Increasingly they’re playing an important role. One of the things for example that we’ve discussed as a group, is the potential value in pre-registered studies as a method of doing science and indeed psychology. These slides expand on the significance of this area as does this article.
Several members of PROSPR have been pre-registering their work. They’ve set up a research design, a plan to ask some scientific questions, but then they’ve pre-registered it. This means that they’ve outlined the design they will use and they have specified what analysis they intend to do with the data once it’s collected. They’ve said what the size of the data set they hope to collect is, what the stopping rule will be for collecting data and so on. Having done all of that preregistered work they’ll then go out and do the experiment, following the protocol that they have specified in advance. They’ll analyse the data primarily on the basis of what they said they would do in the pre-registered report. Any additional work they do, they then explicitly signal as being exploratory or non-planned work. Increasingly PROSPR members have been implementing pre-registration because they think it’s a good way to do science. It’s transparent and it’s open.
Pre-registration has also been implemented by some researchers because certain journals have a pre-registration route to publication. So, the idea is that the journal will evaluate the pre-registered plans and will make a decision about whether they think that’s the right experiment to be doing at that point in time and should lead to illuminating answers. In that case, they will give an in principal acceptance of publication before the data are collected. That means that you can collect the data, analyse it and write it up without feeling that the value to a journal is predicated on a particular answer or a particular significant result. Again, it works towards transparency and openness and avoiding file-draw problems, and indeed other problems that are known to be out there in psychology and indeed science. This is an initiative that a number of people have been engaging in and what PROSPR members are finding is, that it’s potentially a good way to do their work.
Q: Do you think that similar initiatives (pre-registered studies) could be valuable for other disciplines?
A: Yes I certainly do.
I’ve personally been involved in some interdisciplinary projects with colleagues in other departments and disciplines and we’ve thought about the value of doing some pre-registered studies there. So, I’ve discussed with them how preregistration might work and why it would be good, in terms of transparency and setting out prediction in advance. Not only is this about these projects themselves, but also as an example of how this might work. There are a number of other disciplines that might find this of value. It should be said that psychology hasn’t just developed this notion independently. The way that psychologists uses pre-registered studies draws from other fields and practices. It’s not psychology specific, even though we often deal with noisy data and have many ways to analyse our findings. So, although psychology has a number of issues to do with the vagaries and the variabilities of data that make this worth thinking about, it’s going to be meaningful in many sciences. We’re not trying to impose it on other fields but it’s something that’s worth thinking about for other disciplines as an organic discussion.
Q: Do you think that similar initiatives (the PROSPR group itself) could be valuable for other disciplines?
A: Again, I think there are some really useful conversations to be had about some of the things that we’ve been doing, as well as learning from others. So for example, PROSPR members have presented ideas in Lancaster data conversations and one of the things that we’ve talked about is the experience of running an open access, free to publish journal for psychology, such as, the Journal of Numerical Cognition and others within the Free Journal Network
We’ve tried to be open and honest about how that has worked and how one can build on that idea to then offer a greater emphasis on providing data for Journal papers that are published. So, that’s one thing that we’ve talked about in PROSPR but I think that would also be relevant to other disciplines who would also find value in open access, free to publish journals.
To give you another example Dermot has contributed to the Data Conversations about the value of making data available and the many different and sometimes surprising ways that people can take up and reuse those data productively. He walked people through examples of a data journey and illustrated how data were being used in different ways. In other words, how making a data deposit had led to a whole variety of positive consequences from that research project and themselves led on to other funded research projects and so on. That story isn’t one that’s just confined to psychology. That was one of the reasons that he was keen to talk about it because again other disciplines will also have opportunities to make data available.
Hearing from people who’ve done that and reflected on how those data were used by others in different ways… it then helped keep momentum going in asking some of the intellectual questions that surrounded those data. That’s very useful to have. Just as it’s been very useful in the data conversations to hear from people in other departments and working in other disciplines, talking about their own issues and then having the chance to reflect about how they apply to psychology too.
Q: What can the university or the library more specifically do to support the PROSPR group?
A: One of the things that PROSPR is very keen on is trying to promote a more open and transparent scientific landscape and that involves being able to deposit data. There are as we know, many different ways that can happen. The university repository is one of those. So, you can deposit data within the university repository, that data can have its own DOI so that it can be cited and linked with research papers. And, that’s a really good system to have available. There are a variety of reasons why people may be using another repository but having that available is a really good thing. We’ve found this to be an important resource not only for research but also for teaching students about open science and open science practices. We can point to the university repository as an example of where data are being held – some of those data being psychological. And then we can invite students to look at that data, inspect it and to use it for their research development or their learning purposes to test their analytic skills and so on. Having that there is really useful. Having other repositories that can be used for other purposes is great but from a student perspective it makes it more distributed and so for students it’s harder to know where to go or to how to keep track of where it might be.
Keeping the conversations going around open science and around data is also really useful. It’s useful for us to be part of that wider university experience. In a whole variety of ways to have that support and positive drive towards considering innovations and ideas that might work is a really productive thing to have.
Q: For your part as a leader in your field how important is it for you, that postgraduate students and early career researchers develop RDM skills at an early stage?
A: For me, that’s really important. In psychology as the landscape changes, what we want to do is to encourage students to see opportunities and positives in open science and the earlier that we can engage students in this the more natural it’s going to be for them to think about this in a positive way. Not merely as a regulatory burden or bureaucracy. So it’s also much easier for them to take on any ideas or initiatives that we have and see ways to make them happen.
One of the other projects that I’ve been involved in with Rob Davies and a student Ben Gooding is to set a data portal for student’s records.
Initially this was conceived to be a master’s level research project resource. That is, developing a system that allows masters students to engage in the process of data management – all the elements such as preparing for data depositing, coding their data, explaining it and so on. With support through different places in the university such as a Faculty Teaching Development Grant we’ve been setting up this online system that displays student records and provides ways to access the data. This is part of a process of training students in data management. To introduce them to the advantages of data deposits in a controlled ecosystem – a safe space or sandbox – rather than in the institutional repository.
Allowing students to explore these issues and to see what others have done and think about what would work for them and their projects. So, what we’ve doing there is to try and make our ideas of open science and ideas of transparency feed through to the curriculum, feed through to the student experience. What we would like is that from the fairly early stages the students get actively involved in what some of these open science ideas mean and they can explore some of the positive issues. And, explore some of the challenges too. Then that becomes part of their journey that means in the long term there will be a cohort of researchers that are not only familiar with issues of open science but familiar with practices as well and therefore will go out and develop their own perspectives and initiatives that are useful to the discipline.
We’ve got the system working and we’ve got the code on GitHub.
Potentially other institutions can download it and set up their own independent version of this. I’ve also had one or two conversations with other psychology departments about them providing us with their datasets and then we might uploading them to our system, ensuring we can tell the institutional source of the projects. So one of the metadata elements we have implemented is that there is a publisher for each project record. At the minute that’s not terribly meaningful because all of the projects are at Lancaster but suddenly, if we start to get 40-50 projects a year from another institution then it becomes pretty useful. It also means that students at different institutions can see what there others are doing. To learn about experiences beyond geographical walls.
Q: Who are the uploaded datasets visible to?
A: How we’ve developed currently, is that within the department we’re collecting the datasets from students but we’re not uploading them in full view. What we’re doing is uploading the project record and the metadata. So if somebody wants the data for their own research or for their own analytic teaching practices they can contact us with the project record information and we will decide if we’re happy to release the data. We made that decision not to upload the data directly because that avoids potential problems of, for example, non-anonymised data that we don’t have the time to check for.
Potentially though, datasets can be made open. So, other people who did feel that they could scrutinise the data or were confident that the data were unproblematic in terms of being openly available and shared and so on can do so. We’ve just made a decision to by-pass that open option at this time.
Have there been any instances of dataset being requested or is it still early days.
There have been instances within the department when supervisors have wanted to go back to a students data but they can’t contact that student. They’ve been able to use the system that we’ve created to go back and check. Because this system is under development I’ve spoken about this project at departmental meetings and so on. But we’ve not reached the point where it’s sufficiently widely known that we’re getting enquiries coming in. That will hopefully happen in the future as we write about and describe the work more.
Q: So this is all part of a wider initiate within Psychology?
So Psychology has had a number of unfortunate issues such as data fraud and retractions of large numbers of papers from individuals who’ve been found to engage in fraud. That has created a obvious problem that the notion of data deposits maybe helps in some ways to combat But we’ve also had analysis that has looked as issues of poor levels of replication of publish studies.
This has brought into focus the potential impact of file draw problems, sample size problems and so on. Pre-registered studies or lab replication studies have been seen as a way of addressing that. Some of open science has certainly been a response to those negative incidents. Placing greater focus on demonstrating the integrity of the work that you’ve done. Some of what PROSPR has done has tried to address that. However, I also think it is also about a more positive agenda which is apparent even aside from these cases where things have gone very wrong. It’s about making the case that depositing data is a good thing because other people can come to use it in ways that you hadn’t anticipated. Data depositing is good because, technology and analysis moves on so in the future people can come along, do new analytic things with the data that you couldn’t have envisaged at the time, and indeed maybe find ways of reinforcing what you’ve done. Unless you’ve made the data available it’s going to be impossible to do. There are also those case where people are joining datasets together for meta-analysis and studies.
The interview was conducted by Joshua Sendall, Research Data Manager.