WEBVTT
Kind: captions
Language: en

00:00:02.560 --> 00:00:09.440
all right welcome everyone it is 10 o'clock so we 
are going to get started uh to make sure we run  

00:00:09.440 --> 00:00:15.120
this in a timely manner um thank you for coming 
and I think this is going to be a really engaging  

00:00:15.120 --> 00:00:20.480
event today, and so i just want to get some 
housekeeping rules out of the way before we start.  

00:00:20.480 --> 00:00:25.920
My name is Rachel Miles and I'm the research 
impact librarian here at Virginia Tech so  

00:00:25.920 --> 00:00:31.440
I was helping coordinate this event today. 
For housekeeping rules uh just keep in mind  

00:00:31.440 --> 00:00:38.640
that when you want to ask a question use the Q and A 
function rather than the chat. This makes it easier  

00:00:38.640 --> 00:00:44.160
for us to keep track of the questions. We also 
encourage you to write your questions into the Q and Q  

00:00:44.160 --> 00:00:50.960
when you think of them rather than at the end of 
a of a talk or the discussion. Uh this helps it  

00:00:50.960 --> 00:00:56.400
so that we can see them coming in, and we can get 
to them in a timely manner rather than populating  

00:00:56.400 --> 00:01:03.760
all at once. The chat is also available to um 
attendees so you can use that to talk amongst  

00:01:03.760 --> 00:01:08.560
yourselves, make comments. It's really there for 
you to just kind of engage with one another,  

00:01:09.120 --> 00:01:15.440
but you can also do so publicly using the hashtag 
#countingOA, so i'll put that in the chat if you're  

00:01:15.440 --> 00:01:22.800
on Twitter, and you would like to do that, feel free. 
Um if you also need to ask any technical questions  

00:01:22.800 --> 00:01:28.960
uh or just questions that are not directly 
for the speaker you can use the chat function  

00:01:30.480 --> 00:01:35.680
sent that to only the panelists. I made the 
mistake myself so #countingOA, it's in the chat.  

00:01:35.680 --> 00:01:40.880
Now to everybody and not just panelists you'll 
want to toggle to make sure that all panelists  

00:01:40.880 --> 00:01:44.880
and attendees are selected or if you just need 
to ask a question - a technical question - you could  

00:01:44.880 --> 00:01:52.160
do all panel all panelists and that'll go to our 
end so that somebody can help you. Jordan Kuneyl is  

00:01:52.160 --> 00:01:55.440
our technical support and he can help 
with any of those types of questions.

00:01:58.480 --> 00:02:02.480
All right, subtitles and captions are also 
available. We have a live captioning service  

00:02:02.480 --> 00:02:07.520
and you'll just go to 'more' and then you can 
click 'show subtitle' and you should be able  

00:02:07.520 --> 00:02:12.960
to see those display on your screen. The event is 
being recorded and it will be archived on Virginia  

00:02:12.960 --> 00:02:17.840
Tech's institutional repository, VTechWorks. A 
link will be sent to all registrants within the  

00:02:17.840 --> 00:02:23.920
next few days and there's also a transcript of 
the event available and we will work on making  

00:02:23.920 --> 00:02:30.080
sure that captions are also on there for the event. 
We have the keynote talk first followed by some  

00:02:30.080 --> 00:02:37.200
questions for the keynote speaker, Dr. Elizabeth 
Gadd, and then we will have a discussion panel with  

00:02:37.200 --> 00:02:42.080
faculty members from Virginia Tech moderated 
by Dean Tyler Walters of University Libraries.

00:02:42.800 --> 00:02:49.360
And before the event starts, we are trying to 
read a land acknowledgement for uh public events  

00:02:49.360 --> 00:02:55.920
and Virginia Tech events that we hold and so I 
will read that statement now. We acknowledge the  

00:02:55.920 --> 00:03:00.960
Tutelo and Monacan people who are the traditional 
custodians of the land on which we work and live, 

00:03:01.520 --> 00:03:06.720
and recognize their continuing connection to the 
land, water, and air that Virginia Tech consumes.  

00:03:07.360 --> 00:03:12.640
We pay respect to the Tutelo and Monacan nations 
and to their elders past, present, and emerging. 

00:03:12.640 --> 00:03:17.600
I'm grateful to the American Indian and Indigenous 
Community Center at Virginia Tech for providing  

00:03:17.600 --> 00:03:24.480
this land acknowledgment statement. Finally I'd 
like to introduce our speaker Dr. Elizabeth Gadd.  

00:03:24.480 --> 00:03:28.720
Dr. Gadd is a scholarly communications specialist 
working as a Research Policy Manager in  

00:03:28.720 --> 00:03:35.120
publications at Loughborough University in the United 
Kingdom. She chairs the INORMS Research Evaluation  

00:03:35.120 --> 00:03:41.200
Working Group, the ARMA Research Evaluation Special 
Interest group, and the LIS-Bibliometrics Forum. She  

00:03:41.200 --> 00:03:46.880
also founded the Bibliomagician Blog and was the 
recipient of the 2020 INORMS award for excellence  

00:03:46.880 --> 00:03:52.240
and research management leadership. she has spent 
most of her career in libraries and doing research  

00:03:52.240 --> 00:03:58.320
into the rights issues related to scholarly 
activities of UK higher education. I know Dr. Gadd  

00:03:58.320 --> 00:04:03.040
as the Chair of the LIS-Bibliometrics Forum 
which I started following early in my career  

00:04:03.040 --> 00:04:09.360
a few years ago, the Bibliomagician Blog, for which I
started as the blog content development officer.

00:04:10.320 --> 00:04:13.360
Rachel can you slow down a little bit?

00:04:13.360 --> 00:04:17.200
Oh yes i'm sorry. Thank you. Yes, yes I'm sorry. 

00:04:18.480 --> 00:04:23.600
Um, I know Dr. Gadd as the chair of the 
LIS-Bibliometrics Forum and the Bibliomagician

00:04:23.600 --> 00:04:29.120
Blog, and I will also copy paste what 
I have here if that's helpful for everybody.

00:04:31.520 --> 00:04:38.320
Um and I'm always grateful for her leadership 
and her courage to stand up and ask the hard  

00:04:38.320 --> 00:04:44.320
questions to researchers, administrators, and 
data providers. Without further ado I present  

00:04:44.320 --> 00:04:49.360
Dr. Gadd and her keynote talk, "Counting what 
counts in recruitment, promotion and tenure."

00:04:49.360 --> 00:04:55.840
I'm going to copy paste that 
over in case anybody misses that.

00:04:59.360 --> 00:05:03.520
Thank you so much Rachel for that lovely 
introduction. I'm hoping you can hear me I'm hoping  

00:05:03.520 --> 00:05:07.840
you can see my slides. I'm hoping that someone will 
be able to let me know if neither of those are  

00:05:07.840 --> 00:05:14.320
happening um so i can see our thumbs aloft which 
is great. So so a very good morning to you from my  

00:05:14.320 --> 00:05:19.680
afternoon here in the UK and thank you very much 
for inviting me to share a few words at your 

00:05:19.680 --> 00:05:24.000
Open Access Week event. I'm going to try and speak 
slowly because I know this is being live  

00:05:24.000 --> 00:05:31.680
captioned which is fantastic. Um okay so I wanted 
to say a little bit about myself but the outset  

00:05:31.680 --> 00:05:37.520
Rachel said some good stuff, some kind stuff, thank 
you. I think it does help to know uh where someone  

00:05:37.520 --> 00:05:43.280
is coming from so i would describe myself as a 
librarian currently working in research policy  

00:05:43.280 --> 00:05:49.040
at Loughborough University in the UK. Loughborough
University is a research intensive institution  

00:05:49.040 --> 00:05:54.240
uh in the midlands of the UK. I was asked just 
to explain where we were, and we have the largest  

00:05:54.240 --> 00:06:01.200
campus in Europe. Um I had up and also a number of 
groups in this kind of research evaluation space  

00:06:01.200 --> 00:06:05.120
uh including the LIS-Bibliometrics forum 
that Rachel's mentioned which is home of the  

00:06:05.120 --> 00:06:12.000
Bibliomagician blog ably managed by Virginia Tech's own 
Rachel Miles and then when I moved into research  

00:06:12.000 --> 00:06:17.680
policy five years ago. I found myself becoming 
the champion of the UK association of research  

00:06:17.680 --> 00:06:23.520
managers and administrators. Research evaluation 
SIG, and then two years ago I was honoured to be  

00:06:23.520 --> 00:06:29.680
invited to form the INORMS Research Evaluation 
working group that consists of 12 international  

00:06:29.680 --> 00:06:35.120
research management societies. I'm going to share 
some of the exciting work that that group is doing  

00:06:35.120 --> 00:06:43.680
today. So my topic today is counting what counts 
in recruitment, promotion and tenure, and if it's  

00:06:43.680 --> 00:06:49.200
not already obvious what the connection is between 
research evaluation and open access, i hope to make  

00:06:49.200 --> 00:06:56.320
that clear. So what I plan to do is to firstly 
identify what the academy currently rewards  

00:06:56.320 --> 00:07:02.400
and the consequences of that, and then to explore 
what the academy really values, and then I'm going  

00:07:02.400 --> 00:07:09.600
to look at 101 ways that we can measure what we 
value rounded up to the nearest 101. Of course  

00:07:10.160 --> 00:07:16.160
before thinking about how we might persuade 
higher education leaders to actually measure  

00:07:16.160 --> 00:07:24.640
what they value. So what does the academy currently 
reward? So earlier this year at the King's College  

00:07:24.640 --> 00:07:30.880
Open Research Conference Marcus Munafo summed up 
the situation brilliantly with this slide. He said  

00:07:30.880 --> 00:07:39.440
scientists are incentivized to publish and obtain 
grant funding, and uh that was the end of the list.

00:07:39.440 --> 00:07:45.360
And of course you could argue that funding only 
really counts because it leads to publications,  

00:07:45.920 --> 00:07:52.160
but we all know it's not just any publications 
that count. It is journal articles, so this chart  

00:07:52.720 --> 00:07:59.600
shows the submissions to the UK Research 
Assessment Exercise between 1990 and 2008 in  

00:07:59.600 --> 00:08:05.920
different fields, and you can see the proportion of 
journal articles. The red bars went up pretty much  

00:08:05.920 --> 00:08:13.440
every time, even in humanities subjects, and we also 
know that overall in 2014, over 80 percent of all  

00:08:13.440 --> 00:08:19.520
outputs submitted to the UK REF as it is now, were 
journal articles, although some recent research has  

00:08:19.520 --> 00:08:25.920
shown the tide might be turning on that, um but to 
date this has been the story, but it's not just any  

00:08:25.920 --> 00:08:31.600
journal articles that we reward. It's those with 
an exciting story to tell. So this study here  

00:08:31.600 --> 00:08:37.520
was fascinating, and it looked at the cumulative 
effect of reporting and citation biases and what  

00:08:37.520 --> 00:08:44.080
eventually ends up in the scholarly record. So they 
started with all of the clinical trials registered  

00:08:44.080 --> 00:08:49.520
relating to the treatment of depression, and the 
red dots on bar on the left hand side show  

00:08:49.520 --> 00:08:54.480
the proportion of negative trials relative to 
those that have positive results. The green dots  

00:08:55.200 --> 00:08:59.120
on the next bar shows the proportion of those 
positive and negative trials that made it  

00:08:59.120 --> 00:09:04.320
into the literature at all. The next bar shows 
a proportion of those published studies that  

00:09:04.320 --> 00:09:10.480
actually focused on the negative findings; the next 
shows a proportion of published studies that put a  

00:09:10.480 --> 00:09:17.280
positive spin on those negative findings, and the 
final bar shows which of those studies ultimately  

00:09:17.280 --> 00:09:23.680
got cited by resizing the dots according to 
citedness. And from this we can conclude that  

00:09:23.680 --> 00:09:29.680
there is absolutely no citation glory. The tiny 
red dots on the right hand side to be found in  

00:09:29.680 --> 00:09:36.640
negative trials no matter how critical they are to 
scholarship, and of course the more sinister issue  

00:09:36.640 --> 00:09:42.800
here is that this can give us a false impression 
of what's true. So this example shows that 95 out  

00:09:42.800 --> 00:09:49.520
of 100 studies found no association between 
autism and brain region, but five did find a  

00:09:49.520 --> 00:09:54.800
false positive result, and of course those five got 
published. So now when you do a search for autism  

00:09:54.800 --> 00:10:01.520
and cerebellum, you find one negative and five 
supporting studies, which is deeply problematic, 

00:10:03.360 --> 00:10:08.400
and of course, one of the reasons we only publish 
positive studies is that the more prestigious  

00:10:08.400 --> 00:10:14.720
journals won't touch negative studies, because they 
don't get cited; we have to publish in prestigious  

00:10:14.720 --> 00:10:20.000
journals, because most hiring committees are 
looking for publications in prestigious journals,  

00:10:20.000 --> 00:10:26.880
as this study of U.S. promotion and 10-year 
criteria found where 40 of U.S. research intensives  

00:10:26.880 --> 00:10:32.000
had requirements around journal impact factors or 
brands, and the only thing that really surprised  

00:10:32.000 --> 00:10:35.920
me about this was that the figure wasn't 
higher. And of course when you factor in those  

00:10:35.920 --> 00:10:41.280
institutions that use them informally but just 
don't write it down, the figure probably is higher. 

00:10:42.400 --> 00:10:47.360
And the problem with prestigious journals 
is that they are mainly closed or hybrid, 

00:10:47.360 --> 00:10:55.120
so Web of Science boasts that 23 of the papers 
they index have a free legal green pure or hybrid  

00:10:55.120 --> 00:11:02.880
gold version, but by my calculation that means 77 
are still closed and of course Web of Science and  

00:11:02.880 --> 00:11:09.200
Scopus are still the databases that provide many 
of the bibliometrics rewarding our publication  

00:11:09.200 --> 00:11:16.560
culture, and those databases are of course heavily 
skewed towards STEM subjects as this study found  

00:11:16.560 --> 00:11:23.840
last year where every database they investigated 
indexed fewer arts and humanities outputs than all  

00:11:23.840 --> 00:11:30.560
other subjects. And of course as you can see 
from the Scopus journal selection criteria here  

00:11:30.560 --> 00:11:37.360
they index titles mainly in English, and then 
the real proof your publication has made it  

00:11:37.360 --> 00:11:42.320
is if it's cited. Uh as you can see here 
for institutions with elsewhere SciVal  

00:11:42.320 --> 00:11:46.320
that their measure of a researcher's 
overall research performance, 

00:11:46.320 --> 00:11:51.840
and notice the title is not publication 
performance its overall research performance.  

00:11:51.840 --> 00:11:58.880
Their indicators for research performance include 
five citation metrics and your publication count, 

00:12:00.240 --> 00:12:05.360
and for those of you who have been following the 
conversations on LIS-Bibliometrics about scite,  

00:12:05.360 --> 00:12:10.800
that's a new service that uses natural language 
processing to try to discern whether a citation  

00:12:10.800 --> 00:12:16.080
supports or disputes the paper it cites, 
you'll know that in future it may not just  

00:12:16.080 --> 00:12:21.120
be citations that will count but whether a 
machine somewhere thinks they're positive, 

00:12:21.120 --> 00:12:27.200
and this example is one of my own papers, and the 
allegedly disputing paper was also written by me  

00:12:27.200 --> 00:12:34.000
and surprisingly does not dispute my earlier 
work in any way. So to recap what we currently  

00:12:34.000 --> 00:12:40.400
incentivize in the academy is journal articles 
reporting positive results in prestigious journals  

00:12:40.400 --> 00:12:46.320
that are mainly closed or hybrid indexed by 
commercial bibliographic databases that are biased  

00:12:46.320 --> 00:12:53.120
towards STEM subjects written in English and that 
are well cited as measured by bibliometric tools. 

00:12:54.400 --> 00:13:00.400
So what might the consequences of only 
rewarding publications and citations be?

00:13:01.680 --> 00:13:06.560
Well surprise, surprise, valuing only 
one thing leads to a lack of diversity.

00:13:06.560 --> 00:13:13.600
Firstly, in the scholarly record itself so far 
from bibliodiversity, we have bibliohomogeneity.

00:13:13.600 --> 00:13:18.720
Smaller independent publishers can't survive.
There's downward pressure on other arguably  

00:13:18.720 --> 00:13:24.880
more important outputs from research, such as data 
code, protocols, everything's written in English, etc.  

00:13:25.920 --> 00:13:30.960
Secondly, it leads to a lack of diversity 
amongst those actually able to access the  

00:13:30.960 --> 00:13:36.880
scholarly record, because it's closed. This is 
Open Access Week. We need to point that out.  

00:13:36.880 --> 00:13:42.320
And of course as we move towards the scholarly 
record becoming open by default, it leads to  

00:13:42.320 --> 00:13:48.080
a lack of diversity amongst those actually 
able to contribute to the scholarly record,  

00:13:48.080 --> 00:13:54.560
because APC's, article processing charges, are 
increasingly aligned with publication metrics.  

00:13:55.360 --> 00:14:01.840
And of course, it also leads to a narrowing of 
those actually rewarded by the scholarly record.  

00:14:02.400 --> 00:14:08.960
So Indonesia recently introduced a new researcher 
ranking system that uses the volume of articles  

00:14:08.960 --> 00:14:15.040
and citations from Google Scholar and Scopus for 
deciding who are the best researchers and for  

00:14:15.040 --> 00:14:22.240
allocating funding to these people. The resulting 
top 10 looks like this: the number of women in the  

00:14:22.240 --> 00:14:27.760
top ten: one; the number of researchers in the 
fields of social sciences arts and humanities: zero;  

00:14:28.320 --> 00:14:35.600
So how do we fix this? You know what's the 
alternative to publications being the single  

00:14:35.600 --> 00:14:42.400
currency of research credit? Well of course it's by 
returning to what we actually value about research:  

00:14:42.400 --> 00:14:49.040
researchers and research institutions. And this 
is where I want to briefly introduce the INORM's  

00:14:49.040 --> 00:14:55.440
SCOPE model for evaluating responsibly. This is 
a simple five-stage process developed by the INORMS  

00:14:55.440 --> 00:15:00.880
Research Evaluation Working Group, and we 
introduced this model because normally research  

00:15:00.880 --> 00:15:06.720
evaluation starts and finishes with number three: 
what are our options for evaluating this thing?  

00:15:06.720 --> 00:15:12.080
Let's go and evaluate this thing. But we believe 
it's critical that we start with number one: what  

00:15:12.080 --> 00:15:20.560
do we actually value about this thing that we 
want to evaluate? So step one of SCOPE is to start  

00:15:20.560 --> 00:15:25.680
with what you value about the thing that you're 
evaluating, and it sounds obvious but all too often  

00:15:25.680 --> 00:15:32.000
we don't start our evaluations here. We start with 
what other people value, so university rankings or  

00:15:32.000 --> 00:15:38.400
funders or with historical values, which might 
have been meaningful 30 to 50 years ago before we  

00:15:38.400 --> 00:15:44.160
understood what the consequences of those values 
might be, and I think it's worth asking ourselves  

00:15:44.160 --> 00:15:51.120
when did we last review our RPT criteria what has 
changed in the last 30 years? And of course the  

00:15:51.120 --> 00:15:57.600
other thing that we do here is to start with the 
data we have available to us and to work backwards.

00:15:58.320 --> 00:16:04.000
This is called the street light effect as 
illustrated by this cartoon. Where did you lose  

00:16:04.000 --> 00:16:10.640
your wallet? In the park. So why are you looking 
here? Well this is where the street light is. And  

00:16:10.640 --> 00:16:15.920
many, many evaluations rely on bibliometric data, 
not because the volume of publications or even  

00:16:15.920 --> 00:16:21.840
their citedness is what they value, but because 
it's so readily available and so easy to use.

00:16:23.760 --> 00:16:28.640
Once we're clear on what we value, we need 
to consider the context for our evaluation.  

00:16:28.640 --> 00:16:34.160
And I hear many arguments in my role about which 
indicators are responsible and which are not, but  

00:16:34.160 --> 00:16:39.920
we can't have that conversation without knowing 
the context, the why and the what in which the  

00:16:39.920 --> 00:16:45.600
indicator is going to be used. Now of course we're 
thinking about recruitment and promotion today,  

00:16:45.600 --> 00:16:51.200
but I think it's worth remembering that there are 
many reasons for evaluating, and this matrix here  

00:16:51.200 --> 00:16:58.000
lists six key purposes alongside various entities 
of different sizes, and attempts to indicate what  

00:16:58.000 --> 00:17:05.200
the associated impacts, and therefore risks, might 
be. So for example, plotting the publication volume  

00:17:05.200 --> 00:17:09.680
of one country against that of another for the 
sole purpose of understanding their relative  

00:17:09.680 --> 00:17:15.120
production levels has very little impact on the 
country's under assessment, so we've labeled that  

00:17:15.120 --> 00:17:21.440
green. However any form of assessment at the level 
of the individual that results in reward has a  

00:17:21.440 --> 00:17:26.800
high impact on those individuals and therefore 
carries a greater degree of risk, and I would  

00:17:26.800 --> 00:17:33.280
say that the use of bibliometrics in these various 
settings maps pretty neatly onto these color codes.  

00:17:35.120 --> 00:17:41.200
And of course the other key context consideration 
is discipline, so a promotion approach that works  

00:17:41.200 --> 00:17:45.760
for folks in one discipline will not necessarily 
work for those in another. And at Loughborough, 

00:17:45.760 --> 00:17:50.160
we're currently looking at this with the 
particular focus on arts and humanities scholars.

00:17:51.680 --> 00:17:55.760
So once we've thought through what we value about 
the research activity we're measuring and the  

00:17:55.760 --> 00:18:02.240
context, it's worth pausing before we launch into 
our options to consider whether we actually need  

00:18:02.240 --> 00:18:08.320
to evaluate at all. Now this might sound like an 
odd thing to say, but this is really with an eye on  

00:18:08.320 --> 00:18:13.920
the increasing use of measuring to incentivize 
certain behaviors, and as this is Open Access  

00:18:13.920 --> 00:18:19.280
Week, I have to say that this is often seen as a 
solution for getting people to engage with Open  

00:18:19.280 --> 00:18:25.680
Access, but we have to remember that incentivizing 
behaviors can be done in many ways, like actually  

00:18:25.680 --> 00:18:32.160
making it so easy for someone to act in that way 
that they don't need any other form of incentive.

00:18:33.120 --> 00:18:38.320
And I'm a massive fan of the approach taken by 
the Alan Turing Institute where their goal is  

00:18:38.320 --> 00:18:46.560
to make data reproducibility too easy not to do 
rather than rank people on their ability to do it.

00:18:48.640 --> 00:18:56.320
But when we do evaluate we should always do so 
with the evaluated. How often do we get feedback  

00:18:56.320 --> 00:19:01.120
from those seeking promotion and tenure or those 
who've recently been through our recruitment  

00:19:01.120 --> 00:19:06.720
processes to understand their experiences and 
to invite them to help co-design a better way  

00:19:06.720 --> 00:19:12.640
of doing this evaluation? - is something we 
should do WITH people and not TO people.

00:19:15.200 --> 00:19:20.800
Then we have other options, and then this is a 
reminder really that you do have options. We  

00:19:20.800 --> 00:19:26.720
don't have to reach for bibliometric data all the 
time, and the important thing to consider here is  

00:19:26.720 --> 00:19:32.960
whether your indicator is a suitable proxy for the 
thing that you're evaluating. We think the rule of  

00:19:32.960 --> 00:19:38.800
thumb here is that generally speaking, quantitative 
measures are generally for quantifiable things,  

00:19:38.800 --> 00:19:44.800
such as citations, publications, money, or students, 
and qualitative measures are for qualifiable  

00:19:44.800 --> 00:19:51.120
things, so excellence and value, and we should 
be especially careful if using quantitative  

00:19:51.120 --> 00:19:57.600
indicators as a proxy for qualitative things. 
A highly cited candidate does not equal  

00:19:57.600 --> 00:20:03.600
a highly suitable candidate. A candidate from 
a highly ranked institution does not equal a  

00:20:03.600 --> 00:20:10.320
better candidate. Often the selection of options 
for evaluating marks the end of the evaluation.  

00:20:10.320 --> 00:20:16.000
Designer folks launch in, and just do their thing, 
but we feel there's a critical probe stage that  

00:20:16.000 --> 00:20:20.560
needs to be considered first, and we think there 
are four questions that need to be answered here, 

00:20:20.560 --> 00:20:26.560
particularly when it comes to RPT processes; so 
we need to ask firstly, who does this discriminate  

00:20:26.560 --> 00:20:34.320
against? How diverse are the candidates regularly 
being rewarded by this process? Secondly, how might  

00:20:34.320 --> 00:20:39.040
this approach be gamed; so how selective 
is that publication list you're looking at, 

00:20:39.040 --> 00:20:43.120
what contribution did this candidate actually 
make to the activities they're listing?  

00:20:44.160 --> 00:20:49.120
Thirdly, what might the unintended consequences 
of this approach be both on the development of  

00:20:49.120 --> 00:20:54.320
your teams internally but ultimately on 
wider community agendas around openness, 

00:20:54.320 --> 00:21:00.560
rigor, societal impact? And finally does 
the cost of measuring outweigh the benefit? 

00:21:01.120 --> 00:21:06.640
And of course, one is seeking to fairly evaluate a 
large number of applicants, the costs can be really  

00:21:06.640 --> 00:21:14.320
high. So how do we get that balance right and the 
'E' of SCOPE is to then run your evaluation and to  

00:21:14.320 --> 00:21:19.120
evaluate it. So that was a very quick fly through 
SCOPE, and there's a lot more on our website.  

00:21:19.120 --> 00:21:25.680
And I've left the URL there for you if you're 
interested in pursuing that. So I've argued that we  

00:21:25.680 --> 00:21:32.480
need to measure what we value in our RPT processes, 
but the question remains as to what we actually do  

00:21:32.480 --> 00:21:37.280
value in the academy, and obviously we all have 
our own answers to that question. That's quite  

00:21:37.280 --> 00:21:43.680
quite rightly, but I think that in order to build 
strong and successful academic communities, we need  

00:21:43.680 --> 00:21:50.240
firstly a broad diversity of people. We know that 
companies with diverse management teams have 19  

00:21:50.240 --> 00:21:56.080
higher revenue and that 87% of the time 
diverse teams make better decisions. 

00:21:57.120 --> 00:22:03.040
We should also value a variety of disciplines 
as COVID has shown us, it takes more than medics  

00:22:03.040 --> 00:22:07.920
to fight a pandemic. We need communicators 
and psychologists and designers and so on. 

00:22:08.720 --> 00:22:13.760
We need a diversity of findings. We need all the 
results, positive and negative, to make the best  

00:22:13.760 --> 00:22:19.760
decisions. We need a diversity of outputs, 
data, design, sculpture, trade journals, etc. 

00:22:21.040 --> 00:22:26.080
I saw this wonderful cartoon on twitter recently 
which sums it up rather beautifully: all modern  

00:22:26.080 --> 00:22:32.160
digital infrastructure rests on a small but highly 
significant project being maintained by a single  

00:22:32.160 --> 00:22:37.680
soul in their bedroom. And of course if they didn't 
write a paper on it, it currently wouldn't count  

00:22:37.680 --> 00:22:44.720
for anything. We value a much broader diversity 
of impacts than we currently reward. Societal  

00:22:44.720 --> 00:22:50.800
economic public engagement. And of course we value 
a far greater diversity of research activities  

00:22:50.800 --> 00:22:58.800
than we reward. We need leadership, collegiality, 
innovation, integrity. And the Hong Kong manifesto  

00:22:58.800 --> 00:23:05.120
for assessing researchers seeks to foster just 
this by calling for responsible research practices, 

00:23:05.120 --> 00:23:10.640
such as reproducibility, transparent reporting, 
across all stages of the research process, 

00:23:10.640 --> 00:23:16.400
and open science, not just open access in 
addition to the diversity we've just talked about.

00:23:18.400 --> 00:23:22.080
Wo how do we measure all of 
these things. Where do we start? 

00:23:22.080 --> 00:23:27.600
Well we have some help here, and you might 
not be able to see this on a small screen, 

00:23:27.600 --> 00:23:33.440
but the DORA website, the Declaration of Research 
Assessment, has started a really helpful list of  

00:23:33.440 --> 00:23:38.640
research evaluation innovations, and I've thrown a 
few of my own in there as well just to celebrate  

00:23:38.640 --> 00:23:44.080
folks really moving away from the journal 
impact factor as a sole indicator of research  

00:23:44.080 --> 00:23:49.840
success. And when you analyze those research 
evaluation innovations as I've done loosely here, 

00:23:50.400 --> 00:23:56.080
you'll see that the vast majority are around how 
to evaluate individuals for jobs or projects for  

00:23:56.080 --> 00:24:01.360
funding, so when it comes to recruitment promotion 
and tenure as we're discussing today, you're pretty  

00:24:01.360 --> 00:24:08.800
much spoiled for choice. So here's a couple of 
my favorite examples so UT Southwestern provides  

00:24:08.800 --> 00:24:13.840
shortlisted candidates with questions prior to 
skype interviews to give them reflection time, 

00:24:13.840 --> 00:24:18.080
and they ask them not only what they've done 
in the past but what their future plans are,

00:24:21.360 --> 00:24:28.720
and we've also got um an example here from the 
Charity University Hospital in Germany, which asks  

00:24:28.720 --> 00:24:34.480
for Biosketches from candidates asking them what 
their contributions to their field are as well as  

00:24:34.480 --> 00:24:42.000
open science activity and so on. And just yesterday 
Yale's molecular biophysics and biochemistry  

00:24:42.000 --> 00:24:47.520
department announced that they were going to ask 
for anonymized applications for upcoming assistant  

00:24:47.520 --> 00:24:55.520
professor jobs to minimize an intended bias and, 
I think these practical examples remind us that  

00:24:55.520 --> 00:25:01.840
we need to not only measure what we value but to 
live what we value in our evaluative approaches. 

00:25:01.840 --> 00:25:07.600
So we need to put researchers and their well-being 
at the heart of everything that we do by making  

00:25:07.600 --> 00:25:13.280
our evaluation criteria clear and sticking to 
it, which is a principle of DORA, and adhering  

00:25:13.280 --> 00:25:20.480
to three other important principles, the first 
of which I think is to seek potential over past  

00:25:20.480 --> 00:25:26.400
performance. Research evaluation is inherently 
backward looking, especially bibliometrics,  

00:25:26.400 --> 00:25:31.840
and we know that the Matthew Effect, the more you 
have the more you get, is rife in science so we  

00:25:31.840 --> 00:25:38.080
have to find a way, as difficult as it might be, of 
valuing potential and not just past performance,

00:25:38.080 --> 00:25:44.800
which may well just be a function of privileged 
as Adam Grant recently tweeted: "the true measure  

00:25:44.800 --> 00:25:50.480
of people's potential is not the height of the 
peak they've reached but how far they've climbed  

00:25:50.480 --> 00:25:57.840
to get there." And if we care about diversity, we 
have to get better at measuring this. Secondly  

00:25:57.840 --> 00:26:03.440
we need to pursue recognition over judgment. No 
one wants to be judged but everyone wants to be  

00:26:03.440 --> 00:26:09.120
recognized for what they bring to the table. RPT
is incredibly competitive but that doesn't mean  

00:26:09.680 --> 00:26:14.640
we can't do it kindly, celebrating strengths 
where we find them, even if they aren't what  

00:26:14.640 --> 00:26:20.480
we're looking for right now. And I think one of 
the key ways that we demonstrate kindness is by  

00:26:20.480 --> 00:26:26.640
providing feedback that is formative evaluation 
that actually shapes someone's future and helps  

00:26:26.640 --> 00:26:33.760
them to grow rather than summative evaluation 
that just gives them a score or a rejection. And  

00:26:33.760 --> 00:26:39.520
Gemma Derek from Lancaster University in the UK is 
currently working with the Wellcome Trust on kinder  

00:26:39.520 --> 00:26:46.480
peer review and how we can actually introduce 
a kinder and more formative uh research culture.

00:26:48.640 --> 00:26:51.760
But you might be sitting there thinking, well 
that's all well and good Lizzie, but it's never  

00:26:51.760 --> 00:26:56.880
going to happen in my institution. You know how 
do I go about creating change where it's needed? 

00:26:57.440 --> 00:27:03.120
Well never fear because another of the outputs 
from the INORMS Research Evaluation Working Group  

00:27:03.120 --> 00:27:09.360
was a short one-page guide containing five 
arguments that you could use to persuade HE  

00:27:09.360 --> 00:27:15.520
leaders to evaluate responsibly. The guide is on 
the INORMS web pages, and obviously it will depend  

00:27:15.520 --> 00:27:20.480
on your local setting as to which these arguments 
you think might have traction. but in a nutshell  

00:27:20.480 --> 00:27:27.520
the five arguments are these. but firstly there's a 
moral argument for responsible research evaluation  

00:27:28.160 --> 00:27:33.600
as I've already mentioned, that it's fairer, and 
that institutions that engage with poor practices  

00:27:33.600 --> 00:27:38.640
put their staff well-being at risk and with 
it expose themselves to reputational risk. 

00:27:39.520 --> 00:27:44.720
So there's been a number of high-profile cases 
internationally where poor evaluation practices  

00:27:44.720 --> 00:27:50.640
have led to tragic outcomes, such as the case 
of Professor Stefan Grimm from Imperial College  

00:27:50.640 --> 00:27:57.760
London who died by suicide after failing to 
meet the expected income target. There's the  

00:27:57.760 --> 00:28:03.360
case of Lucy Zowe at the University of Technology 
Sydney; she was sacked after she failed to meet a  

00:28:03.360 --> 00:28:09.120
publication target, and the court subsequently 
ordered her institution to reinstate her. 

00:28:10.880 --> 00:28:16.480
And of course the ongoing unhappiness in 
the US about the use of Academic Analytics. 

00:28:17.920 --> 00:28:22.720
There is undoubtedly a growing audit culture 
in higher education, just as there is globally, 

00:28:22.720 --> 00:28:30.080
which is having a negative impact on our academic 
colleagues who are already facing a higher mental  

00:28:30.080 --> 00:28:36.480
health risk than other professions. So to my mind 
there is a strong argument that higher education  

00:28:36.480 --> 00:28:43.840
leaders should engage with responsible RPT 
practices simply to discharge their duty of care.  

00:28:45.040 --> 00:28:50.000
However, ultimately responsible research and 
evaluation, to my mind, simply leads to better  

00:28:50.000 --> 00:28:55.520
decision making. So Campbell's Law tells us 
that we get what we measure effectively by  

00:28:55.520 --> 00:29:01.360
measuring what matters, and with indicators that 
actually indicate those things that matter to them. 

00:29:01.360 --> 00:29:06.080
Institutions are simply more likely to 
make the best decisions that lead to the  

00:29:06.080 --> 00:29:12.960
best outcomes. Excuse me. As somebody wants to 
put it, play stupid games win stupid prizes.

00:29:15.760 --> 00:29:20.000
I mean you can recruit on h-index 
if you like but you might find up - 

00:29:20.000 --> 00:29:26.960
find out - that you end up with a lot of elderly male 
physicists on your staff. Of course when you're  

00:29:26.960 --> 00:29:32.640
potentially investing a lot of money in research, 
staff systems, and activities, institutions also  

00:29:32.640 --> 00:29:37.680
need to ensure that they are investing wisely in 
meaningful evaluations that are actually going  

00:29:37.680 --> 00:29:43.040
to give them valid answers to their questions 
and not waste money on shortcuts that won't.  

00:29:44.960 --> 00:29:49.200
We've talked already about the competition 
for jobs in academia being really high, which  

00:29:49.200 --> 00:29:54.480
is painful for candidates but also makes 
the job very difficult for recruiters  

00:29:54.480 --> 00:29:59.520
and very expensive, but not as 
expensive as hiring the wrong person.

00:30:02.240 --> 00:30:08.080
Our fourth argument points leaders at the 
increased focus on responsible research evaluation. 

00:30:08.080 --> 00:30:14.320
as a result of sector engenders, including 
open research improving research culture and  

00:30:14.320 --> 00:30:20.000
responsible research and - evaluate - innovation. 
Responsible research evaluation is now a  

00:30:20.000 --> 00:30:26.880
requirement of Plan S, and the Wellcome Trust's 
Open Access policy, and rather excitingly the  

00:30:26.880 --> 00:30:32.480
Global Research Council, is taking an interest 
in this area; that's an international group of  

00:30:32.480 --> 00:30:38.560
research funders, and they're holding an online 
event in November on this topic with a view to, 

00:30:38.560 --> 00:30:44.960
and I quote, "consider collectively approaches 
to research assessment criteria and processes  

00:30:44.960 --> 00:30:51.760
to facilitate real shifts towards a better 
research and innovation culture." So institutions  

00:30:51.760 --> 00:30:56.800
need to make sure that they're operationally 
ready for whatever results from this process.

00:30:59.120 --> 00:31:03.760
However, whilst there are external drivers, 
ultimately we believe that responsible research  

00:31:03.760 --> 00:31:09.520
evaluation gives institutions an opportunity 
to maintain their institutional autonomy  

00:31:09.520 --> 00:31:16.160
as mission and value-led organizations and 
enabling them to set and pursue their own goals,  

00:31:16.160 --> 00:31:20.640
recruit their own people according to 
their own ambitions, and measure progress  

00:31:20.640 --> 00:31:26.800
against their own goals and not others, such as 
university rankings. And you may have seen that  

00:31:26.800 --> 00:31:33.200
one of the other outputs from the INORMS research 
Evaluation Working Group was our university rank  

00:31:33.200 --> 00:31:39.680
ratings, launched just last week, which highlighted 
just how deficient the university rankers are when  

00:31:39.680 --> 00:31:45.360
it comes to things that matter to the higher 
education community, such as good governance, 

00:31:45.360 --> 00:31:50.880
transparency, and rigor, with their worst 
performance levels on measuring what  

00:31:50.880 --> 00:31:57.200
matters, which must lead us to think if they're 
not counting what counts, why should it count?

00:31:59.280 --> 00:32:04.960
In summary, what we reward and what we value in 
the academy are actually quite different things, 

00:32:04.960 --> 00:32:11.360
and this doesn't serve anyone very well. It doesn't 
serve our scholars, our institutions, or scholarship  

00:32:11.360 --> 00:32:17.280
itself, and it certainly doesn't lead to structural 
equalities or openness, as celebrated by this  

00:32:17.280 --> 00:32:25.680
year's Open Access Week. So we need to both assess 
and live by our values through our RPT processes. 

00:32:25.680 --> 00:32:29.920
And there are an increasing number of innovations 
in the space that we can learn from, which are  

00:32:29.920 --> 00:32:35.200
leading to significantly more successful 
appointments, so it stands to reason that  

00:32:35.200 --> 00:32:41.120
we should all use our influence to achieve this. 
And I hope that some of the INORMS work can assist  

00:32:41.120 --> 00:32:49.280
in helping us all to count what counts to the 
benefit, ultimately, I think of all humanity. So that  

00:32:49.280 --> 00:32:55.520
is it from me. Thank you very much for listening. 
I'm going to pass back to our host now. I look  

00:32:55.520 --> 00:32:59.920
forward to hearing the panel discussion 
and to continuing the conversation with you.  

00:33:01.680 --> 00:33:08.560
Hey thank you Dr. Gadd, and I'm going to uh give you 
some questions. We have some questions in the Q&A.  

00:33:09.280 --> 00:33:16.560
Um so first question is that: "a value emerges 
from communicating rather than publishing.  

00:33:16.560 --> 00:33:20.400
Can we refocus the discussion?" And

00:33:23.440 --> 00:33:25.200
I have an um question but what do you think?

00:33:26.640 --> 00:33:35.040
Value emerges, emerges from communicating 
rather than publishing. Yes it does. Um and I think  

00:33:35.040 --> 00:33:43.360
that, um, there's a lot of innovations now which are 
seeking to just re-evaluate the whole scholarly  

00:33:43.360 --> 00:33:48.400
communication process and not just incrementally 
improve um journals, to make them a little bit more  

00:33:48.400 --> 00:33:53.840
open, a little bit more transparent, and include 
additional data, but to re rethink the journal as  

00:33:53.840 --> 00:33:59.520
the - as the means of communicating in a scholarly 
way. So there's the - I'm a big fan of the octopus  

00:33:59.520 --> 00:34:07.360
project Alexandra Freeman has um uh developed, 
where she proposes that you publish um your  

00:34:08.240 --> 00:34:15.360
your hypothesis, your methods, your uh data, your 
kind of findings, your conclusions as separate  

00:34:16.160 --> 00:34:22.000
um entities as their separate outputs, all of which 
could be kind of assessed through open peer review. 

00:34:22.560 --> 00:34:27.280
Um and so there are there are other much better 
ways of doing it but it's um I think a lot of  

00:34:27.280 --> 00:34:31.520
the frustrations certainly from early career 
researchers and Martin is I think I know he's  

00:34:31.520 --> 00:34:39.680
a representative of those is um that they want 
to leap from here to there overnight and I and I 

00:34:39.680 --> 00:34:44.800
understand that frustration, and I would share 
that excitement if we could do that but I don't  

00:34:44.800 --> 00:34:50.400
think it's possible. I think we're gonna have to 
move there incrementally. I think um and to engage  

00:34:51.040 --> 00:35:00.480
probably with funders to really um power that that 
change um, so so yes you're absolutely right we do  

00:35:00.480 --> 00:35:06.240
need to move in that direction. Um but I, I think it 
might be a bit slower than most of us would like.  

00:35:08.240 --> 00:35:13.680
Yeah, in the octopus project um doing a simple 
Google search for it I have to give her name  

00:35:13.680 --> 00:35:17.600
because I'd like to link it in the chat. 
Uh do you know? Alexandra freeman, I think.  

00:35:19.360 --> 00:35:26.400
Freeman, I'll get that pulled up, but uh moving 
on to the next question: "how do we account for  

00:35:26.400 --> 00:35:33.600
diversity without locking up researchers in 
disciplines which is merely a social construct?"

00:35:36.080 --> 00:35:40.880
Well I guess there's different forms of diversity 
isn't there? And and discipline is one of those  

00:35:41.520 --> 00:35:47.840
but, well, if we had a good answer to that it 
would be very happy people and it's a it's um  

00:35:48.640 --> 00:35:54.160
a discussion topic that even the greatest and 
the best of us are scratching their heads over.

00:35:54.160 --> 00:35:58.400
You know what is a discipline exactly, you  
know there's a lot written on that um how  

00:35:58.400 --> 00:36:03.680
we define it where the boundaries are so yeah, 
it's a very good point I don't have a solution.

00:36:03.680 --> 00:36:08.320
I don't know whether Martin has one that he 
wants to propose but um it's a good point.

00:36:11.680 --> 00:36:19.520
Okay: "could you elaborately elaborate a bit on 
how the SCOPE INORMS framework compares to DORA, 

00:36:19.520 --> 00:36:25.200
the Leiden Manifesto and the Hong Kong Manifesto. 
Do we need to look at all four? What are their  

00:36:25.200 --> 00:36:33.760
specific strengths? That's a good question 
um so we see um DORA, the Leiden Manifesto um  

00:36:33.760 --> 00:36:40.000
particularly as the high level principles 
around responsible research evaluation. Um  

00:36:40.800 --> 00:36:50.320
they are older so DORA was um created in 2012, Leiden
Manifesto in 2015, and so we're we're five or eight  

00:36:50.320 --> 00:36:57.840
years on from those now. We see SCOPE as more of 
a uh a road map rather than a highway code if you  

00:36:57.840 --> 00:37:03.440
like, that's that's the distinction in our minds. 
So for practitioners like us, um principles are  

00:37:03.440 --> 00:37:09.600
very helpful, but how do you actually go about 
developing a responsible evaluation? And that's  

00:37:09.600 --> 00:37:14.640
where we think SCOPE comes in, because it does 
clearly say right we'll start with start here  

00:37:14.640 --> 00:37:18.960
with what you value, then consider your con- your 
context then you can think about your options.  

00:37:18.960 --> 00:37:26.720
and i think in terms of the Hong Kong Manifesto, 
that provides helpful guidance if your the values  

00:37:26.720 --> 00:37:32.320
that you come up with under the S of SCOPE map 
onto the values that they're seeking to propagate  

00:37:32.320 --> 00:37:37.520
through that assessment framework, so that's how 
I'd see those fitting together, but there was a a  

00:37:37.520 --> 00:37:44.240
conference last week of the Nordic Bibliometrics 
Workshop, and somebody um had developed a mapping  

00:37:44.240 --> 00:37:50.400
of dora, Leiden Manifesto, and the Metric Tide, 
which is the UK's um independent investigation  

00:37:50.400 --> 00:37:56.400
interest responsible research evaluation. Again 
from 2015 and so it's and I came up with five  

00:37:56.400 --> 00:38:01.200
principles for responsible research evaluation, 
so that's quite a nice little mapping and if I

00:38:01.200 --> 00:38:05.360
um after this Q&A, I'll try and find that and 
put it in the chat so people can have a look.

00:38:07.520 --> 00:38:10.400
Okay, all right, next question we have:  

00:38:10.960 --> 00:38:15.920
"Looking forward, what do you think the role of 
altmetrics will be for scholarly evaluation?" 

00:38:17.600 --> 00:38:25.200
that's a really good question. i think the beauty 
of altmetrics at the moment is the contextual data  

00:38:25.200 --> 00:38:31.280
that they provide. so at Loughborough, we encourage 
individuals to supply altmetrics where they're  

00:38:31.280 --> 00:38:38.000
aware of them that highlight alternative impacts 
that their work has had on an individual output by  

00:38:38.000 --> 00:38:45.600
output basis. What altmetrics can't currently do is 
to um well it couldn't the last time I looked so  

00:38:45.600 --> 00:38:53.200
correct me if I'm wrong, is to kind of allow us 
to benchmark. So how how is Loughborough chemistry  

00:38:53.200 --> 00:38:57.840
department doing on altmetrics relative uh to 
Virginia Tech, for example. You can't currently  

00:38:57.840 --> 00:39:02.720
do that level of analysis, so and I don't know 
if they - I think they've probably deliberately  

00:39:02.720 --> 00:39:06.560
steered away from that; I can't speak for them, 
but that's my understanding from conversations  

00:39:06.560 --> 00:39:13.120
with old metrics folks, so I think that the beauty 
of altmetrics is in that kind of rich contextual.

00:39:13.120 --> 00:39:18.000
You're not just looking at numbers you're looking 
at the actual tweets, the actual blog posts, 

00:39:18.000 --> 00:39:23.440
um the actual demographics of folk who've 
engaged with your work, and you as an individual  

00:39:23.440 --> 00:39:30.320
can identify where that's meaningful, but beyond 
that it's it's a bit more challenging for folks  

00:39:30.320 --> 00:39:34.640
like me who are trying to do analysis at 
a like at a higher level of granularity.

00:39:37.920 --> 00:39:43.760
Okay uh next question, and this has a lot of 
upvotes uh so it's getting pushed to the top here,

00:39:43.760 --> 00:39:48.800
"Is the notion that we have we have to persuade 
higher education leaders to make changes  

00:39:49.360 --> 00:39:55.520
a sign that the roadblocks to change are much 
bigger? Can these changes be made in a hierarchical  

00:39:55.520 --> 00:40:02.160
and anti-democratic workplace?" Yeah I mean that's 
why we've offered these - we've offered these five  

00:40:02.160 --> 00:40:06.160
arguments but not in any particular 
order because we recognize that some  

00:40:06.160 --> 00:40:12.320
institutions, you know culturally, eat strategy for 
breakfast doesn't it? That's the expression and  

00:40:12.320 --> 00:40:16.960
it's so true. Um and if you've got a culture that 
is hierarchical and anti-democratic, you're going  

00:40:16.960 --> 00:40:21.920
to struggle a lot more than if you're working in 
an institution like mine which is open, liberal,  

00:40:21.920 --> 00:40:28.880
welcomes um any opportunity to support a broader 
diversity of um scholars and scholarships, so  

00:40:30.240 --> 00:40:35.280
yes it is going to be it is more challenging
of course in those sorts of organizations,  

00:40:35.280 --> 00:40:41.360
but that's why we put as one of our arguments 
you know that these demands are coming. You're  

00:40:41.360 --> 00:40:45.520
going to be evaluated on this sooner or later, 
you'd have to get operationally ready for this,  

00:40:45.520 --> 00:40:51.840
because whether you like it or not, it is 
coming so um yeah it's the challenge.  

00:40:53.040 --> 00:40:59.200
Okay okay: "Given that researcher metrics are 
also wrapped up and used to rank and evaluate  

00:40:59.200 --> 00:41:05.840
institutions, what would convince a high-ranked 
university to risk its valued global ranking?"  

00:41:07.360 --> 00:41:14.880
Yeah um what would convince high ranked
university to risk its valued global ranking? 

00:41:14.880 --> 00:41:19.360
That's the problem, that university leaders 
have is that they are what I call it - excuse  

00:41:19.360 --> 00:41:24.240
the expression - but in the UK we have a reality tv 
program called Snog, Marry, Avoid. We might have got  

00:41:24.240 --> 00:41:29.200
it from the U.S, I don't know, um, but I think 
that your higher education leaders have this  

00:41:29.760 --> 00:41:34.080
problem when it comes to engaging with 
university rankers, because most understand  

00:41:34.080 --> 00:41:40.640
that they are methodologically problematic, um, 
but they either kind of jump into bed with them -

00:41:41.200 --> 00:41:47.520
to use a clinical expression - um by their day to 
host their conferences and really seek to decline  

00:41:47.520 --> 00:41:53.200
the rankings, or they kind of make the best use of 
them where they're shown in a good light, but then  

00:41:53.200 --> 00:41:57.600
keep quiet about it most of the time, or they avoid 
them all together. But none of these positions  

00:41:58.320 --> 00:42:02.800
actually challenges the dominance of the 
university rankings, um which is why we introduced  

00:42:02.800 --> 00:42:11.440
this ranker rate - ranker rating at the right round um -
to try and kind of introduce a new apex predator  

00:42:11.440 --> 00:42:15.920
in the food chain if you like, to kind of say, "No 
we want the research community to ultimately be  

00:42:15.920 --> 00:42:21.200
the judge of what we what we value and how we're 
assessed, and we would like you university rankers  

00:42:21.200 --> 00:42:27.200
to answer to us as a research community, and we 
want to expose, you know, how well you're doing on  

00:42:27.200 --> 00:42:32.320
the values of the higher education community, um at 
the moment, and hopefully encourage them to change. 

00:42:32.880 --> 00:42:39.280
So um there's so many stakeholders in research 
evaluation, and in the kind of university rankings  

00:42:39.280 --> 00:42:43.120
game, it's not just university rankers that use 
them, it's their students. You know, you can't, 

00:42:43.120 --> 00:42:49.360
it'd be financial reputational suicide to um 
stop promoting your university ranking, um if  

00:42:49.360 --> 00:42:54.240
it's a good one, because then global students 
are going to go elsewhere, and we know that's -

00:42:54.240 --> 00:43:00.160
that's problematic for those institutions that 
rely on uh international students, so it it's a  

00:43:00.160 --> 00:43:04.720
it's a challenging conundrum um - 
I've outlined our solution to that.

00:43:04.720 --> 00:43:08.880
Uh it needs to be sustainable though recognize 
it's just a project at the moment and so if  

00:43:08.880 --> 00:43:13.280
you've got any great ideas as to how we can make 
that more sustainable then happy to to hear those.

00:43:16.160 --> 00:43:19.760
Um all right so we are almost at time. 
I think we'll do one more question  

00:43:19.760 --> 00:43:22.880
and we'll try to get to the other 
questions, uh after the discussion. 

00:43:23.520 --> 00:43:29.600
Um so I apologize if we didn't get to your 
question yet, uh but the next question and I'm

00:43:29.600 --> 00:43:34.080
gonna go with the one with the most upvotes 
and I feel like I'm ranking the comments.

00:43:37.840 --> 00:43:41.440
"Isn't publishing the dissemination of 
research findings after peer review  

00:43:42.240 --> 00:43:46.720
critical to sharing quote 'trustworthy' 
research findings that others  

00:43:46.720 --> 00:43:49.680
can build upon so - not quote 
but emphasize *trustworthy*  

00:43:50.240 --> 00:43:53.600
that others can build upon communication 
without any sort of checks or balances  

00:43:53.600 --> 00:43:58.160
leads to the dissemination of misinformation as
we - as we've seen in the case of COVID?"

00:44:01.200 --> 00:44:08.560
Yes um I don't think anything I've 
said would um mitigate against that. 

00:44:09.200 --> 00:44:14.640
Um you know obviously there is 
an increase in the sharing of um  

00:44:15.840 --> 00:44:22.400
research results through pre-prints 
through the COVID era which has proved um  

00:44:22.400 --> 00:44:27.120
both successful and problematic in probably 
equal proportions, so there is - there is a big  

00:44:27.120 --> 00:44:32.800
question here undoubtedly. uh I'm not sure I'm 
entirely qualified to to say any more on that.

00:44:35.680 --> 00:44:40.720
Okay um it looks like it's 10:45, so we're gonna 
move on. I still have three questions in here,  

00:44:40.720 --> 00:44:46.720
but hopefully we'll get to them at the very end, 
and I'm going to go ahead and start introducing  

00:44:47.440 --> 00:44:57.680
um dr. uh Tyler Walters. Um Tyler Walters is 
the Dean of University Libraries, and he'll  

00:44:57.680 --> 00:45:02.400
be moderating this panel. He is also a member 
of the Advancing Public Access to Research Data  

00:45:02.400 --> 00:45:08.560
Steering Committee, co-sponsored by the AAU and 
APLU, he was also recently elected as a Member  

00:45:08.560 --> 00:45:12.880
of the Board of the Association of Research 
Libraries, and I will go ahead and turn it over  

00:45:12.880 --> 00:45:17.840
to Tyler. I will start sharing my screen so we 
have the panelists up here for everybody to see.

00:45:25.680 --> 00:45:32.480
All right, all right, great, thank you very much 
Rachel. So real quick let you know that we have  

00:45:32.480 --> 00:45:37.040
uh we're going to be efficient this this morning 
or afternoon depending on where you're at. We have  

00:45:37.600 --> 00:45:41.840
four panelists. We have about 35 minutes and a 
number of questions that we're going to try to  

00:45:41.840 --> 00:45:47.360
get through, but let me introduce our panelists. 
To begin with, first we have Tom Ewing who is  

00:45:47.360 --> 00:45:51.600
a Professor in the Department of History and 
Associate Dean for Graduate Studies and Research  

00:45:52.240 --> 00:45:56.960
at Virginia Tech's College of Liberal Arts 
and Human Sciences. and next we have Carla  

00:45:56.960 --> 00:46:01.840
Finkelstein, who's an Associate Professor in 
the Fralin Biomedical Research institute  

00:46:01.840 --> 00:46:04.800
and Department of Biological 
Sciences in the College of Science and

00:46:05.360 --> 00:46:10.640
also holds affiliate positions in the Department 
of Surgery and the Faculty of Health Sciences in  

00:46:10.640 --> 00:46:15.040
the Virginia Tech Carillion School of Medicine; 
she's a Principal Investigator of the Integrated  

00:46:15.040 --> 00:46:20.320
Cellular Responses Lab and Scientific Director 
of the Molecular Diagnostics Lab in the Fralin  

00:46:20.320 --> 00:46:27.440
Biomedical Research Institute in Roanoke. Third we 
have Bikum Gill, who is an associate professor in  

00:46:27.440 --> 00:46:32.640
the Department of Political Science in the College 
of Liberal Arts and Human Sciences; he's also the  

00:46:32.640 --> 00:46:38.880
co-co-investigator of the Project for Stories and 
about Food Sovereignty, for which his team received  

00:46:38.880 --> 00:46:45.040
a Social Sciences and Humanities Research Council 
Partnership Development Grant. And fourth we have  

00:46:45.040 --> 00:46:49.760
Sylvester Johnson, who is the founding Director 
of the Virginia Tech Center for Humanities and 

00:46:49.760 --> 00:46:54.240
internationally recognized humanities scholars 
specializing in the study of technology,  

00:46:54.240 --> 00:46:59.200
race, religion, and national security. He's also 
our Assistant Vice Provost for the Humanities  

00:46:59.200 --> 00:47:04.720
at Virginia Tech, and executive director of the 
university's Tech for Humanities Initiative. 

00:47:06.000 --> 00:47:12.080
So we have four wonderful panelists and we'd 
like to begin again with some you know brief  

00:47:12.080 --> 00:47:18.960
responses for you all from Dr. Gadd's talk. So what 
what about her talk resonated most with you and  

00:47:18.960 --> 00:47:23.840
with your own background and experience.
Anybody like to go ahead and address that?

00:47:26.560 --> 00:47:32.640
First I would like to thank uh Dr. Gadd for um her 
presentation it was fantastic and you hit really  

00:47:32.640 --> 00:47:38.560
all the points. I mean um I was really thinking 
on you know ahead and you your nexus like what  

00:47:38.560 --> 00:47:44.080
is what's exactly my my you know the answer 
to my question pretty much um I - I - think what  

00:47:44.080 --> 00:47:51.200
resonated is, to me, the most is the concept of that 
evaluation has to be with people and not to people.

00:47:52.240 --> 00:48:00.160
I think that that is something that in my 
field at least is - is it's clearly not the case.

00:48:05.440 --> 00:48:11.440
I just want to pass this to my colleagues yeah. 
Tom would you like to go next? Sure no and I uh  

00:48:11.440 --> 00:48:16.880
also thank Dr. Gadd for for for presenting um. 
As a historian all of this sounded you know  

00:48:17.520 --> 00:48:22.640
very familiar. Um you know that there are real 
challenges with evaluating scholarship across  

00:48:22.640 --> 00:48:28.320
the humanities. Um she mentioned some and I can add 
a few more. Books count more than articles. Citation  

00:48:28.320 --> 00:48:32.480
citation counts are not reliable usually.
They don't even count citations in books only  

00:48:32.480 --> 00:48:38.240
in articles. Um there's certainly a bias towards 
science technology engineering and mathematics.

00:48:38.240 --> 00:48:43.360
Um funding is much less uh there's less 
funding available in the humanities, um and  

00:48:43.360 --> 00:48:47.040
one particular challenge here at Virginia Tech at 
land grant university but this is true of a lot of  

00:48:47.840 --> 00:48:54.240
other universities: how do you balance um research 
that that's disseminated within a scholarly field  

00:48:54.240 --> 00:48:58.720
um as opposed to research that's disseminated 
to the public, um you know how do you - how do  

00:48:58.720 --> 00:49:05.680
you balance those two things? Um i would say we 
do a better job, um at evaluating scholarship  

00:49:05.680 --> 00:49:10.560
at the local level uh within the department you 
know. I think that that we have a pretty good  

00:49:10.560 --> 00:49:17.440
um uh process for doing that. The challenge comes 
when you move that out and try and you know make  

00:49:17.440 --> 00:49:22.960
an argument for the quality of scholarship um 
across the university or particularly outside the  

00:49:22.960 --> 00:49:27.760
university in comparison to other universities, 
and this is clearly the the seductive but  

00:49:27.760 --> 00:49:33.120
false promise of Academic Analytics, and also 
global rankings as you heard from from Dr. Gadd.  

00:49:33.840 --> 00:49:38.000
Um and I must agree with Dr Finkelstein in 
terms of this idea of you know evaluating  

00:49:38.000 --> 00:49:44.080
with people rather than to people. Um the case I 
would make is that you know what we need to do  

00:49:44.080 --> 00:49:49.200
is actually use the skills of the liberal arts to 
evaluate research. You know we teach our students  

00:49:49.920 --> 00:49:53.920
we should model this behavior ourselves um we 
need to think critically about what we're doing  

00:49:53.920 --> 00:49:59.120
and how we do it. We need to persuade others um in 
our writing and our speaking of why this research  

00:49:59.120 --> 00:50:03.360
matters. Um we need to think about the context 
you know in which we're doing this research and  

00:50:03.360 --> 00:50:09.600
how we're evaluating it, um and how do we you know 
talk about what we're doing in a way, in a language,  

00:50:09.600 --> 00:50:13.840
that makes sense to others, um you know, and these 
are things we do you know in our teaching. We need  

00:50:13.840 --> 00:50:19.120
to think about how we do those um in our research, 
but the challenge is to do that you know at each  

00:50:19.120 --> 00:50:28.640
successful - at each successive and expanded level. 
Thanks um would anybody else like to address that?

00:50:31.520 --> 00:50:38.640
Uh yes I can I can go um I just think 
just briefly a couple of comments on  

00:50:39.200 --> 00:50:46.320
what resonated most from the talk. I - there was 
a an emphasis on towards the end of putting  

00:50:46.320 --> 00:50:51.440
researchers and their well-beings at the center, and 
there was one point that was raised that I found  

00:50:51.440 --> 00:50:56.080
really interesting, was around potential over past 
performance, and I know that's a very difficult  

00:50:56.080 --> 00:51:01.440
thing to measure, and to get into, but I appreciated 
that point because uh as somebody who's like  

00:51:01.440 --> 00:51:08.080
a first generation uh university student, you 
know, my parents didn't even finish high school. 

00:51:08.080 --> 00:51:14.160
I know like from the beginning I always felt 
like I was scrambling to keep up with those and  

00:51:14.160 --> 00:51:18.560
you know even myself now being faculty, and having 
children, and seeing how much easier it is for them.

00:51:19.440 --> 00:51:25.440
This notion of, I think the quote by Adam Grant, 
was "how far you have climbed" uh to recognize that  

00:51:25.440 --> 00:51:30.800
rather than simply um conventional metrics that 
people who kind of are starting ahead already  

00:51:30.800 --> 00:51:36.160
have quite an advantage so I appreciated 
that point, and then something else that um  

00:51:36.160 --> 00:51:42.080
that spoke to me quite a bit was this question 
of the problem of incentivization. I know as an  

00:51:42.080 --> 00:51:48.080
early kind of career researcher pre-tenure and 
even somebody just came off the the job search  

00:51:49.600 --> 00:51:54.400
you're often told you've got to publish publish 
publish and then that can impact how you move  

00:51:54.400 --> 00:52:00.480
through the peer review, how you're responding 
to the criticisms of your work, and you - it can  

00:52:00.480 --> 00:52:05.120
end up shaping kind of the methods you use and 
the arguments you're willing to put forward and  

00:52:05.120 --> 00:52:10.800
not because you're just racing to publish, so 
yeah I appreciated those points, yeah thank you.

00:52:14.400 --> 00:52:21.440
Perhaps we can go on to our next question. Can 
I add a little bit uh something Tyler, so um  

00:52:22.880 --> 00:52:27.760
one thing that I would like to bring about the 
whole publication issue to consideration, and it's  

00:52:27.760 --> 00:52:34.560
true in the field of science, is as the graph was 
perfectly uh accurate in the sense that everything  

00:52:34.560 --> 00:52:42.560
really goes to to journals in our in our field 
and and that has to do a lot with the fact that um  

00:52:42.560 --> 00:52:48.000
by the time that books get usually published the 
technology that we discussed, there is usually  

00:52:48.880 --> 00:52:54.480
not leading technology anymore, so publishing we 
we know that, I mean it's different in other fields, 

00:52:54.480 --> 00:52:59.680
especially in humanities, but in in our field that 
that's the that's a conundrum right? I mean it's  

00:52:59.680 --> 00:53:08.320
just you know, you, if you accept to publish a 
book then it's very likely that whatever you  

00:53:08.320 --> 00:53:15.280
put in there, it will be really not obsolete but 
it will be definitely not cutting edge anymore.  

00:53:15.280 --> 00:53:20.400
By the by the time that it gets published so it's 
really discouraged in some way. Plus you put a lot  

00:53:20.400 --> 00:53:26.880
of effort in in in publishing in a book. The other 
thing that I just want to bring to consideration  

00:53:26.880 --> 00:53:32.800
is what bitcoin says is true. I mean you need 
to publish publish publish right? But then  

00:53:32.800 --> 00:53:37.440
that brings the other issue of the predatory 
journals that populate at least in my field  

00:53:38.000 --> 00:53:45.280
and everywhere it's like suddenly you get five six 
emails per day of offerings to publishing whatever  

00:53:45.280 --> 00:53:51.200
research you have as long as you pay of course.
Pretty soon so that will increase your number  

00:53:51.200 --> 00:53:55.840
and then that would probably increase 
your either citations or you know  

00:53:57.520 --> 00:54:01.920
um that's another problem the other the 
third problem about the publication is true  

00:54:02.480 --> 00:54:07.120
that we we're usually encouraged to publish 
in high profile journals and we all want to  

00:54:07.120 --> 00:54:13.520
do that to be honest. It would be unfair to say 
the opposite but the the reality is also that  

00:54:13.520 --> 00:54:16.880
the number of papers published in 
our field in high-profile journals  

00:54:17.920 --> 00:54:25.360
is is reduced because it costs a lot of money 
to publish research that will go to one of those  

00:54:25.360 --> 00:54:31.360
journals. I mean the amount of research that is 
needed for publishing in a top tier journal is  

00:54:31.360 --> 00:54:36.400
unbelievable. Hundreds, thousands of dollars and 
you also need a lot of people working on those  

00:54:37.040 --> 00:54:43.200
papers, so there is a there's a conflicting thing 
here right? I mean it's just that it's probably um  

00:54:44.080 --> 00:54:48.880
you know due to different is it's different 
in different disciplines, but I just want to  

00:54:48.880 --> 00:54:54.000
bring all this into consideration because that's 
the reason because you see some of those charts.

00:54:57.760 --> 00:54:59.920
Let's move on to our next question for our panel

00:55:03.120 --> 00:55:08.560
So who or what do you perceive as having power 
to define and implement value systems around  

00:55:08.560 --> 00:55:16.320
research activities? Anybody like to start out? 
Yeah I'll get us started on that I think the  

00:55:17.840 --> 00:55:23.600
certainly was a really amazing and incisive 
presentation, so just want to echo my appreciation  

00:55:24.320 --> 00:55:29.600
for what we were able to hear this morning 
or this afternoon depending on where you are.

00:55:30.240 --> 00:55:36.160
So the the control in one sense as one 
panelist said they're at the local very  

00:55:36.160 --> 00:55:41.840
small departmental level for example program 
level uh disciplinary level there's a lot of  

00:55:42.640 --> 00:55:49.360
ability to be more granular and flexible 
in theory at least in evaluating work  

00:55:50.240 --> 00:55:58.400
um but it doesn't necessarily mean that we can 
always or even usually get the dynamism that  

00:55:58.400 --> 00:56:04.480
we might need in order to address the kinds of 
of aims and important objectives that we heard  

00:56:04.480 --> 00:56:12.480
outlined this morning. Also disciplinary 
departments tend to be very traditional  

00:56:12.480 --> 00:56:18.640
in looking at what qualifies for tenure and 
promotion, and and it's good that departments  

00:56:18.640 --> 00:56:23.600
can define that themselves so they do have the 
power and the incentive to say this is what  

00:56:23.600 --> 00:56:27.920
we're looking for if we have a hiring committee. 
For example this is how we're going to define  

00:56:28.720 --> 00:56:34.640
the the excellent candidate or if it's tenured 
promotion this is how we're going to define  

00:56:34.640 --> 00:56:41.200
the fulfillment of needing research excellence in 
our department or in our in our field or subfield  

00:56:42.000 --> 00:56:49.200
and the pure, a 10-year promotion review process, 
is often typically shaped by sending those  

00:56:49.200 --> 00:56:55.840
criteria that a department gets to define to peer 
reviewers or evaluators who are then looking at  

00:56:55.840 --> 00:57:01.440
a dossier and trying to judge it and also doing 
so with a larger field of view so I'll give one  

00:57:01.440 --> 00:57:07.840
example of some of the challenges here Virginia 
Tech is part of the public interest technology  

00:57:07.840 --> 00:57:15.760
university network, and this university network is 
fairly new, as three years old, it is focusing on  

00:57:16.560 --> 00:57:20.880
encouraging technology talent, and that's 
broadly defined so that's not just STEM,  

00:57:20.880 --> 00:57:26.720
that's that's comprehensive you know it's 
social science, arts, humanities, creatives,  

00:57:26.720 --> 00:57:31.680
as well as math and technology, engineering, 
but getting technology talent to support  

00:57:32.480 --> 00:57:38.720
positive impact of technology on a human 
society through comprehensive approaches and  

00:57:39.520 --> 00:57:44.880
one of the things that that university network 
is trying to address is precisely this problem. 

00:57:44.880 --> 00:57:50.960
So this public good or public interest work is 
transdisciplinary, it means that people who are  

00:57:50.960 --> 00:57:56.880
in name your discipline have to collaborate with 
people who are not going to be in their department - 

00:57:56.880 --> 00:58:01.760
who are not going to be in their discipline to 
tackle the big challenges that we love to say.  

00:58:02.560 --> 00:58:07.840
Our disciplinary work is urgent 
and invaluable for tackling,

00:58:09.440 --> 00:58:16.160
but if you are pre-tenure and you're doing this 
public interest technology work, you are putting  

00:58:16.160 --> 00:58:21.520
yourself at risk of not being legible to the 
people in your department or in your discipline  

00:58:21.520 --> 00:58:26.160
who are going to judge your work to see how you're 
presenting and publishing and disseminating your  

00:58:26.160 --> 00:58:32.240
research, and so what we're trying to do now is 
to figure out how to incorporate some of the  

00:58:32.240 --> 00:58:37.440
aims and objectives that we saw beautifully 
articulated this morning in the presentation  

00:58:38.320 --> 00:58:44.240
in order to to move the needle somewhat on that 
review process, so it is definitely a lot of  

00:58:44.240 --> 00:58:50.640
power at the level of of departments that get to 
say this is what counts for promotion or tenure. 

00:58:50.640 --> 00:58:54.080
Uh it's definitely on hiring committees. 
It's not the only place that exists  

00:58:54.720 --> 00:59:03.840
but those are two places 
that exert a lot of influence.

00:59:07.040 --> 00:59:11.520
And I'm muted of course. Thank you Sylvester. 
Anybody else like to address this topic?

00:59:14.480 --> 00:59:21.520
Um yeah I would love to have the opportunity to 
address it um I think um I I wanted to jump in on  

00:59:21.520 --> 00:59:25.680
this question because it has a word in there that 
i think is very significant here and that's um  

00:59:25.680 --> 00:59:32.960
power um, and I appreciate many of 
Sylvester's kind of comments and like  

00:59:32.960 --> 00:59:40.320
I think in this discussion around um you know how 
we value research and kind of questions of metrics, 

00:59:40.320 --> 00:59:46.160
I think it's important that we move the discussion 
also away from a purely technocratic discussion of  

00:59:46.160 --> 00:59:53.520
how do we come up with better techniques and 
standards and norms and think very critically  

00:59:53.520 --> 00:59:58.640
and seriously about the role that power plays 
in shaping how things are evaluated, because  

00:59:58.640 --> 01:00:02.800
there was a great question in the Q&A about 
well how so far can universities address this  

01:00:03.360 --> 01:00:08.000
and this question of anti-democratic workplace 
comes up now. This has not been my experience at  

01:00:08.000 --> 01:00:14.560
say Virginia Tech. Folks like Sylvester and Tom 
have been so helpful in supporting kind of our  

01:00:14.560 --> 01:00:19.200
research initiatives, but when you think about when 
you're submitting something for publication or a  

01:00:19.200 --> 01:00:24.320
grant application you're often told you know 
uh you need to de-radicalize your critique of  

01:00:24.320 --> 01:00:28.400
systems of power, right? like you you need to 
couch this in a different set of languages, 

01:00:29.520 --> 01:00:34.640
and if you want to get into the high impact 
journals, there is a certain set of questions  

01:00:34.640 --> 01:00:39.680
you can and you cannot ask right? And so there 
are entrenched systems of power that have a set  

01:00:39.680 --> 01:00:46.000
of interests in maintaining these systems as well, 
so it's not a simply a technical question, let's  

01:00:46.000 --> 01:00:52.400
say, but a question on how do you contest uh these 
existing uh systems of power, right? And I know like  

01:00:53.840 --> 01:00:57.760
as an example, one thing that's very 
challenging is if suppose one is doing  

01:00:58.960 --> 01:01:04.560
uh research on something like uh you know the 
relationship of military industries to research  

01:01:04.560 --> 01:01:11.600
right? And how far you can contest um the role 
that militarism plays in the united states, and  

01:01:11.600 --> 01:01:17.520
in shaping social science research, like how do you 
apply for a grant where can you apply and so that  

01:01:17.520 --> 01:01:23.440
often will read out or it will be biased against 
research that really challenges at a fundamental  

01:01:23.440 --> 01:01:28.560
level uh systems of power, right? So I think that's 
one thing that I have found very challenging. 

01:01:28.560 --> 01:01:33.440
Uh moving through kind of institutions is because 
as a graduate student you often start with a  

01:01:33.440 --> 01:01:38.800
really - and the humanities and social sciences are 
supposed to take a very critical stance towards  

01:01:38.800 --> 01:01:43.680
power - speak truth back to power, right? tut the kind 
of institutional arrangements of evaluation work  

01:01:43.680 --> 01:01:49.040
against that in many ways, and that's something -
that's why you often find grad student conferences  

01:01:49.600 --> 01:01:54.000
will often be asking some of the more critical 
questions, because they have not yet entered  

01:01:54.000 --> 01:01:59.120
onto the job market or the pre-tenure path so just 
kind of pointing that out as a challenge and maybe  

01:01:59.120 --> 01:02:03.440
something to keep in mind in the 
discussion, actually, what you just  

01:02:03.440 --> 01:02:09.040
post in your field also happens in the field of 
science maybe is is hidden in a different way. 

01:02:09.040 --> 01:02:15.280
For example, when you try to publish a paper that's 
supposed to shake the dogma, right? I mean it's like  

01:02:15.920 --> 01:02:22.720
you you have this fantastic research out there 
and it's so there's some kind of well-established  

01:02:23.440 --> 01:02:31.040
you know um way of things, the uh way of mechanisms 
that they work, and you just have something that  

01:02:31.040 --> 01:02:35.600
completely challenge that - that dogma and then 
of course you send it to a top journal for  

01:02:35.600 --> 01:02:41.840
publication, and it's like yeah this is too risky, 
or this is this is too revolutionary for the field.  

01:02:41.840 --> 01:02:47.760
That's something that I got once, and as like you 
and uh publishing in the paper, that ended up being  

01:02:47.760 --> 01:02:54.160
a lower tier just to be sure that is published, and 
peer review just because even the top journals,

01:02:54.160 --> 01:02:58.880
they don't they don't want to take that risk. 
Okay, and in fact they probably ask you for way  

01:02:58.880 --> 01:03:04.560
much more evidences than any other standard paper, 
so those level of powers exist in all disciplines.

01:03:04.560 --> 01:03:10.640
Right maybe they are hidden in a different way, 
in fact if you go back to the history of the most  

01:03:10.640 --> 01:03:18.000
relevant work that was awarded with whatever 
metrics you use for a word of top research, 

01:03:18.000 --> 01:03:23.520
call it Nobel Prize or whatever those are usually 
published in in in journals that are not top  

01:03:23.520 --> 01:03:29.840
journals simply because they were very afraid of 
you know later retractions and things like that.

01:03:32.800 --> 01:03:34.560
Okay let's move on to our next question. 

01:03:36.400 --> 01:03:41.760
How do or don't existing reward and evaluation 
systems recognize and value different types  

01:03:41.760 --> 01:03:45.680
of research? This could be research in your 
fields or across fields that you're aware of.

01:03:50.400 --> 01:03:56.640
So I'll give a couple of examples. One I just put 
into the chat is digital humanities scholarship. 

01:03:57.200 --> 01:04:02.960
Uh this is a huge challenge for 
digital humanists who face a very  

01:04:04.560 --> 01:04:10.960
inflexible very narrow way of interpreting 
what counts as public scholarship, and so if  

01:04:10.960 --> 01:04:17.440
it's a digital project that's just it's usually 
a situation where there's tremendous resistance  

01:04:17.440 --> 01:04:23.680
if not outright dismissal that doesn't count 
for tenure. You can wait seven or eight more  

01:04:23.680 --> 01:04:28.880
years once you get tenure, then you can do that 
stuff. Another example: one of the things that  

01:04:28.880 --> 01:04:35.360
we faced here at Virginia Tech is the Destination 
Areas uh trying to promote transdisciplinary work  

01:04:36.080 --> 01:04:42.400
and and dealing with what is typically for 
departments. This is not unique to Virginia Tech.

01:04:42.400 --> 01:04:48.080
It's just something we dealt with here and that 
is a narrow reading of of what counts if you're  

01:04:48.080 --> 01:04:56.480
a sociologist or a religion and culture person or, 
whatever the department may be on a Destination  

01:04:56.480 --> 01:05:02.480
Area team or if your Destination Area higher, there 
has been tremendous pressure to just focus in on

01:05:07.360 --> 01:05:10.000
when you're disappointed. You do have 
tinder, then you can go explore that  

01:05:10.000 --> 01:05:13.600
other stuff, so those are a couple of 
examples that we need to figure out.

01:05:17.360 --> 01:05:18.880
Tyler if I can follow up on that.

01:05:22.400 --> 01:05:29.440
Yes go ahead. Okay sorry, it's the world we live 
in there's there's the Q&A screen. There's the  

01:05:29.440 --> 01:05:35.680
chat screen, and then there's the you know other 
screen. Um and very much appreciate all of the  

01:05:35.680 --> 01:05:40.880
hundred and so people out there listening 
to this um yeah I mean I think that  

01:05:42.080 --> 01:05:48.560
you know the question is how does the -
how is the field changing at the same  

01:05:48.560 --> 01:05:54.000
time that the evaluation is happening? Um you 
know, and I can certainly think of examples. 

01:05:54.000 --> 01:05:59.280
Um following up on on what Sylvester just said, uh 
but I can think of contrary examples as well from  

01:05:59.280 --> 01:06:05.040
Virginia Tech and elsewhere where you know the 
hiring process, the promotion and tenure process,

01:06:05.040 --> 01:06:12.160
the institutional rewards, um do in fact recognize 
the value of interdisciplinary scholarship  

01:06:13.200 --> 01:06:19.280
of alternative forms of publication, um you know 
to take the example of the digital humanities. 

01:06:19.280 --> 01:06:25.600
The potential there um you know to not only to 
reach a broader audience than a journal article or  

01:06:25.600 --> 01:06:33.360
or a monograph um but you know really to to 
connect in some cases with um you know a public  

01:06:33.360 --> 01:06:38.560
audience that cares deeply about the topic um you 
know, and - and - I you know I can think of ways in  

01:06:38.560 --> 01:06:46.160
which that exactly that um evaluation and reward 
system is is in fact in in in place um I have to  

01:06:46.160 --> 01:06:51.040
admit Lizzie, I'm trying to figure out if I know 
who that guy in Nebraska is that's maintaining  

01:06:51.040 --> 01:06:56.720
that that corner of the digital infrastructure,
um because it does raise a different question  

01:06:56.720 --> 01:07:01.520
and and this follows up a little bit uh 
carla with what you were talking about um  

01:07:02.320 --> 01:07:10.160
which is how do you evaluate and and reward 
scholarship that lasts you know, that makes  

01:07:10.160 --> 01:07:15.440
a long-term significance, has a long-term 
significance, um and I certainly appreciate  

01:07:16.800 --> 01:07:22.000
you know that this is different across 
fields um in a manner's field like history  

01:07:22.000 --> 01:07:28.160
you know you again most of the value is 
associated with the publication of a book um  

01:07:28.160 --> 01:07:32.560
You know, and so if you publish a monograph 
most of the market is going to be libraries. 

01:07:33.440 --> 01:07:39.280
Um the book will stay in the library uh hopefully 
10 years from now, 20 years from now, 30 years from  

01:07:39.280 --> 01:07:43.920
now, you know someone could go into that library 
and either take the book off the shelf or  

01:07:44.480 --> 01:07:50.560
summon it from the storage facility um and 
have a copy you know. That's a long-term impact  

01:07:51.120 --> 01:07:57.520
that's certainly very different from a blog or an 
Open Access journal that you know may or may not  

01:07:58.240 --> 01:08:02.880
be available in five years, ten years, not to 
mention the fact that we may or may not have  

01:08:02.880 --> 01:08:09.520
the technology to actually read whatever format 
that that exists in um you know so so these are  

01:08:09.520 --> 01:08:15.280
the kinds of questions I could think to go back 
to what um Dr. Gadd was saying, you know, we really  

01:08:15.280 --> 01:08:22.000
have to think about what do we value um and a part 
of what we value we and we can valuable things you  

01:08:22.000 --> 01:08:27.920
know we can value short term. How do you get this 
to engage a public audience but we can and should  

01:08:27.920 --> 01:08:37.840
also value you know how is this going to continue 
to matter in the field um looking ahead? Thanks.  

01:08:38.560 --> 01:08:42.240
Great, okay, let's move on to our next question. 
We've got a few more if we can get to them. 

01:08:43.680 --> 01:08:47.760
So if we all could start from scratch, 
what recommendations would you have  

01:08:47.760 --> 01:08:52.320
for creating value-led evaluation of 
and rewards for research activities? 

01:08:52.320 --> 01:08:56.320
And that could be at Virginia Tech, in 
your field, by funders, or other groups.

01:09:02.960 --> 01:09:10.720
So at the risk of exacerbation problems it's it's 
I'm glad we have this question because it's not  

01:09:10.720 --> 01:09:17.680
enough to talk about the challenges, but also to 
imagine some solutions. Uh I think I think we need  

01:09:17.680 --> 01:09:24.880
to have room for as as long as we have tenure 
and promotion. I think we need to have room for  

01:09:24.880 --> 01:09:34.400
evaluation of that - that is not just a department 
level also back to the example of Destination  

01:09:34.400 --> 01:09:40.640
Area hires. If - if there's at any institution an 
effort to recruit people to be part of something  

01:09:41.280 --> 01:09:48.880
that by its very nature can't reside inside of a 
department that moves beyond it, then I think it - it  

01:09:48.880 --> 01:09:56.800
would actually make a lot of sense to include the 
room for the evaluation of that person's promotion  

01:09:56.800 --> 01:10:04.320
or tenure or their research to to be by a 
committee that is itself transdisciplinary  

01:10:05.040 --> 01:10:11.520
uh because it will address what is 
typically the big hurdle of dealing with  

01:10:12.160 --> 01:10:19.120
the the discipline itself as a primary way of 
defining what the research outcome should be. 

01:10:19.120 --> 01:10:25.760
I think we have to address the level of training, 
the future of research talent, so that means that  

01:10:25.760 --> 01:10:35.520
I think we need to also include room if we could 
start from scratch of making the capstone project  

01:10:36.400 --> 01:10:45.600
For a PhD, something that doesn't always have to be 
in in humanities disciplines, for example, doesn't  

01:10:45.600 --> 01:10:53.360
always have to be a long-form booklink manuscript, 
so the example of digital humanities project is  

01:10:53.360 --> 01:11:04.080
a perfect example or maybe someone is studying um 
let's say gender or religion or race and what they  

01:11:04.800 --> 01:11:10.800
really want to do is to address this at a 
societal level through organizing some kind of  

01:11:10.800 --> 01:11:18.800
equitable outcomes so could they could, they have 
a capstone project that is focusing on having some  

01:11:19.440 --> 01:11:24.080
uh social entrepreneurship organization 
where they have to put together. 

01:11:24.080 --> 01:11:30.000
The business plan for the public impact models 
the collaborations that they would have uh since  

01:11:30.000 --> 01:11:35.360
they want to have that kind of public effect as 
part of their their scholarship so if we could  

01:11:35.360 --> 01:11:40.320
actually build in different ways of disseminating 
research into the way that we credential people, 

01:11:41.120 --> 01:11:46.560
and the way that we train the future of talent, 
I think that would need to be an essential part  

01:11:47.280 --> 01:11:50.880
of whatever this new model would be 
if we were going to start from scratch.

01:11:52.160 --> 01:11:56.720
Yeah I agree I think we have the same issue in 
science. I mean nowadays we're doing a lot of  

01:11:57.760 --> 01:12:03.520
interdisciplinary work, and even trans-disciplinary 
work and when it comes to the evaluation, the  

01:12:03.520 --> 01:12:07.920
person is evaluated in one department by 
their peers that they don't necessarily  

01:12:07.920 --> 01:12:13.760
do that kind of research, or they don't apply 
to the same kind of funding agencies, because  

01:12:14.400 --> 01:12:18.560
I'm not supporting that particular 
transdisciplinary research.

01:12:18.560 --> 01:12:25.760
So then it comes to what is the weight that that 
uh transdisciplinary evaluation from other peers  

01:12:26.880 --> 01:12:32.560
is on the final evaluation for promotion. For 
example, but I totally agree. I mean nowadays I

01:12:32.560 --> 01:12:38.480
personally consider departments like a placeholder. 
I mean as scientists we are scientists, we use  

01:12:38.480 --> 01:12:46.400
all the all the resources around to answer one 
particular question so is is reduction is to think  

01:12:46.400 --> 01:12:55.840
that just one single uh um expertise from one 
department can really fairly judge the work that  

01:12:55.840 --> 01:13:03.840
someone is - someone else is doing in their field 
when is totally transdisciplinary. I agree with you.

01:13:10.000 --> 01:13:14.640
All right so let's go to our very last question. 
We've got about six minutes or so left for our  

01:13:14.640 --> 01:13:20.480
panel before general questions at the end. So this 
question is for panelists who have been involved  

01:13:20.480 --> 01:13:26.240
in the valuation side of recruitment promotion 
and tenure processes. So which of the concepts  

01:13:26.240 --> 01:13:28.960
tools or approaches presented by Dr. Gadd might  

01:13:29.680 --> 01:13:34.000
- might you consider for your own units or your 
own institutions promotion and tenure process.

01:13:38.480 --> 01:13:42.320
I can address that quickly I - I  
think I really like this idea of  

01:13:43.760 --> 01:13:50.320
evaluating what we value um rather than than 
what counts um you know, and I think this is a um  

01:13:50.960 --> 01:13:55.600
an argument again that can be made at 
different levels. Um it has to be made to  

01:13:55.600 --> 01:14:01.440
different audiences, um and it was it was implicit 
in Dr. Gadd's presentation but I appreciate Dr. Gill  

01:14:01.440 --> 01:14:07.760
bringing it out more explicitly, um, which is to 
make the the kind of power dynamics more visible.  

01:14:07.760 --> 01:14:12.480
Um you know, as we go through the process, 
um and look for ways to to recognize that  

01:14:13.760 --> 01:14:16.400
in each of these kind of 
stages of evaluation, thank you.

01:14:19.280 --> 01:14:26.800
Yeah I'll briefly jump in as well. I think the 
the suggestions to start with questions such  

01:14:26.800 --> 01:14:35.600
as whom do we exclude, what's the cost benefit 
analysis, and how does this individual demonstrate  

01:14:36.720 --> 01:14:43.520
hurdles, overcome or barriers that they've 
been able to move beyond as opposed to just  

01:14:43.520 --> 01:14:50.400
looking for highest peaks those are I think 
very important uh equity focused approaches  

01:14:50.400 --> 01:14:57.120
that are especially essential to addressing the 
the issues of power, uh the the lack of dynamism  

01:14:57.760 --> 01:15:06.720
in shaping the standards for evaluating excellence, 
and and trying to achieve an outcome that's just  

01:15:06.720 --> 01:15:11.440
going to be more inclusive. It's going to have 
more talent and different kinds of talent.

01:15:16.160 --> 01:15:23.520
Yeah it's a - it's really a complicated question, 
because really our field is very driven by what  

01:15:23.520 --> 01:15:30.240
you publish, and the resources that you find to - to 
actually fund that research, but there's certainly  

01:15:30.240 --> 01:15:36.800
a part that is tremendously missing that is 
which one is the societal - societal benefit  

01:15:36.800 --> 01:15:45.520
benefit of all our research. I mean, I think 
society is the ultimate beneficiary of our work,  

01:15:46.240 --> 01:15:54.800
and that is definitely not evaluated in any way 
in any promotion or recruitment in our field.

01:16:00.080 --> 01:16:07.760
I have not been involved in the evaluation 
side of P&T or anything like that. My experience  

01:16:07.760 --> 01:16:13.600
more is being evaluated, but I think uh 
just the one thing that was in Dr. Gadd's  

01:16:14.160 --> 01:16:20.160
presentation, and it builds on something 
Dr. Johnson was just mentioning. Uh I think  

01:16:21.440 --> 01:16:25.680
maybe a consideration of the different research 
communities that people are coming out of as  

01:16:25.680 --> 01:16:30.800
well so like, you know when many people will move 
through graduate studies and beyond, we all find  

01:16:30.800 --> 01:16:34.000
research communities that are asking 
particular questions are centered on certain  

01:16:34.000 --> 01:16:40.320
questions. Again, very often in terms of thinking 
about how particular systems of power negatively  

01:16:40.320 --> 01:16:45.040
impact certain constituencies or communities, 
so we locate ourselves in research communities  

01:16:45.040 --> 01:16:50.160
and these are our peers to which we look for 
kind of reading our papers and getting back to  

01:16:50.160 --> 01:16:55.040
us, and so those, but those communities 
are often not involved in evaluating  

01:16:55.680 --> 01:17:02.080
our uh research when it comes to going up for 
10-year um or at an institutional level, so I think  

01:17:03.200 --> 01:17:09.200
some sort of collaborative kind of process 
through which the holders of power within kind  

01:17:09.200 --> 01:17:14.880
of departments or more broadly are cognizant 
of - in recognizing the variety of research  

01:17:14.880 --> 01:17:20.320
communities that individual faculty are coming 
out of and be being kind of made to be aware of  

01:17:20.880 --> 01:17:25.040
let's say not standards of evaluation but what 
kind of counts and matters in those specific  

01:17:25.040 --> 01:17:34.960
research communities. Great okay, thank you very 
much. So I think we're going to move into a period  

01:17:34.960 --> 01:17:39.920
of general questions uh to our panelists as well 
as our keynote speaker, and Rachel Miles is going  

01:17:39.920 --> 01:17:43.840
to take over that part, but I'd like to thank all 
of our panelists for your participation thus far. 

01:17:44.800 --> 01:17:48.720
Great responses and great insights. Thank 
you very much for sharing. So Rachel, if  

01:17:48.720 --> 01:17:54.880
you'd like to take it over. Yeah uh thank you 
and let me stop share on this really quick. 

01:17:56.560 --> 01:18:02.960
Um - and we have some questions for panelists as 
well, so uh let's start out with uh one of the  

01:18:02.960 --> 01:18:10.160
ones we didn't get to before. So this will be for 
Dr. Gadd. Um so how to balance the need to perform  

01:18:11.040 --> 01:18:18.160
um to some KPIs for national funding versus the 
more supporting way of evaluating researchers? 

01:18:18.960 --> 01:18:23.360
Um so I think that's just um 
kind of a how do you balance that? 

01:18:25.040 --> 01:18:29.440
It's a difficult balancing act, um I think 
the way that we do it at Loughborough is to  

01:18:30.160 --> 01:18:33.520
make sure that we - honestly this depends where 
you are in the world, because not everywhere in  

01:18:33.520 --> 01:18:40.480
the world has national kind of funding schemes.
Um but in the UK we tend to - um Loughborough at  

01:18:40.480 --> 01:18:47.840
least - try to align ourselves with the researchers 
and see the national kind of funding dictats as  

01:18:47.840 --> 01:18:54.000
the necessary evil, and not the other way around, so 
we're not kind of focusing on the national funding  

01:18:54.000 --> 01:18:59.520
um requirements and seeing our researchers as 
the necessary evil to get that funding, and I 

01:18:59.520 --> 01:19:04.720
think that's that sounds a bit silly, but I think 
there are institutions where that is upside down.  

01:19:04.720 --> 01:19:08.000
Um I think we do. It comes back to this - 
putting researchers at the heart of everything. 

01:19:08.000 --> 01:19:13.600
You do as kind of research policy - focus senior 
managers in institutions. You have to absolutely  

01:19:13.600 --> 01:19:18.080
know where your priorities lie, and that's with 
your - your people - with your researchers, and yes  

01:19:18.080 --> 01:19:25.120
we have to play these games. We have to um kind 
of submit to national funding exercises etc. We  

01:19:25.120 --> 01:19:32.080
have to to stay alive, um, but our priority, our 
alignment is with our academic colleagues and  

01:19:32.080 --> 01:19:35.840
I think we just - you just need to make that very, 
very clear and consistent in all of your messaging.

01:19:38.160 --> 01:19:45.920
Okay thank you Dr. Gadd. Um let's see. Next one:
"Dr. Finkelstein said we all want to publish in  

01:19:45.920 --> 01:19:51.920
high profile journals, to be honest. Is this just 
because of the impact factor or some other reason?"

01:19:54.160 --> 01:20:00.400
And if somebody else wants to address this as 
well, that's also fine. Well multiple reasons. One  

01:20:00.400 --> 01:20:04.720
is certainly a reason of prestige. It's really 
good research that has been published in those  

01:20:04.720 --> 01:20:10.640
journals, and you know it's certainly a good 
recognition by your peers to get published  

01:20:10.640 --> 01:20:20.000
in one of those journals. Also, it really impacts 
your funding when you submit your grants for um  

01:20:20.720 --> 01:20:25.360
- for funding. The fact that they see that 
they are publishing your research is  

01:20:25.360 --> 01:20:30.480
being published in - in those high-profile 
journals certainly impacts the reviewers. 

01:20:31.120 --> 01:20:38.880
Um, I think that it comes from that end. 
Um also we do see that on evaluations  

01:20:38.880 --> 01:20:44.560
in - in - in my field. I mean Stack is not the 
same, you know, the same to - to publish in, and  

01:20:46.160 --> 01:20:51.040
journalists that they have very low impact 
factor and very many articles; they're having one  

01:20:51.040 --> 01:20:55.760
published in a well-recognized journal. It's 
really a combination of all of them. Sometimes -

01:20:56.560 --> 01:21:02.640
sometimes uh for example, we pick not to publish 
in a high impact journal, because we know that  

01:21:02.640 --> 01:21:07.920
it takes really long time to publish, and we 
decide, 'okay instead of going here, we go here.' 

01:21:07.920 --> 01:21:13.840
But we get our research published faster. That 
sometimes is it's a game that you need to play, 

01:21:14.480 --> 01:21:21.440
and it's unfortunately to be honest with 
you - um - and you know, sometimes you have  

01:21:21.440 --> 01:21:26.720
journals that because of the interdisciplinary 
nature of your research, even if the findings  

01:21:26.720 --> 01:21:32.720
are super interesting, they might not understand 
that it might serve different communities, and  

01:21:32.720 --> 01:21:37.360
it might not be the right journal to send. So 
I mean it's a number of factors that play.  

01:21:37.360 --> 01:21:43.840
I would say probably the most relevant is -
is truly prestige, at least in our field.

01:21:48.240 --> 01:21:52.400
Does anybody else want to address that 
question. If not we can just move on.

01:21:55.200 --> 01:22:00.080
Okay no one else wants to jump in. I'll just say 
a few things. Um I think we do need to really  

01:22:00.080 --> 01:22:04.720
understand what we value about journals better. 
We - it's become so much more complicated for  

01:22:04.720 --> 01:22:09.440
researchers. Now we're talking about developing 
a matrix at Loughborough University to help  

01:22:09.440 --> 01:22:14.560
academics to choose where to publish. I mean, 
10 years ago you - you knew where to publish.

01:22:14.560 --> 01:22:18.240
You published in your favorite journal. You 
knew the editor. You always published there, you  

01:22:18.240 --> 01:22:22.800
know. Now with - there's so many things to take into 
consideration, and I think there's - there is a right  

01:22:23.760 --> 01:22:28.800
- the argument that high impact factor journals 
don't equal quality is absolutely right and true,

01:22:28.800 --> 01:22:33.360
but I - it's interesting that when DORA, 
the Declaration of Research Assessment,  

01:22:33.360 --> 01:22:40.000
uh wanted to communicate, uh the declaration,
they did so in Nature, and when the top factor  

01:22:40.000 --> 01:22:44.880
I wanted to - um - discipline information about this 
alternative, so that's a transparency and openness  

01:22:45.440 --> 01:22:50.640
kind of factor for journals that could challenge 
the impact factor. They published in science, so  

01:22:50.640 --> 01:22:55.760
they're publishing in these high impact journals 
to criticize high impact journals, but the reason  

01:22:55.760 --> 01:23:00.240
they're doing that is not because they think that 
they're higher quality of the work that's submitted  

01:23:00.240 --> 01:23:04.400
to those journals - is higher quality - but they 
get higher visibility, and of course then you'll  

01:23:04.400 --> 01:23:09.920
get into this catch-22 that, you know, I've got to 
publish there, because it's visible. Um but then  

01:23:10.640 --> 01:23:15.120
you know, if people don't - if people stopped um 
submitting to those journals and reading those  

01:23:15.120 --> 01:23:18.640
journals and citing those journals, they would 
be less visible, and therefore the chain would  

01:23:18.640 --> 01:23:24.080
be broken. Um so we kind of, I think we need 
to look elsewhere than to kind of criticize  

01:23:24.080 --> 01:23:29.680
academics for publishing in high impact journals.
We need to break that chain, but we, but we can't  

01:23:29.680 --> 01:23:34.880
rely on individual academics to do it. It's just 
not - not fair or feasible. Yeah, this the - the truth  

01:23:34.880 --> 01:23:40.560
is also the citation increases, right? I think the 
citation of your article is - is also more likely  

01:23:40.560 --> 01:23:44.720
to increase if you publish in one of those high 
profile journals, that if you pri - publish it in  

01:23:44.720 --> 01:23:50.320
another one, because it looks - it looks better 
to cite a juror when you send your next paper. 

01:23:50.320 --> 01:23:55.120
And it's better to cite a journal that has - that is 
very prestigious among the the ones that you cite  

01:23:55.120 --> 01:24:03.760
in your own article. I mean it's just - it's yeah 
it's a ball really. It's super sad, but it's true.

01:24:08.080 --> 01:24:14.400
Okay, let's go to the next question. So uh this 
is a question for Dr. Gadd: "If we are to measure  

01:24:14.400 --> 01:24:20.160
what matters within our institutions, what is it 
specifically that faculty leadership is determined  

01:24:20.160 --> 01:24:26.080
needs to matter for achieving promotion and 
tenure? Are productivity and impact not critical?"

01:24:29.520 --> 01:24:35.280
Okay there's two questions there, um, I think the 
first part of the question, which is you know what  

01:24:35.280 --> 01:24:40.480
is it specifically that faculty leadership, uh, 
have determined needs to matter is up to faculty  

01:24:40.480 --> 01:24:45.280
leadership to respond to. Hence the SCOPE model 
you - you need to decide as an evaluator what it  

01:24:45.280 --> 01:24:50.000
is that matters to you about the thing that you're 
evaluating and I can say what I think matters to  

01:24:50.000 --> 01:24:55.760
scholarship, uh give my opinion on that, but as 
some of the panelists um mentioned we need to  

01:24:55.760 --> 01:25:00.640
do this at a local level. It's much more sensibly 
done there, but in terms of our productivity and  

01:25:00.640 --> 01:25:09.120
impact not critical, yes they are critical, but how 
do we how - how do you articulate what productivity  

01:25:09.120 --> 01:25:15.600
and impact are, and how do you measure what those 
things are, and do we just measure those things. So  

01:25:15.600 --> 01:25:21.600
the Wellcome Trust report, um, that came out recently 
on research culture was very much focused on um  

01:25:22.640 --> 01:25:28.560
looking at how we do our research, not what 
research is done, um, and that is just so much  

01:25:28.560 --> 01:25:34.320
more important. How are we working collegiately or 
are we providing a fair - uh - electrical environment  

01:25:34.320 --> 01:25:40.320
for um scholars to do their best work, you know? 
Or are they working under enormous pressures and  

01:25:40.320 --> 01:25:45.680
um publishing what they shouldn't be publishing?
And also all sorts of perverse incentives that  

01:25:45.680 --> 01:25:52.560
the current system is kind of perpetuating. So yes 
productivity and impact are critical. How do you  

01:25:52.560 --> 01:25:57.520
define them? How do you measure them? And are we 
just measuring those things? - is what I would say.

01:26:06.880 --> 01:26:14.080
Rachel, you're muted, yeah. I was muted, okay, let's 
see here - um there was a follow-up question for  

01:26:14.080 --> 01:26:18.800
Dr. Finkelstein. "You noted that the prestige of 
a journal matters for deciding where to publish.

01:26:19.440 --> 01:26:23.920
Why is it a problem to consider journal 
impact during evaluation of a reach  

01:26:23.920 --> 01:26:28.240
of a researcher's body of research?"
I don't think there's a problem  

01:26:28.240 --> 01:26:32.800
on considering that. I think the problem is that 
when you give too much weight to that single  

01:26:34.480 --> 01:26:43.280
parameter for evaluation - I - you know - it's just it 
seems - it seems to be um overweighting any other  

01:26:44.080 --> 01:26:51.280
form of evaluation. I mean, it's okay, it should be 
there. It should be included, um, but it shouldn't be

01:26:53.360 --> 01:27:01.520
the sole factor, or the - or - or should it be the one 
of the top two main factors for evaluating anybody.

01:27:02.240 --> 01:27:07.360
I mean there are other contributions that they are 
equally relevant, and they are not being taken into  

01:27:07.360 --> 01:27:19.600
consideration, so which goes back to the power of -
you know - what is important. Okay, let's see, we have -  

01:27:21.120 --> 01:27:27.920
"Isn't a major challenge the limited amount of 
time available for a) the evaluation process, and   

01:27:28.560 --> 01:27:31.840
b) the time of managers to absorb information."

01:27:35.200 --> 01:27:39.920
Might be more for Dr. Gadd, 
since you're in management. 

01:27:43.120 --> 01:27:48.400
Yes, but the decisions that we're making, using this 
data, are very important decisions, and you know,  

01:27:48.400 --> 01:27:55.600
a good evaluation process does take time and money,
and you have to factor that in if you care about  

01:27:55.600 --> 01:27:59.200
the decisions that you're making on the back of 
that evaluation. I think sometimes we evaluate it  

01:27:59.200 --> 01:28:04.160
too much. We love our data. We slice it this way 
and that way. We want to see what it can tell us. 

01:28:04.160 --> 01:28:10.800
Um rather than kind of taking a kind of more - um -
you know hypothesis-based approach to - to what we  

01:28:10.800 --> 01:28:17.040
want to actually discover from this data - and that -
can that can um take less time if you're - you're 

01:28:17.040 --> 01:28:24.160
very, very careful, and about what you actually do 
evaluate, um but yeah, it is time consuming in terms  

01:28:24.160 --> 01:28:30.960
of um time taken for uh managers to interpret the 
data. Yes, I mean, that is just - that's the art of the  

01:28:30.960 --> 01:28:36.480
policy maker- to kind of communicate what needs 
to be communicated in an efficient and effective  

01:28:36.480 --> 01:28:44.880
way, but um I - I think it can be done. Okay 
we're almost at time, but uh we only have two  

01:28:44.880 --> 01:28:49.280
questions left, possibly. I have a couple from our 
panelists, but I don't know if we'll get to them.

01:28:49.280 --> 01:28:54.560
So: "academic journals tend to be read by academics.
Is that the stakeholder group we should be serving?

01:28:54.560 --> 01:28:58.240
If we should be serving other 
stakeholders, what metrics might we value?"

01:29:01.520 --> 01:29:04.640
I can respond to that, because I also 
want to talk about the time question.

01:29:04.640 --> 01:29:09.040
We have to be very attentive to the amount of time 
we're asking for those who are being evaluated, and  

01:29:09.040 --> 01:29:14.400
it's - it's easy to say: 'do this and this and this 
and this' without recognizing the cost and benefit.

01:29:15.200 --> 01:29:20.000
But you know in terms of this question, it may be 
appropriate, you know, in working with those being  

01:29:20.000 --> 01:29:25.520
evaluated to say um you know be selective 
in what you're doing, and maybe instead of  

01:29:25.520 --> 01:29:29.680
two more journal articles, you know, you think 
about an Open Access publication or something  

01:29:29.680 --> 01:29:34.560
aimed at a public audience that then brings 
attention - more attention to your work. Um  

01:29:34.560 --> 01:29:39.440
and I appreciate that that is also a saying: 
'do more,' but there may be ways to say 'do less'  

01:29:39.440 --> 01:29:44.720
in certain areas and 'do more' in others, and that 
will um support your evaluation throughout the  

01:29:44.720 --> 01:29:49.680
process, as long as it's clear that it's with the 
person being evaluated, not about them or to them.

01:29:49.680 --> 01:29:56.240
Thanks, okay, we're at time but I'll try to get this 
last question, and "is there a chance that changing  

01:29:56.240 --> 01:30:01.680
the system will disenfranchise a different set of 
scholars? How can we mitigate against that impact?"

01:30:03.360 --> 01:30:09.760
Yeah, I just want to emphasize - uh - what the panel 
is - is suggesting in different ways is actually  

01:30:10.560 --> 01:30:17.200
having a more dynamic and inclusive approach 
to assessing and determining what counts, so  

01:30:17.200 --> 01:30:23.280
that's not disenfranchising anyone. It's - it's 
creating more space. It's like uh well, I won't  

01:30:23.280 --> 01:30:28.000
use that example, because I don't want this 
to sound political, but it's - we're not arguing  

01:30:28.960 --> 01:30:37.040
for excluding anybody. What we're arguing is that 
we already have a narrow system that is inflexible, 

01:30:37.680 --> 01:30:44.400
that is largely arbitrary, and that we've inherited,
and there's some simple steps that we can take  

01:30:45.200 --> 01:30:54.560
to create more space to include different ways, 
of a set of determining what counts. And so if we  

01:30:54.560 --> 01:31:00.480
do that, then we're not disenfranchised. If we're 
just going to say - I think one of the questions  

01:31:00.480 --> 01:31:06.880
assumed that we were against counting publications, 
so that would be an example of disenfranchising. No  

01:31:06.880 --> 01:31:12.480
one - no one on this panel is claiming that, but - but 
if you took that approach, then obviously you would  

01:31:12.480 --> 01:31:17.200
disenfranchise people who published, but - but no 
one is suggesting that what we're saying is that.  

01:31:17.200 --> 01:31:21.440
It makes so much sense to - to have a much more 
comprehensive approach to determining what -

01:31:21.440 --> 01:31:25.520
what counts as a publication for 10 
year promotion, if we're assessing talent.

01:31:28.560 --> 01:31:35.840
Um okay. Can I - okay - just very briefly I think - um -
to build on Dr. Johnson's comments uh around um -

01:31:35.840 --> 01:31:41.600
yeah not - not necessarily excluding any - necessarily 
existing metrics - or like - uh - but to say that not  

01:31:41.600 --> 01:31:46.640
only publications count, but I think something also 
to consider alongside that - that maybe we haven't  

01:31:46.640 --> 01:31:50.960
discussed yet, and I'm curious maybe I should have 
brought this up sooner, but I think - um - you know  

01:31:50.960 --> 01:31:55.920
something like the structure of public funding
for universities - I think is really important to  

01:31:55.920 --> 01:31:59.600
a discussion like this, because the - you know -
the creeping and growing and long-standing  

01:31:59.600 --> 01:32:05.200
commodification of education, you know, when you're 
hired into an institution, you join, the first thing  

01:32:05.200 --> 01:32:10.400
that is taken for granted and assumed as natural 
is that universities are facing a funding squeeze,

01:32:10.400 --> 01:32:14.720
a funding cut. There's always this question of -
we have these problems, and so go out and get  

01:32:14.720 --> 01:32:20.080
that funding, and it's often private funders or 
funding that compromises your research, rather  

01:32:20.080 --> 01:32:25.600
than asking this question: 'how can we advocate 
for more robust and sustained public funding  

01:32:25.600 --> 01:32:30.480
that can allow for more independent approaches 
to evaluation?' Because a lot of the journals  

01:32:30.480 --> 01:32:34.560
who are they being funded by? Who sits on the 
boards? And are they on the board because they've  

01:32:34.560 --> 01:32:40.240
established a certain reputation - do - resulting 
to how they can draw on particular funders?

01:32:40.240 --> 01:32:44.080
So and who are the owners of the journals, right?
These are some political economy questions, like  

01:32:44.080 --> 01:32:49.200
who is actually concretely owning and managing 
these journals? And a lot of these issues can be  

01:32:49.200 --> 01:32:55.520
addressed by advocating for and questioning why we 
accept this ongoing structural decline in public  

01:32:55.520 --> 01:33:01.600
funding that is important for autonomous research 
and autonomous evaluation for public universities.

01:33:05.520 --> 01:33:11.280
Those are some really excellent points. Um we 
do have a question from one of the panelists. Um  

01:33:11.840 --> 01:33:14.640
do you all want to stay on two 
more minutes, or do we wanna -  

01:33:15.920 --> 01:33:19.760
See a couple people nodding. If you have to go,
panelists, just give a wave and I'll totally  

01:33:19.760 --> 01:33:25.280
understand. Otherwise I wanted to ask this last 
question, uh from the - from one of the panelists, 

01:33:25.280 --> 01:33:30.800
um since she's involved in open education. Uh 
"where do highly impactful teaching materials,

01:33:30.800 --> 01:33:35.840
especially open educational resources, 
fit into recruitment, promotion, and tenure?"

01:33:40.000 --> 01:33:47.040
Should I kick off on that one sure I- I- I think 
you know if you value them, you recognize them.

01:33:47.760 --> 01:33:52.400
You know, that's - that's - that's the importance of um 
starting with what you value, if you really value  

01:33:52.400 --> 01:33:58.480
highly impactful teaching materials, um, then 
you should um specify that in your RPT criteria.

01:33:58.480 --> 01:34:03.280
whatever it is, your - your framework you're - you're 
working on, uh make it clear how that's going to  

01:34:03.280 --> 01:34:08.400
be assessed. You know at Loughborough, this is where 
altmetrics are useful, because um you can start  

01:34:08.400 --> 01:34:15.760
looking at where, for example, textbooks are um made 
available on reading lists around the world, and  

01:34:15.760 --> 01:34:21.840
that's quite an interesting indicator, and quite an 
easy one to to supply. Other teaching materials, a  

01:34:21.840 --> 01:34:34.400
little bit more difficult to measure, but that's 
one way of doing that. Any other panelists? Um,  

01:34:35.840 --> 01:34:42.560
okay, all right, well I know we went over by five 
minutes, and, uh, I apologize for that but I want  

01:34:42.560 --> 01:34:48.000
to respect everybody's time for the rest of the 
day morning, afternoon, wherever you are, and thank  

01:34:48.000 --> 01:34:54.240
all the panelists very much for giving us their 
time today and speaking honestly about, um, you know,  

01:34:54.240 --> 01:34:59.440
their experiences and so forth, and also thanking 
Dr. Gadd for giving the keynote. That was really  

01:34:59.440 --> 01:35:05.600
enjoyable, and we had a lot of people, so thank 
you everyone, and hope you all have a great rest  

01:35:05.600 --> 01:35:13.840
of your day. Thank you very much for the invitation.
Yeah, yes, thanks, Rachel. Thanks all. Thank you.