all right welcome everyone it is 10 o'clock so we are going to get started uh to make sure we run
this in a timely manner um thank you for coming and I think this is going to be a really engaging
event today, and so i just want to get some housekeeping rules out of the way before we start.
My name is Rachel Miles and I'm the research impact librarian here at Virginia Tech so
I was helping coordinate this event today. For housekeeping rules uh just keep in mind
that when you want to ask a question use the Q and A function rather than the chat. This makes it easier
for us to keep track of the questions. We also encourage you to write your questions into the Q and Q
when you think of them rather than at the end of a of a talk or the discussion. Uh this helps it
so that we can see them coming in, and we can get to them in a timely manner rather than populating
all at once. The chat is also available to um attendees so you can use that to talk amongst
yourselves, make comments. It's really there for you to just kind of engage with one another,
but you can also do so publicly using the hashtag #countingOA, so i'll put that in the chat if you're
on Twitter, and you would like to do that, feel free. Um if you also need to ask any technical questions
uh or just questions that are not directly for the speaker you can use the chat function
sent that to only the panelists. I made the mistake myself so #countingOA, it's in the chat.
Now to everybody and not just panelists you'll want to toggle to make sure that all panelists
and attendees are selected or if you just need to ask a question - a technical question - you could
do all panel all panelists and that'll go to our end so that somebody can help you. Jordan Kuneyl is
our technical support and he can help with any of those types of questions.
All right, subtitles and captions are also available. We have a live captioning service
and you'll just go to 'more' and then you can click 'show subtitle' and you should be able
to see those display on your screen. The event is being recorded and it will be archived on Virginia
Tech's institutional repository, VTechWorks. A link will be sent to all registrants within the
next few days and there's also a transcript of the event available and we will work on making
sure that captions are also on there for the event. We have the keynote talk first followed by some
questions for the keynote speaker, Dr. Elizabeth Gadd, and then we will have a discussion panel with
faculty members from Virginia Tech moderated by Dean Tyler Walters of University Libraries.
And before the event starts, we are trying to read a land acknowledgement for uh public events
and Virginia Tech events that we hold and so I will read that statement now. We acknowledge the
Tutelo and Monacan people who are the traditional custodians of the land on which we work and live,
and recognize their continuing connection to the land, water, and air that Virginia Tech consumes.
We pay respect to the Tutelo and Monacan nations and to their elders past, present, and emerging.
I'm grateful to the American Indian and Indigenous Community Center at Virginia Tech for providing
this land acknowledgment statement. Finally I'd like to introduce our speaker Dr. Elizabeth Gadd.
Dr. Gadd is a scholarly communications specialist working as a Research Policy Manager in
publications at Loughborough University in the United Kingdom. She chairs the INORMS Research Evaluation
Working Group, the ARMA Research Evaluation Special Interest group, and the LIS-Bibliometrics Forum. She
also founded the Bibliomagician Blog and was the recipient of the 2020 INORMS award for excellence
and research management leadership. she has spent most of her career in libraries and doing research
into the rights issues related to scholarly activities of UK higher education. I know Dr. Gadd
as the Chair of the LIS-Bibliometrics Forum which I started following early in my career
a few years ago, the Bibliomagician Blog, for which I started as the blog content development officer.
Rachel can you slow down a little bit?
Oh yes i'm sorry. Thank you. Yes, yes I'm sorry.
Um, I know Dr. Gadd as the chair of the LIS-Bibliometrics Forum and the Bibliomagician
Blog, and I will also copy paste what I have here if that's helpful for everybody.
Um and I'm always grateful for her leadership and her courage to stand up and ask the hard
questions to researchers, administrators, and data providers. Without further ado I present
Dr. Gadd and her keynote talk, "Counting what counts in recruitment, promotion and tenure."
I'm going to copy paste that over in case anybody misses that.
Thank you so much Rachel for that lovely introduction. I'm hoping you can hear me I'm hoping
you can see my slides. I'm hoping that someone will be able to let me know if neither of those are
happening um so i can see our thumbs aloft which is great. So so a very good morning to you from my
afternoon here in the UK and thank you very much for inviting me to share a few words at your
Open Access Week event. I'm going to try and speak slowly because I know this is being live
captioned which is fantastic. Um okay so I wanted to say a little bit about myself but the outset
Rachel said some good stuff, some kind stuff, thank you. I think it does help to know uh where someone
is coming from so i would describe myself as a librarian currently working in research policy
at Loughborough University in the UK. Loughborough University is a research intensive institution
uh in the midlands of the UK. I was asked just to explain where we were, and we have the largest
campus in Europe. Um I had up and also a number of groups in this kind of research evaluation space
uh including the LIS-Bibliometrics forum that Rachel's mentioned which is home of the
Bibliomagician blog ably managed by Virginia Tech's own Rachel Miles and then when I moved into research
policy five years ago. I found myself becoming the champion of the UK association of research
managers and administrators. Research evaluation SIG, and then two years ago I was honoured to be
invited to form the INORMS Research Evaluation working group that consists of 12 international
research management societies. I'm going to share some of the exciting work that that group is doing
today. So my topic today is counting what counts in recruitment, promotion and tenure, and if it's
not already obvious what the connection is between research evaluation and open access, i hope to make
that clear. So what I plan to do is to firstly identify what the academy currently rewards
and the consequences of that, and then to explore what the academy really values, and then I'm going
to look at 101 ways that we can measure what we value rounded up to the nearest 101. Of course
before thinking about how we might persuade higher education leaders to actually measure
what they value. So what does the academy currently reward? So earlier this year at the King's College
Open Research Conference Marcus Munafo summed up the situation brilliantly with this slide. He said
scientists are incentivized to publish and obtain grant funding, and uh that was the end of the list.
And of course you could argue that funding only really counts because it leads to publications,
but we all know it's not just any publications that count. It is journal articles, so this chart
shows the submissions to the UK Research Assessment Exercise between 1990 and 2008 in
different fields, and you can see the proportion of journal articles. The red bars went up pretty much
every time, even in humanities subjects, and we also know that overall in 2014, over 80 percent of all
outputs submitted to the UK REF as it is now, were journal articles, although some recent research has
shown the tide might be turning on that, um but to date this has been the story, but it's not just any
journal articles that we reward. It's those with an exciting story to tell. So this study here
was fascinating, and it looked at the cumulative effect of reporting and citation biases and what
eventually ends up in the scholarly record. So they started with all of the clinical trials registered
relating to the treatment of depression, and the red dots on bar on the left hand side show
the proportion of negative trials relative to those that have positive results. The green dots
on the next bar shows the proportion of those positive and negative trials that made it
into the literature at all. The next bar shows a proportion of those published studies that
actually focused on the negative findings; the next shows a proportion of published studies that put a
positive spin on those negative findings, and the final bar shows which of those studies ultimately
got cited by resizing the dots according to citedness. And from this we can conclude that
there is absolutely no citation glory. The tiny red dots on the right hand side to be found in
negative trials no matter how critical they are to scholarship, and of course the more sinister issue
here is that this can give us a false impression of what's true. So this example shows that 95 out
of 100 studies found no association between autism and brain region, but five did find a
false positive result, and of course those five got published. So now when you do a search for autism
and cerebellum, you find one negative and five supporting studies, which is deeply problematic,
and of course, one of the reasons we only publish positive studies is that the more prestigious
journals won't touch negative studies, because they don't get cited; we have to publish in prestigious
journals, because most hiring committees are looking for publications in prestigious journals,
as this study of U.S. promotion and 10-year criteria found where 40 of U.S. research intensives
had requirements around journal impact factors or brands, and the only thing that really surprised
me about this was that the figure wasn't higher. And of course when you factor in those
institutions that use them informally but just don't write it down, the figure probably is higher.
And the problem with prestigious journals is that they are mainly closed or hybrid,
so Web of Science boasts that 23 of the papers they index have a free legal green pure or hybrid
gold version, but by my calculation that means 77 are still closed and of course Web of Science and
Scopus are still the databases that provide many of the bibliometrics rewarding our publication
culture, and those databases are of course heavily skewed towards STEM subjects as this study found
last year where every database they investigated indexed fewer arts and humanities outputs than all
other subjects. And of course as you can see from the Scopus journal selection criteria here
they index titles mainly in English, and then the real proof your publication has made it
is if it's cited. Uh as you can see here for institutions with elsewhere SciVal
that their measure of a researcher's overall research performance,
and notice the title is not publication performance its overall research performance.
Their indicators for research performance include five citation metrics and your publication count,
and for those of you who have been following the conversations on LIS-Bibliometrics about scite,
that's a new service that uses natural language processing to try to discern whether a citation
supports or disputes the paper it cites, you'll know that in future it may not just
be citations that will count but whether a machine somewhere thinks they're positive,
and this example is one of my own papers, and the allegedly disputing paper was also written by me
and surprisingly does not dispute my earlier work in any way. So to recap what we currently
incentivize in the academy is journal articles reporting positive results in prestigious journals
that are mainly closed or hybrid indexed by commercial bibliographic databases that are biased
towards STEM subjects written in English and that are well cited as measured by bibliometric tools.
So what might the consequences of only rewarding publications and citations be?
Well surprise, surprise, valuing only one thing leads to a lack of diversity.
Firstly, in the scholarly record itself so far from bibliodiversity, we have bibliohomogeneity.
Smaller independent publishers can't survive. There's downward pressure on other arguably
more important outputs from research, such as data code, protocols, everything's written in English, etc.
Secondly, it leads to a lack of diversity amongst those actually able to access the
scholarly record, because it's closed. This is Open Access Week. We need to point that out.
And of course as we move towards the scholarly record becoming open by default, it leads to
a lack of diversity amongst those actually able to contribute to the scholarly record,
because APC's, article processing charges, are increasingly aligned with publication metrics.
And of course, it also leads to a narrowing of those actually rewarded by the scholarly record.
So Indonesia recently introduced a new researcher ranking system that uses the volume of articles
and citations from Google Scholar and Scopus for deciding who are the best researchers and for
allocating funding to these people. The resulting top 10 looks like this: the number of women in the
top ten: one; the number of researchers in the fields of social sciences arts and humanities: zero;
So how do we fix this? You know what's the alternative to publications being the single
currency of research credit? Well of course it's by returning to what we actually value about research:
researchers and research institutions. And this is where I want to briefly introduce the INORM's
SCOPE model for evaluating responsibly. This is a simple five-stage process developed by the INORMS
Research Evaluation Working Group, and we introduced this model because normally research
evaluation starts and finishes with number three: what are our options for evaluating this thing?
Let's go and evaluate this thing. But we believe it's critical that we start with number one: what
do we actually value about this thing that we want to evaluate? So step one of SCOPE is to start
with what you value about the thing that you're evaluating, and it sounds obvious but all too often
we don't start our evaluations here. We start with what other people value, so university rankings or
funders or with historical values, which might have been meaningful 30 to 50 years ago before we
understood what the consequences of those values might be, and I think it's worth asking ourselves
when did we last review our RPT criteria what has changed in the last 30 years? And of course the
other thing that we do here is to start with the data we have available to us and to work backwards.
This is called the street light effect as illustrated by this cartoon. Where did you lose
your wallet? In the park. So why are you looking here? Well this is where the street light is. And
many, many evaluations rely on bibliometric data, not because the volume of publications or even
their citedness is what they value, but because it's so readily available and so easy to use.
Once we're clear on what we value, we need to consider the context for our evaluation.
And I hear many arguments in my role about which indicators are responsible and which are not, but
we can't have that conversation without knowing the context, the why and the what in which the
indicator is going to be used. Now of course we're thinking about recruitment and promotion today,
but I think it's worth remembering that there are many reasons for evaluating, and this matrix here
lists six key purposes alongside various entities of different sizes, and attempts to indicate what
the associated impacts, and therefore risks, might be. So for example, plotting the publication volume
of one country against that of another for the sole purpose of understanding their relative
production levels has very little impact on the country's under assessment, so we've labeled that
green. However any form of assessment at the level of the individual that results in reward has a
high impact on those individuals and therefore carries a greater degree of risk, and I would
say that the use of bibliometrics in these various settings maps pretty neatly onto these color codes.
And of course the other key context consideration is discipline, so a promotion approach that works
for folks in one discipline will not necessarily work for those in another. And at Loughborough,
we're currently looking at this with the particular focus on arts and humanities scholars.
So once we've thought through what we value about the research activity we're measuring and the
context, it's worth pausing before we launch into our options to consider whether we actually need
to evaluate at all. Now this might sound like an odd thing to say, but this is really with an eye on
the increasing use of measuring to incentivize certain behaviors, and as this is Open Access
Week, I have to say that this is often seen as a solution for getting people to engage with Open
Access, but we have to remember that incentivizing behaviors can be done in many ways, like actually
making it so easy for someone to act in that way that they don't need any other form of incentive.
And I'm a massive fan of the approach taken by the Alan Turing Institute where their goal is
to make data reproducibility too easy not to do rather than rank people on their ability to do it.
But when we do evaluate we should always do so with the evaluated. How often do we get feedback
from those seeking promotion and tenure or those who've recently been through our recruitment
processes to understand their experiences and to invite them to help co-design a better way
of doing this evaluation? - is something we should do WITH people and not TO people.
Then we have other options, and then this is a reminder really that you do have options. We
don't have to reach for bibliometric data all the time, and the important thing to consider here is
whether your indicator is a suitable proxy for the thing that you're evaluating. We think the rule of
thumb here is that generally speaking, quantitative measures are generally for quantifiable things,
such as citations, publications, money, or students, and qualitative measures are for qualifiable
things, so excellence and value, and we should be especially careful if using quantitative
indicators as a proxy for qualitative things. A highly cited candidate does not equal
a highly suitable candidate. A candidate from a highly ranked institution does not equal a
better candidate. Often the selection of options for evaluating marks the end of the evaluation.
Designer folks launch in, and just do their thing, but we feel there's a critical probe stage that
needs to be considered first, and we think there are four questions that need to be answered here,
particularly when it comes to RPT processes; so we need to ask firstly, who does this discriminate
against? How diverse are the candidates regularly being rewarded by this process? Secondly, how might
this approach be gamed; so how selective is that publication list you're looking at,
what contribution did this candidate actually make to the activities they're listing?
Thirdly, what might the unintended consequences of this approach be both on the development of
your teams internally but ultimately on wider community agendas around openness,
rigor, societal impact? And finally does the cost of measuring outweigh the benefit?
And of course, one is seeking to fairly evaluate a large number of applicants, the costs can be really
high. So how do we get that balance right and the 'E' of SCOPE is to then run your evaluation and to
evaluate it. So that was a very quick fly through SCOPE, and there's a lot more on our website.
And I've left the URL there for you if you're interested in pursuing that. So I've argued that we
need to measure what we value in our RPT processes, but the question remains as to what we actually do
value in the academy, and obviously we all have our own answers to that question. That's quite
quite rightly, but I think that in order to build strong and successful academic communities, we need
firstly a broad diversity of people. We know that companies with diverse management teams have 19
higher revenue and that 87% of the time diverse teams make better decisions.
We should also value a variety of disciplines as COVID has shown us, it takes more than medics
to fight a pandemic. We need communicators and psychologists and designers and so on.
We need a diversity of findings. We need all the results, positive and negative, to make the best
decisions. We need a diversity of outputs, data, design, sculpture, trade journals, etc.
I saw this wonderful cartoon on twitter recently which sums it up rather beautifully: all modern
digital infrastructure rests on a small but highly significant project being maintained by a single
soul in their bedroom. And of course if they didn't write a paper on it, it currently wouldn't count
for anything. We value a much broader diversity of impacts than we currently reward. Societal
economic public engagement. And of course we value a far greater diversity of research activities
than we reward. We need leadership, collegiality, innovation, integrity. And the Hong Kong manifesto
for assessing researchers seeks to foster just this by calling for responsible research practices,
such as reproducibility, transparent reporting, across all stages of the research process,
and open science, not just open access in addition to the diversity we've just talked about.
Wo how do we measure all of these things. Where do we start?
Well we have some help here, and you might not be able to see this on a small screen,
but the DORA website, the Declaration of Research Assessment, has started a really helpful list of
research evaluation innovations, and I've thrown a few of my own in there as well just to celebrate
folks really moving away from the journal impact factor as a sole indicator of research
success. And when you analyze those research evaluation innovations as I've done loosely here,
you'll see that the vast majority are around how to evaluate individuals for jobs or projects for
funding, so when it comes to recruitment promotion and tenure as we're discussing today, you're pretty
much spoiled for choice. So here's a couple of my favorite examples so UT Southwestern provides
shortlisted candidates with questions prior to skype interviews to give them reflection time,
and they ask them not only what they've done in the past but what their future plans are,
and we've also got um an example here from the Charity University Hospital in Germany, which asks
for Biosketches from candidates asking them what their contributions to their field are as well as
open science activity and so on. And just yesterday Yale's molecular biophysics and biochemistry
department announced that they were going to ask for anonymized applications for upcoming assistant
professor jobs to minimize an intended bias and, I think these practical examples remind us that
we need to not only measure what we value but to live what we value in our evaluative approaches.
So we need to put researchers and their well-being at the heart of everything that we do by making
our evaluation criteria clear and sticking to it, which is a principle of DORA, and adhering
to three other important principles, the first of which I think is to seek potential over past
performance. Research evaluation is inherently backward looking, especially bibliometrics,
and we know that the Matthew Effect, the more you have the more you get, is rife in science so we
have to find a way, as difficult as it might be, of valuing potential and not just past performance,
which may well just be a function of privileged as Adam Grant recently tweeted: "the true measure
of people's potential is not the height of the peak they've reached but how far they've climbed
to get there." And if we care about diversity, we have to get better at measuring this. Secondly
we need to pursue recognition over judgment. No one wants to be judged but everyone wants to be
recognized for what they bring to the table. RPT is incredibly competitive but that doesn't mean
we can't do it kindly, celebrating strengths where we find them, even if they aren't what
we're looking for right now. And I think one of the key ways that we demonstrate kindness is by
providing feedback that is formative evaluation that actually shapes someone's future and helps
them to grow rather than summative evaluation that just gives them a score or a rejection. And
Gemma Derek from Lancaster University in the UK is currently working with the Wellcome Trust on kinder
peer review and how we can actually introduce a kinder and more formative uh research culture.
But you might be sitting there thinking, well that's all well and good Lizzie, but it's never
going to happen in my institution. You know how do I go about creating change where it's needed?
Well never fear because another of the outputs from the INORMS Research Evaluation Working Group
was a short one-page guide containing five arguments that you could use to persuade HE
leaders to evaluate responsibly. The guide is on the INORMS web pages, and obviously it will depend
on your local setting as to which these arguments you think might have traction. but in a nutshell
the five arguments are these. but firstly there's a moral argument for responsible research evaluation
as I've already mentioned, that it's fairer, and that institutions that engage with poor practices
put their staff well-being at risk and with it expose themselves to reputational risk.
So there's been a number of high-profile cases internationally where poor evaluation practices
have led to tragic outcomes, such as the case of Professor Stefan Grimm from Imperial College
London who died by suicide after failing to meet the expected income target. There's the
case of Lucy Zowe at the University of Technology Sydney; she was sacked after she failed to meet a
publication target, and the court subsequently ordered her institution to reinstate her.
And of course the ongoing unhappiness in the US about the use of Academic Analytics.
There is undoubtedly a growing audit culture in higher education, just as there is globally,
which is having a negative impact on our academic colleagues who are already facing a higher mental
health risk than other professions. So to my mind there is a strong argument that higher education
leaders should engage with responsible RPT practices simply to discharge their duty of care.
However, ultimately responsible research and evaluation, to my mind, simply leads to better
decision making. So Campbell's Law tells us that we get what we measure effectively by
measuring what matters, and with indicators that actually indicate those things that matter to them.
Institutions are simply more likely to make the best decisions that lead to the
best outcomes. Excuse me. As somebody wants to put it, play stupid games win stupid prizes.
I mean you can recruit on h-index if you like but you might find up -
find out - that you end up with a lot of elderly male physicists on your staff. Of course when you're
potentially investing a lot of money in research, staff systems, and activities, institutions also
need to ensure that they are investing wisely in meaningful evaluations that are actually going
to give them valid answers to their questions and not waste money on shortcuts that won't.
We've talked already about the competition for jobs in academia being really high, which
is painful for candidates but also makes the job very difficult for recruiters
and very expensive, but not as expensive as hiring the wrong person.
Our fourth argument points leaders at the increased focus on responsible research evaluation.
as a result of sector engenders, including open research improving research culture and
responsible research and - evaluate - innovation. Responsible research evaluation is now a
requirement of Plan S, and the Wellcome Trust's Open Access policy, and rather excitingly the
Global Research Council, is taking an interest in this area; that's an international group of
research funders, and they're holding an online event in November on this topic with a view to,
and I quote, "consider collectively approaches to research assessment criteria and processes
to facilitate real shifts towards a better research and innovation culture." So institutions
need to make sure that they're operationally ready for whatever results from this process.
However, whilst there are external drivers, ultimately we believe that responsible research
evaluation gives institutions an opportunity to maintain their institutional autonomy
as mission and value-led organizations and enabling them to set and pursue their own goals,
recruit their own people according to their own ambitions, and measure progress
against their own goals and not others, such as university rankings. And you may have seen that
one of the other outputs from the INORMS research Evaluation Working Group was our university rank
ratings, launched just last week, which highlighted just how deficient the university rankers are when
it comes to things that matter to the higher education community, such as good governance,
transparency, and rigor, with their worst performance levels on measuring what
matters, which must lead us to think if they're not counting what counts, why should it count?
In summary, what we reward and what we value in the academy are actually quite different things,
and this doesn't serve anyone very well. It doesn't serve our scholars, our institutions, or scholarship
itself, and it certainly doesn't lead to structural equalities or openness, as celebrated by this
year's Open Access Week. So we need to both assess and live by our values through our RPT processes.
And there are an increasing number of innovations in the space that we can learn from, which are
leading to significantly more successful appointments, so it stands to reason that
we should all use our influence to achieve this. And I hope that some of the INORMS work can assist
in helping us all to count what counts to the benefit, ultimately, I think of all humanity. So that
is it from me. Thank you very much for listening. I'm going to pass back to our host now. I look
forward to hearing the panel discussion and to continuing the conversation with you.
Hey thank you Dr. Gadd, and I'm going to uh give you some questions. We have some questions in the Q&A.
Um so first question is that: "a value emerges from communicating rather than publishing.
Can we refocus the discussion?" And
I have an um question but what do you think?
Value emerges, emerges from communicating rather than publishing. Yes it does. Um and I think
that, um, there's a lot of innovations now which are seeking to just re-evaluate the whole scholarly
communication process and not just incrementally improve um journals, to make them a little bit more
open, a little bit more transparent, and include additional data, but to re rethink the journal as
the - as the means of communicating in a scholarly way. So there's the - I'm a big fan of the octopus
project Alexandra Freeman has um uh developed, where she proposes that you publish um your
your hypothesis, your methods, your uh data, your kind of findings, your conclusions as separate
um entities as their separate outputs, all of which could be kind of assessed through open peer review.
Um and so there are there are other much better ways of doing it but it's um I think a lot of
the frustrations certainly from early career researchers and Martin is I think I know he's
a representative of those is um that they want to leap from here to there overnight and I and I
understand that frustration, and I would share that excitement if we could do that but I don't
think it's possible. I think we're gonna have to move there incrementally. I think um and to engage
probably with funders to really um power that that change um, so so yes you're absolutely right we do
need to move in that direction. Um but I, I think it might be a bit slower than most of us would like.
Yeah, in the octopus project um doing a simple Google search for it I have to give her name
because I'd like to link it in the chat. Uh do you know? Alexandra freeman, I think.
Freeman, I'll get that pulled up, but uh moving on to the next question: "how do we account for
diversity without locking up researchers in disciplines which is merely a social construct?"
Well I guess there's different forms of diversity isn't there? And and discipline is one of those
but, well, if we had a good answer to that it would be very happy people and it's a it's um
a discussion topic that even the greatest and the best of us are scratching their heads over.
You know what is a discipline exactly, you know there's a lot written on that um how
we define it where the boundaries are so yeah, it's a very good point I don't have a solution.
I don't know whether Martin has one that he wants to propose but um it's a good point.
Okay: "could you elaborately elaborate a bit on how the SCOPE INORMS framework compares to DORA,
the Leiden Manifesto and the Hong Kong Manifesto. Do we need to look at all four? What are their
specific strengths? That's a good question um so we see um DORA, the Leiden Manifesto um
particularly as the high level principles around responsible research evaluation. Um
they are older so DORA was um created in 2012, Leiden Manifesto in 2015, and so we're we're five or eight
years on from those now. We see SCOPE as more of a uh a road map rather than a highway code if you
like, that's that's the distinction in our minds. So for practitioners like us, um principles are
very helpful, but how do you actually go about developing a responsible evaluation? And that's
where we think SCOPE comes in, because it does clearly say right we'll start with start here
with what you value, then consider your con- your context then you can think about your options.
and i think in terms of the Hong Kong Manifesto, that provides helpful guidance if your the values
that you come up with under the S of SCOPE map onto the values that they're seeking to propagate
through that assessment framework, so that's how I'd see those fitting together, but there was a a
conference last week of the Nordic Bibliometrics Workshop, and somebody um had developed a mapping
of dora, Leiden Manifesto, and the Metric Tide, which is the UK's um independent investigation
interest responsible research evaluation. Again from 2015 and so it's and I came up with five
principles for responsible research evaluation, so that's quite a nice little mapping and if I
um after this Q&A, I'll try and find that and put it in the chat so people can have a look.
Okay, all right, next question we have:
"Looking forward, what do you think the role of altmetrics will be for scholarly evaluation?"
that's a really good question. i think the beauty of altmetrics at the moment is the contextual data
that they provide. so at Loughborough, we encourage individuals to supply altmetrics where they're
aware of them that highlight alternative impacts that their work has had on an individual output by
output basis. What altmetrics can't currently do is to um well it couldn't the last time I looked so
correct me if I'm wrong, is to kind of allow us to benchmark. So how how is Loughborough chemistry
department doing on altmetrics relative uh to Virginia Tech, for example. You can't currently
do that level of analysis, so and I don't know if they - I think they've probably deliberately
steered away from that; I can't speak for them, but that's my understanding from conversations
with old metrics folks, so I think that the beauty of altmetrics is in that kind of rich contextual.
You're not just looking at numbers you're looking at the actual tweets, the actual blog posts,
um the actual demographics of folk who've engaged with your work, and you as an individual
can identify where that's meaningful, but beyond that it's it's a bit more challenging for folks
like me who are trying to do analysis at a like at a higher level of granularity.
Okay uh next question, and this has a lot of upvotes uh so it's getting pushed to the top here,
"Is the notion that we have we have to persuade higher education leaders to make changes
a sign that the roadblocks to change are much bigger? Can these changes be made in a hierarchical
and anti-democratic workplace?" Yeah I mean that's why we've offered these - we've offered these five
arguments but not in any particular order because we recognize that some
institutions, you know culturally, eat strategy for breakfast doesn't it? That's the expression and
it's so true. Um and if you've got a culture that is hierarchical and anti-democratic, you're going
to struggle a lot more than if you're working in an institution like mine which is open, liberal,
welcomes um any opportunity to support a broader diversity of um scholars and scholarships, so
yes it is going to be it is more challenging of course in those sorts of organizations,
but that's why we put as one of our arguments you know that these demands are coming. You're
going to be evaluated on this sooner or later, you'd have to get operationally ready for this,
because whether you like it or not, it is coming so um yeah it's the challenge.
Okay okay: "Given that researcher metrics are also wrapped up and used to rank and evaluate
institutions, what would convince a high-ranked university to risk its valued global ranking?"
Yeah um what would convince high ranked university to risk its valued global ranking?
That's the problem, that university leaders have is that they are what I call it - excuse
the expression - but in the UK we have a reality tv program called Snog, Marry, Avoid. We might have got
it from the U.S, I don't know, um, but I think that your higher education leaders have this
problem when it comes to engaging with university rankers, because most understand
that they are methodologically problematic, um, but they either kind of jump into bed with them -
to use a clinical expression - um by their day to host their conferences and really seek to decline
the rankings, or they kind of make the best use of them where they're shown in a good light, but then
keep quiet about it most of the time, or they avoid them all together. But none of these positions
actually challenges the dominance of the university rankings, um which is why we introduced
this ranker rate - ranker rating at the right round um - to try and kind of introduce a new apex predator
in the food chain if you like, to kind of say, "No we want the research community to ultimately be
the judge of what we what we value and how we're assessed, and we would like you university rankers
to answer to us as a research community, and we want to expose, you know, how well you're doing on
the values of the higher education community, um at the moment, and hopefully encourage them to change.
So um there's so many stakeholders in research evaluation, and in the kind of university rankings
game, it's not just university rankers that use them, it's their students. You know, you can't,
it'd be financial reputational suicide to um stop promoting your university ranking, um if
it's a good one, because then global students are going to go elsewhere, and we know that's -
that's problematic for those institutions that rely on uh international students, so it it's a
it's a challenging conundrum um - I've outlined our solution to that.
Uh it needs to be sustainable though recognize it's just a project at the moment and so if
you've got any great ideas as to how we can make that more sustainable then happy to to hear those.
Um all right so we are almost at time. I think we'll do one more question
and we'll try to get to the other questions, uh after the discussion.
Um so I apologize if we didn't get to your question yet, uh but the next question and I'm
gonna go with the one with the most upvotes and I feel like I'm ranking the comments.
"Isn't publishing the dissemination of research findings after peer review
critical to sharing quote 'trustworthy' research findings that others
can build upon so - not quote but emphasize *trustworthy*
that others can build upon communication without any sort of checks or balances
leads to the dissemination of misinformation as we - as we've seen in the case of COVID?"
Yes um I don't think anything I've said would um mitigate against that.
Um you know obviously there is an increase in the sharing of um
research results through pre-prints through the COVID era which has proved um
both successful and problematic in probably equal proportions, so there is - there is a big
question here undoubtedly. uh I'm not sure I'm entirely qualified to to say any more on that.
Okay um it looks like it's 10:45, so we're gonna move on. I still have three questions in here,
but hopefully we'll get to them at the very end, and I'm going to go ahead and start introducing
um dr. uh Tyler Walters. Um Tyler Walters is the Dean of University Libraries, and he'll
be moderating this panel. He is also a member of the Advancing Public Access to Research Data
Steering Committee, co-sponsored by the AAU and APLU, he was also recently elected as a Member
of the Board of the Association of Research Libraries, and I will go ahead and turn it over
to Tyler. I will start sharing my screen so we have the panelists up here for everybody to see.
All right, all right, great, thank you very much Rachel. So real quick let you know that we have
uh we're going to be efficient this this morning or afternoon depending on where you're at. We have
four panelists. We have about 35 minutes and a number of questions that we're going to try to
get through, but let me introduce our panelists. To begin with, first we have Tom Ewing who is
a Professor in the Department of History and Associate Dean for Graduate Studies and Research
at Virginia Tech's College of Liberal Arts and Human Sciences. and next we have Carla
Finkelstein, who's an Associate Professor in the Fralin Biomedical Research institute
and Department of Biological Sciences in the College of Science and
also holds affiliate positions in the Department of Surgery and the Faculty of Health Sciences in
the Virginia Tech Carillion School of Medicine; she's a Principal Investigator of the Integrated
Cellular Responses Lab and Scientific Director of the Molecular Diagnostics Lab in the Fralin
Biomedical Research Institute in Roanoke. Third we have Bikum Gill, who is an associate professor in
the Department of Political Science in the College of Liberal Arts and Human Sciences; he's also the
co-co-investigator of the Project for Stories and about Food Sovereignty, for which his team received
a Social Sciences and Humanities Research Council Partnership Development Grant. And fourth we have
Sylvester Johnson, who is the founding Director of the Virginia Tech Center for Humanities and
internationally recognized humanities scholars specializing in the study of technology,
race, religion, and national security. He's also our Assistant Vice Provost for the Humanities
at Virginia Tech, and executive director of the university's Tech for Humanities Initiative.
So we have four wonderful panelists and we'd like to begin again with some you know brief
responses for you all from Dr. Gadd's talk. So what what about her talk resonated most with you and
with your own background and experience. Anybody like to go ahead and address that?
First I would like to thank uh Dr. Gadd for um her presentation it was fantastic and you hit really
all the points. I mean um I was really thinking on you know ahead and you your nexus like what
is what's exactly my my you know the answer to my question pretty much um I - I - think what
resonated is, to me, the most is the concept of that evaluation has to be with people and not to people.
I think that that is something that in my field at least is - is it's clearly not the case.
I just want to pass this to my colleagues yeah. Tom would you like to go next? Sure no and I uh
also thank Dr. Gadd for for for presenting um. As a historian all of this sounded you know
very familiar. Um you know that there are real challenges with evaluating scholarship across
the humanities. Um she mentioned some and I can add a few more. Books count more than articles. Citation
citation counts are not reliable usually. They don't even count citations in books only
in articles. Um there's certainly a bias towards science technology engineering and mathematics.
Um funding is much less uh there's less funding available in the humanities, um and
one particular challenge here at Virginia Tech at land grant university but this is true of a lot of
other universities: how do you balance um research that that's disseminated within a scholarly field
um as opposed to research that's disseminated to the public, um you know how do you - how do
you balance those two things? Um i would say we do a better job, um at evaluating scholarship
at the local level uh within the department you know. I think that that we have a pretty good
um uh process for doing that. The challenge comes when you move that out and try and you know make
an argument for the quality of scholarship um across the university or particularly outside the
university in comparison to other universities, and this is clearly the the seductive but
false promise of Academic Analytics, and also global rankings as you heard from from Dr. Gadd.
Um and I must agree with Dr Finkelstein in terms of this idea of you know evaluating
with people rather than to people. Um the case I would make is that you know what we need to do
is actually use the skills of the liberal arts to evaluate research. You know we teach our students
we should model this behavior ourselves um we need to think critically about what we're doing
and how we do it. We need to persuade others um in our writing and our speaking of why this research
matters. Um we need to think about the context you know in which we're doing this research and
how we're evaluating it, um and how do we you know talk about what we're doing in a way, in a language,
that makes sense to others, um you know, and these are things we do you know in our teaching. We need
to think about how we do those um in our research, but the challenge is to do that you know at each
successful - at each successive and expanded level. Thanks um would anybody else like to address that?
Uh yes I can I can go um I just think just briefly a couple of comments on
what resonated most from the talk. I - there was a an emphasis on towards the end of putting
researchers and their well-beings at the center, and there was one point that was raised that I found
really interesting, was around potential over past performance, and I know that's a very difficult
thing to measure, and to get into, but I appreciated that point because uh as somebody who's like
a first generation uh university student, you know, my parents didn't even finish high school.
I know like from the beginning I always felt like I was scrambling to keep up with those and
you know even myself now being faculty, and having children, and seeing how much easier it is for them.
This notion of, I think the quote by Adam Grant, was "how far you have climbed" uh to recognize that
rather than simply um conventional metrics that people who kind of are starting ahead already
have quite an advantage so I appreciated that point, and then something else that um
that spoke to me quite a bit was this question of the problem of incentivization. I know as an
early kind of career researcher pre-tenure and even somebody just came off the the job search
you're often told you've got to publish publish publish and then that can impact how you move
through the peer review, how you're responding to the criticisms of your work, and you - it can
end up shaping kind of the methods you use and the arguments you're willing to put forward and
not because you're just racing to publish, so yeah I appreciated those points, yeah thank you.
Perhaps we can go on to our next question. Can I add a little bit uh something Tyler, so um
one thing that I would like to bring about the whole publication issue to consideration, and it's
true in the field of science, is as the graph was perfectly uh accurate in the sense that everything
really goes to to journals in our in our field and and that has to do a lot with the fact that um
by the time that books get usually published the technology that we discussed, there is usually
not leading technology anymore, so publishing we we know that, I mean it's different in other fields,
especially in humanities, but in in our field that that's the that's a conundrum right? I mean it's
just you know, you, if you accept to publish a book then it's very likely that whatever you
put in there, it will be really not obsolete but it will be definitely not cutting edge anymore.
By the by the time that it gets published so it's really discouraged in some way. Plus you put a lot
of effort in in in publishing in a book. The other thing that I just want to bring to consideration
is what bitcoin says is true. I mean you need to publish publish publish right? But then
that brings the other issue of the predatory journals that populate at least in my field
and everywhere it's like suddenly you get five six emails per day of offerings to publishing whatever
research you have as long as you pay of course. Pretty soon so that will increase your number
and then that would probably increase your either citations or you know
um that's another problem the other the third problem about the publication is true
that we we're usually encouraged to publish in high profile journals and we all want to
do that to be honest. It would be unfair to say the opposite but the the reality is also that
the number of papers published in our field in high-profile journals
is is reduced because it costs a lot of money to publish research that will go to one of those
journals. I mean the amount of research that is needed for publishing in a top tier journal is
unbelievable. Hundreds, thousands of dollars and you also need a lot of people working on those
papers, so there is a there's a conflicting thing here right? I mean it's just that it's probably um
you know due to different is it's different in different disciplines, but I just want to
bring all this into consideration because that's the reason because you see some of those charts.
Let's move on to our next question for our panel
So who or what do you perceive as having power to define and implement value systems around
research activities? Anybody like to start out? Yeah I'll get us started on that I think the
certainly was a really amazing and incisive presentation, so just want to echo my appreciation
for what we were able to hear this morning or this afternoon depending on where you are.
So the the control in one sense as one panelist said they're at the local very
small departmental level for example program level uh disciplinary level there's a lot of
ability to be more granular and flexible in theory at least in evaluating work
um but it doesn't necessarily mean that we can always or even usually get the dynamism that
we might need in order to address the kinds of of aims and important objectives that we heard
outlined this morning. Also disciplinary departments tend to be very traditional
in looking at what qualifies for tenure and promotion, and and it's good that departments
can define that themselves so they do have the power and the incentive to say this is what
we're looking for if we have a hiring committee. For example this is how we're going to define
the the excellent candidate or if it's tenured promotion this is how we're going to define
the fulfillment of needing research excellence in our department or in our in our field or subfield
and the pure, a 10-year promotion review process, is often typically shaped by sending those
criteria that a department gets to define to peer reviewers or evaluators who are then looking at
a dossier and trying to judge it and also doing so with a larger field of view so I'll give one
example of some of the challenges here Virginia Tech is part of the public interest technology
university network, and this university network is fairly new, as three years old, it is focusing on
encouraging technology talent, and that's broadly defined so that's not just STEM,
that's that's comprehensive you know it's social science, arts, humanities, creatives,
as well as math and technology, engineering, but getting technology talent to support
positive impact of technology on a human society through comprehensive approaches and
one of the things that that university network is trying to address is precisely this problem.
So this public good or public interest work is transdisciplinary, it means that people who are
in name your discipline have to collaborate with people who are not going to be in their department -
who are not going to be in their discipline to tackle the big challenges that we love to say.
Our disciplinary work is urgent and invaluable for tackling,
but if you are pre-tenure and you're doing this public interest technology work, you are putting
yourself at risk of not being legible to the people in your department or in your discipline
who are going to judge your work to see how you're presenting and publishing and disseminating your
research, and so what we're trying to do now is to figure out how to incorporate some of the
aims and objectives that we saw beautifully articulated this morning in the presentation
in order to to move the needle somewhat on that review process, so it is definitely a lot of
power at the level of of departments that get to say this is what counts for promotion or tenure.
Uh it's definitely on hiring committees. It's not the only place that exists
but those are two places that exert a lot of influence.
And I'm muted of course. Thank you Sylvester. Anybody else like to address this topic?
Um yeah I would love to have the opportunity to address it um I think um I I wanted to jump in on
this question because it has a word in there that i think is very significant here and that's um
power um, and I appreciate many of Sylvester's kind of comments and like
I think in this discussion around um you know how we value research and kind of questions of metrics,
I think it's important that we move the discussion also away from a purely technocratic discussion of
how do we come up with better techniques and standards and norms and think very critically
and seriously about the role that power plays in shaping how things are evaluated, because
there was a great question in the Q&A about well how so far can universities address this
and this question of anti-democratic workplace comes up now. This has not been my experience at
say Virginia Tech. Folks like Sylvester and Tom have been so helpful in supporting kind of our
research initiatives, but when you think about when you're submitting something for publication or a
grant application you're often told you know uh you need to de-radicalize your critique of
systems of power, right? like you you need to couch this in a different set of languages,
and if you want to get into the high impact journals, there is a certain set of questions
you can and you cannot ask right? And so there are entrenched systems of power that have a set
of interests in maintaining these systems as well, so it's not a simply a technical question, let's
say, but a question on how do you contest uh these existing uh systems of power, right? And I know like
as an example, one thing that's very challenging is if suppose one is doing
uh research on something like uh you know the relationship of military industries to research
right? And how far you can contest um the role that militarism plays in the united states, and
in shaping social science research, like how do you apply for a grant where can you apply and so that
often will read out or it will be biased against research that really challenges at a fundamental
level uh systems of power, right? So I think that's one thing that I have found very challenging.
Uh moving through kind of institutions is because as a graduate student you often start with a
really - and the humanities and social sciences are supposed to take a very critical stance towards
power - speak truth back to power, right? tut the kind of institutional arrangements of evaluation work
against that in many ways, and that's something - that's why you often find grad student conferences
will often be asking some of the more critical questions, because they have not yet entered
onto the job market or the pre-tenure path so just kind of pointing that out as a challenge and maybe
something to keep in mind in the discussion, actually, what you just
post in your field also happens in the field of science maybe is is hidden in a different way.
For example, when you try to publish a paper that's supposed to shake the dogma, right? I mean it's like
you you have this fantastic research out there and it's so there's some kind of well-established
you know um way of things, the uh way of mechanisms that they work, and you just have something that
completely challenge that - that dogma and then of course you send it to a top journal for
publication, and it's like yeah this is too risky, or this is this is too revolutionary for the field.
That's something that I got once, and as like you and uh publishing in the paper, that ended up being
a lower tier just to be sure that is published, and peer review just because even the top journals,
they don't they don't want to take that risk. Okay, and in fact they probably ask you for way
much more evidences than any other standard paper, so those level of powers exist in all disciplines.
Right maybe they are hidden in a different way, in fact if you go back to the history of the most
relevant work that was awarded with whatever metrics you use for a word of top research,
call it Nobel Prize or whatever those are usually published in in in journals that are not top
journals simply because they were very afraid of you know later retractions and things like that.
Okay let's move on to our next question.
How do or don't existing reward and evaluation systems recognize and value different types
of research? This could be research in your fields or across fields that you're aware of.
So I'll give a couple of examples. One I just put into the chat is digital humanities scholarship.
Uh this is a huge challenge for digital humanists who face a very
inflexible very narrow way of interpreting what counts as public scholarship, and so if
it's a digital project that's just it's usually a situation where there's tremendous resistance
if not outright dismissal that doesn't count for tenure. You can wait seven or eight more
years once you get tenure, then you can do that stuff. Another example: one of the things that
we faced here at Virginia Tech is the Destination Areas uh trying to promote transdisciplinary work
and and dealing with what is typically for departments. This is not unique to Virginia Tech.
It's just something we dealt with here and that is a narrow reading of of what counts if you're
a sociologist or a religion and culture person or, whatever the department may be on a Destination
Area team or if your Destination Area higher, there has been tremendous pressure to just focus in on
when you're disappointed. You do have tinder, then you can go explore that
other stuff, so those are a couple of examples that we need to figure out.
Tyler if I can follow up on that.
Yes go ahead. Okay sorry, it's the world we live in there's there's the Q&A screen. There's the
chat screen, and then there's the you know other screen. Um and very much appreciate all of the
hundred and so people out there listening to this um yeah I mean I think that
you know the question is how does the - how is the field changing at the same
time that the evaluation is happening? Um you know, and I can certainly think of examples.
Um following up on on what Sylvester just said, uh but I can think of contrary examples as well from
Virginia Tech and elsewhere where you know the hiring process, the promotion and tenure process,
the institutional rewards, um do in fact recognize the value of interdisciplinary scholarship
of alternative forms of publication, um you know to take the example of the digital humanities.
The potential there um you know to not only to reach a broader audience than a journal article or
or a monograph um but you know really to to connect in some cases with um you know a public
audience that cares deeply about the topic um you know, and - and - I you know I can think of ways in
which that exactly that um evaluation and reward system is is in fact in in in place um I have to
admit Lizzie, I'm trying to figure out if I know who that guy in Nebraska is that's maintaining
that that corner of the digital infrastructure, um because it does raise a different question
and and this follows up a little bit uh carla with what you were talking about um
which is how do you evaluate and and reward scholarship that lasts you know, that makes
a long-term significance, has a long-term significance, um and I certainly appreciate
you know that this is different across fields um in a manner's field like history
you know you again most of the value is associated with the publication of a book um
You know, and so if you publish a monograph most of the market is going to be libraries.
Um the book will stay in the library uh hopefully 10 years from now, 20 years from now, 30 years from
now, you know someone could go into that library and either take the book off the shelf or
summon it from the storage facility um and have a copy you know. That's a long-term impact
that's certainly very different from a blog or an Open Access journal that you know may or may not
be available in five years, ten years, not to mention the fact that we may or may not have
the technology to actually read whatever format that that exists in um you know so so these are
the kinds of questions I could think to go back to what um Dr. Gadd was saying, you know, we really
have to think about what do we value um and a part of what we value we and we can valuable things you
know we can value short term. How do you get this to engage a public audience but we can and should
also value you know how is this going to continue to matter in the field um looking ahead? Thanks.
Great, okay, let's move on to our next question. We've got a few more if we can get to them.
So if we all could start from scratch, what recommendations would you have
for creating value-led evaluation of and rewards for research activities?
And that could be at Virginia Tech, in your field, by funders, or other groups.
So at the risk of exacerbation problems it's it's I'm glad we have this question because it's not
enough to talk about the challenges, but also to imagine some solutions. Uh I think I think we need
to have room for as as long as we have tenure and promotion. I think we need to have room for
evaluation of that - that is not just a department level also back to the example of Destination
Area hires. If - if there's at any institution an effort to recruit people to be part of something
that by its very nature can't reside inside of a department that moves beyond it, then I think it - it
would actually make a lot of sense to include the room for the evaluation of that person's promotion
or tenure or their research to to be by a committee that is itself transdisciplinary
uh because it will address what is typically the big hurdle of dealing with
the the discipline itself as a primary way of defining what the research outcome should be.
I think we have to address the level of training, the future of research talent, so that means that
I think we need to also include room if we could start from scratch of making the capstone project
For a PhD, something that doesn't always have to be in in humanities disciplines, for example, doesn't
always have to be a long-form booklink manuscript, so the example of digital humanities project is
a perfect example or maybe someone is studying um let's say gender or religion or race and what they
really want to do is to address this at a societal level through organizing some kind of
equitable outcomes so could they could, they have a capstone project that is focusing on having some
uh social entrepreneurship organization where they have to put together.
The business plan for the public impact models the collaborations that they would have uh since
they want to have that kind of public effect as part of their their scholarship so if we could
actually build in different ways of disseminating research into the way that we credential people,
and the way that we train the future of talent, I think that would need to be an essential part
of whatever this new model would be if we were going to start from scratch.
Yeah I agree I think we have the same issue in science. I mean nowadays we're doing a lot of
interdisciplinary work, and even trans-disciplinary work and when it comes to the evaluation, the
person is evaluated in one department by their peers that they don't necessarily
do that kind of research, or they don't apply to the same kind of funding agencies, because
I'm not supporting that particular transdisciplinary research.
So then it comes to what is the weight that that uh transdisciplinary evaluation from other peers
is on the final evaluation for promotion. For example, but I totally agree. I mean nowadays I
personally consider departments like a placeholder. I mean as scientists we are scientists, we use
all the all the resources around to answer one particular question so is is reduction is to think
that just one single uh um expertise from one department can really fairly judge the work that
someone is - someone else is doing in their field when is totally transdisciplinary. I agree with you.
All right so let's go to our very last question. We've got about six minutes or so left for our
panel before general questions at the end. So this question is for panelists who have been involved
in the valuation side of recruitment promotion and tenure processes. So which of the concepts
tools or approaches presented by Dr. Gadd might
- might you consider for your own units or your own institutions promotion and tenure process.
I can address that quickly I - I think I really like this idea of
evaluating what we value um rather than than what counts um you know, and I think this is a um
an argument again that can be made at different levels. Um it has to be made to
different audiences, um and it was it was implicit in Dr. Gadd's presentation but I appreciate Dr. Gill
bringing it out more explicitly, um, which is to make the the kind of power dynamics more visible.
Um you know, as we go through the process, um and look for ways to to recognize that
in each of these kind of stages of evaluation, thank you.
Yeah I'll briefly jump in as well. I think the the suggestions to start with questions such
as whom do we exclude, what's the cost benefit analysis, and how does this individual demonstrate
hurdles, overcome or barriers that they've been able to move beyond as opposed to just
looking for highest peaks those are I think very important uh equity focused approaches
that are especially essential to addressing the the issues of power, uh the the lack of dynamism
in shaping the standards for evaluating excellence, and and trying to achieve an outcome that's just
going to be more inclusive. It's going to have more talent and different kinds of talent.
Yeah it's a - it's really a complicated question, because really our field is very driven by what
you publish, and the resources that you find to - to actually fund that research, but there's certainly
a part that is tremendously missing that is which one is the societal - societal benefit
benefit of all our research. I mean, I think society is the ultimate beneficiary of our work,
and that is definitely not evaluated in any way in any promotion or recruitment in our field.
I have not been involved in the evaluation side of P&T or anything like that. My experience
more is being evaluated, but I think uh just the one thing that was in Dr. Gadd's
presentation, and it builds on something Dr. Johnson was just mentioning. Uh I think
maybe a consideration of the different research communities that people are coming out of as
well so like, you know when many people will move through graduate studies and beyond, we all find
research communities that are asking particular questions are centered on certain
questions. Again, very often in terms of thinking about how particular systems of power negatively
impact certain constituencies or communities, so we locate ourselves in research communities
and these are our peers to which we look for kind of reading our papers and getting back to
us, and so those, but those communities are often not involved in evaluating
our uh research when it comes to going up for 10-year um or at an institutional level, so I think
some sort of collaborative kind of process through which the holders of power within kind
of departments or more broadly are cognizant of - in recognizing the variety of research
communities that individual faculty are coming out of and be being kind of made to be aware of
let's say not standards of evaluation but what kind of counts and matters in those specific
research communities. Great okay, thank you very much. So I think we're going to move into a period
of general questions uh to our panelists as well as our keynote speaker, and Rachel Miles is going
to take over that part, but I'd like to thank all of our panelists for your participation thus far.
Great responses and great insights. Thank you very much for sharing. So Rachel, if
you'd like to take it over. Yeah uh thank you and let me stop share on this really quick.
Um - and we have some questions for panelists as well, so uh let's start out with uh one of the
ones we didn't get to before. So this will be for Dr. Gadd. Um so how to balance the need to perform
um to some KPIs for national funding versus the more supporting way of evaluating researchers?
Um so I think that's just um kind of a how do you balance that?
It's a difficult balancing act, um I think the way that we do it at Loughborough is to
make sure that we - honestly this depends where you are in the world, because not everywhere in
the world has national kind of funding schemes. Um but in the UK we tend to - um Loughborough at
least - try to align ourselves with the researchers and see the national kind of funding dictats as
the necessary evil, and not the other way around, so we're not kind of focusing on the national funding
um requirements and seeing our researchers as the necessary evil to get that funding, and I
think that's that sounds a bit silly, but I think there are institutions where that is upside down.
Um I think we do. It comes back to this - putting researchers at the heart of everything.
You do as kind of research policy - focus senior managers in institutions. You have to absolutely
know where your priorities lie, and that's with your - your people - with your researchers, and yes
we have to play these games. We have to um kind of submit to national funding exercises etc. We
have to to stay alive, um, but our priority, our alignment is with our academic colleagues and
I think we just - you just need to make that very, very clear and consistent in all of your messaging.
Okay thank you Dr. Gadd. Um let's see. Next one: "Dr. Finkelstein said we all want to publish in
high profile journals, to be honest. Is this just because of the impact factor or some other reason?"
And if somebody else wants to address this as well, that's also fine. Well multiple reasons. One
is certainly a reason of prestige. It's really good research that has been published in those
journals, and you know it's certainly a good recognition by your peers to get published
in one of those journals. Also, it really impacts your funding when you submit your grants for um
- for funding. The fact that they see that they are publishing your research is
being published in - in those high-profile journals certainly impacts the reviewers.
Um, I think that it comes from that end. Um also we do see that on evaluations
in - in - in my field. I mean Stack is not the same, you know, the same to - to publish in, and
journalists that they have very low impact factor and very many articles; they're having one
published in a well-recognized journal. It's really a combination of all of them. Sometimes -
sometimes uh for example, we pick not to publish in a high impact journal, because we know that
it takes really long time to publish, and we decide, 'okay instead of going here, we go here.'
But we get our research published faster. That sometimes is it's a game that you need to play,
and it's unfortunately to be honest with you - um - and you know, sometimes you have
journals that because of the interdisciplinary nature of your research, even if the findings
are super interesting, they might not understand that it might serve different communities, and
it might not be the right journal to send. So I mean it's a number of factors that play.
I would say probably the most relevant is - is truly prestige, at least in our field.
Does anybody else want to address that question. If not we can just move on.
Okay no one else wants to jump in. I'll just say a few things. Um I think we do need to really
understand what we value about journals better. We - it's become so much more complicated for
researchers. Now we're talking about developing a matrix at Loughborough University to help
academics to choose where to publish. I mean, 10 years ago you - you knew where to publish.
You published in your favorite journal. You knew the editor. You always published there, you
know. Now with - there's so many things to take into consideration, and I think there's - there is a right
- the argument that high impact factor journals don't equal quality is absolutely right and true,
but I - it's interesting that when DORA, the Declaration of Research Assessment,
uh wanted to communicate, uh the declaration, they did so in Nature, and when the top factor
I wanted to - um - discipline information about this alternative, so that's a transparency and openness
kind of factor for journals that could challenge the impact factor. They published in science, so
they're publishing in these high impact journals to criticize high impact journals, but the reason
they're doing that is not because they think that they're higher quality of the work that's submitted
to those journals - is higher quality - but they get higher visibility, and of course then you'll
get into this catch-22 that, you know, I've got to publish there, because it's visible. Um but then
you know, if people don't - if people stopped um submitting to those journals and reading those
journals and citing those journals, they would be less visible, and therefore the chain would
be broken. Um so we kind of, I think we need to look elsewhere than to kind of criticize
academics for publishing in high impact journals. We need to break that chain, but we, but we can't
rely on individual academics to do it. It's just not - not fair or feasible. Yeah, this the - the truth
is also the citation increases, right? I think the citation of your article is - is also more likely
to increase if you publish in one of those high profile journals, that if you pri - publish it in
another one, because it looks - it looks better to cite a juror when you send your next paper.
And it's better to cite a journal that has - that is very prestigious among the the ones that you cite
in your own article. I mean it's just - it's yeah it's a ball really. It's super sad, but it's true.
Okay, let's go to the next question. So uh this is a question for Dr. Gadd: "If we are to measure
what matters within our institutions, what is it specifically that faculty leadership is determined
needs to matter for achieving promotion and tenure? Are productivity and impact not critical?"
Okay there's two questions there, um, I think the first part of the question, which is you know what
is it specifically that faculty leadership, uh, have determined needs to matter is up to faculty
leadership to respond to. Hence the SCOPE model you - you need to decide as an evaluator what it
is that matters to you about the thing that you're evaluating and I can say what I think matters to
scholarship, uh give my opinion on that, but as some of the panelists um mentioned we need to
do this at a local level. It's much more sensibly done there, but in terms of our productivity and
impact not critical, yes they are critical, but how do we how - how do you articulate what productivity
and impact are, and how do you measure what those things are, and do we just measure those things. So
the Wellcome Trust report, um, that came out recently on research culture was very much focused on um
looking at how we do our research, not what research is done, um, and that is just so much
more important. How are we working collegiately or are we providing a fair - uh - electrical environment
for um scholars to do their best work, you know? Or are they working under enormous pressures and
um publishing what they shouldn't be publishing? And also all sorts of perverse incentives that
the current system is kind of perpetuating. So yes productivity and impact are critical. How do you
define them? How do you measure them? And are we just measuring those things? - is what I would say.
Rachel, you're muted, yeah. I was muted, okay, let's see here - um there was a follow-up question for
Dr. Finkelstein. "You noted that the prestige of a journal matters for deciding where to publish.
Why is it a problem to consider journal impact during evaluation of a reach
of a researcher's body of research?" I don't think there's a problem
on considering that. I think the problem is that when you give too much weight to that single
parameter for evaluation - I - you know - it's just it seems - it seems to be um overweighting any other
form of evaluation. I mean, it's okay, it should be there. It should be included, um, but it shouldn't be
the sole factor, or the - or - or should it be the one of the top two main factors for evaluating anybody.
I mean there are other contributions that they are equally relevant, and they are not being taken into
consideration, so which goes back to the power of - you know - what is important. Okay, let's see, we have -
"Isn't a major challenge the limited amount of time available for a) the evaluation process, and
b) the time of managers to absorb information."
Might be more for Dr. Gadd, since you're in management.
Yes, but the decisions that we're making, using this data, are very important decisions, and you know,
a good evaluation process does take time and money, and you have to factor that in if you care about
the decisions that you're making on the back of that evaluation. I think sometimes we evaluate it
too much. We love our data. We slice it this way and that way. We want to see what it can tell us.
Um rather than kind of taking a kind of more - um - you know hypothesis-based approach to - to what we
want to actually discover from this data - and that - can that can um take less time if you're - you're
very, very careful, and about what you actually do evaluate, um but yeah, it is time consuming in terms
of um time taken for uh managers to interpret the data. Yes, I mean, that is just - that's the art of the
policy maker- to kind of communicate what needs to be communicated in an efficient and effective
way, but um I - I think it can be done. Okay we're almost at time, but uh we only have two
questions left, possibly. I have a couple from our panelists, but I don't know if we'll get to them.
So: "academic journals tend to be read by academics. Is that the stakeholder group we should be serving?
If we should be serving other stakeholders, what metrics might we value?"
I can respond to that, because I also want to talk about the time question.
We have to be very attentive to the amount of time we're asking for those who are being evaluated, and
it's - it's easy to say: 'do this and this and this and this' without recognizing the cost and benefit.
But you know in terms of this question, it may be appropriate, you know, in working with those being
evaluated to say um you know be selective in what you're doing, and maybe instead of
two more journal articles, you know, you think about an Open Access publication or something
aimed at a public audience that then brings attention - more attention to your work. Um
and I appreciate that that is also a saying: 'do more,' but there may be ways to say 'do less'
in certain areas and 'do more' in others, and that will um support your evaluation throughout the
process, as long as it's clear that it's with the person being evaluated, not about them or to them.
Thanks, okay, we're at time but I'll try to get this last question, and "is there a chance that changing
the system will disenfranchise a different set of scholars? How can we mitigate against that impact?"
Yeah, I just want to emphasize - uh - what the panel is - is suggesting in different ways is actually
having a more dynamic and inclusive approach to assessing and determining what counts, so
that's not disenfranchising anyone. It's - it's creating more space. It's like uh well, I won't
use that example, because I don't want this to sound political, but it's - we're not arguing
for excluding anybody. What we're arguing is that we already have a narrow system that is inflexible,
that is largely arbitrary, and that we've inherited, and there's some simple steps that we can take
to create more space to include different ways, of a set of determining what counts. And so if we
do that, then we're not disenfranchised. If we're just going to say - I think one of the questions
assumed that we were against counting publications, so that would be an example of disenfranchising. No
one - no one on this panel is claiming that, but - but if you took that approach, then obviously you would
disenfranchise people who published, but - but no one is suggesting that what we're saying is that.
It makes so much sense to - to have a much more comprehensive approach to determining what -
what counts as a publication for 10 year promotion, if we're assessing talent.
Um okay. Can I - okay - just very briefly I think - um - to build on Dr. Johnson's comments uh around um -
yeah not - not necessarily excluding any - necessarily existing metrics - or like - uh - but to say that not
only publications count, but I think something also to consider alongside that - that maybe we haven't
discussed yet, and I'm curious maybe I should have brought this up sooner, but I think - um - you know
something like the structure of public funding for universities - I think is really important to
a discussion like this, because the - you know - the creeping and growing and long-standing
commodification of education, you know, when you're hired into an institution, you join, the first thing
that is taken for granted and assumed as natural is that universities are facing a funding squeeze,
a funding cut. There's always this question of - we have these problems, and so go out and get
that funding, and it's often private funders or funding that compromises your research, rather
than asking this question: 'how can we advocate for more robust and sustained public funding
that can allow for more independent approaches to evaluation?' Because a lot of the journals
who are they being funded by? Who sits on the boards? And are they on the board because they've
established a certain reputation - do - resulting to how they can draw on particular funders?
So and who are the owners of the journals, right? These are some political economy questions, like
who is actually concretely owning and managing these journals? And a lot of these issues can be
addressed by advocating for and questioning why we accept this ongoing structural decline in public
funding that is important for autonomous research and autonomous evaluation for public universities.
Those are some really excellent points. Um we do have a question from one of the panelists. Um
do you all want to stay on two more minutes, or do we wanna -
See a couple people nodding. If you have to go, panelists, just give a wave and I'll totally
understand. Otherwise I wanted to ask this last question, uh from the - from one of the panelists,
um since she's involved in open education. Uh "where do highly impactful teaching materials,
especially open educational resources, fit into recruitment, promotion, and tenure?"
Should I kick off on that one sure I- I- I think you know if you value them, you recognize them.
You know, that's - that's - that's the importance of um starting with what you value, if you really value
highly impactful teaching materials, um, then you should um specify that in your RPT criteria.
whatever it is, your - your framework you're - you're working on, uh make it clear how that's going to
be assessed. You know at Loughborough, this is where altmetrics are useful, because um you can start
looking at where, for example, textbooks are um made available on reading lists around the world, and
that's quite an interesting indicator, and quite an easy one to to supply. Other teaching materials, a
little bit more difficult to measure, but that's one way of doing that. Any other panelists? Um,
okay, all right, well I know we went over by five minutes, and, uh, I apologize for that but I want
to respect everybody's time for the rest of the day morning, afternoon, wherever you are, and thank
all the panelists very much for giving us their time today and speaking honestly about, um, you know,
their experiences and so forth, and also thanking Dr. Gadd for giving the keynote. That was really
enjoyable, and we had a lot of people, so thank you everyone, and hope you all have a great rest
of your day. Thank you very much for the invitation. Yeah, yes, thanks, Rachel. Thanks all. Thank you.