So Okay, what do I actually mean by that?
Let's take one step up. Mhm.
And I'll talk first about one of these two conceptual
changes. The first one of those changes is essentially
that are old model of publication is dead. The
existing journal publication system is a bunch of zombies shambling
around and falling apart. That system of controlling access
and charging for subscription. Just simply will not scale
and will not operate. But if you're in the
business of making content shareable, if you're in the
business of trying to spread it as far as you
possibly can and get it to the right people and
that's a good business to be in because there are
real opportunities to their to be there. And again,
it's a service business. So the second conceptual
change is this issue about filters. Okay, so
now I'm just repeating myself. So again, let
me try and take you through this as an example.
This is the bit that's conceptually a little bit
harder to grasp. But, if you bought into the
whole open access thing already than you'll find up till
now, this is the bit that starts to get
a little bit more uncomfortable. So a peer review
is a filter. The filter, in its traditional
form that we apply prior to publication as a mechanism
for ensuring quality. So called in our published literature,
filters block, that's what they're there for.
That's what they do. But blocks of friction and
we're trying to reduce friction. So that's a problem.
If that block, if that filter is on
the supply side, if I'm trying to prevent and
trying to control something and prevent it getting online,
then I'm causing friction. And we can argue about whether
peer review is effective or not, but that's not
actually the problem we're talking about here because whether peer
review is a good filter, or a bad filter,
it cannot always be the right filter when we're talking
about monolithic publication objects and it certainly can't be the
right filter for everyone. The paper that got rejected
today because the conclusions aren't supported by the data is
the paper I need because I want to understand the
methodology. The paper that you reject tomorrow because the
methodology is flawed is a paper I need for
the data. And maybe I want that data to
build a filter to take out the papers from my
incoming stream that have bad methodology. Maybe I need
the rubbish to help me build better filters to deal
with what I want to filter out, not what
some anonymous group of faceless people behind their desks who
I don't even know who they are making decisions about
what I should see. And every time we take
something away from the web, we break connections and
we lose the scale of potential networks. We reduce
the opportunities to generate impact and we reduce the opportunities
to connect people with ideas and resources that might be
useful to them. So we have fewer opportunities to
build networks. But of course if we throw everything
onto the web, then we've got another problem.
Because how are we going to choose what to look
at and what to pay attention to? So the
answer to that for me is that we need to
actually do our filtering on the demand side. I
need to choose how I configure the filter for
the problem I have for the needs I have right
at this moment in time. Because we need this level
of control and personalization over the incoming information streams because
we can no longer rely on the peer review literature.
Because half the information we need is in Wikipedia
or online in some other form or in a database
or in some other source. And we need to
be able to aggregate all of that information together in
a way that we can control modify for ourselves,
for the needs we have for the moment in which
we're trying to solve a specific problem. Readers want
this control. This is the problem we have with
information overload at the moment is we don't feel we
have that control over understanding what's important to us.
What do I need? Why can't the information seek
me out when I need it? And I would
argue this is the biggest opportunity we actually have ever
had to advance research communications because at the end of
the day, this is what we do. We
critique study, analyze aggregate knowledge and we outsourced it
to commercial enterprises who won't even tell us who
they're bringing in to do the job. So what
can you guys do about it? What can we
do to change how we approach this different systems?
So ask yourself some questions. What barriers are you
creating to the re use of your work? And
it might be research work and it might be scholarly
work of another type. But what barriers are you
putting in place to stopping other people use it?
Where is their friction in the system? Where are
the things that are difficult? How easy is that
? Transfer information from one institutional system to another?
From the HR system to the institutional repository? From
external bibliographies? Sources into either of those? And
where do these systems lack scale again? That sort
of question. If you had to take what you
were doing today and multiplied a hundredfold. Even if
you have the resources with the systems cope, how
much manual copy and pasting are you doing? And
can you really do that if you want to scale
that out to 100 fold? Because if you want
to compete in the research world in two years,
five years, six months next week's time, you've
got to be scaling at that kind of rape.
So share what you can and where you can and
contribute and annotate and comment and be involved in the
building of connections and networks. All of the resources
you create need to be open where they can not
because it's a nice thing to do not because even
we should give taxpayers access to what they're paid for
. But because it's necessary to build your work into
the global network and if your work is not in
the global network, it does not exist. If
it's not in the 1st 10 Google Hits, you
don't exist. How many of you have cracked open
an encyclopedia in the last 10 years? Physical book
. If you gain people, How many of you
haven't used a search engine in the last 24 hours
? We've got one brave person up the back.
You've been offline, right. Uh huh. You
already know this. You already know this is true
. So the bottom line is that if you want
to be a player in the research world, if
you want to be part of a service industry that
actually delivers outcomes to our customers, then we need
to set the default practice to being open whether for
publications, whether for data, whether for ideas,
whether for educational content and other resources. That means
things need to be accessible. Yes. But we
also need to ensure that we have scale that we
lack friction. So it's not just enough to put
things online. We need to ensure that people have
the legal rights and understand they have the legal rights
to use your work for things that you've never thought
of before because asking permission time after time does not
scale. You may think it's okay. Someone comes
and asks you once a week if they can use
your data set, how many datasets do you generate
and if that question gets asked every day or every
hour or every minute or every second, how is
that going to scale? You have to enable not
restrict. We need to worry a lot more about
the technical capacity and the technical usability of the systems
we have. We're not good at standardized data formats
. We're not good at recording where data came from
and where it came from, where it's going to
. We're not good at writing research papers that actually
allow text mining to work efficiently and pull the data
out and aggregate and manage those things for us.
And if we did have those things, there will
be a lot better at finding information. So that
probably feels like a bit of a jump into the
unknown. Um, and I want to pull back
a bit. You don't need to take the whole
leap right off the pier immediately. There is a
thing here about finding a balance between where you are
now and where the world perhaps should be in a
perfect case. But what I want to do is
suggest that balances further to the side of sharing than
you might think. You have. A couple of
examples of that one is again from the perspective of
funders interest. This is an interesting study now,
a couple of years old, done on a natural
experiment where the NIH contracted with Merck to make available
a set of mouse strains that had to do with
study of Alzheimer's and they did this in such a
way that the reached through was very minimal. So
people were basically free to use these mice for research
um and development. And then there were another set
of mice also to do with Alzheimer's, that we're
not in this trance that were not made open,
that were not made available. Now, our classic
industrial and economic theory tells us that as soon as
you gave everything away, it would be impossible to
innovate, impossible to make money, impossible to do
further development and all the economic and development activity would
happen with the mice are still under strong ip restrictions
turns out the opposite is true. The mice that
were given away generated more research papers that generated more
diversity of research. They generated more patents and more
products in the clinic than the mice for which things
were not available. From a funding perspective, the
evidence for this kind of thing is becoming compelling and
very compelling. Human genome project for every dollar spent
in the U. S. Generated 100 and$41
in economic activity. The closed human genome project also
generated economic activity. Also generated more than it costs
. But it generated a whole lot less. If
your concern is creating jobs or building the capacity for
innovation, I can tell you that's one of the
very deep concerns of the NSF NIH and the Office
of Science and Technology Policy. Then you want to
make sure those results are getting out there. But
that's the funder perspective. What about the researcher?
I think this is also a very interesting study from
other people's work where she shows that where a data
set is shared alongside a publication that those publications get
more citations, 70% more citations. So by sharing
your work by making it more usable and reusable by
making it more useful for other people and easier to
use for other people who might want to use that
data to do further analysis, solve other problems.
You get more citations, more citations higher H index
, better chance of that tenure track position. It's
a balance and it's a balance between the realities of
the world we find ourselves in with the incentives and
the tenure and promotion panels that exists and the demands
of funders, taxpayers, patients, innovators and entrepreneurs
to do more out of the research that we do
. And the money we spend at the end of
the day outputs become outcomes when the right person connects
and sometimes making the connection providing the metadata by leaving
the comment can actually be more valuable that our new
resource. So those are also valuable contributions when you
can't share immediately or can't share. Holy by making
the metadata the surrounding information available. At least saying
that the resources are available in some sense when you're
building these networks, if you build open platforms.
And I think it's really interesting to think of institutions
as platforms to support innovation and enable the unexpected uses
and reuses of research and scholarship and supporting communities,
supporting students and researchers and fellows and peas and professors
to build their networks to build out the sets of
connections that may become valuable. That may be critical
in solving some particular problems. Yeah. So some
up I said there were three areas in which I
think as a service industry we need to deliver,
we need to provide scale and connectivity of networks that
drive these kinds of connections that drive the ability to
exploit networks to create the kind of quality of change
, qualitative change in research capacity. They talked about
we need to lower the friction of transfer standardized materials
transfer agreements easy. Get rid of two lawyers in
every single institution in the country. Sorry. The
lawyers in the audience because if they were standardized,
we wouldn't need to argue over the terms of them
every time. Yeah. And we need to think
very seriously about where we put the filters in our
systems. Where do we choose to lock down?
Where we choose to change and tighten down on the
flow of information Because if we do that on the
wrong side of the system, then we also break
connections and lose capacity. So then I said there
were two conceptual changes and the first of those was
this old model, a broadcast and subscription is over
. It is dead and the second is to actually
deal with that again. We need to think very
carefully about where and how we filter and what kinds
of tools and systems we need to be able to
enable them and all of that for me wraps up
in one central principle. One way that I find
is the most effective way of understanding all of these
issues and understanding the fundamental design principles and that is
to think at the scale of the network where you
have it. And this requires thinking in different ways
. It requires realizing that its scale those connections are
probabilistic, not really things you can design. It
requires understanding. But the most interesting thing anyone will
ever do with your work is the thing you never
thought of and realizing that you have to enable them
to figure that out, even though you have no
idea who they are. At the scale of networks
, at the scale of the networks we now have
. We manufacture serendipity through the design and architecture of
the systems we build. It doesn't mean that one
person will do well. It doesn't mean that one
spaces where it will all happen. But when we
design for this at scale, what we can say
is that it will happen somewhere at some time.
And the bottom line is that we can't build a
system on the truths from yesterday. We can't continue
to use the assumptions and systems that arise from a
paper world with physical distribution of information. We could
maintain a systems on on the way the world looks
today for a while, but it won't last.
You build for the future by being ahead of the
game by understanding how the world is changing and being
ahead of that curve at the end of the day
, innovators don't follow markets. Tracker funds are not
the way to beat the market. Innovators build markets
, they create new spaces and understanding that. And
understanding how the network impacts all of our futures is
the way to actually deliver on our obligation customers who
ultimately we serve. Thank you. Does anyone have
any questions? There are some microphones somewhere. What
do you think of the implications that for free,
it's research. Um, so I've seen people doing
freelance researching, um, on a small scale.
I think there's a huge potential for it. It's
not clear to me. Sorry, turn around.
I think it's, I think it's reasonably clear that
for certain types of research, we require less physical
infrastructure. Um, the compelling need to be part
of an institution is decreasing and uh, the major
reasons why some of the people I talked to choose
to be parts of institutions because you need that to
be able to accept funding from particular bodies that you
need it before a publisher will take you seriously.
Um, and also simply that the cash flow systems
don't really support the kind of freelance contract work that
might might work. It's difficult. It's difficult to
put freelancer on a grant and not be laughed at
. That. Said, there are organizations as a
startup in Palo alto science exchange and what it does
is essentially enables people to trade laboratory capacity. So
they're focused on core centers, so by informatics,
cause genomics cause those kinds of things and they enable
people to buy access to those analytical facilities through a
sort of centralist bidding platform. Almost so once you
have that kind of um amazon for research capabilities,
then it's kind of interesting to wonder whether you need
anything at all in terms of in terms of running
a lab and I could give you one specific example
. I'm actually working with someone who runs a virtual
biotech, so he runs biotech company developing new clinical
devices. Um his equipment consists of his back office
and his phone and his laptop and he outsources every
stage of the process. So he came to me
with a specific problem, he's getting some materials made
somewhere on the east coast of the US. They're
going to be shipped to a lab in, sorry
. Whereas collaborators, collaborators of mine work and when
we've used them to make a particular device that's going
to be sent off to what is going to be
sent Japan, I think Japan or Germany, I
don't even know to actually be tested. So people
are doing this and they're doing it in really interesting
spaces and if you combine that with things like the
bio community, who are you doing, showing how
you can do basic genetic testing in your, In
a Porta Cabin in Boston at the moment. You
know you can build a PCR machine for$150,
you can build a gel tank um with pcr and
a gel tank, you can do genetic tests.
Um so people looking at their own snips um in
in May labs and then think about expanding that up
so I don't have the slide. Um JOHn Wilbanks
shows this wonderful slide of what looks like a brand
new lab with a liquid handler and a sequence of
some description and it's a little bit data baby.
But it looks good and then it sort of pans
out and you see it, someone's garage and the
point is not, you can buy these things on
Ebay, you can buy an old capillary electrophoresis sequencing
machine for hundreds, not even thousands these days.
Um, and so the point, the point is
not that it's possible to do it. The point
is that it's cheap if you've got the expertise and
it's relatively easy to do, it's not cutting edge
equipment. It's not, but you know, a
lot of the most interesting innovation doesn't come from the
cutting edge equipment comes from people with doing new new
ideas with cheap tools. So I think, I
think we're going to see a resurgence in it.
I think we're going to see a massive backlash against
it. Um, you combine the capacity for getting
cheap research equipment with the ingenuity of being able to
build stuff for yourself with three D. Printing.
Um with there's a very interesting decade coming up and
you can be sure there will be concerns and regulations
that try and clamp down on it and I'm pretty
sure they will fail in the same way that you
information finds its way out into the spaces Anyway.
One of the interesting things is um someone I was
talking to hear someone in Canada um um NSF have
been actually really quite engaged with the D. O
. Y. D. O. Y. Bio
community and looking at the safety protocols and things like
that. Um and so they're shattered a bunch of
these groups for about 12 months and identified their safety
, the safety protocols and the culture of safe use
of materials in that D. I. Y.
Community was vastly superior to what the NSF actually required
. So sometimes it's interesting to see where the where
the priorities like, but I think it's going to
be, it's going to be interesting. It will
happen, it will obviously happen and is happening in
the spaces where people need less infrastructure unless tooling.
Um the interesting bit is how rapidly that what we
think of as complex tools that only exist inside research
labs are becoming readily available. There's one up here
. The first question is what sort of Yes,
the second. Yeah. Galaxy. They eliminate users
who consistently. Yeah. Uh huh. Um Okay
, so in terms of three d printing, I
don't I don't I I always feel that my imagination
is too limited because I don't think of things that
are cool enough. What I really want, I
have to say is printed electronics. So I want
to be able to I don't want to be stuck
with buying the phone that some manufacturer tells me I
can have. I want I want a dual sim
android smartphone with near field communication and only really need
a five megapixel camera. But I really do want
it to be quite band and that doesn't exist.
Um in fact there are no deal seem uh smartphones
which strikes me as a whole great hold on the
market. Um So I think one of the most
interesting things is gonna be the module authorization of products
and much more choice. I hope much more choice
about the kinds of things you can build then.
Think about applying that scientific equipment and think about how
cheap that can be. But at the moment the
smartest thing I ever did was suggest three D.
Printing little holder so that the tube that didn't quite
fit in the machine would fit better. So what
was your second question you for that? Oh right
, sorry. So, so the advantage, the
advantage, of course of having 300,000 people doing the
whole thing four or five times over was they get
they got quite a lot of data on each,
on each galaxy. Um, and so they did
some. The additional approach was just to use a
statistical approach to look for outliers. So they cluster
the cluster the results. One of the interesting things
about the processes that as you go through the process
of a community growing around these projects and have done
many of these projects. And the pattern is always
very similar. You get groups of people who become
expert in the space and the people who are excellent
sort of top 5% of the community that hangs around
and does things is invariably better than the professionals,
really. If they can become really focused on this
problem, they become really good at it. And
a lot of the story of Galaxies who is actually
that group of really dedicated people starting to notice other
things that are interesting and asking the professional researchers questions
about them, if you talk to the team,
they'll they'll say consistently you got this thing of,
we never would have seen that, but because these
people are focused on it, they know that that
bar is actually really quite odd and you don't see
many of those or that funny green shape is unusual
. Um so they've worked quite hard at structuring communities
so that the community self reinforces and that's probably most
interestingly played out. Another game called Turner. Turner
is quite interesting because what they do is they have
a community of people, this is about RNA folding
. Um, so the question is, can the
, can the community and can people design RNA molecules
that will fold into a particular kind of shape?
Um And this is a design problem that from a
computational perspective we haven't solved any really rational way.
But the really interesting thing about this is they don't
test this in some computational fashion. They get the
community to vote and agree on a particular design.
And then every week they make it, they see
they do an animal strong structure of it and they
see what comes out the other end, what the
structure looks like, at least in terms of the
overall fold. And that community is really interesting because
there are people have taken on the responsibility of training
people to not say silly things. Um and to
and provide the resources and educational materials that tell people
you just don't do this because it never works.
Um You have to bear in mind that this community
started from zero, so they had no prior information
incorporated into this thing beyond age matched T. S
and G's match sees, um so that communities becomes
self reinforcing, it's a community of practice, and
that's a pattern that actually emerges in other places on
the web as well. We effectively build good communities
that are self reinforcing, for which there is some
sort of penalty for bad behavior, which can include
just being sent to Coventry and ignored by the rest
of the community in some cases. Um So it's
it's a, I think the evidence pretty strong now
that the technical systems are okay for a while and
you can do a whole lot of stuff with clustering
, and that's really, it's a really good start
, but actually the community has become far more expert
and can become far more self regulating, um and
that's when you really gain the real power of crowdsourcing
that was. Yes. Yeah. Thanks for a
fascinating how in fact here at the university a lot
of what um what do you say the fact?
No see the value. So yeah we're putting their
product out for a public um. Yeah clean.
They need to respond to this where their promotional intended
. Yes and asking them to spend more dissemination occasions
. Yeah. Yes. It really well for uh
what he what he said. Well obviously the glib
answer is it should be more rewarded. Um But
that's not always straightforward. Um So so where I
start with those conversations is first appealing to self interest
. So it's it's things like the the data availability
tied to citations, things that really immediately speak to
the question of this is actually going to make a
difference potentially to actually your conventional promotions and it's going
to make a difference to your H. Index.
And if that's what people care about and then that's
the game you can play so that that's at one
level. Um I think the other sort of the
other appeal to ego is actually we mostly like having
our work recognized by the wider world. Yeah we
like like having our face on tv. We like
to get, yeah I haven't yet met a researcher
who wasn't at least a bit tickled by getting a
bit of media coverage of one sort or another.
And so so that's one element that plays out in
some cases and then the other is as you start
to explain how you can tell who's looking at your
research that that can offer new opportunities for thinking about
spaces to move into new collaborations, those kinds of
things that if you imagine that you start to look
at, um the information that can tell you who
is who and what kind of people are actually looking
at your research, then that can guide you down
there now where all of that leads ultimately is two
systems measurements. Proxies that tell you about how your
research is being used. And so what I'm arguing
at the other end and I argue this to funders
and a government is that that's what they care about
. They care about research being used, they want
to see evidence of it being used and they're going
to use that to make judgments about strategic decision making
. Um so if there's a kind of carrot out
this way in terms of personal ego and other things
, then there's the hammer coming down from the top
that in the end we have no choice. In
the end we have to engage with the impact agenda
. We have to engage with government and with funders
, um, because if we don't do that,
they will simply decide it's better to build another hospital
. And I think we believe as a community that
while another hospital is critical and important, that actually
there are huge benefits that come out of research and
we have to make that case. And if we
don't make that case, then we deserve to get
the front, we'll get the funding we deserve.
And there's a whole bunch of middle ground stuff around
how you can help institutions to start look at looking
at their strategic priorities. You know, I understand
that you're reporting to the Legislature, that doesn't necessarily
understand the details of how you can talk about your
research and why it's important and why it's high quality
, you know, that's that's difficult and it's not
straight not straight forward to cut through. Um But
at the end of the day if an institution has
to take a view that it sits on its own
and believes in its own strategic priorities because an institution
that doesn't have a faith in its mission and doesn't
have a faith in its own strategic decision making process
, it's not going to succeed either. So there's
lots of different parts of this and it's going to
be a combination of of carrot and stick. But
if you engage with a stick you might find out
it's a big carrot. Put that friend for one
or 2 more. Mhm. Mhm. What?
Okay. What in there? Yeah. Through my
point is is on the open. Yeah. Mhm
. Um So that's a lot of there's a lot
of answers to that question. I mean the first
the first answer I give is that open access per
se the publication of research through an open access mechanism
, be it through a repository or through an open
access journal can be done without any modification to peer
review whatsoever. And we can we adopt and plus
does adopt very conventional straightforward peer review with academic editors
, um referees and all of that very standard process
. Could we do better? Could we do more
efficiently? Yes. Absolutely. And we're going to
have to make some hard choices because this is an
expensive process. We're spending Probably at least 5 to
10 billion globally on peer review through investigating at the
time of people doing the peer review. Um And
we're getting more and more research papers submitted. And
I think we have to ask the question, does
that scale scale out? And I would argue that
it didn't and we know very little about what peer
review is good for and where it works and where
it doesn't. There are very, very few good
studies, there an awful lot of fairly bad studies
um that really dig into the question of where peer
review works and where it doesn't and what it's good
for telling and what um some things it's not good
for telling is whether an article is fraudulent or whether
it's important, peer reviewers are amongst the worst predictors
of the number of citations of paper will receive.
Um I think there is some evidence that shows there's
some emerging evidence and I was I was skeptical about
this. There's a paper in science actually just the
other day, which suggests that peer review does on
average increase the written quality, the readability of papers
, the comprehensive bility of papers. Um so I
think I want to look more closely at that because
I'm still a bit skeptical about that and I think
there's probably some good evidence that at least when you
have very specific technical questions and you ask the right
referee that you can get good a good assessment of
of the technical correctness of an approach of paper.
Now, I think the question I'd like to ask
is should we do that prior to publication or after
publication? And should we put every paper through the
same rigmarole or should we only apply the degree of
attention to certain things that we think are worth the
effort? I think that's a that's a discussion we
should we should have. I think as we move
towards the publication of data, we need to ask
ourselves some really serious questions about whether we can afford
the time, what it even means to peer review
data. Um and I think there are some other
opportunities we can look at, particularly. There's lots
of computational validation that could be done in particular cases
. Um There's lots of information to be garnered from
actually looking at whether people use a piece of research
or not. If no one's using it, then
it's probably not much use. So someone has written
a methods paper and it's never been sighted, a
fairly good sign that it probably wasn't worth publishing.
Um or maybe it was worth publishing but it wasn't
worth the methodology isn't. Yeah, maybe it will
inspire someone to create a better method. Um Hopefully
they would cite the fact they were inspired by create
china's attribution. So I think the question, the
question, the place where I will start is it
seems clear to me that critical review of our knowledge
and critical review of our beliefs is at the core
of what makes research actually function. And I think
the question given that's a really valuable resource, given
that the attention of experts in Michael Nielsen's phrase is
the most valuable resource we have as a community,
we should work very hard at making sure we're applying
it in the right places and at the right times
and to the right thing. Um and I don't
think we have the data yet to really inform those
decisions. Mhm.