We are doubly honored today to have Dr. Leon Osterweil
and Dr. Lori Clarke both from University of Massachusetts
and both distinguished lecturers in our program
eminent folks in the software engineering community. So let
me embarrass them by saying a few things about their
achievements and I'm just cherry picking here. Their bios are
much more impressive and much longer. So, Dr. Clarke is a
current chair of the Department of Computer Science at the
University of Massachusetts. She is a fellow of the
ACM and of the IEEE. Her most recent award
already mentioned one, is uh, the University of Massachusetts
Outstanding Accomplishments in Research and Creativity award given in 2011
And she's won other awards for her work.
Uh she is a board member and former co-chair
of the Computing Research Association's Committee on the Status of
Women and has done a lot of good things for
diversity in the community. And um we definitely honored because actually both
of these individuals are awardees in of a very high award in the SIGSOFT, ACM SIGSOFT community which is the Special Interest Group on Software Engineering for ACM.
They have both won outstanding research awards for their research in software engineering. Leon won it a few years earlier than Lori did. He's older, because he's older.
Let's talk a little bit about Dr. Osterweil, Dr. Osterweil is a professor in
the Department of Computer Science at University of Massachusetts,
and he has also served as dean of the College
of Natural Sciences and Mathematics for four years at Massachusetts
In addition to that, chaired the computer science
departments at UC Irvine and the University of Colorado
in Boulder. Uh and I mentioned, um,
he's uh has has received the outstanding research award from SIGSOFT
often, many other rewards. Both of them are
ACM fellows. Uh, and Dr. Osterweil also
has recently got received SIGSOFT's The Most Influential Educator award
So again, I'm cherry picking among the rewards
I'm very happy to have them visiting with us
today and I think we're going to be interested in
the applications that they're making out of some of their software
engineering research. Thank you, Barbara. I
think one of the things that Lori and I are
most proud about is that yes, we've both won
the outstanding research award, but it's not for work
we've done together. It's for work we've done separately
So each of us has done a body of
work that was recognized separately and individually. But what
we're going to talk about today is something that we
are doing together. And as you can see it
is all about using process modeling and analysis to reduce
errors in healthcare. So many of you may be
aware of some of these statistics, but they're jarring
statistics. Uh, there was a study by the
Institute of Medicine in 1999 that estimated that 100,000 people
die in us hospitals from preventable errors and notice the
word avoidable, preventable is emphasized. So this is
not incorrect diagnoses or a surgeon whose hand slipped or
something like that. This is people getting the wrong
type of blood, People having one cancerous lung
left in while the healthy lung is removed and things
of that sort 100,000 per year in the US.
It is widely believed that this is an underestimate by
about a factor of two. That would make this
one of the leading causes of death in the United
States far more than the number of people who die
from auto accidents and job related injuries and so on
and so forth. This does not count near misses
where people have been maimed perhaps for life or subjected
to a great deal of pain and aggravation and agony
and hundreds of billions of dollars wasted. So this
is an enormous problem. That number basically is told
people and people don't quite exactly internalize it the way
we would like Um, consider one fully loaded Boeing
747 crashing every day of the year, 365 days
. And that's the number of people. And clearly
if there was a 747 that went down every day
killing everybody on board, there would be a tremendous
human cry about the safety issues and what are we
gonna do about it and so on and so forth
. And uh, but that sort of hew and
cry has, has not yet arisen, although the
health care community is now very focused on trying to
do something about this. There was a follow up
study 10 years later in 2000 and nine to see
how they, how things seem to be going.
And there was no evidence that things had improved,
even though there had been 10 years of focus on
this problem and we're now a few years further down
This is a quote from a 2009 National Research Council
report and the thing that is emphasized here inherent intellectual
complexity really starts to get into what it is we
focus on and what it is we think is compelling
for computer scientists. The plain fact is that health
care has become an extremely complicated enterprise these days.
And it involves the interplay of many different people,
different kinds of people. Many devices. There has
been the advent of all kinds of software innovations like
electronic health records and so on. And each of
these is an intricate player in its own right.
The interplay of them has created systems that are far
beyond the capabilities of the people who are participants to
understand. And as a consequence, we now have
doctors who are very well trained in the medical arts
but are not systems people, but they are now
in meshed in these systems and trying to wield them
. So, um, that takes us to the
larger issue which is of interest to us. And
that is the fact that such systems are really what
runs our society these days. So, health care
is a glaring example. And this estimate about 100,000
deaths is extremely compelling. But pretty much every other
large scale system is the same kind of a thing
. A system that involves the participation and interaction of
lots of different kinds of people with increasingly complicated devices
. So one of our other studies is on elections
where we're trying to look into why elections don't work
as well as they should. And we're discovering the
same thing. Too many people doing too many uh
intricately interconnected kinds of things supported by devices which think
they're going to make all the problems go away,
but which somehow seem to just be complicating the issues
. So we call these human intensive systems. They
involve collaborations among people software, hardware devices and there
are some examples flight control, which is well studied
emergency response, how an emergency room works voting devices
, the entire electric power grids. All of these
things are examples of human intensive systems and they all
seem to be at the point where they are taxing
the ability of the human participants to deal with them
. So our approach which we call continuous process improvement
is basically a riff on continuous process improvement as practiced
by industrial engineers and and often in business management so
on. And it says basically if you want to
understand what's actually going on for the purpose of improving
it, making it better, making it safer and
so and so forth. One way you can do
that is to take the processes these systems we think
of as being composites of lots of different processes.
Take these processes, model them take the models and
evaluate the models and here what we're going to be
talking about today is the arsenal of analytic techniques that
we're throwing at these process definitions and after you discover
some defect or some shortcoming, fix the defect and
go back. So this is basically for those of
use our study industrial engineering, the shoe heart cycle
or the deming cycle of continuous process improvement. Notice
however, that we now have a couple of things
we put on the bottom because what's happening here is
a mostly intellectual exercise and taking a model improving the
model, making the model better and so on and
so forth. But the model has not yet engaged
the world. And the goal of course is to
save 100,000 lives a year and that involves engaging the
world. So the next loop is after you get
to the point where the model has been improved sufficiently
you believe then actually deploy these models in the real
context and evaluate them in the real context in this
case, the clinical setting and from the experience in
the clinical setting, this is outside loop. So
we have been mostly engaged in the inner loop and
now we are with our latest research grant moving into
this outer loop where we're intending to actually deploy these
things in the world. Okay, so the focus
of the talk is on the technologies so we are
not doctors, although we're learning a lot about health
care. One of the things we're learning is do
not ever go to an emergency room and of course
if you need an emergency room you sort of have
to go to an emergency room. But the idea
when we ask emergency docs, what do you fear
most? And they say mostly we fear about ever
having to be a patient in our own emergency room
. Uh but mostly we're going to give you the
systems approach and the system approach basically is summarized here
, starting out by uh using a language which was
a home grown language which will describe uh just a
modest amount of detail called little Jill, which is
a process language and is designed to model processes of
these exact kinds with the diversity of different kinds of
performers that are interacting with each other in all kinds
of complicated ways, throwing exceptions and working in peril
and so on so forth. Uh that is to
capture the process. Then we have a tool called
propel which Laurie will be talking about, which is
an exquisitely uh precise tool for capturing exactly what kinds
of behaviors you would like to see. And Laurie
will have some examples of that. Then having captured
the process with little Jill and the decide arata with
propel. Then there is a model checking tool which
is also something that's been built at u mass um
called flavors and we use flavors to determine. Is
it possible to execute this process in such a way
that that the constraints are violated. Um there's another
kind of analysis. We do fault tree analysis which
is different from the model checking. The model checking
basically says here are a set of tasks if they're
done right, will there be incorrect and worrisome sequences
fault tree analysis says supposing some of these tasks were
done wrong. What would be the ramifications? And
in doing this we're hoping to identify for example single
points of failure tests that are done which need to
be right which have some strong possibility being done well
. And then as time permits we may also talk
about a discrete event simulator which we built whose job
is to study the optimal allocation of all of these
diverse resources in order to make these processes run better
. So there is a battery of things that we're
doing taking different looks at these processes. So this
does involve working with medical professionals which is interesting.
Um There's a lot we we will be saying about
this as we go along but these are the three
areas that we have actually applied these technologies to.
First is chemotherapy example, breast cancer, chemotherapy.
We're working with mostly with colleagues at the Bay State
Medical Center which is a very large tertiary care facility
that operates one of the largest emergency rooms in the
U. S. Northeast. Um We've also working
with a professor of nursing Beth Henman on the in
patient blood transfusion process and we have a very well
articulated specification of the process and properties that should adhere
to and then we are doing emergency department patient flow
trying to determine How can you move patients through the
emergency room faster. Where we've discovered is the average
waiting time in an emergency room is anywhere from 6-8
hours. And at first it seems as a very
simple uh problem to fix you just throw more resources
at it and then the emergency room doctor said no
, no, no, you can't continue to throw
resources at it because we will lose money. So
it actually is a complicated optimization problems. So this
is something of a sort of a roadmap of what
we're trying to do. We're trying to apply these
technologies to make big improvements in these areas with the
expectations that what we have learned in the approaches that
we're taking will apply to other processing and other domains
. Indeed. So what we've discovered about these processes
is that they are incredibly complex. So very often
people ask this language which you're about to see little
Jill, you invented this took a lot of effort
, why didn't you use something off the shelf?
And what we discovered is that the languages that exist
on the shelf to define processes really lack the semantic
richness that's needed to really capture what real processes are
like. So here are just some of the descriptors
. The languages the processes themselves are complex. They
tend to be concurrent. They tend to be exception
rich. So when we told our nurse we have
a language that deals with exceptions. Is that something
that's important? She said exceptions are my whole life
. So most processed languages do a poor job of
dealing with exceptions. They need to support human choice
and flexibility tasks may involve multi processing and multitasking.
And the list of the separator for this language goes
on and on and on. At the end you
wind up with a very, very complicated language which
really needs to borrow a lot from programming languages.
And that's why we invented our own. And in
fact the work we've done and applying it has indicated
that the language itself still doesn't have all the stuff
it needs and has led to improvements, complexity.
The languages has a strong semantic basis. It's defined
in terms of finite state machines. And the main
reason for that is if we're gonna do definitive analyses
, we need a precisely defined language Has timing,
resource utilization specifications and so on and so forth.
And then at the end the last one is probably
the hardest. You know, some of you may
be thinking about 10th order logics that could do all
of these things. It needs to be understandable by
a medical professional because at the end we have this
process and we're about to study it and maybe deployed
. Is this your process? Is this what you
do? And medical professionals need to see something visual
. So the language is visual in a moment,
you'll see an example of it. It's centered on
this idea of a step and instead of thinking of
this as being a step think of this as being
a procedure. So the language is in fact a
hierarchical decomposition language with some things added pre and post
conditions interface badge that describes the flow of artifacts through
it. Uh it is scoped language so it has
scoped exception handlers and then it finally has a flow
of control specify here that says the order in which
the sub steps can be executed. Instead of talking
about the picture here is the impatient blood transfusion process
. It's the beginning of a hierarchical decomposition and it
says basically patient blood transfusion process. Uh inpatient is
hidden under this triangle is a sequential execution of a
cleaning plus number of instances of carry out physician order
for transfusion which consists of the sequential execution of check
for the type, prepare the documentation for blood.
Pick up, pick up the blood, do single
unit transfusion cleaning plus number of times, do a
follow through check. And then up here we have
an exception handler and as they said exceptions are very
important. So you notice that two of these steps
have these green triangles and those are preconditions checks the
first one. Check to be sure that this has
been prescribed by a physician. So when you actually
get into this and you listen to people telling you
what it is the process entails. Very often they
leave out these important points until you pull them out
of them. So for example, we always say
since the language as pre and post conditions, is
there a precondition? Well of course the physician needs
to prescribe this blood transfusion. And if not then
this precondition guard kicks you out of this entire process
. There's another one before you carry out the transfusion
confirmed that the patient has filled out a consent.
So this is legally important and it's also pragmatically important
. But in this case and we can see that
this exception gets thrown up to apparent where there is
a matching exception handler and this icon says continue.
Don't do any of this stuff. Don't do the
transfusion without permission. Okay, so I said it's
a hierarchical decomposition language. So just briefly we will
decompose the single unit transfusion process into this. And
you can see how things start to get quite complicated
. So the entire transfusion process that we have is
100 and 50 or 202 150 steps and it gets
down to quite a complex, quite a detailed level
and we stop here. We don't really go too
far. I just wanted to give you a sense
um uh there's a parallel step over here which says
discard transfusion materials and record infusion info. Both have
to happen but the order which they happen is not
important. Uh there's a suspected transfusion reaction process and
this is an example of while the blood transfusion is
going on. The nurse is supposed to be checking
to be sure that everything is going okay. If
not an exception is thrown. And this black bar
means that there's a whole long decomposition of what the
nurse does and artifacts get passed along with this.
So the nurse has information what actually went wrong so
that hopefully the nurse does the right thing. Okay
, so we're actually rather proud of the language.
Some of us are very, very proud of the
language, but it has been pointed out that the
pictures are a little hard to draw, a little
hard to grasp. So Lori and some of the
students a while back said what we really need is
some way of projecting this in english or in some
structured form of english. So there is now a
narrative view of this and this is the narrative rendering
of that process. It's a narrative rendering of a
small part of that process. And on the left
, what you see is a sort of a table
of contents with these live links and you can click
on these links and that will take you to different
parts of the process and what we've discovered is that
some medical professionals really like this view, some medical
professionals really like that view. But all medical professionals
like the fact that there are both and of course
since this is automatically generated, we know that there
is coherence between this textual view and that pictorial view
. Okay, so I'm gonna finish up talking about
the process modeling quickly and turned it over. Glory
to talk about analysis. We've discovered quite a lot
in doing this mostly. Um, one of a
couple of the main points are indicated there first,
the processes are not terribly well understood by the performers
. Since there's so many performers, each performer understands
what it is they do. The doctor knows what
the doctor does and the doctor has expectations about what
a nurse does. And then when you ask the
nurse, the nurse very often doesn't do that.
We had a number of psychodramas between our doctor and
nurse domain experts where the nurse said, okay,
now the doctor does this and the doctor said,
no, no, no, I don't do that
, I do this, this and this, and
then the nurse does this. And then the nurse
popped up and said, no, no, I
don't do that. And then there's the pharmacist and
then there are the people who push the journeys and
each one has a view of what they do and
Presumptions about what other people do. And sometimes those
Presumptions are wrong. So in order to tie these
processes down, there is a tremendous amount of time
spent interviewing and going over and over and over again
eliciting details in order to do that. It's really
important to tie down the terminology. Um there is
the word transfused which is used in connection with blood
and you would think it would be well understood what
transfused is. But sometimes the word infused is used
. Sometimes a distant difference matters. Sometimes it doesn't
, there is a word check, There's a word
verify, there is a word compare and these words
until we started interacting with these folks were used in
a very casual sort of away and it was never
quite clear whether they were different or not. They
are much more precise and crisp and the discourse now
and in fact our professor of nursing says that the
process orientation has caused her to teach nursing differently to
think about it differently and to teach it differently,
which is one of the more gratifying things. Takes
many iterations to define a process. And uh,
we probably actually are involved in an illicit ation of
a new process right now at the University of Massachusetts
Medical Center and we're actually measuring how long it takes
, how many iterations and how many hours it takes
to do this. But it's a lot. So
a bunch of observations. Medical professionals tend to think
in terms of war stories and this actually sort of
causes the whole thing to go longer but more interestingly
. Well, we try to get details of the
process is, they tell us about grotesque things that
have happened. It's a scary to do this.
But these are typically negative scenarios when what we would
like is this sort of the positive, what is
it you are trying to do? Um They're initially
scared by the terminology by the fact that they're dealing
with the systems people but after a while we do
get comfortable with each other but it does take a
while. Um We have not yet found a simple
process. So we are working on a process called
verify I. D. And have been working on
this for a couple of years, verify ID happens
all over the medical domain. And verify ideas.
Just simply do we have the right person here?
Do we have the right blood bag here? And
while you seem like they must be very very simple
things when you think about all the complexities that are
possible, they're emergencies situations that are different situations where
the patient can talk, where the patient can't talk
where the patient can talk but not in english and
on and on and on. So every one of
these processes is amazingly complex. And when you extrapolate
this to the way our world works it's almost a
wonder that anything works because every simple process turns out
to be not simple at all. And that's where
I think one of the features of the language,
the fact that the language has these tags connected to
it. Basically our students are now they know when
they come and ask somebody for information about the process
. They want to know what is the sequence in
which things happen. What are the exceptions, what
are the preconditions where the post conditions and just the
asking of these questions actually causes the solicitation to go
better because the questions are suggested. So there is
no best process. There are only processes that are
good at meeting their requirements or meeting the needs of
the domain experts. And when you interview the domain
experts, we've discovered there are two kinds of questions
you ask, what are you doing and why are
you doing that or what is the goal or what
are you trying to achieve by doing that? And
both of these questions are unfamiliar and difficult, but
on the other hand, what we discover is you've
asked, what do you do? They will leave
out things that come out when you say what are
you trying to do and vice versa. So this
has been an exciting thing. We now have some
processes that I think are really quite remarkable. But
the purpose of doing this was to create a baseline
so that we can then take those processes and study
them and infer things about them, hopefully finding defects
that will cause us to be able to improve them
. And at at this point I'd like to turn
it over the lorry because she's the one that's going
to tell you the story about verification. Mhm.
Uh huh. Mhm. Mhm. Oh. Mhm
. Okay. So after spending a lot of time
creating these models, then there's a couple of questions
that you might ask about these models? So one
is is the model consistent with the real process and
in fact there's a lot of work that we do
in terms of having different individuals look at the look
at the process models asking, giving them scenarios and
then getting to a certain point in saying and then
what would happen or doing shadowing if people wear eye
tracking devices and that's a whole part of this validating
the model that I'm not going to talk about because
I'm going to talk about the other question. And
that is let's assume that the model is a pretty
realistic representation of the process is the process, right
? Does it do what it's supposed to do?
And are there different kinds of errors that could be
in that process? And so that's what I'm going
to talk about is what we can do to try
to find errors in the process model. So we
do a number of different things and we talked about
some of them. The model checking looks to see
does this process model satisfy the requirements as it do
what it's supposed to do, then we do some
what we call safety analysis and that is um if
things aren't done right, where is the process vulnerable
for catastrophes? We also can use the models to
drive simulation and we've also done some work on requirements
generation. So if you're going to have devices within
the process coordinating with the people? What are the
requirements that you might have on that device if it's
going to work within the process or what are the
requirements on the process of? It's going to work
within that device. And these are examples of the
analysis now to support this analysis we basically have an
environment of analysis capabilities and I I hope you can
see the arrow here. So here's this process model
that lee was describing and um so we have the
editor to help you create that model but then we
have a range of analysis tools that's what's in the
turquoise down this this column here that analyzed the process
model and usually with the analysis there might be some
secondary information that we have to provide and then we
get this feedback reports and in terms of a process
improvement environment, the view is that we would get
this feedback, we take that back to the medical
professionals and then there would be an iteration where we
would improve the process and repeat this. So I'm
gonna start talking about some of these analysis techniques in
a little more detail and um leary described one and
that is just creating alternative representations. And he showed
you the narrative view and we found that very often
just looking at the two different views helps discover errors
in in the process. Mhm Whoops here it is
. Okay. But what I want to talk about
now is the model checking. So how many people
here are familiar with model checking? Oh, not
too many. So um let me just give you
a high level overview. And basically what model checking
does um is it takes the representation and creates a
relatively concise model that allows you to reason about all
the possible ways that you could actually execute that model
. So it was originally developed for hardware systems and
then moved to software systems. And in fact both
lee and I and Barb at times have worked on
um developing these techniques where you can look at all
the possible, execute double paths through the program and
determine if they meet the requirements. And so there's
a whole lot of work that's done in terms of
how you can recognize equivalence classes and how you can
keep this so it can be optimized and unlike Regular
, the're improving where you can make very strong statements
about what you want to prove, you're a little
more restricted in what you can prove. But if
you can show that it's satisfied on all paths,
you've actually proved it. So it's a very strong
, very nice technique. Um But one of the
things that people sometimes forget to mention is that coming
up with what you're going to actually prove is hard
and people get it wrong because since this is we're
doing this in an automated way, you have to
say it very, very precisely and we're dealing with
medical professionals. So it's important to be able to
capture that precisely. So one of the projects that
we've worked on is represented here, the model checker
. And I'm not gonna go into the details of
the model checker but I am going to talk about
how do we represent what we want to prove.
And this is a system that we developed called propel
and propel um basically tries to um give people hints
. So there was some work that was done that
by Dwyer of Runnin and Corbett on property patterns.
And this was for verifying software systems and they discovered
that a lot of these requirements of which out when
you think about them in terms of smaller statements,
a collection of properties would be would be the requirements
for a system. If you look at these small
concise statements, a lot of them fall into a
restricted number of patterns. But it turns out that
each one of those patterns is very hard to really
get it right. There's a lot of details.
And what propelled does is it goes through and it
tries to help you figure out what are your options
, what has to be determined And it does this
with three different representations. A question tree, a
natural language, pull down menus or with an extended
view of a finite state automata. I'll give you
an example of what this looks like. So here's
an example of a property. The patient's identification must
be verified prior to transfusing each unit of blood product
. Okay so what we're really talking about are two
events. That's verifying the patient ID. And transfusing
blood and how they're related to each other. So
if we look at the question tree view um it
turns out most properties have primary only one or two
primary events. So the first question after you've sort
of filled in what the parameter names are is are
you dealing with one event or two events? And
in this case we have two events. And if
you're dealing with two events then the next question is
well how do they relate to each other? If
you verify patient ID must you transfused blood and you
know you might decide not to do the transfusion for
some reason. So you would pick the second choice
here that just says you can't transfuse blood until you've
done to verify the patient I. D. And
this is the question tree. And it goes on
asking these questions well what's happening at the same time
? And what corresponds to this? Is having a
finite state automata representation? And these questions really correspond
to where we sort of have these dashed lines.
Those are options we have to decide can you repeat
this. Is there a loop? Is there something
that can happen between verify patient ID and transfused blood
? Um and so on and so forth. So
those questions just correspond to filling out this information here
. This is an example of of having different labels
on a transition. So it's not only is there
a transition if there's a transition, what is the
label? So for example, we can ask do
you have to verify patient ID? And the answer
is no. And that means that's an accepting state
. In terms of the finite state automata and the
finite state automata is going to use the the basis
for doing the model checking the verification. So I'm
showing you the question tree. You could do it
in terms of filling out the templates on the finite
state automata. Or you can use these english language
expressions and here you have a a template and then
you can choose which one you want, you can
select it and it fills it out. And the
thing is when you're all done, you have a
finite state of thomas, a representation verification. And
you have an english description. Okay. So we
were talking this morning about english descriptions and if you
look at this, this is really very detailed and
you wouldn't expect that any uh english description that you
picked up would really provide all this information, you
really have to be driven by knowing that you need
these details in order to do the verification. So
, especially when you're working with domain experts that aren't
in computer science, you really need something to help
them think about all these details. So in terms
of our process and modeling and analysis, we talked
about developing the process model, we also have to
develop these requirements to do the verification. And so
like for the blood transfusion we have about 60 some
properties that we then want to verify and see if
they're true for the process model. So using the
system called flavors, which I'm, I'm not going
to describe, we do that verification. So some
observations about doing this in terms of specifying the properties
, sometimes just specifying the properties, we realized that
there was problems in the process model. They would
say, oh this is very, very important and
then we'd realize that the process didn't have that information
in it and very often it didn't go down to
the level of detail that we needed. So doing
the properties, finding out what are these requirements helped
us determine what should be the granularity of the process
model. What should be the scope, what should
be covered because there's always these questions about what do
you want to talk about? So for example,
you're going to fill out a document. Well,
do you really have to say answer question one,
answer question two. Not unless you're going to reason
about it. You don't need to go down to
that level of detail. But if there was something
that was really important, then you would want to
make sure that was in the process model. Um
The other thing is lee talked about how important exceptions
are. Well that means that the properties also have
to deal with that. If something is supposed to
always be true unless the patient dies or the patient
is unconscious or whatever your exceptions are, then you
have to capture that because if you're going to really
do this verification, it has to be an accurate
statement of what you're doing. The other point here
is that for the medical professionals it was pretty hard
for them to understand the difference between um having a
particular scenario and he was talking about negative scenarios,
these war stories. But even, you know,
having a particular scenario that talked about what happened versus
having a process model that described everything that was supposed
to happen. That's like a program where you talk
about all the cases versus a requirement or a property
that's really talking about what's supposed to be true no
matter how you actually implement or represent the process.
And this is something as computer scientists that we deal
with all the time. But for people who are
outside of computer science, these are hard concepts and
I don't know if people have been following the kinds
of what people again this afternoon Jeanette wing have been
talking about computational thinking and how that has so much
influence in other areas. But this is just really
a perfect example of where we're taking things that we
do in computer science that we learn when we learn
programming and software development and how it applies to other
domains. Um I won't go into the to the
model checking except to just say here's this high level
view of what it looks like. Um So we
have now we have the property that we want to
verify that we created using propel we have the process
definition that was from the process editor. Then we
do that. Um the model checking, finance state
verification which really basically says is there any way I
could execute this process that didn't satisfy the property And
if so then you see the counter example and in
fact in doing this in creating the models we found
a lot of errors. Um Now first what happens
is we tend to find errors in the model or
in the properties you know we've made mistakes we didn't
think of you know we didn't think about particular issues
but after we clean that up and we actually found
a really important errors in the in the processes themselves
. For example we found deadlock. Now we never
expected to find a deadlock example in processes with people
where people would really get stuck and when we found
it we went to them and we thought of this
we must have made a mistake somewhere but they when
they showed them the counter example and in fact they
said yeah that in fact does happen. So the
situation was a patient would come in and depending how
they entered the hospital but if they entered in a
particular way there'd be a request for their blood and
the blood bank would be waiting for some additional information
and the nurse would be waiting for the blood and
after a while someone would notice that this poor patient
has been lying there for a long time and someone
pick up the phone and break the deadlock. Now
the processes were complicated enough that even though they knew
this happened, they didn't quite know how to fix
it. And in fact the first fix that they
proposed when we could show them, the counter example
ended up breaking other properties. And that's because when
you're dealing with concurrency and exceptions, things get really
complicated and it's hard for someone to really see all
the implications just like with the program. Um We
had other examples. Another example was mm a missed
event where it was possible for someone to go through
and not really have there and wait updated. And
when you're doing chemotherapy, the drug is really based
on height and weight and so if you don't have
that up to date, that's a really big problem
. And other other examples. Okay, so in
terms of overall observations, um what we did find
a lot of areas just doing the modeling even before
we had to apply the model checking. Um And
as I said initially we found errors were our errors
in the process modeling or in the properties. But
after that we found real errors in the system.
Um since there weren't experts here about model checking other
than barb maybe? Um uh some of the things
that make model checking hard didn't occur. Um In
fact we had few false positives and we did have
to worry about optimizations, but things seem to work
out pretty well. Um But the other thing is
that the process is, so one of my takeaways
from all this is if you really have a process
that's complicated enough that you need a modeling language to
represent it to reason about it, then you really
need this automated analysis because the errors are really very
, very hard to find. And I think there's
a lot of people who are advocating uml um which
is not a very well defined language and doesn't have
very strong analysis capabilities. And I think without that
it's really gonna be very hard to get correct models
. Okay, so let me go back to this
picture. I was talking about the model checking.
Let me talk about these two analysis techniques which are
safety analysis and um lee mentioned this uh what the
safety analysis is versus the model checking. So the
model checking if you say transfused blood or or um
or check their idea. It assumes that if that's
what the task was. That's what you did.
But when you're dealing with humans, sometimes they don't
do it right, when you're dealing with hardware,
sometimes it doesn't do it. Right. So the
safety analysis says what happens if something isn't done?
Quite right. Are you going to detect that?
Is there enough checking that goes on? And is
it done in the right place that you would detect
those problems so that you didn't really have errors reaching
the patients? So one of these the failure modes
and effects analysis basically says if one thing goes wrong
here in the process how would that propagate through the
process? And what they do with failure mode effect
analysis is they basically create take a a failure and
they track it through the system showing what would go
wrong. So if your input to a particular task
was wrong you'd assume that your output would be wrong
and in that outfit gets used and so on and
so forth. And here this is done. I
don't know if you can read it but here this
is showing a particular trace through the system. And
one thing is if you let's say you put in
the wrong bed number for patient then what could happen
and you trace that through and I won't go through
the details but basically you end up with the wrong
patient receiving the blood. So that's an example of
how you could trace that through the system. The
other let's say you put in the wrong blood type
and you might then have the right patient but the
right patient is receiving the wrong blood and that's traced
through that's what this trace through here through the process
shows. So by looking at this you can see
what would be the hazards that could arise. So
um failure modes and effects analysis has been used in
by industrial engineers. And what we can do is
we automatically create these traces to show what could happen
. An alternative is to think about the hazard.
This is the condition that might cause loss of life
or some kind of serious injury and say how could
this happen? And basically again, this is an
industrial engineering approach. Usually you bring together experts about
the process and they talk about the hazard and then
they create what's called a fault tree that says,
well this hazard could only occur if these things happened
and then you look at each one of those things
, how could they occur and you go continue on
down until you get to the level where you feel
. You have enough detail and then you can use
that paltry to do some reasoning about how likely are
collections of things, how likely are certain collections of
things to occur that could cause the hazard. So
what we've done is automate that based on the process
model and use the process model to drive that well
what could go wrong and to create the fault tree
automatically. So to to see an example of this
, I need a little more information in that the
blood transfusion process that you were looking at before and
here just it's just annotated with the flow of artifacts
. And here for example we have the blood unit
that's going in to perform transfusion and that was uh
output from pick up blood from the blood bank that
got the blood type information and there's more information there
. But you can just see a little bit of
that from that. You can sort of can you
read this or not? So it's sort of it
starts off with okay um with the hazard did it
with the help. I'll use this with the hazard
at the top here. Um basically saying the blood
unit performed is wrong. That goes to perform transfusion
. I'll take that. Okay. And and have
it doesn't like me. Uh And how could that
happen? And this is the blood unit from pick
up blood from blood banks one. Well how could
that happen in here here we have an or gate
and that could happen if this happens or if that
happens and so on and so forth. Okay.
And you create this tree. Now this has created
automatically from the process model. Knowing about what the
artifact flow is and The the events here are labeled
E one. E 2. And what we're first
going to do is create just network flow equations perhaps
. Yeah, these network flow equations. And if
you look at this, you know E one,
there was just one thing. So E one equals
E. To you look at E two. Well
that's an ore of E. Three or on E
four. So that's that representation here and so on
and so forth. So you have all these equations
and we want to we want to everyone as our
top events. So E one equals E. To
and we do the substitution and we put it in
disjunctive normal form and that gives us what are called
um um minimal cut sets. And basically if any
one of these are true, then the hazard could
occur. And so this says E for all by
itself, which is the input blood type is correct
. But you picked up the wrong blood unit.
Okay. That all by itself gets the patient the
wrong type of blood. Okay. And this and
this would be a single point of failure. If
this went wrong all by itself, you would have
this hazard. Here's another single point of failure.
Here's a combination. Well, if these you can
have a collection of things but if there were likely
to happen, um the collection was likely to happen
or had a relatively high probability of happening. That
means the hazard could occur. And this kind of
analysis shows you where the process is vulnerable and we
did that kind of process analysis and in fact we
found places where the processes we were modeling were vulnerable
to these kinds of errors and the medical professionals were
quite horrified and then change their processes to reduce the
probability that these errors would occur. Um, so
let me just quickly because I'm running out of time
, say something about this. So these techniques,
the medical professionals were in fact familiar especially with the
failure modes and effects analysis, the analysis being able
to create these trees, these fault trees automatically.
It was really a big savings because in fact industrial
engineers usually do this manually takes a lot of time
. And you really have to have a deep understanding
of the process, You can say well for our
process model and you have to have a deep understanding
of the process and that's true. But for each
process model, we can create a number of the
fall trees for a number of different hazards and we
can change the process easily and then recreate the fault
trees and to give you a sense of how hard
this is. These fall trees are in fact not
small. Okay, here's one for the blood transfusion
and in talking to industrial engineers, they say yes
, they look like this. Can you imagine creating
something like this manually and having any faith in it
. Okay, so, um, you know,
really being able to do this automatically is really very
, very important. Okay. I'm I think we're
not going to talk about simulations just say we do
it. Okay. Just to say that, you
know, one of the things that we're trying to
do is if you spend the time to create that
process model, which we realized, you know,
takes a lot of time and effort. We want
to leverage that. And so I've talked about finding
errors and properties, talked about doing safety analysis.
The other thing we can do is um simulations and
we can use those models to dr simulations so you
can determine what happens when you change the resources,
when you're executing a process model, for example.
So I'm going to skip over all these wonderful slides
about what we learned about simulations. Um There's a
lot of work in simulations and there's a lot of
commercial simulation systems that the medical community is used to
using. The thing is that the models are usually
based on queuing theory. And so they can't do
the fine grained representations that we can do in our
process modeling. And that's why we believe and we're
seeing, we seem to be getting initial results that
show that we get we get better predictive information.
Okay, so, overall observations, we found really
several important errors in the work that in the work
that we did. Um Initially we found errors in
our models and our properties. But correcting these is
really important before you actually use this to tell you
anything about the real processes. We found really important
errors in the processes that we did model and in
the chemotherapy process They saw a 70% reduction in the
errors that reach the patients. So they had been
gathering information on errors were they worked with us and
then the the errors that reach patients went down drastically
after they improve their processes. So that's really very
very strong results. Just let me just quickly say
something about the future. Um So what what we
have been really describing is this view of creating a
process model, doing static analysis and this feedback.
But we're really where we're going is taking those models
now after they've been validated using them to provide guidance
to medical professionals while they're executing those processes and then
we're going to be able to gather information about what
actually happened. Uh We talked about using probabilities and
the fault tree analysis. They don't really know what
those probabilities are, they're just estimating them. But
if we were actually following them while they were actually
executing the processes we start to get that information.
Then we could do some probabilistic analysis and again improve
their process models and then expand the loop. So
we have a static loop as well as a dynamic
loop. And that's actually what we're what we're working
on now. So I think based on the time
I'll end and open it up for questions. Yes
. So one of the examples that you you found
focused on a deadlock scenario but it wasn't quite clear
how your body can currency in the form that you're
using. I saw in the example language you have
this parallel bars. Yes. But you're also using
the finance state machine, which isn't concurrent but you
have an extended version of that so that they have
more Petrie net like Okay, so the in the
process language itself, the equal sign that says you
can do this in parallel is like a fork and
join. But there's also a general message passing.
So you can talk about synchronized and unseen. Quran
is a NSYNC rin ized um control and communication.
Okay. The properties themselves that was the finite set
of tomato and so that's basically the requirements and that's
really talking about the ordering of events and when we
do the analysis. Um, basically we then create
an interleague model of the concurrency to see if the
property is going to be violated. Which is basically
equivalent if you were going to use a Petri net
. I mean you could do this with a Petri
net model and do the concurrency analysis that you would
do over a Petri net explosion. No, I
didn't show you what that model looks like. Black
. Yeah, but again, you know, we
do lots and lots of optimizations. So a property
is only talking about so many events and we take
the model and we get rid of anything that doesn't
impact those events. But yes, the model gets
very large fault tree. One is is large.
Just again because the real model is much larger and
there's so many different things that could happen. I
wondered if manual car sex are unique that they may
not be and they they wouldn't be explanation on the
side of the street common affection so theoretically perhaps.
But these are real processes and the processes we looked
at really basically the cut sets don't tend to be
one or two. So the medical processes that we
looked at tend to be robust, but they may
have four or five and there may be a couple
of there, maybe a dozen or two of them
, but they're not thousands. And that's because these
are real processes. They're not random graphs. And
also you're really concerned that when the terms have a
few events in them because those are the ones that
means that only a couple of things need to go
wrong. If you have a term that has lots
of things in it and they'd all have to be
done wrong, then it's a safer situation. If
I would like to use a little Jill to model
different processes. What's the best way to start?
I'm thinking a particular objective Is uh, for cybersecurity
in 30 minutes. But you know, the system
admins network admins, they could open new accounts for
users purchase new devices, putting in other things they
may affect the security. So little Jill is downloadable
. So you can go to our site and fill
out the form and you can download. There's a
visual editor and there's a tool set and those are
all downloadable and we in fact have, as I
said earlier, we have an election project which is
looking at security, the security of elections. And
we actually have a student who's we have a student
who's working on automatically synthesizing attacks. So it does
appear that this is possible to take the analysis technologies
. We have an automatically synthesized attacks on a system
that's defined in little joke. And um what was
the other application remember the other application asked about but
it's been applied to a lot of different app.
We have working with some people in Australia who are
operating a cloud computing facility and they're very concerned about
where eras originate, how they propagate, how to
find the steps that need to be done better and
so on. It really seems to be a fairly
robust toolset that is applicable to a lot of different
problem domains and a lot of different kinds of issues
in those domains. So our website will guide you
through downloading the little Jill editor for start and then
after that probably we should interact. Right? So
your work is an excellent example of country take nothing
. Computer science constant salt office And so if we're
taking this concept a little bit the privacy of our
lives and self process. So traditional software engineering,
it seems that the traditional processes right as exemplified by
the waterfall model. The problem to be insufficiently flexible
model collection, a real world software development as a
result, uh actual physical being embraced. Right?
So where the process is more intuitive and incremental hasn't
been more observations that they want. A job process
model would be more appropriate for the government training song
. No, we have we have not found that
basically what we have is a mechanism for determining whether
processes if they're suitably precisely defined can be analyzed in
a variety of ways. So there is in this
scenario you're talking about, there are different development environments
and different development environments basically levy different requirements on the
processes in order to determine whether you have a process
that is suitable for your environment. You might take
these tools and do this kind of analysis. So
we actually have some papers where we start to do
some analysis of hypothetical scrum process for example. Well
there is no definition of a scrum process. So
we hypothesized different variations on the scrum process. Try
to infer the kinds of characteristics those processes had and
then it's up to you as a sort of a
development manager. Do you want to process that?
Has these characteristics or not? We've also taken the
basic waterfall chart and we've elaborated it so that instead
of seeing lots of back groups with no semantics on
them, we now have iteration generally in the form
of Rikers asian that shows how context gets carried from
step to step and when you go back to a
previous step, how that context gets carried with it
and how that if done right can support the rework
that causes a water foolish kind of a world to
work better. And in reasoning about all these different
kinds of processes, we hope to create enough knowledge
so that people can intelligently decide which processes they should
be using as opposed to just simply saying, I
hear scrum is good and then making up what that
means to them. So in the context of other
checking, Tyler or software, typically it's very easy
, it's not very easy to build a model from
from the program. For example, you have to
know all the programs have in order to do a
very accurate. You have you we all start learning
and all that in this context. We're designing a
question here uh to the medical professionals. And so
so my question is when do you, when do
you stop and say, you know, the model
is never Yes. I mean, you know,
it's like a program. In fact it's probably never
totally accurate and you probably forgot some things. I
mean in terms if you don't want to reason about
it, we wouldn't know that. Um so yeah
, it's an ongoing. So we see this.
These are living documents, not something you create and
throw away. And I think that's moving to online
guidance keeps them alive. I mean as you're going
to continue to use them. So I think it's
it's something that you will continue to evolve the process
and medical processes always change. There's new, they
get new findings and you have to put that into
the processes so you want to update them. So
it's an ongoing living effort and the properties continue to
live to because usually the properties don't change too much
when the process models change, we think of processes
as software. So there was a paper that get
written like that. They have requirements, they have
architectures, they have designs, they have code which
you've just seen and they evolve. So that suggests
that the things that software engineers have learned about how
to do that should apply to processes as well.
But the problems also get carried over to this domain
as well. And what you've just asked about is
among the problems that software engineers have grappled with for
a long time. So applying them to the process
domain seems to have created some benefits. And then
one thing I wanted to be sure to mention is
that we really regard this as a two way street
. So, as software engineers were always ask,
you, can you can you write a tool for
this, can you do that? And yeah,
software engineers can always take what it is. We
know, go to other domains and help people with
their problems by writing code. We are doing that
to some extent. We're writing an analyzing process code
. But in doing so, we're discovering things about
software and software engineering, which software engineers don't know
and which enriches the study of software engineering. So
one of the things we didn't spend much time on
here is that a process definition is very rich and
specification about resources. Most processes are very heavily focused
on resource specification utilization. And we have technologies to
quite a good job of that. And then we
go back to software engineering and we say, how
come there isn't a focus on resources and software engineering
and you discover that there is a certain impoverishment of
certain kinds of things that software engineers try to do
, which could be fixed if there was more of
a focus on resources. So this really is a
two way street where we're learning about software engineering by
applying it to these application domains while also enriching those
donates. Maybe have time for one more question in
here. Um so uh your inner nature of the
mission that want to work on uh huh. I
want to know this place needs to add up if
you have any thoughts. Well, so again,
we are actually spending time working on scrum. Yeah
, so that's 11 of one of the agile methods
, there are others that we could look into.
But we're taking scrum as an example. So for
example, you know the scrum has this burned down
list and people come into the morning, take things
off the burned down list. If you had a
discipline on how people did that, would it actually
cause that to work better? So we have created
the scrum process definition, we've loaded it up with
resource specifications, specifications of job mixes, and we're
running discrete events simulations and we're discovering, you know
, surprisingly to me that it seems to make a
huge difference how these tasks actually get assigned. And
there are some scrum environments where people worry about this
somewhere, they don't, it seems to matter.
So this is something which we believe can help people
to learn about software processes. And in fact,
that was the original application domain. And the main
reason we're working with medical people and election officials is
because software engineers really didn't seem to like this idea
at first and but they're now warming up to it
. So I just want to remind everybody that was
recession battered and fried in 106 from 4 to 5
. And I know there's some graduate students meeting with
dr Esther Rolle and Dr Clarke this afternoon at K
. W. Two. So let's thank our son 1154 01:06:18.190 --> A:middle L:90% . Yeah.