I am now at,
and I'll talk a bit more about this later.
But I'm now at Smarter Balanced,
which is affiliated with UCLA.
It was one of two organizations that was funded in
2010 by the US Department of Education
to create computer adaptive assessments,
which are geared towards
testing students in 3rd-8th grade and 11th grade.
What's really fascinating about
this organization is that it
gives us the opportunity
to do some really innovative work.
You might remember when you were in school,
you use Scantrons once a year to take this tests,
wasn't always fun, but
unfortunately you can't get away from assessment.
What we're trying to do now developed
an automated system that adapts to our users,
that allows the students to
progress if they're being
proficient and if they're struggling,
then it provides other kinds of questions,
and other kinds of content scenarios
that allow them to succeed.
It really cutting edge,
this is as far as I'm concerned,
because we don't have something like this right now,
and we're currently in the testing phase.
The member states there are
deploying this test with their students,
and we're actually obtaining data now.
I'll talk more about what I see as future work.
I just started this opportunity in January of this year,
so I'm looking forward to a lot of applied research,
and a lot of knowledge of how
we might move forward in this phase of education.
But before we get there,
I want to spend some time talking
about my journey effectively.
I am honored to be considered and invited as
a distinguished visitor, a distinguished lecture.
I will be the first to concede that
my story isn't that long, I graduated 2007.
But perhaps that's just a testament to the fact that we
want to encourage our lumps to do some great work.
I'll consider myself in good company,
even though I know I've got a lot more growing to do.
I'd like to talk to you a little bit about my journey in
the sense and the the umbrella
of multidisciplinary computer science,
my degree here wasn't Computer Science and I
focus on human-computer interaction or HCI.
How many of you are in or studying
HCI and are familiar at least with this?
Excellent, Perfect. I'll be
singing your tune to a large extent.
But others of you are in other areas,
networking, software engineering,
operating systems, all of which are important to
this idea of how do we work
together to solve problems within the day.
That's what computer science does, problem solving.
I argue that we solve
the world's toughest challenges
by working together across disciplines.
I had this year,
not just as an opportunity,
to show you a picture, without word.
But as we spend some time talking together,
let's think a little bit about what it
is you want for yourself?
What image do you want to project in the world,
but what do you want to be?
What problems do you want to solve?
How do you see yourself
being that change you wish to see in the world?
I think having that lens,
trying to figure out, even if
you don't quite figure it out,
what you want to do when you grow up or
at least what space you want to be in,
helps you to think more broadly
about the impact you can make in the world.
At the end of the day, that's what we all want do.
Or at least I hope we wanna do want to make some impact.
I think having that as our motivation,
can drive a lot of the work that we do. It has for me.
What's really great about
multidisciplinary computer science is that it
combines the field of
one discipline into the problems of another.
I don't expect to this as a new idea for you,
but at least in the context of how you can model
your careers across a range of opportunity.
It's really the focus here, at least that's the way I'm
crafting my message this morning.
The idea of being able to apply computer science and
compete competing principles to
a range of problems is not new.
As a takeaway, I want you to again think about
what impression you want to make as computer scientists.
What are the most pressing issues
that require a multidisciplinary approach?
I argue that the bulk of
those issues require a host of problem sets,
and in tackling those problems set.
I like this tag cloud because it's
an example of computer science
integrated with educational software design,
whether or not you familiar with that area.
It's a really great visualization of the many topics,
the many areas that are part and parcel to this area.
Looking at this cloud,
and a number of you can see a handful of
topics and disciplines that meet your own experiences.
You can also see that there are areas that maybe don't,
but that's the beauty of
working in multidisciplinary teams.
Having that opportunity to work across with
your own skill sets are
to solve problems along with other people.
Again, I really like this because it allows for
a really tangible example of how
your experience is important when
you're working on some really world has problems.
Another area in which it really makes sense for us
to think multidisciplinary as it relates to computing,
relates to cyber security and cyber technology.
More and more especially in
the government we're looking for,
or the government is looking for individuals who were
skilled in the ability to counterattacks,
to monitor our systems,
to monitor our software,
to ensure that we are protected.
Or at least there's a level of protection,
a barrier against malware
or against those who would attack us and against
those who would otherwise render us
unable to use systems to complete our work.
This quote actually from
the Department of Homeland Security,
describes the need for cybersecurity as follows,
our daily life, economic vitality and
national security depend on stable,
safe and resilient cyberspace.
We rely on this vast array of
networks to communicate and travel,
power our homes, run our economy,
and provide government services,
so that covers a lot.
But to bring things into perspective,
let's consider that cyberattacks
are really a way of life these days.
None of these should be a surprise to you,
and might jog your memory about
some of the most recent attacks we've
experienced in terms of our technical systems.
You might remember the Target breach
last year that rendered a 148,
I didn't realize it's a $148 million in damages.
Whereas people lost money on their credit cards,
lost revenue, lost resources in millions of dollars.
Not to mention
the 110 million consumers who were impacted.
Who's going to be stopping these attacks?
How do we figure
out when these attacks are going to occur?
How can we better arm ourself?
But Home Depot, a similar breach,
60 million cards stolen from customers,
and will continue to see similar trends.
It just amplifies the importance of us being able to
understand how we meet in
the middle of solving this problem.
Of course, we can't forget,
at least most of us can't forget
that celebrity iCloud account that were hacked,
and there were
a lot too personal and private pictures that were
shared to the children
of the people whose accounts were affected.
Thinking holistically, not just about
being on the receiving end of these issue,
but how do we combat them is really important.
Let's pick a more broadly about what it takes,
to be in cybersecurity.
Just yell out for me an area,
a discipline that you think is important
for working in cybersecurity.
A discipline, a scale, Just call them out for me,
cryptography, [BACKGROUND] networking,
psychology, absolutely.
Others.
[BACKGROUND] Natural language right about it that.
I know, you're burning to share,
this is the participation.
Visualization.
Oh, Visualization. Absolutely. Other non professors?
[LAUGHTER] Its okay. There are no wrong answers,
at least I will penalize you for them.
Other ideas, other thoughts.
Law.
Law. Politics, that's a good one.
Other topics, other disciplines, other skills?
Two more and I'll stop. Two more willing souls
[LAUGHTER] and we'll move forward.
[NOISE] Physical security, that's a great one.
One more. [NOISE] Education.
Absolutely. I heard another one and I'm going
to give a bonus [NOISE].
Sorry? [NOISE] Real-time systems.
All really great answers, some of
which I didn't think about.
But the idea is, it takes a whole lot,
including the ones that you listed that are not here.
The idea is that we
maintain a level of
holistic thinking about what it means,
not just to be a computer scientist,
but to really solve important problems in the world.
Less and less, we're able to do it in a silo.
That's the point we're making here.
Speaking of making a point that brings us to
the focus of our talk because we want
to build this together,
and make it make sense in a way
that ends in a pretty bow.
We're building a case here,
not just multidisciplinary computer science.
But from my own experiences,
I've tried to produce evidence-based research and
evidence-based solutions for how
we design and develop socio-technical systems.
That's what I'm talking about today.
I want to bring you along as
I go back through memory lane,
but really give you an idea of the kinds of
work that really require
cross collaboration across disciplines.
Let's start with this idea of community,
which is not something
you think about when you think of computer science.
But is where I want to start.
So let's start with computing for community,
specifically in empowering non-profit groups.
When I left Virginia Tech in 2007,
I was fortunate enough to
participate or take a postdoc at Penn State,
where I worked with John M. Carroll and Mary Beth Rosson,
who were former Virginia Tech Professors.
In many ways for me, it was full circle
because when I started at Virginia Tech,
Mary Beth Rosson was my advisor.
I had the great fortune to go work in state college.
My goal over the course of my postdoc was to study
non-profits but in the context
of community wireless networks.
You may not remember or
know much about community wireless networks.
But at the time, they were a thing.
They were really popular.
But there were inherent issues with
maintaining these networks because they were cost issues.
How does the state sustain this kind of connectivity?
Are there broadband issues that they have to pay for?
At the time, there was
talk that these wireless networks would be free.
Again, who's going to bear the cost of that?
What kind of infrastructure are we going
to be able to build to
support this kind of interconnectivity?
Before we even get to the issue of
how we actually approach
these non-profits to do the work,
let's take a step back and think
about what it means to be part of the community,
what it means to believe
that you're contributing to your community.
I focus a lot on Albert Bandura,
who is a socio- psychologist,
who describes this idea of self-efficacy.
Which effectively allows us to believe,
not just believe, but to
know that we can accomplish something.
If you think about when you were a kid,
when you were younger, at least for me.
I grew up believing,
if you can believe it, you can achieve it.
As a converse to that,
you think of The Little Engine That Could. "
I think I can, I think I can."
In any case, they're not necessarily the same.
It may sound the same.
But if you think about it,
"To believe you can achieve it,
then you can achieve it," is
actually based in epistemology,
which is the theory of knowledge,
which is based in truth.
Often, when we talk about self-efficacy,
we're talking about having
experienced something or having
accomplished something in the past,
and because you've accomplished it,
you know that you can do it again.
Or you know you can do something similar.
Or you know you can develop or grow in
a similar performance task or
some activity that allows you to feel empowered,
because you've got history on your side.
That was very different from saying,
"Okay, I've never done this before,
I wonder if I can do it."
No, it's believing that you can.
That's the whole point of self-efficacy,
and to think of self-efficacy in
the sense of communities.
Well, there's also the construct
of social cognitive theory,
which is this idea of human agency,
but human agency that extends
personal self-efficacy and takes
us into what we call proxy.
If I tell someone else to do the work
or ensure that they
perform some task that I know is going to get done,
so I absolve myself of performing those activities.
But you know it's going to get done.
Then the third area, which is again,
is part of the story that I'm building,
is this idea of collective efficacy.
As a community, we can make this happen.
You and I are in this together.
That's really what we focused on for this idea
of community collective efficacy,
which is an idea that we termed and worked for,
that we believed would allow us to empower non-profit,
empower the local community
to be the change they wish to see in their community,
using only community wireless networks.
It seemed a bit far-fetched and it
required a lot of discussion
with organizations, with groups.
I had lots of meetings,
and interviews and research.
Meetings with organizations like United Way,
Girls & Boys Club,
even local food banks,
with the idea of empowering
them to be leaders in their communities.
Now, non-profits don't have the most resources.
They're often understaffed.
They don't have a lot of funding.
But they have a whole list of
constituents who need a lot of support.
But we believed that providing an opportunity for them to
be proponents of a system that would be free,
that is a wireless network that would be free,
would enable them to not
just accomplish their bottom line,
but to be leaders in the community,
to encourage and empower their constituents
to be the change they wish to see in
the world, to help themselves.
The idea is if we develop
a sense of community among the non-profits,
then we were leaning on this idea
of communities want to help
themselves get better because
they are all in this together.
It sounds touchy-feely.
But when you're dealing with people,
when you're dealing with socio-technical systems,
that's the problem we're solving.
How do we integrate these highly technical interactions
and highly technical systems,
and couple those with the people who use them?
At the end of the day,
the common denominator is the user, is the individual.
That's been the course of a lot of my work.
My work focuses on user research.
In this stage of my life,
it was highly pure research
to the extent that we wanted to
really talk with individuals,
talk with groups, do behavioral studies,
do user studies to really understand the impact of,
first of all, what their current state of the art is,
because as you'll learn if you have it already,
it's hard to make change and create
change if you don't
really understand socio-technical change.
If you don't understand the current landscape.
If you don't understand why
people use the system that they use,
or why they don't use the system that they use.
That's the first part of this process,
to indulge in user research to understand the impact.
Now again, user research is
an opportunity to focus on the user and to impact design.
It's been a large part
of what it takes in
my mind to really develop
software that meets students where they are,
or meets users where they are.
We followed this suit. In so doing,
the idea was to make
these community groups civic contributors,
make them part of the local government.
Ensure that they were able to have that voice,
enable them to have
that voice because now they were able to connect.
Now, there were other issues.
If you followed CWNs or community wireless networks,
or even just look them up during the course of this talk,
you'll find that they're virtually non-existent
now because of issues that we talked about, resources.
But on the flip side of that, even
in empowering these non-profits,
they had infrastructure issues.
Even if they wanted to get online more strategically,
they weren't always able.
There was one organization that I worked with,
that was housed in a cement embossed building.
Even the cellphones wouldn't work.
But the building was accessible.
The group had been there for years.
They didn't see themselves leaving.
In order to connect them to, again,
this structure that was soon to be,
that didn't yet exist, we had to talk
about infrastructure changes, and that was an expense.
That's part of the challenge
in encouraging users to adopt new technologies.
But it also falls along
the lines of those things we don't always expect.
Yeah, we'd like to all be connected,
we'd like to provide this kind of exposure,
but it isn't always possible because of
the limitations that we just didn't account for.
I'll talk little bit about this in terms of
the tool that I use,
and others, maybe some of you use in terms of really
determining what the actual needs are.
If you're in software engineering, you
might call this needs analysis,
if you're in HCI,
you might call this usability engineering.
But it goes a step further because it doesn't
assume that we know what the users need,
and that's, by definition,
the purpose of user research.
We actually do the work called attitudinal.
What are the attitudes that users
have about the technology itself,
about the interaction, about
the opportunities that are available to them,
but also what are the behaviors?
How do they actually use the current state of the art?
How do they think they'll use
the new systems when they become available?
So it requires a multilevel approach
to really understanding how
to meet the user where they are.
I'm going to say that a lot,
probably because it's ingrained in me.
I work with a lot of users.
I've worked with a lot of different individuals
who represent different sets of users.
But, at the end of the day,
that's lesson number 1.
The user is still at the center of any innovation.
You can create the next killer app,
but if nobody ever uses it,
it's of no consequence.
So that's the really important part of this.
Again, it will be
strange to hear your computer scientist
talking to like this.
I remember when I was getting my degree, I worried.
I'm a computer scientist. Is it okay for me to learn
sociology and anthropology and
the psychology of technology,
in the sense that that's not what I
thought a typical computer scientist did.
But I was gratified because
computer scientists problem solve,
and what I really appreciated about
my degree experience is that it
allowed me to see what
those different avenues were for a computer scientist.
Now, it didn't mean that I didn't do
like to share coding and development,
which is part of the game,
if you will, but to take it to the next level,
I always wanted to be the person who was telling
everybody else what to develop, what to design.
So maybe I'm living the dream,
but the idea is to decide what it is
you want to accomplish and make it happen,
so user research is an important part
of what is in the arsenal.
This graphic actually is from Nielsen/Norman Group,
who many of you may be aware of,
a forefront in user research,
so I always find this to be a graphic,
handy reminder of the tools that are available,
as we think about the kinds of designs that are
available to us as scientists.
Here's an example of some of the designs
that we created in terms of our work with our groups.
One of the groups
that we're working with was an autistic group.
In fact, we worked with the behavior therapist who had
clients whose kids struggled
with autism and struggled with being
able to be out in the world because
there were certain behavioral issues
that kept them from being able to go
out and experience life or
just barriers to social connectivity.
What we decided to do was to
talk with this behavior therapist,
understand exactly what he wanted to
accomplish from a technical system,
and we designed this network community
that had this give-and-take between
families who were dealing with autism
to students who are autistic themselves at Penn State,
and it allowed for an exchange of experiences.
Now, this is different from any other online or
other network community in
that there was this give-and-take,
and there was this sharing of
a very personal experience
that you don't always see in communities.
Autistic students at Penn State or any place are very
unique and often have
challenges that are unlike any other students',
but what it ended up being was
an opportunity for students and
kids and their parents to see what
was possible for those students, and vice versa.
There was this opportunity to mentor the students,
and because students and individuals on
the autistic spectrum struggle
with social kinds of behaviors,
it was an amazing opportunity to see them in action,
to have that opportunity to interact,
grow themselves, and develop
themselves in a way that doesn't come naturally.
Again, this design afforded the opportunity,
at least we wished, to enable
that interaction that doesn't always come naturally.
Moving on to the world
of computing and multidisciplinary work,
I'd like to talk a little bit about
the work I did during my time at DARPA.
I was not a program manager,
but I was recruited to work with, I think, Capt.
Russell Shilling, who was the Program Manager.
I remember, very clearly,
the day that we interviewed each other,
and I say that lovingly because,
not only was he my colleague,
he's now my friend, and he's actually now
the Executive Director of
STEM at the US Department of Education.
The opportunity of working with him allowed me to try to,
not just see how work was done in the military,
how it's done in the DOD,
but when we interviewed each other,
he said he wanted me to be his second hit,
his second brain, which for me was
amazing because it meant that I'd be
able to work really closely with this person.
He called the shots, obviously,
but to have that opportunity was not something that you
see a lot and that you get a lot of opportunity doing,
so it meant a lot to be able to have that relationship.
But one of the programs that we
worked on was called the DCAPS program,
and if you agree with anything military,
you know that we use acronyms.
The DCAPS program stands for
the Detection and Computation
Analysis of Psychological Signals, DCAPS.
The idea of the program was to develop
technology that allowed us
to monitor psychological distress.
Now, you might know,
but I'll remind you,
that we have military veterans
who would come back from Iran and
Afghanistan experiencing episodes of
PTSD at almost 25 percent at the most,
and suicide rates were not that much better.
So this program was an opportunity for us
to do something about that.
The idea was, again,
to merge computer science with
psychology with development and
medical health and mental health to devise
solutions that would allow us
to support our military veterans.
So what did that mean? This is DARPA again.
So there had to be something driving the research.
We decided to use what were called honest signals,
which is an idea originally fronted by Sandy Pentland,
who is a professor at MIT Media Labs,
and what these honest signals mean is that there are
certain behavioral cues that are obvious,
but you may not know them by looking at them.
For example, if I'm depressed,
I'm going to exhibit non-verbal cues
that you may not recognize as my being distressed,
and what we're able to find is,
in the course of developing these technologies,
there were a host of these kinds of
behavioral and non-verbal cues that
allowed us to determine not necessarily that
someone was depressed or
that someone was suffering from PTSD,
but perhaps that person needed to go see
a clinician or monitor their personal health.
I want to say very clearly that
a paramount part of this program was privacy,
so if we're talking about
devices that can monitor your behavioral health,
anything about you, we're
talking about personally identifiable information.
We're talking about those cues
that you probably wouldn't
want the average person to get a hold of,
so aside from the fact that these systems were
very secure, users opted in.
In fact, a very important part of
this project was ensuring that
we follow the strictest IRB protocol,
so I want to assure
you that that was part of the project.
But more generally, we weren't talking
about making a diagnosis for anybody.
We're not talking Doc-in-a-box,
which one of our researchers would often say,
but we're talking about providing support.
I'm going to show you a video that one of
our developer teams created that showcases the research.
We'll talk about it afterwards,
but the idea is that there's a host of opportunity and
tools that you'll see when you see the video
that dictate the method
of problem-solving that we were going for.
I also want to say that
the person in the video is not an actual patient,
he's an actor, so that's an important distinction.
I'll also call out that
the work that I'm showing was led by
Skip Rizzo and Louis-Phillipe Morency who are at USC.
In fact, Louis-Phillipe is now at Carnegie Mellon.
I'll like to show you
a clip of the system that they created,
which is an avatar-based system.
The idea of this particular system is that it's able to
read behavior cues on an individual sitting right across,
and it did that by training itself,
based on many real life interviews
that individuals had that were videotaped,
and based on those video analysis,
we were able to compile a list of queues that allowed
us to know what to look for when we were looking for
depression or emotional distress.
Keep that in the back of your mind as you watch this.
I'm going to try to elegantly pause
this and move to the video.
[NOISE]
Hi, I'm Ellie.
Thanks for coming in today.
I was created to talk to people in
a safe and secure environment.
I'm not a therapist,
but I'm here to learn about people
and would love to learn about you.
I'll ask a few questions to get us
started and please feel free to tell me anything.
Your answers are totally confidential.
Are you okay with this?
Yes.
How are you doing today?
I'm doing well.
That's good. Where are you from originally?
I'm from Los Angeles.
I'm from LA myself.
When was the last time you felt really happy?
When was the last time? I don't know.
I don't have any real high highs.
I feel like I'm a level person, it's just happy.
I guess to answer your question,
probably a couple months ago.
I noticed you were hesitant on that one.
Would do you say you are generally a happy person?
I'm generally happy, just lately
there are things that's keeping me down.
Can you tell me more about that?
What was the last time?
You can actually do this on YouTube,
but you notice, hopefully, you notice a few things,
the least of which is displayed across the person's face.
The team used the connect system to actually model
how the individual appeared on the screen.
You'll also notice that there are the other cues,
the other parts of the interaction that are being
measured and analyzed in real-time.
By the way, after this interaction happens,
the data are eliminated.
There's no storage of these data.
But what you're noticing at the top right of
the black screen here, this smile level.
Smiles are one of those non-behavior cues
that people will show,
sometimes despite being depressed.
There are things called social smiles that
will indicate that we're
smiling and you might think
we're happy, but we're really not.
We're hiding some things.
It allows us to measure smile level
again as a construct that
might indicate psychological distress.
Speaking fractals, how the mouth is situated again,
looking at the smiles,
looking at the movement of the face as it
relates to showing or indicating some level of distress.
There are also things like horizontal gaze
and moving forward and backward.
Through this work, the team was able to determine that
folks who actually were depressed, fidget a lot,
and tend to move forward,
forward and backward, something that we may
not pick up on when we're talking to individuals.
Some of you might, but most part we don't.
Before you laugh, because I laughed too,
frankly, at the idea that
this avatar was trying to establish rapport,
we've learned that individual suffering from PTSD,
especially those in the military,
are less likely to see a clinician or see
a counselor for various reasons: stigma,
uncomfortable, just don't want to go.
But they're more likely to interact with this avatar,
which if it helps one or two soldiers,
it's done, it's measured good.
Still under development, but I wanted to show you
that as an example.
Again, I want to recap,
privacy and security are so important
here because we're talking
about DARPA work and
research that relates to individuals.
I actually had to participate in a privacy panel.
I appeared before lawyers
and Head of Research and other DARPA personnel
to ensure that we're going to ensure
that proper safeguards are
followed when we're working with individuals,
especially those from particularly sensitive populations.
If your familiar with IRB protocols,
Institutional Review Board that ensure that
the appropriate level of care is
given to subjects in your research,
you'll know that some populations
are more vulnerable than others and so
we have to address the risk or
ensure that participation is minimal risk.
We had to do that very efficiently.
In fact, I did that through
my first month at DARPA. That's how important it was.
Let's shuffle along now to talk about
computing and computer science and data analytics.
This is another program that I had the privilege
of working on, the ENGAGE program.
This is actually not an acronym.
It just stands for engagement,
the opportunity to engage our users.
So got a lot of questions about that and we just
tend to write a capitalize it is what it is.
But the idea is to develop
these data-intensive training methods, that scale.
Some of you might be familiar
with a similar term called big data,
where you take lots and lots of information.
You look for patterns, you look
for very interesting stories,
and you allow that data to inform your design,
your problem set, your solution,
whatever you're moving toward accomplishing.
For this particular project,
we were working on K through
12 as our user group, that is,
how could we impact training and
instruction for students at the K-12 level,
especially in concepts as they
relate it to development of
problem-solving and conceptual reasoning
that might help them as they become adults,
and maybe they'll pursue
technical careers as a by-product.
Because we're talking about scaling,
we're talking about using data
for thousands and tens of thousands of users,
we're talking about kids, another vulnerable population,
so again, strictest IRB protocols are followed.
In fact, for this project
and the last project that I discussed,
the Navy and the Army were
IRB gateways to ensure
that any work done within the military,
within DoD, was appropriately managed.
The inspiration for the ENGAGE program
was called the Fold-It program.
I think one of your previous speakers talked about
Fold-It from University of Washington
as being a really amazing,
effectively program that allowed
novice users to work on
problems that typically experts work on,
especially if it's about biochemistry problems or
other activities that typically are
done by those who are well-versed in the subject matter.
But the benefit and the beauty of using these kinds of
systems is not just that the novice becomes the expert,
but they can be expert without really realizing
that they're doing a lot of
work and not putting the work in,
that they're actually training.
That's the benefit of these kinds of systems.
That's the model we want to follow in the ENGAGE program.
Effectively crowd-sourcing from players to ensure that
the game creates these super users,
as it were, but not just super users,
the super knowledge holders
of information and capability.
Imagine what we could do with kids by
teaching them constructs like problem-solving,
computational thinking, without them
thinking that those are actually what they're learning.
Make it fun. So another name
for instructional technologies is?
A game. I want to show you,
again, a couple of these examples.
[BACKGROUND]
[NOISE] There we go.
Thank you and I'm
wearing glasses if you can believe it. All right.
So here's refraction, the first game.
Refraction is great because it uses what's
called a Playtracer to
actually trace a player's activities in a day.
[MUSIC] Refraction tends to be really fun.
But what it allows players to do is,
progress through the game and it actually
teaches fractions by splitting and dicing through.
It actually captures the player's moves
during the game and it logs
it by two groups.
The path that they follow to success,
to reaching a level and conquering a level,
that is figuring out how to solve the fractions
correctly or the fails.
How long did it take them to get there,
if they got there at all?
This effective database, this
store of all this data across many, many students,
across many, many players,
allows us to determine
the level at which actual learning happens.
Where are the breakdowns,
where are the pain points,
and how can we fix this within
the next players or the next similar kind of player?
Does it have as much difficulty or at least get
the proper support in
game to actually learn the concept sooner?
I'm going to just play a couple of levels
here and I also want to say that this was
developed at
the University of Washington under Zoran Popović,
again, who is the lead for
the hold at work and at the center of game science.
With these games, you can see you can play online
and experience for yourself.
I'm going to split this ray quite easily.
So that was easy. No, they're not all that easy.
But depending on how well you do,
it presents to you the next set of problems.
So obviously this is not going to work and you see that
it just doesn't allow the user to move forward,
but it logs that move so that it can determine
how often this person to play and makes this
move and whether or not the person might need help.
You notice a little pop-up appears,
which is typical for games, right?
You want to provide that level of feedback.
These are pretty easy and it's
a fun game, but I wanted to, again,
break the monotony here,
but feel free to go online and play
these Treefrog Treasure or similar game.
It allows the capture of students' play,
and it adapts to the users as necessary.
Again, it becomes this big data problem,
but we're capturing all this data,
capturing all this information,
and learning from it in the sense that
we can improve the game and makes it even better.
Let us take a closer look at
how the Playtracer actually works.
I'm going to look here so that I'm
not twisted behind you.
What you're looking at is
the way the Playtracer actually works.
Under this first box here,
you're actually looking at the start state,
which is shown in yellow,
and the goal state, that is where you want to end up
in that level, in green.
The second box, the middle box,
which is the second iteration,
as a player progresses through the game,
shows you the observed states.
It is where those states that
the player visits and those states are shown in black.
The more states or the more ways, the more problems,
the more content that a player visits,
the larger the circle,
the larger that particular state.
It shows an example of how we're
capturing this information player
by player in aggregate
so that we can go back and analyze it.
The 3rd picture actually
shows the state transitions as shown by the arrows.
So if you're familiar with database modeling,
Finite Automata that you
might learn in theory, this is similar to that.
We're talking about movement within a game across states.
What this particular algorithm captures,
the Playtracer, which is pretty novel,
is an opportunity to
transit between states within a game,
within a level, and determine how often
students or players are succeeding and failing.
More to the point, here's output
from about 30 people, 30 players.
The blue arrows, which I hope they show up blue,
show you the level of success.
That is the amount
of times that players actually succeeded.
They've got the fractions [inaudible] correct,
or they figured out the right answer to the fraction.
The red arrows actually show level failure.
So looking at this,
it's a visualization of the problem space.
How do we model this?
Well, this is the way we do it at least in a visual way.
It's an amazing opportunity to look
at how our players play and learn,
in this case, for infractions and
apply it back to the game.
In-depth versions of the game can adapt to the user,
adapt to the player, and that's what we want.
I don't want to get ahead of myself, but imagine if we
can deploy these algorithms,
this kind of structure that allows us
to educate and instruct all kinds
of learners how truly personal that experience would be.
Again, the novelty of this approach, the true benefit,
it gives us the opportunity to see how
well or poorly players played.
It gives us insight into those pain points,
again, that I mentioned.
Where are their breakdowns?
Were users messing up?
How are they missing the content?
But also, where did they get back on track?
How do we take that information,
learn from it, and pipe it back into the game?
This visualization, this algorithm allows us to do that.
The two takeaways from this, especially lessons learned,
as we think about the broader impact
of big data for education,
it's entirely possible to
improve instruction through adaptation,
by adapting the technology.
That's not always easy, but it's also important.
It also makes a very personal learning experience.
Applying both individual steps
to what's a success or what's a failure for
a learner applied across
hundreds of thousands of users is amazing information.
Again, this idea of
making what we might consider the individual,
the big data problem becomes enormous for us.
Thinking about education, thinking about computing,
and how we might teach even kids how to think more
critically about what it means to
be a computational thinker
without them knowing that they're doing it.
It made me think a little bit more
too about how do we even
get more individuals interested in
math or computer science?
More broadly, what we've learned about being in
any kind of technical situation or
technical group is that the diversity of ideas,
not just from the ideas that come from the individual,
but from the individual themselves,
will they make the most dynamic teams?
So how can we make an impact?
How can I make an impact on better
understanding the landscape of what it
means to learn computer science at
whatever age and how can we make
an impact and increasing those numbers?
If any of you have done any outreach
or any student outreach,
or understanding of what it means
to encourage inclusion and computing,
the best time to really start is middle-school.
If we started to recruit at college,
then we're often too late and those individuals who we
recruit decide to move into the major often leave.
I wanted to think more broadly about what it
meant to actually do
something about increasing the pipeline
or making an impact in the pipeline early on.
Looking at these data,
these actually were made
available by NSF in January of this year,
I apologize, it's not very clear,
but it certainly shows
the proportion of women, minorities,
and persons with disabilities in science and engineering
across different disciplines including psychology,
social sciences, biological sciences,
physical sciences, math and
statistics, CS and engineering.
The point I want to make here,
if you look at the bottom two lines,
the first is engineering,
and the blue there, computer science, 4.8.
The data show, from previous years,
if those numbers are holding steady,
but they're certainly not increasing.
So the question becomes, how do we make an impact?
How did we increase those numbers?
Or at least attempt to
better understand why we're not seeing
a more diverse set of
computer scientists later in the pipeline.
It motivated a colleague and me to
create a program called
Motivate that's actually the name of the program,
where it's a five-level process that
incorporates mentorship, after-school programming,
which falls under what we call
informal education and training,
as well as formal training where we provide
in-course or course-related training for computing,
as well as including parental involvement.
How do we engage the parents?
Because a very important part of what we've learned about
minority groups is
understanding what the family structure is,
understanding what some of those impacts
are to those barriers are to
entry fall often with
parents just not being aware of the opportunities,
so including parents as a part of that process,
as well as this idea of making Computer Science real,
applying it in everyday life so that students
understand how they can see
Computer Science in anything that they do.
We did that as an effort to
remove what we call the mystique of computing.
Often for under-represented students,
there's this belief or there's not a belief or
understanding about what Computer Science
is or what a computer scientist looks like.
What's more, the fact
that computer scientist doesn't look like them.
We wanted to measure whether or not attitude
shifted pre and post-exposition to these program ideas.
How do we learn,
first of all, what their attitudes are?
We talk about user research, remember?
Attitudes, behaviors as it relates to what
a user's propensity towards
a certain activity or technology is.
This is the same idea.
We had a peer host survey of
these students to better understand [BACKGROUND]
what they know about computing
and whether or not they would
pursue it based on their experiences in our camp.
We actually did this through a summer camp,
which was our informal
training session for these students.
We did this through a couple of methods.
We provided an e-textile workshop,
which is a part of this wearable computing movement,
especially for young people.
They were able to sow
these technical systems into
bags and clothes and other kinds of things
just to see how they work and also use
logic switching to see how
the logic and be at electronics
of the system actually operated.
There's also a 3D-printing part of
the workshop where students were able to learn CAD,
and then print their designs,
which was a lot of fun for the students.
This is a pilot program that we
did a couple of years ago and it
allowed us to see that there was more work to be done.
Obviously, with a small sample of 15,
there weren't a lot of people.
But it was great because it allowed us to
get a sense of where these students were.
The really nice part about it,
it seemed for this sample anyway,
is that six have participated in prior STEM experiences,
which is great because the minute they came
in with a certain level of expectation,
certain experience that hopefully matches,
impacted their experience in the program,
but others as well,
and it might have been part of what I might call
this invisible collective efficacy.
If you're around people who can
do things and maybe you think you can do it too.
Because what we found is that
no student actually said
maybe there were a couple of maybe who said,
that they didn't like Computer Science.
But no one actually said, or computing,
but no one actually said I
can't do it. Which was really important.
It helped us to see that maybe
there's something here to removing this mystique,
removing what these barriers to
the idea of what it
means to be a computer scientist
is for a lot of these students.
That was encouraging, though
there's still more work to do there.
Another important part was this idea that
if we can train or help students to
understand that if you wear your clothes or
if you wear the watch or you create a 3D-printed device,
that that can become part of your everyday life or
whatever you're developing, whatever you're designing.
There's no reason why if you didn't create it
and you didn't divine some use for it,
that it can't be applied to you for your everyday living.
That was an important part.
Students were actually very gratified that they
were able to create these devices or
these clothes and I'll show
you actually here at the top left,
this monster-looking thing is actually your purse.
What we found is that the girls couldn't actually
sew which is it is what it is.
But more and more, a few of us can sew.
I can sew on a button or whatnot,
but I can't create clothes like my mom and her mom could.
Part of it is generational,
but that was a very interesting response.
But at least they were able to
create and at the very top next to
the pilot text is an example of one of
the CAD drawings that one of the students
actually were able to create through a 3D-printer.
What was really great about this,
I think there was a session this morning
that talked a little bit about this,
was again, the girls could see
how this worked in everyday life.
If there is a design which you've
divined for a 3D-printer,
then you can create it and just yesterday,
I saw on the news and again
this morning that we can actually
create these kinds of
devices for everyday use for certain people.
The girls were able to understand that
and to see that there really is
application for this combining computing
with providing a real need
for real users is really powerful.
To finish off this idea of
what it means to provide opportunities for
computing to those who may not be inclined to
pursue with this idea of what it means to
persist in computing this
is a great book if you've never read it,
I highly recommended, How Children Succeed by Paul Tough,
who was a reporter for the New York Times.
He did quite a bit of research for
understanding what it takes for a lot of kids,
many of whom were born in poverty to persist and succeed.
Some of those traits are grit,
perseverance, discipline, even self-control.
But in the context of an individual
who's born in poverty,
who often doesn't know a lot of other options,
becomes magnanimous in helping
them succeed when given the opportunity.
One of the questions that I ask is,
"Do these apply to instruction in computing,
maybe even learning computing?
If we encourage a try and try and try mentality,
maybe that'll go a long way
towards encouraging students again,
who might not be inclined to computing to succeed."
Fascinating book, I highly recommend it.
I'm nearly done. I noticed some of you
looked up because there was that excitement
of almost being done, but that's okay.
I want to take a little bit further.
I want to finish with a little bit of what I'm doing now.
I again, I'm at Smarter Balanced which
is recently moved to UCLA.
In fact, Smarter Balanced was
an independent organization affiliated with
the State of Washington up until last December.
The benefit of moving to UCLA is that we're housed with
some additional world-class researchers who can
help us better understand
the information that we're generating,
the data that we're generating,
and how to create the systems that again,
we want to be able to provide great assessments.
But what else can we do to
improve this experience for students,
teachers, administrators,
and even in preparing the parents?
What's really amazing, again in
this multidisciplinary discussion is that
I'm working with a lot of folks that
I would not normally work with.
I work with psychometricians,
the individuals who create
the statistics to determine how accurate the test
is or how correct an item is in the exam itself.
That help create the content,
the math, and the language arts pieces.
The folks who are actually
the district lease the member lease.
All of these folks are stakeholders and as the designer,
it's important to me to ensure
that the most important voices are heard,
and we can't forget those students
who are from those unrepresented groups,
not necessarily those who have not used
technology or those who suffer physical disabilities.
They're are also important to
this community of test-takers.
How do we take into account all
these user needs in creating
a computer-adapted assessment and
allow it to be effective and
valid and that's the challenge.
But I argue that this
is a step towards the right direction.
How awesome would it be if we could
create assessments that allowed us to
operationalize this process of
what it means to learn through an assessment?
You probably don't think about
it when you take an exam, you take a test.
But sometimes you learn something,
maybe you are something you didn't know,
you learn something you forgot.
But we'd like to be able create an exam and
assessment that allows you to learn in
the process of taking it and how
amazing would it be if we could
operationalize what that meant,
how that looked, and we could apply it to it to a range,
an array of instruction technologies.
Or maybe if we marry this idea of what it
means to assess an individual,
but also at the individual level.
If we took this idea of
this play tracer that we saw in our engaged games
and ported it as an algorithm in
the backend of a system
that actually performed assessments.
Now there are challenges there
because obviously assessment
is very personalized, right?
We had to be very careful
about how that process comes to bear.
But what would it mean if we can make that happen?
Could we be looking at a situation where
an individual learner walks away from
an exam having learned more than they thought that they
would and not just learned it, they can apply it.
How amazing would it be
if we could teach computational thinking?
This idea of what it means to learn computing,
not computing actually computational thinking is not computing.
But if we can teach these concepts as
ways to problem-solving and conceptual thinking and
understanding in a way that is separate
from Computer Science but it still is a way of
thinking that force these young learners
to adopt the very important part of what it means to
be a successful person in life and
hopefully in some technical field.
I [inaudible] on these questions because I
think that they're motivating me
in my current position and I think
they provide for a lot of questions,
a lot of difficulty maybe, but a lot of fun.
I appreciate your time and before I let you go,
I want to end with this idea of
these are exactly the kinds of questions that make
the work that we all wish to do or should be doing,
most certain that I do multi-disciplinary.
I've tried to pull out for you,
for I think are the most salient part
of this whole process of
combining computing across a lot of
different areas and I
hope you've enjoyed your journey with me.
I know it's been a long one, but I hope
that I get the opportunity to answer
any questions but talk with you more and
also about your own thoughts and ideas moving forward.
I thank you for your time.
[APPLAUSE]