WEBVTT 00:00:04.490 --> 00:00:06.510 I am now at, 00:00:06.510 --> 00:00:08.190 and I'll talk a bit more about this later. 00:00:08.190 --> 00:00:09.600 But I'm now at Smarter Balanced, 00:00:09.600 --> 00:00:12.340 which is affiliated with UCLA. 00:00:12.340 --> 00:00:15.525 It was one of two organizations that was funded in 00:00:15.525 --> 00:00:18.270 2010 by the US Department of Education 00:00:18.270 --> 00:00:20.820 to create computer adaptive assessments, 00:00:20.820 --> 00:00:23.400 which are geared towards 00:00:23.400 --> 00:00:26.385 testing students in 3rd-8th grade and 11th grade. 00:00:26.385 --> 00:00:28.110 What's really fascinating about 00:00:28.110 --> 00:00:29.250 this organization is that it 00:00:29.250 --> 00:00:30.390 gives us the opportunity 00:00:30.390 --> 00:00:31.740 to do some really innovative work. 00:00:31.740 --> 00:00:33.660 You might remember when you were in school, 00:00:33.660 --> 00:00:37.305 you use Scantrons once a year to take this tests, 00:00:37.305 --> 00:00:39.060 wasn't always fun, but 00:00:39.060 --> 00:00:40.740 unfortunately you can't get away from assessment. 00:00:40.740 --> 00:00:43.115 What we're trying to do now developed 00:00:43.115 --> 00:00:46.430 an automated system that adapts to our users, 00:00:46.430 --> 00:00:48.650 that allows the students to 00:00:48.650 --> 00:00:50.090 progress if they're being 00:00:50.090 --> 00:00:51.815 proficient and if they're struggling, 00:00:51.815 --> 00:00:54.710 then it provides other kinds of questions, 00:00:54.710 --> 00:00:56.990 and other kinds of content scenarios 00:00:56.990 --> 00:00:58.600 that allow them to succeed. 00:00:58.600 --> 00:01:00.690 It really cutting edge, 00:01:00.690 --> 00:01:02.225 this is as far as I'm concerned, 00:01:02.225 --> 00:01:04.415 because we don't have something like this right now, 00:01:04.415 --> 00:01:06.740 and we're currently in the testing phase. 00:01:06.740 --> 00:01:08.480 The member states there are 00:01:08.480 --> 00:01:09.950 deploying this test with their students, 00:01:09.950 --> 00:01:12.050 and we're actually obtaining data now. 00:01:12.050 --> 00:01:14.795 I'll talk more about what I see as future work. 00:01:14.795 --> 00:01:18.080 I just started this opportunity in January of this year, 00:01:18.080 --> 00:01:22.010 so I'm looking forward to a lot of applied research, 00:01:22.010 --> 00:01:23.780 and a lot of knowledge of how 00:01:23.780 --> 00:01:25.730 we might move forward in this phase of education. 00:01:25.730 --> 00:01:27.410 But before we get there, 00:01:27.410 --> 00:01:29.390 I want to spend some time talking 00:01:29.390 --> 00:01:31.655 about my journey effectively. 00:01:31.655 --> 00:01:35.510 I am honored to be considered and invited as 00:01:35.510 --> 00:01:39.585 a distinguished visitor, a distinguished lecture. 00:01:39.585 --> 00:01:41.540 I will be the first to concede that 00:01:41.540 --> 00:01:45.260 my story isn't that long, I graduated 2007. 00:01:45.260 --> 00:01:48.680 But perhaps that's just a testament to the fact that we 00:01:48.680 --> 00:01:52.280 want to encourage our lumps to do some great work. 00:01:52.280 --> 00:01:54.080 I'll consider myself in good company, 00:01:54.080 --> 00:01:56.635 even though I know I've got a lot more growing to do. 00:01:56.635 --> 00:01:59.960 I'd like to talk to you a little bit about my journey in 00:01:59.960 --> 00:02:02.030 the sense and the the umbrella 00:02:02.030 --> 00:02:04.610 of multidisciplinary computer science, 00:02:04.610 --> 00:02:07.430 my degree here wasn't Computer Science and I 00:02:07.430 --> 00:02:10.055 focus on human-computer interaction or HCI. 00:02:10.055 --> 00:02:12.320 How many of you are in or studying 00:02:12.320 --> 00:02:14.635 HCI and are familiar at least with this? 00:02:14.635 --> 00:02:16.985 Excellent, Perfect. I'll be 00:02:16.985 --> 00:02:19.225 singing your tune to a large extent. 00:02:19.225 --> 00:02:21.290 But others of you are in other areas, 00:02:21.290 --> 00:02:23.750 networking, software engineering, 00:02:23.750 --> 00:02:26.810 operating systems, all of which are important to 00:02:26.810 --> 00:02:28.340 this idea of how do we work 00:02:28.340 --> 00:02:30.815 together to solve problems within the day. 00:02:30.815 --> 00:02:33.310 That's what computer science does, problem solving. 00:02:33.310 --> 00:02:35.310 I argue that we solve 00:02:35.310 --> 00:02:38.795 the world's toughest challenges 00:02:38.795 --> 00:02:41.930 by working together across disciplines. 00:02:42.110 --> 00:02:44.595 I had this year, 00:02:44.595 --> 00:02:46.335 not just as an opportunity, 00:02:46.335 --> 00:02:50.025 to show you a picture, without word. 00:02:50.025 --> 00:02:53.480 But as we spend some time talking together, 00:02:53.480 --> 00:02:55.400 let's think a little bit about what it 00:02:55.400 --> 00:02:57.775 is you want for yourself? 00:02:57.775 --> 00:03:00.815 What image do you want to project in the world, 00:03:00.815 --> 00:03:02.405 but what do you want to be? 00:03:02.405 --> 00:03:04.430 What problems do you want to solve? 00:03:04.430 --> 00:03:06.035 How do you see yourself 00:03:06.035 --> 00:03:08.585 being that change you wish to see in the world? 00:03:08.585 --> 00:03:10.910 I think having that lens, 00:03:10.910 --> 00:03:12.350 trying to figure out, even if 00:03:12.350 --> 00:03:13.790 you don't quite figure it out, 00:03:13.790 --> 00:03:14.990 what you want to do when you grow up or 00:03:14.990 --> 00:03:16.685 at least what space you want to be in, 00:03:16.685 --> 00:03:18.170 helps you to think more broadly 00:03:18.170 --> 00:03:19.820 about the impact you can make in the world. 00:03:19.820 --> 00:03:21.710 At the end of the day, that's what we all want do. 00:03:21.710 --> 00:03:24.625 Or at least I hope we wanna do want to make some impact. 00:03:24.625 --> 00:03:28.455 I think having that as our motivation, 00:03:28.455 --> 00:03:32.860 can drive a lot of the work that we do. It has for me. 00:03:34.040 --> 00:03:36.530 What's really great about 00:03:36.530 --> 00:03:38.210 multidisciplinary computer science is that it 00:03:38.210 --> 00:03:40.910 combines the field of 00:03:40.910 --> 00:03:43.295 one discipline into the problems of another. 00:03:43.295 --> 00:03:46.760 I don't expect to this as a new idea for you, 00:03:46.760 --> 00:03:50.975 but at least in the context of how you can model 00:03:50.975 --> 00:03:55.250 your careers across a range of opportunity. 00:03:55.250 --> 00:03:57.350 It's really the focus here, at least that's the way I'm 00:03:57.350 --> 00:04:01.100 crafting my message this morning. 00:04:01.100 --> 00:04:05.135 The idea of being able to apply computer science and 00:04:05.135 --> 00:04:07.190 compete competing principles to 00:04:07.190 --> 00:04:09.955 a range of problems is not new. 00:04:09.955 --> 00:04:13.850 As a takeaway, I want you to again think about 00:04:13.850 --> 00:04:17.495 what impression you want to make as computer scientists. 00:04:17.495 --> 00:04:20.885 What are the most pressing issues 00:04:20.885 --> 00:04:25.060 that require a multidisciplinary approach? 00:04:25.060 --> 00:04:27.920 I argue that the bulk of 00:04:27.920 --> 00:04:32.375 those issues require a host of problem sets, 00:04:32.375 --> 00:04:34.760 and in tackling those problems set. 00:04:34.760 --> 00:04:38.420 I like this tag cloud because it's 00:04:38.420 --> 00:04:41.435 an example of computer science 00:04:41.435 --> 00:04:44.340 integrated with educational software design, 00:04:44.340 --> 00:04:47.000 whether or not you familiar with that area. 00:04:47.000 --> 00:04:52.880 It's a really great visualization of the many topics, 00:04:52.880 --> 00:04:58.870 the many areas that are part and parcel to this area. 00:04:58.870 --> 00:05:01.260 Looking at this cloud, 00:05:01.260 --> 00:05:04.985 and a number of you can see a handful of 00:05:04.985 --> 00:05:09.260 topics and disciplines that meet your own experiences. 00:05:09.260 --> 00:05:12.455 You can also see that there are areas that maybe don't, 00:05:12.455 --> 00:05:13.640 but that's the beauty of 00:05:13.640 --> 00:05:16.010 working in multidisciplinary teams. 00:05:16.010 --> 00:05:19.160 Having that opportunity to work across with 00:05:19.160 --> 00:05:20.555 your own skill sets are 00:05:20.555 --> 00:05:23.470 to solve problems along with other people. 00:05:23.470 --> 00:05:26.750 Again, I really like this because it allows for 00:05:26.750 --> 00:05:30.440 a really tangible example of how 00:05:30.440 --> 00:05:32.420 your experience is important when 00:05:32.420 --> 00:05:35.550 you're working on some really world has problems. 00:05:38.840 --> 00:05:42.880 Another area in which it really makes sense for us 00:05:42.880 --> 00:05:46.315 to think multidisciplinary as it relates to computing, 00:05:46.315 --> 00:05:49.610 relates to cyber security and cyber technology. 00:05:49.610 --> 00:05:51.580 More and more especially in 00:05:51.580 --> 00:05:53.860 the government we're looking for, 00:05:53.860 --> 00:05:56.425 or the government is looking for individuals who were 00:05:56.425 --> 00:06:00.850 skilled in the ability to counterattacks, 00:06:00.850 --> 00:06:02.845 to monitor our systems, 00:06:02.845 --> 00:06:04.314 to monitor our software, 00:06:04.314 --> 00:06:06.220 to ensure that we are protected. 00:06:06.220 --> 00:06:07.765 Or at least there's a level of protection, 00:06:07.765 --> 00:06:10.405 a barrier against malware 00:06:10.405 --> 00:06:12.910 or against those who would attack us and against 00:06:12.910 --> 00:06:14.785 those who would otherwise render us 00:06:14.785 --> 00:06:18.590 unable to use systems to complete our work. 00:06:18.590 --> 00:06:20.060 This quote actually from 00:06:20.060 --> 00:06:21.440 the Department of Homeland Security, 00:06:21.440 --> 00:06:25.040 describes the need for cybersecurity as follows, 00:06:25.040 --> 00:06:27.800 our daily life, economic vitality and 00:06:27.800 --> 00:06:30.950 national security depend on stable, 00:06:30.950 --> 00:06:33.430 safe and resilient cyberspace. 00:06:33.430 --> 00:06:35.390 We rely on this vast array of 00:06:35.390 --> 00:06:37.280 networks to communicate and travel, 00:06:37.280 --> 00:06:40.370 power our homes, run our economy, 00:06:40.370 --> 00:06:42.319 and provide government services, 00:06:42.319 --> 00:06:43.925 so that covers a lot. 00:06:43.925 --> 00:06:46.445 But to bring things into perspective, 00:06:46.445 --> 00:06:50.135 let's consider that cyberattacks 00:06:50.135 --> 00:06:52.385 are really a way of life these days. 00:06:52.385 --> 00:06:54.290 None of these should be a surprise to you, 00:06:54.290 --> 00:06:55.670 and might jog your memory about 00:06:55.670 --> 00:06:57.080 some of the most recent attacks we've 00:06:57.080 --> 00:07:00.640 experienced in terms of our technical systems. 00:07:00.640 --> 00:07:03.545 You might remember the Target breach 00:07:03.545 --> 00:07:07.240 last year that rendered a 148, 00:07:07.240 --> 00:07:10.640 I didn't realize it's a $148 million in damages. 00:07:10.640 --> 00:07:13.805 Whereas people lost money on their credit cards, 00:07:13.805 --> 00:07:18.965 lost revenue, lost resources in millions of dollars. 00:07:18.965 --> 00:07:20.195 Not to mention 00:07:20.195 --> 00:07:23.695 the 110 million consumers who were impacted. 00:07:23.695 --> 00:07:26.310 Who's going to be stopping these attacks? 00:07:26.310 --> 00:07:27.800 How do we figure 00:07:27.800 --> 00:07:29.960 out when these attacks are going to occur? 00:07:29.960 --> 00:07:32.090 How can we better arm ourself? 00:07:32.090 --> 00:07:36.840 But Home Depot, a similar breach, 00:07:36.840 --> 00:07:40.455 60 million cards stolen from customers, 00:07:40.455 --> 00:07:44.415 and will continue to see similar trends. 00:07:44.415 --> 00:07:48.290 It just amplifies the importance of us being able to 00:07:48.290 --> 00:07:50.180 understand how we meet in 00:07:50.180 --> 00:07:52.580 the middle of solving this problem. 00:07:52.580 --> 00:07:54.320 Of course, we can't forget, 00:07:54.320 --> 00:07:55.760 at least most of us can't forget 00:07:55.760 --> 00:07:59.110 that celebrity iCloud account that were hacked, 00:07:59.110 --> 00:08:00.590 and there were 00:08:00.590 --> 00:08:03.350 a lot too personal and private pictures that were 00:08:03.350 --> 00:08:05.660 shared to the children 00:08:05.660 --> 00:08:08.170 of the people whose accounts were affected. 00:08:08.170 --> 00:08:10.940 Thinking holistically, not just about 00:08:10.940 --> 00:08:13.820 being on the receiving end of these issue, 00:08:13.820 --> 00:08:16.340 but how do we combat them is really important. 00:08:16.340 --> 00:08:19.855 Let's pick a more broadly about what it takes, 00:08:19.855 --> 00:08:22.140 to be in cybersecurity. 00:08:22.140 --> 00:08:24.560 Just yell out for me an area, 00:08:24.560 --> 00:08:26.510 a discipline that you think is important 00:08:26.510 --> 00:08:29.105 for working in cybersecurity. 00:08:29.105 --> 00:08:33.420 A discipline, a scale, Just call them out for me, 00:08:34.480 --> 00:08:39.270 cryptography, [BACKGROUND] networking, 00:08:39.710 --> 00:08:43.650 psychology, absolutely. 00:08:43.650 --> 00:08:44.850 Others. 00:08:44.850 --> 00:08:50.575 [BACKGROUND] Natural language right about it that. 00:08:50.575 --> 00:08:53.525 I know, you're burning to share, 00:08:53.525 --> 00:08:55.700 this is the participation. 00:08:55.700 --> 00:08:57.065 Visualization. 00:08:57.065 --> 00:09:02.160 Oh, Visualization. Absolutely. Other non professors? 00:09:02.160 --> 00:09:07.700 [LAUGHTER] Its okay. There are no wrong answers, 00:09:07.700 --> 00:09:09.870 at least I will penalize you for them. 00:09:10.040 --> 00:09:12.390 Other ideas, other thoughts. 00:09:12.390 --> 00:09:18.950 Law. 00:09:18.950 --> 00:09:20.610 Law. Politics, that's a good one. 00:09:20.850 --> 00:09:25.280 Other topics, other disciplines, other skills? 00:09:25.800 --> 00:09:29.500 Two more and I'll stop. Two more willing souls 00:09:29.500 --> 00:09:31.510 [LAUGHTER] and we'll move forward. 00:09:31.510 --> 00:09:36.070 [NOISE] Physical security, that's a great one. 00:09:36.070 --> 00:09:40.645 One more. [NOISE] Education. 00:09:40.645 --> 00:09:41.830 Absolutely. I heard another one and I'm going 00:09:41.830 --> 00:09:43.810 to give a bonus [NOISE]. 00:09:43.810 --> 00:09:46.060 Sorry? [NOISE] Real-time systems. 00:09:46.060 --> 00:09:47.140 All really great answers, some of 00:09:47.140 --> 00:09:48.715 which I didn't think about. 00:09:48.715 --> 00:09:52.285 But the idea is, it takes a whole lot, 00:09:52.285 --> 00:09:56.320 including the ones that you listed that are not here. 00:09:56.320 --> 00:09:59.335 The idea is that we 00:09:59.335 --> 00:10:00.430 maintain a level of 00:10:00.430 --> 00:10:02.230 holistic thinking about what it means, 00:10:02.230 --> 00:10:03.895 not just to be a computer scientist, 00:10:03.895 --> 00:10:06.700 but to really solve important problems in the world. 00:10:06.700 --> 00:10:09.880 Less and less, we're able to do it in a silo. 00:10:09.880 --> 00:10:12.580 That's the point we're making here. 00:10:12.580 --> 00:10:15.670 Speaking of making a point that brings us to 00:10:15.670 --> 00:10:16.870 the focus of our talk because we want 00:10:16.870 --> 00:10:18.220 to build this together, 00:10:18.220 --> 00:10:19.660 and make it make sense in a way 00:10:19.660 --> 00:10:22.090 that ends in a pretty bow. 00:10:22.090 --> 00:10:24.265 We're building a case here, 00:10:24.265 --> 00:10:27.190 not just multidisciplinary computer science. 00:10:27.190 --> 00:10:28.780 But from my own experiences, 00:10:28.780 --> 00:10:32.890 I've tried to produce evidence-based research and 00:10:32.890 --> 00:10:36.340 evidence-based solutions for how 00:10:36.340 --> 00:10:39.805 we design and develop socio-technical systems. 00:10:39.805 --> 00:10:41.380 That's what I'm talking about today. 00:10:41.380 --> 00:10:43.000 I want to bring you along as 00:10:43.000 --> 00:10:45.190 I go back through memory lane, 00:10:45.190 --> 00:10:48.640 but really give you an idea of the kinds of 00:10:48.640 --> 00:10:50.725 work that really require 00:10:50.725 --> 00:10:53.590 cross collaboration across disciplines. 00:10:53.590 --> 00:10:57.819 Let's start with this idea of community, 00:10:57.819 --> 00:11:00.040 which is not something 00:11:00.040 --> 00:11:01.930 you think about when you think of computer science. 00:11:01.930 --> 00:11:03.580 But is where I want to start. 00:11:03.580 --> 00:11:06.685 So let's start with computing for community, 00:11:06.685 --> 00:11:09.745 specifically in empowering non-profit groups. 00:11:09.745 --> 00:11:12.865 When I left Virginia Tech in 2007, 00:11:12.865 --> 00:11:14.980 I was fortunate enough to 00:11:14.980 --> 00:11:17.695 participate or take a postdoc at Penn State, 00:11:17.695 --> 00:11:22.180 where I worked with John M. Carroll and Mary Beth Rosson, 00:11:22.180 --> 00:11:24.475 who were former Virginia Tech Professors. 00:11:24.475 --> 00:11:26.440 In many ways for me, it was full circle 00:11:26.440 --> 00:11:28.165 because when I started at Virginia Tech, 00:11:28.165 --> 00:11:30.370 Mary Beth Rosson was my advisor. 00:11:30.370 --> 00:11:34.750 I had the great fortune to go work in state college. 00:11:34.750 --> 00:11:38.020 My goal over the course of my postdoc was to study 00:11:38.020 --> 00:11:40.030 non-profits but in the context 00:11:40.030 --> 00:11:42.025 of community wireless networks. 00:11:42.025 --> 00:11:44.095 You may not remember or 00:11:44.095 --> 00:11:46.120 know much about community wireless networks. 00:11:46.120 --> 00:11:47.725 But at the time, they were a thing. 00:11:47.725 --> 00:11:49.420 They were really popular. 00:11:49.420 --> 00:11:51.820 But there were inherent issues with 00:11:51.820 --> 00:11:54.760 maintaining these networks because they were cost issues. 00:11:54.760 --> 00:11:58.180 How does the state sustain this kind of connectivity? 00:11:58.180 --> 00:12:01.235 Are there broadband issues that they have to pay for? 00:12:01.235 --> 00:12:03.360 At the time, there was 00:12:03.360 --> 00:12:05.775 talk that these wireless networks would be free. 00:12:05.775 --> 00:12:07.935 Again, who's going to bear the cost of that? 00:12:07.935 --> 00:12:09.210 What kind of infrastructure are we going 00:12:09.210 --> 00:12:10.410 to be able to build to 00:12:10.410 --> 00:12:13.870 support this kind of interconnectivity? 00:12:15.810 --> 00:12:19.420 Before we even get to the issue of 00:12:19.420 --> 00:12:20.890 how we actually approach 00:12:20.890 --> 00:12:22.450 these non-profits to do the work, 00:12:22.450 --> 00:12:24.730 let's take a step back and think 00:12:24.730 --> 00:12:27.415 about what it means to be part of the community, 00:12:27.415 --> 00:12:29.185 what it means to believe 00:12:29.185 --> 00:12:31.555 that you're contributing to your community. 00:12:31.555 --> 00:12:35.350 I focus a lot on Albert Bandura, 00:12:35.350 --> 00:12:38.140 who is a socio- psychologist, 00:12:38.140 --> 00:12:42.265 who describes this idea of self-efficacy. 00:12:42.265 --> 00:12:46.300 Which effectively allows us to believe, 00:12:46.300 --> 00:12:47.650 not just believe, but to 00:12:47.650 --> 00:12:49.675 know that we can accomplish something. 00:12:49.675 --> 00:12:52.825 If you think about when you were a kid, 00:12:52.825 --> 00:12:54.430 when you were younger, at least for me. 00:12:54.430 --> 00:12:55.870 I grew up believing, 00:12:55.870 --> 00:12:58.945 if you can believe it, you can achieve it. 00:12:58.945 --> 00:13:01.570 As a converse to that, 00:13:01.570 --> 00:13:03.040 you think of The Little Engine That Could. " 00:13:03.040 --> 00:13:05.170 I think I can, I think I can." 00:13:05.170 --> 00:13:08.770 In any case, they're not necessarily the same. 00:13:08.770 --> 00:13:10.030 It may sound the same. 00:13:10.030 --> 00:13:11.530 But if you think about it, 00:13:11.530 --> 00:13:13.900 "To believe you can achieve it, 00:13:13.900 --> 00:13:15.490 then you can achieve it," is 00:13:15.490 --> 00:13:17.410 actually based in epistemology, 00:13:17.410 --> 00:13:21.415 which is the theory of knowledge, 00:13:21.415 --> 00:13:22.780 which is based in truth. 00:13:22.780 --> 00:13:24.550 Often, when we talk about self-efficacy, 00:13:24.550 --> 00:13:25.900 we're talking about having 00:13:25.900 --> 00:13:26.980 experienced something or having 00:13:26.980 --> 00:13:28.480 accomplished something in the past, 00:13:28.480 --> 00:13:30.145 and because you've accomplished it, 00:13:30.145 --> 00:13:31.555 you know that you can do it again. 00:13:31.555 --> 00:13:32.980 Or you know you can do something similar. 00:13:32.980 --> 00:13:35.560 Or you know you can develop or grow in 00:13:35.560 --> 00:13:36.910 a similar performance task or 00:13:36.910 --> 00:13:40.075 some activity that allows you to feel empowered, 00:13:40.075 --> 00:13:41.695 because you've got history on your side. 00:13:41.695 --> 00:13:43.840 That was very different from saying, 00:13:43.840 --> 00:13:44.980 "Okay, I've never done this before, 00:13:44.980 --> 00:13:46.420 I wonder if I can do it." 00:13:46.420 --> 00:13:48.070 No, it's believing that you can. 00:13:48.070 --> 00:13:49.794 That's the whole point of self-efficacy, 00:13:49.794 --> 00:13:51.610 and to think of self-efficacy in 00:13:51.610 --> 00:13:54.265 the sense of communities. 00:13:54.265 --> 00:13:56.365 Well, there's also the construct 00:13:56.365 --> 00:13:59.080 of social cognitive theory, 00:13:59.080 --> 00:14:01.120 which is this idea of human agency, 00:14:01.120 --> 00:14:03.100 but human agency that extends 00:14:03.100 --> 00:14:05.380 personal self-efficacy and takes 00:14:05.380 --> 00:14:07.625 us into what we call proxy. 00:14:07.625 --> 00:14:12.525 If I tell someone else to do the work 00:14:12.525 --> 00:14:14.100 or ensure that they 00:14:14.100 --> 00:14:17.865 perform some task that I know is going to get done, 00:14:17.865 --> 00:14:20.970 so I absolve myself of performing those activities. 00:14:20.970 --> 00:14:22.635 But you know it's going to get done. 00:14:22.635 --> 00:14:24.840 Then the third area, which is again, 00:14:24.840 --> 00:14:26.565 is part of the story that I'm building, 00:14:26.565 --> 00:14:30.105 is this idea of collective efficacy. 00:14:30.105 --> 00:14:32.915 As a community, we can make this happen. 00:14:32.915 --> 00:14:35.395 You and I are in this together. 00:14:35.395 --> 00:14:37.840 That's really what we focused on for this idea 00:14:37.840 --> 00:14:40.525 of community collective efficacy, 00:14:40.525 --> 00:14:43.510 which is an idea that we termed and worked for, 00:14:43.510 --> 00:14:47.695 that we believed would allow us to empower non-profit, 00:14:47.695 --> 00:14:49.809 empower the local community 00:14:49.809 --> 00:14:52.750 to be the change they wish to see in their community, 00:14:52.750 --> 00:14:55.465 using only community wireless networks. 00:14:55.465 --> 00:14:58.600 It seemed a bit far-fetched and it 00:14:58.600 --> 00:14:59.980 required a lot of discussion 00:14:59.980 --> 00:15:01.495 with organizations, with groups. 00:15:01.495 --> 00:15:02.980 I had lots of meetings, 00:15:02.980 --> 00:15:04.705 and interviews and research. 00:15:04.705 --> 00:15:08.230 Meetings with organizations like United Way, 00:15:08.230 --> 00:15:09.880 Girls & Boys Club, 00:15:09.880 --> 00:15:11.770 even local food banks, 00:15:11.770 --> 00:15:14.980 with the idea of empowering 00:15:14.980 --> 00:15:18.235 them to be leaders in their communities. 00:15:18.235 --> 00:15:21.820 Now, non-profits don't have the most resources. 00:15:21.820 --> 00:15:23.290 They're often understaffed. 00:15:23.290 --> 00:15:25.150 They don't have a lot of funding. 00:15:25.150 --> 00:15:27.040 But they have a whole list of 00:15:27.040 --> 00:15:29.530 constituents who need a lot of support. 00:15:29.530 --> 00:15:33.130 But we believed that providing an opportunity for them to 00:15:33.130 --> 00:15:36.400 be proponents of a system that would be free, 00:15:36.400 --> 00:15:38.860 that is a wireless network that would be free, 00:15:38.860 --> 00:15:40.240 would enable them to not 00:15:40.240 --> 00:15:44.110 just accomplish their bottom line, 00:15:44.110 --> 00:15:46.195 but to be leaders in the community, 00:15:46.195 --> 00:15:48.955 to encourage and empower their constituents 00:15:48.955 --> 00:15:50.530 to be the change they wish to see in 00:15:50.530 --> 00:15:52.105 the world, to help themselves. 00:15:52.105 --> 00:15:54.940 The idea is if we develop 00:15:54.940 --> 00:15:58.645 a sense of community among the non-profits, 00:15:58.645 --> 00:16:01.390 then we were leaning on this idea 00:16:01.390 --> 00:16:03.700 of communities want to help 00:16:03.700 --> 00:16:07.330 themselves get better because 00:16:07.330 --> 00:16:09.010 they are all in this together. 00:16:09.010 --> 00:16:11.755 It sounds touchy-feely. 00:16:11.755 --> 00:16:13.120 But when you're dealing with people, 00:16:13.120 --> 00:16:15.610 when you're dealing with socio-technical systems, 00:16:15.610 --> 00:16:18.010 that's the problem we're solving. 00:16:18.010 --> 00:16:22.209 How do we integrate these highly technical interactions 00:16:22.209 --> 00:16:23.740 and highly technical systems, 00:16:23.740 --> 00:16:26.695 and couple those with the people who use them? 00:16:26.695 --> 00:16:28.330 At the end of the day, 00:16:28.330 --> 00:16:32.275 the common denominator is the user, is the individual. 00:16:32.275 --> 00:16:34.360 That's been the course of a lot of my work. 00:16:34.360 --> 00:16:36.880 My work focuses on user research. 00:16:36.880 --> 00:16:39.205 In this stage of my life, 00:16:39.205 --> 00:16:41.530 it was highly pure research 00:16:41.530 --> 00:16:42.700 to the extent that we wanted to 00:16:42.700 --> 00:16:44.050 really talk with individuals, 00:16:44.050 --> 00:16:47.200 talk with groups, do behavioral studies, 00:16:47.200 --> 00:16:50.140 do user studies to really understand the impact of, 00:16:50.140 --> 00:16:52.075 first of all, what their current state of the art is, 00:16:52.075 --> 00:16:54.475 because as you'll learn if you have it already, 00:16:54.475 --> 00:16:56.500 it's hard to make change and create 00:16:56.500 --> 00:16:57.520 change if you don't 00:16:57.520 --> 00:16:59.680 really understand socio-technical change. 00:16:59.680 --> 00:17:01.825 If you don't understand the current landscape. 00:17:01.825 --> 00:17:03.880 If you don't understand why 00:17:03.880 --> 00:17:05.740 people use the system that they use, 00:17:05.740 --> 00:17:08.245 or why they don't use the system that they use. 00:17:08.245 --> 00:17:10.420 That's the first part of this process, 00:17:10.420 --> 00:17:14.590 to indulge in user research to understand the impact. 00:17:14.590 --> 00:17:16.990 Now again, user research is 00:17:16.990 --> 00:17:20.290 an opportunity to focus on the user and to impact design. 00:17:20.290 --> 00:17:22.945 It's been a large part 00:17:22.945 --> 00:17:25.570 of what it takes in 00:17:25.570 --> 00:17:26.950 my mind to really develop 00:17:26.950 --> 00:17:28.855 software that meets students where they are, 00:17:28.855 --> 00:17:30.685 or meets users where they are. 00:17:30.685 --> 00:17:34.825 We followed this suit. In so doing, 00:17:34.825 --> 00:17:36.565 the idea was to make 00:17:36.565 --> 00:17:40.134 these community groups civic contributors, 00:17:40.134 --> 00:17:42.640 make them part of the local government. 00:17:42.640 --> 00:17:45.115 Ensure that they were able to have that voice, 00:17:45.115 --> 00:17:46.600 enable them to have 00:17:46.600 --> 00:17:49.570 that voice because now they were able to connect. 00:17:49.570 --> 00:17:51.010 Now, there were other issues. 00:17:51.010 --> 00:17:54.550 If you followed CWNs or community wireless networks, 00:17:54.550 --> 00:17:56.860 or even just look them up during the course of this talk, 00:17:56.860 --> 00:17:58.600 you'll find that they're virtually non-existent 00:17:58.600 --> 00:18:01.570 now because of issues that we talked about, resources. 00:18:01.570 --> 00:18:03.100 But on the flip side of that, even 00:18:03.100 --> 00:18:05.290 in empowering these non-profits, 00:18:05.290 --> 00:18:07.225 they had infrastructure issues. 00:18:07.225 --> 00:18:11.810 Even if they wanted to get online more strategically, 00:18:11.810 --> 00:18:13.470 they weren't always able. 00:18:13.470 --> 00:18:15.885 There was one organization that I worked with, 00:18:15.885 --> 00:18:21.440 that was housed in a cement embossed building. 00:18:21.440 --> 00:18:23.620 Even the cellphones wouldn't work. 00:18:23.620 --> 00:18:27.215 But the building was accessible. 00:18:27.215 --> 00:18:29.445 The group had been there for years. 00:18:29.445 --> 00:18:31.035 They didn't see themselves leaving. 00:18:31.035 --> 00:18:33.300 In order to connect them to, again, 00:18:33.300 --> 00:18:35.310 this structure that was soon to be, 00:18:35.310 --> 00:18:37.530 that didn't yet exist, we had to talk 00:18:37.530 --> 00:18:40.310 about infrastructure changes, and that was an expense. 00:18:40.310 --> 00:18:42.594 That's part of the challenge 00:18:42.594 --> 00:18:45.950 in encouraging users to adopt new technologies. 00:18:45.950 --> 00:18:47.210 But it also falls along 00:18:47.210 --> 00:18:49.140 the lines of those things we don't always expect. 00:18:49.140 --> 00:18:50.950 Yeah, we'd like to all be connected, 00:18:50.950 --> 00:18:53.000 we'd like to provide this kind of exposure, 00:18:53.000 --> 00:18:55.010 but it isn't always possible because of 00:18:55.010 --> 00:18:58.320 the limitations that we just didn't account for. 00:18:59.640 --> 00:19:02.260 I'll talk little bit about this in terms of 00:19:02.260 --> 00:19:04.825 the tool that I use, 00:19:04.825 --> 00:19:07.570 and others, maybe some of you use in terms of really 00:19:07.570 --> 00:19:10.075 determining what the actual needs are. 00:19:10.075 --> 00:19:11.290 If you're in software engineering, you 00:19:11.290 --> 00:19:12.745 might call this needs analysis, 00:19:12.745 --> 00:19:13.990 if you're in HCI, 00:19:13.990 --> 00:19:15.790 you might call this usability engineering. 00:19:15.790 --> 00:19:18.100 But it goes a step further because it doesn't 00:19:18.100 --> 00:19:20.860 assume that we know what the users need, 00:19:20.860 --> 00:19:22.210 and that's, by definition, 00:19:22.210 --> 00:19:23.695 the purpose of user research. 00:19:23.695 --> 00:19:26.230 We actually do the work called attitudinal. 00:19:26.230 --> 00:19:28.480 What are the attitudes that users 00:19:28.480 --> 00:19:30.820 have about the technology itself, 00:19:30.820 --> 00:19:32.590 about the interaction, about 00:19:32.590 --> 00:19:34.930 the opportunities that are available to them, 00:19:34.930 --> 00:19:36.700 but also what are the behaviors? 00:19:36.700 --> 00:19:39.175 How do they actually use the current state of the art? 00:19:39.175 --> 00:19:40.885 How do they think they'll use 00:19:40.885 --> 00:19:43.195 the new systems when they become available? 00:19:43.195 --> 00:19:46.959 So it requires a multilevel approach 00:19:46.959 --> 00:19:48.970 to really understanding how 00:19:48.970 --> 00:19:50.125 to meet the user where they are. 00:19:50.125 --> 00:19:51.400 I'm going to say that a lot, 00:19:51.400 --> 00:19:52.810 probably because it's ingrained in me. 00:19:52.810 --> 00:19:53.920 I work with a lot of users. 00:19:53.920 --> 00:19:55.539 I've worked with a lot of different individuals 00:19:55.539 --> 00:19:57.400 who represent different sets of users. 00:19:57.400 --> 00:19:58.750 But, at the end of the day, 00:19:58.750 --> 00:20:00.100 that's lesson number 1. 00:20:00.100 --> 00:20:03.400 The user is still at the center of any innovation. 00:20:03.400 --> 00:20:05.095 You can create the next killer app, 00:20:05.095 --> 00:20:07.235 but if nobody ever uses it, 00:20:07.235 --> 00:20:09.920 it's of no consequence. 00:20:09.920 --> 00:20:12.400 So that's the really important part of this. 00:20:12.400 --> 00:20:13.840 Again, it will be 00:20:13.840 --> 00:20:14.980 strange to hear your computer scientist 00:20:14.980 --> 00:20:15.790 talking to like this. 00:20:15.790 --> 00:20:19.765 I remember when I was getting my degree, I worried. 00:20:19.765 --> 00:20:23.155 I'm a computer scientist. Is it okay for me to learn 00:20:23.155 --> 00:20:25.465 sociology and anthropology and 00:20:25.465 --> 00:20:27.820 the psychology of technology, 00:20:27.820 --> 00:20:30.190 in the sense that that's not what I 00:20:30.190 --> 00:20:33.505 thought a typical computer scientist did. 00:20:33.505 --> 00:20:35.110 But I was gratified because 00:20:35.110 --> 00:20:37.555 computer scientists problem solve, 00:20:37.555 --> 00:20:39.310 and what I really appreciated about 00:20:39.310 --> 00:20:41.710 my degree experience is that it 00:20:41.710 --> 00:20:43.180 allowed me to see what 00:20:43.180 --> 00:20:46.045 those different avenues were for a computer scientist. 00:20:46.045 --> 00:20:47.800 Now, it didn't mean that I didn't do 00:20:47.800 --> 00:20:50.290 like to share coding and development, 00:20:50.290 --> 00:20:51.880 which is part of the game, 00:20:51.880 --> 00:20:54.610 if you will, but to take it to the next level, 00:20:54.610 --> 00:20:56.770 I always wanted to be the person who was telling 00:20:56.770 --> 00:20:59.560 everybody else what to develop, what to design. 00:20:59.560 --> 00:21:01.780 So maybe I'm living the dream, 00:21:01.780 --> 00:21:05.200 but the idea is to decide what it is 00:21:05.200 --> 00:21:08.215 you want to accomplish and make it happen, 00:21:08.215 --> 00:21:11.050 so user research is an important part 00:21:11.050 --> 00:21:13.840 of what is in the arsenal. 00:21:13.840 --> 00:21:17.500 This graphic actually is from Nielsen/Norman Group, 00:21:17.500 --> 00:21:18.835 who many of you may be aware of, 00:21:18.835 --> 00:21:22.180 a forefront in user research, 00:21:22.180 --> 00:21:25.165 so I always find this to be a graphic, 00:21:25.165 --> 00:21:27.565 handy reminder of the tools that are available, 00:21:27.565 --> 00:21:29.200 as we think about the kinds of designs that are 00:21:29.200 --> 00:21:31.810 available to us as scientists. 00:21:31.810 --> 00:21:35.800 Here's an example of some of the designs 00:21:35.800 --> 00:21:39.100 that we created in terms of our work with our groups. 00:21:39.100 --> 00:21:40.930 One of the groups 00:21:40.930 --> 00:21:42.850 that we're working with was an autistic group. 00:21:42.850 --> 00:21:46.120 In fact, we worked with the behavior therapist who had 00:21:46.120 --> 00:21:50.410 clients whose kids struggled 00:21:50.410 --> 00:21:52.930 with autism and struggled with being 00:21:52.930 --> 00:21:55.420 able to be out in the world because 00:21:55.420 --> 00:21:57.130 there were certain behavioral issues 00:21:57.130 --> 00:21:59.470 that kept them from being able to go 00:21:59.470 --> 00:22:01.720 out and experience life or 00:22:01.720 --> 00:22:04.780 just barriers to social connectivity. 00:22:04.780 --> 00:22:06.910 What we decided to do was to 00:22:06.910 --> 00:22:09.025 talk with this behavior therapist, 00:22:09.025 --> 00:22:10.840 understand exactly what he wanted to 00:22:10.840 --> 00:22:12.745 accomplish from a technical system, 00:22:12.745 --> 00:22:16.975 and we designed this network community 00:22:16.975 --> 00:22:19.810 that had this give-and-take between 00:22:19.810 --> 00:22:23.695 families who were dealing with autism 00:22:23.695 --> 00:22:29.125 to students who are autistic themselves at Penn State, 00:22:29.125 --> 00:22:33.595 and it allowed for an exchange of experiences. 00:22:33.595 --> 00:22:36.820 Now, this is different from any other online or 00:22:36.820 --> 00:22:38.650 other network community in 00:22:38.650 --> 00:22:40.225 that there was this give-and-take, 00:22:40.225 --> 00:22:41.830 and there was this sharing of 00:22:41.830 --> 00:22:43.375 a very personal experience 00:22:43.375 --> 00:22:45.610 that you don't always see in communities. 00:22:45.610 --> 00:22:49.090 Autistic students at Penn State or any place are very 00:22:49.090 --> 00:22:50.740 unique and often have 00:22:50.740 --> 00:22:53.860 challenges that are unlike any other students', 00:22:53.860 --> 00:22:55.930 but what it ended up being was 00:22:55.930 --> 00:22:58.420 an opportunity for students and 00:22:58.420 --> 00:23:01.510 kids and their parents to see what 00:23:01.510 --> 00:23:04.600 was possible for those students, and vice versa. 00:23:04.600 --> 00:23:07.615 There was this opportunity to mentor the students, 00:23:07.615 --> 00:23:10.120 and because students and individuals on 00:23:10.120 --> 00:23:11.860 the autistic spectrum struggle 00:23:11.860 --> 00:23:13.794 with social kinds of behaviors, 00:23:13.794 --> 00:23:17.635 it was an amazing opportunity to see them in action, 00:23:17.635 --> 00:23:19.885 to have that opportunity to interact, 00:23:19.885 --> 00:23:21.400 grow themselves, and develop 00:23:21.400 --> 00:23:24.440 themselves in a way that doesn't come naturally. 00:23:24.440 --> 00:23:29.220 Again, this design afforded the opportunity, 00:23:29.220 --> 00:23:31.110 at least we wished, to enable 00:23:31.110 --> 00:23:34.720 that interaction that doesn't always come naturally. 00:23:37.260 --> 00:23:40.300 Moving on to the world 00:23:40.300 --> 00:23:43.030 of computing and multidisciplinary work, 00:23:43.030 --> 00:23:44.620 I'd like to talk a little bit about 00:23:44.620 --> 00:23:46.435 the work I did during my time at DARPA. 00:23:46.435 --> 00:23:48.545 I was not a program manager, 00:23:48.545 --> 00:23:52.050 but I was recruited to work with, I think, Capt. 00:23:52.050 --> 00:23:54.585 Russell Shilling, who was the Program Manager. 00:23:54.585 --> 00:23:56.145 I remember, very clearly, 00:23:56.145 --> 00:23:57.860 the day that we interviewed each other, 00:23:57.860 --> 00:23:59.380 and I say that lovingly because, 00:23:59.380 --> 00:24:00.370 not only was he my colleague, 00:24:00.370 --> 00:24:02.620 he's now my friend, and he's actually now 00:24:02.620 --> 00:24:04.345 the Executive Director of 00:24:04.345 --> 00:24:07.300 STEM at the US Department of Education. 00:24:07.300 --> 00:24:11.050 The opportunity of working with him allowed me to try to, 00:24:11.050 --> 00:24:14.035 not just see how work was done in the military, 00:24:14.035 --> 00:24:15.670 how it's done in the DOD, 00:24:15.670 --> 00:24:17.590 but when we interviewed each other, 00:24:17.590 --> 00:24:21.235 he said he wanted me to be his second hit, 00:24:21.235 --> 00:24:24.160 his second brain, which for me was 00:24:24.160 --> 00:24:25.930 amazing because it meant that I'd be 00:24:25.930 --> 00:24:27.790 able to work really closely with this person. 00:24:27.790 --> 00:24:29.830 He called the shots, obviously, 00:24:29.830 --> 00:24:32.110 but to have that opportunity was not something that you 00:24:32.110 --> 00:24:34.525 see a lot and that you get a lot of opportunity doing, 00:24:34.525 --> 00:24:37.945 so it meant a lot to be able to have that relationship. 00:24:37.945 --> 00:24:40.390 But one of the programs that we 00:24:40.390 --> 00:24:43.720 worked on was called the DCAPS program, 00:24:43.720 --> 00:24:45.460 and if you agree with anything military, 00:24:45.460 --> 00:24:47.170 you know that we use acronyms. 00:24:47.170 --> 00:24:48.850 The DCAPS program stands for 00:24:48.850 --> 00:24:50.710 the Detection and Computation 00:24:50.710 --> 00:24:54.325 Analysis of Psychological Signals, DCAPS. 00:24:54.325 --> 00:24:57.205 The idea of the program was to develop 00:24:57.205 --> 00:24:59.290 technology that allowed us 00:24:59.290 --> 00:25:02.079 to monitor psychological distress. 00:25:02.079 --> 00:25:04.600 Now, you might know, 00:25:04.600 --> 00:25:06.025 but I'll remind you, 00:25:06.025 --> 00:25:08.500 that we have military veterans 00:25:08.500 --> 00:25:10.780 who would come back from Iran and 00:25:10.780 --> 00:25:14.680 Afghanistan experiencing episodes of 00:25:14.680 --> 00:25:17.845 PTSD at almost 25 percent at the most, 00:25:17.845 --> 00:25:21.760 and suicide rates were not that much better. 00:25:21.760 --> 00:25:23.830 So this program was an opportunity for us 00:25:23.830 --> 00:25:26.035 to do something about that. 00:25:26.035 --> 00:25:28.180 The idea was, again, 00:25:28.180 --> 00:25:31.450 to merge computer science with 00:25:31.450 --> 00:25:35.665 psychology with development and 00:25:35.665 --> 00:25:39.910 medical health and mental health to devise 00:25:39.910 --> 00:25:41.785 solutions that would allow us 00:25:41.785 --> 00:25:45.115 to support our military veterans. 00:25:45.115 --> 00:25:50.470 So what did that mean? This is DARPA again. 00:25:50.470 --> 00:25:53.440 So there had to be something driving the research. 00:25:53.440 --> 00:25:56.335 We decided to use what were called honest signals, 00:25:56.335 --> 00:26:00.340 which is an idea originally fronted by Sandy Pentland, 00:26:00.340 --> 00:26:03.730 who is a professor at MIT Media Labs, 00:26:03.730 --> 00:26:06.490 and what these honest signals mean is that there are 00:26:06.490 --> 00:26:11.335 certain behavioral cues that are obvious, 00:26:11.335 --> 00:26:14.605 but you may not know them by looking at them. 00:26:14.605 --> 00:26:17.005 For example, if I'm depressed, 00:26:17.005 --> 00:26:20.050 I'm going to exhibit non-verbal cues 00:26:20.050 --> 00:26:23.980 that you may not recognize as my being distressed, 00:26:23.980 --> 00:26:25.660 and what we're able to find is, 00:26:25.660 --> 00:26:27.715 in the course of developing these technologies, 00:26:27.715 --> 00:26:30.475 there were a host of these kinds of 00:26:30.475 --> 00:26:34.390 behavioral and non-verbal cues that 00:26:34.390 --> 00:26:37.735 allowed us to determine not necessarily that 00:26:37.735 --> 00:26:39.580 someone was depressed or 00:26:39.580 --> 00:26:41.785 that someone was suffering from PTSD, 00:26:41.785 --> 00:26:45.070 but perhaps that person needed to go see 00:26:45.070 --> 00:26:49.135 a clinician or monitor their personal health. 00:26:49.135 --> 00:26:52.180 I want to say very clearly that 00:26:52.180 --> 00:26:55.765 a paramount part of this program was privacy, 00:26:55.765 --> 00:26:57.745 so if we're talking about 00:26:57.745 --> 00:27:01.945 devices that can monitor your behavioral health, 00:27:01.945 --> 00:27:03.880 anything about you, we're 00:27:03.880 --> 00:27:05.695 talking about personally identifiable information. 00:27:05.695 --> 00:27:08.215 We're talking about those cues 00:27:08.215 --> 00:27:09.550 that you probably wouldn't 00:27:09.550 --> 00:27:11.890 want the average person to get a hold of, 00:27:11.890 --> 00:27:15.100 so aside from the fact that these systems were 00:27:15.100 --> 00:27:18.205 very secure, users opted in. 00:27:18.205 --> 00:27:21.430 In fact, a very important part of 00:27:21.430 --> 00:27:23.830 this project was ensuring that 00:27:23.830 --> 00:27:26.860 we follow the strictest IRB protocol, 00:27:26.860 --> 00:27:28.510 so I want to assure 00:27:28.510 --> 00:27:31.180 you that that was part of the project. 00:27:31.180 --> 00:27:33.070 But more generally, we weren't talking 00:27:33.070 --> 00:27:35.470 about making a diagnosis for anybody. 00:27:35.470 --> 00:27:37.105 We're not talking Doc-in-a-box, 00:27:37.105 --> 00:27:40.520 which one of our researchers would often say, 00:27:40.520 --> 00:27:43.305 but we're talking about providing support. 00:27:43.305 --> 00:27:46.560 I'm going to show you a video that one of 00:27:46.560 --> 00:27:49.980 our developer teams created that showcases the research. 00:27:49.980 --> 00:27:51.765 We'll talk about it afterwards, 00:27:51.765 --> 00:27:58.840 but the idea is that there's a host of opportunity and 00:27:58.840 --> 00:28:02.080 tools that you'll see when you see the video 00:28:02.080 --> 00:28:04.030 that dictate the method 00:28:04.030 --> 00:28:06.505 of problem-solving that we were going for. 00:28:06.505 --> 00:28:08.710 I also want to say that 00:28:08.710 --> 00:28:11.830 the person in the video is not an actual patient, 00:28:11.830 --> 00:28:14.740 he's an actor, so that's an important distinction. 00:28:14.740 --> 00:28:16.780 I'll also call out that 00:28:16.780 --> 00:28:19.315 the work that I'm showing was led by 00:28:19.315 --> 00:28:25.660 Skip Rizzo and Louis-Phillipe Morency who are at USC. 00:28:25.660 --> 00:28:28.420 In fact, Louis-Phillipe is now at Carnegie Mellon. 00:28:28.420 --> 00:28:30.055 I'll like to show you 00:28:30.055 --> 00:28:33.670 a clip of the system that they created, 00:28:33.670 --> 00:28:35.575 which is an avatar-based system. 00:28:35.575 --> 00:28:39.190 The idea of this particular system is that it's able to 00:28:39.190 --> 00:28:43.795 read behavior cues on an individual sitting right across, 00:28:43.795 --> 00:28:46.765 and it did that by training itself, 00:28:46.765 --> 00:28:50.289 based on many real life interviews 00:28:50.289 --> 00:28:53.065 that individuals had that were videotaped, 00:28:53.065 --> 00:28:55.750 and based on those video analysis, 00:28:55.750 --> 00:29:01.480 we were able to compile a list of queues that allowed 00:29:01.480 --> 00:29:04.060 us to know what to look for when we were looking for 00:29:04.060 --> 00:29:08.155 depression or emotional distress. 00:29:08.155 --> 00:29:12.790 Keep that in the back of your mind as you watch this. 00:29:12.790 --> 00:29:16.260 I'm going to try to elegantly pause 00:29:16.260 --> 00:29:19.720 this and move to the video. 00:29:19.720 --> 00:29:53.095 [NOISE] 00:29:53.095 --> 00:29:55.225 Hi, I'm Ellie. 00:29:55.225 --> 00:29:57.010 Thanks for coming in today. 00:29:57.010 --> 00:29:59.050 I was created to talk to people in 00:29:59.050 --> 00:30:01.195 a safe and secure environment. 00:30:01.195 --> 00:30:02.680 I'm not a therapist, 00:30:02.680 --> 00:30:04.180 but I'm here to learn about people 00:30:04.180 --> 00:30:05.815 and would love to learn about you. 00:30:05.815 --> 00:30:07.600 I'll ask a few questions to get us 00:30:07.600 --> 00:30:11.005 started and please feel free to tell me anything. 00:30:11.005 --> 00:30:13.705 Your answers are totally confidential. 00:30:13.705 --> 00:30:15.980 Are you okay with this? 00:30:15.990 --> 00:30:18.295 Yes. 00:30:18.295 --> 00:30:21.350 How are you doing today? 00:30:21.390 --> 00:30:23.515 I'm doing well. 00:30:23.515 --> 00:30:29.270 That's good. Where are you from originally? 00:30:29.280 --> 00:30:32.065 I'm from Los Angeles. 00:30:32.065 --> 00:30:34.940 I'm from LA myself. 00:30:35.250 --> 00:30:39.920 When was the last time you felt really happy? 00:30:42.090 --> 00:30:50.990 When was the last time? I don't know. 00:30:53.970 --> 00:30:58.195 I don't have any real high highs. 00:30:58.195 --> 00:31:05.510 I feel like I'm a level person, it's just happy. 00:31:06.990 --> 00:31:10.700 I guess to answer your question, 00:31:13.680 --> 00:31:17.450 probably a couple months ago. 00:31:17.640 --> 00:31:21.220 I noticed you were hesitant on that one. 00:31:21.220 --> 00:31:24.910 Would do you say you are generally a happy person? 00:31:24.910 --> 00:31:30.730 I'm generally happy, just lately 00:31:30.730 --> 00:31:40.040 there are things that's keeping me down. 00:31:40.800 --> 00:31:44.210 Can you tell me more about that? 00:31:44.970 --> 00:31:50.440 What was the last time? 00:31:50.440 --> 00:31:52.870 You can actually do this on YouTube, 00:31:52.870 --> 00:31:55.435 but you notice, hopefully, you notice a few things, 00:31:55.435 --> 00:32:00.265 the least of which is displayed across the person's face. 00:32:00.265 --> 00:32:03.295 The team used the connect system to actually model 00:32:03.295 --> 00:32:06.925 how the individual appeared on the screen. 00:32:06.925 --> 00:32:09.670 You'll also notice that there are the other cues, 00:32:09.670 --> 00:32:11.380 the other parts of the interaction that are being 00:32:11.380 --> 00:32:13.465 measured and analyzed in real-time. 00:32:13.465 --> 00:32:15.415 By the way, after this interaction happens, 00:32:15.415 --> 00:32:17.305 the data are eliminated. 00:32:17.305 --> 00:32:19.480 There's no storage of these data. 00:32:19.480 --> 00:32:21.850 But what you're noticing at the top right of 00:32:21.850 --> 00:32:24.130 the black screen here, this smile level. 00:32:24.130 --> 00:32:26.980 Smiles are one of those non-behavior cues 00:32:26.980 --> 00:32:28.735 that people will show, 00:32:28.735 --> 00:32:31.570 sometimes despite being depressed. 00:32:31.570 --> 00:32:33.580 There are things called social smiles that 00:32:33.580 --> 00:32:35.590 will indicate that we're 00:32:35.590 --> 00:32:36.640 smiling and you might think 00:32:36.640 --> 00:32:38.320 we're happy, but we're really not. 00:32:38.320 --> 00:32:40.300 We're hiding some things. 00:32:40.300 --> 00:32:44.305 It allows us to measure smile level 00:32:44.305 --> 00:32:46.480 again as a construct that 00:32:46.480 --> 00:32:49.450 might indicate psychological distress. 00:32:49.450 --> 00:32:55.180 Speaking fractals, how the mouth is situated again, 00:32:55.180 --> 00:32:56.380 looking at the smiles, 00:32:56.380 --> 00:32:59.440 looking at the movement of the face as it 00:32:59.440 --> 00:33:02.875 relates to showing or indicating some level of distress. 00:33:02.875 --> 00:33:05.619 There are also things like horizontal gaze 00:33:05.619 --> 00:33:09.175 and moving forward and backward. 00:33:09.175 --> 00:33:11.590 Through this work, the team was able to determine that 00:33:11.590 --> 00:33:14.335 folks who actually were depressed, fidget a lot, 00:33:14.335 --> 00:33:16.300 and tend to move forward, 00:33:16.300 --> 00:33:19.240 forward and backward, something that we may 00:33:19.240 --> 00:33:22.105 not pick up on when we're talking to individuals. 00:33:22.105 --> 00:33:24.655 Some of you might, but most part we don't. 00:33:24.655 --> 00:33:27.535 Before you laugh, because I laughed too, 00:33:27.535 --> 00:33:29.410 frankly, at the idea that 00:33:29.410 --> 00:33:31.810 this avatar was trying to establish rapport, 00:33:31.810 --> 00:33:35.320 we've learned that individual suffering from PTSD, 00:33:35.320 --> 00:33:36.700 especially those in the military, 00:33:36.700 --> 00:33:39.310 are less likely to see a clinician or see 00:33:39.310 --> 00:33:42.835 a counselor for various reasons: stigma, 00:33:42.835 --> 00:33:45.640 uncomfortable, just don't want to go. 00:33:45.640 --> 00:33:49.195 But they're more likely to interact with this avatar, 00:33:49.195 --> 00:33:53.080 which if it helps one or two soldiers, 00:33:53.080 --> 00:33:55.540 it's done, it's measured good. 00:33:55.540 --> 00:33:57.820 Still under development, but I wanted to show you 00:33:57.820 --> 00:34:01.250 that as an example. 00:34:06.750 --> 00:34:09.685 Again, I want to recap, 00:34:09.685 --> 00:34:12.370 privacy and security are so important 00:34:12.370 --> 00:34:15.210 here because we're talking 00:34:15.210 --> 00:34:16.290 about DARPA work and 00:34:16.290 --> 00:34:18.464 research that relates to individuals. 00:34:18.464 --> 00:34:21.665 I actually had to participate in a privacy panel. 00:34:21.665 --> 00:34:23.875 I appeared before lawyers 00:34:23.875 --> 00:34:28.285 and Head of Research and other DARPA personnel 00:34:28.285 --> 00:34:31.420 to ensure that we're going to ensure 00:34:31.420 --> 00:34:33.670 that proper safeguards are 00:34:33.670 --> 00:34:35.455 followed when we're working with individuals, 00:34:35.455 --> 00:34:41.050 especially those from particularly sensitive populations. 00:34:41.050 --> 00:34:43.210 If your familiar with IRB protocols, 00:34:43.210 --> 00:34:45.550 Institutional Review Board that ensure that 00:34:45.550 --> 00:34:47.320 the appropriate level of care is 00:34:47.320 --> 00:34:49.404 given to subjects in your research, 00:34:49.404 --> 00:34:51.940 you'll know that some populations 00:34:51.940 --> 00:34:53.620 are more vulnerable than others and so 00:34:53.620 --> 00:34:55.450 we have to address the risk or 00:34:55.450 --> 00:34:58.045 ensure that participation is minimal risk. 00:34:58.045 --> 00:34:59.965 We had to do that very efficiently. 00:34:59.965 --> 00:35:02.170 In fact, I did that through 00:35:02.170 --> 00:35:05.960 my first month at DARPA. That's how important it was. 00:35:07.320 --> 00:35:10.330 Let's shuffle along now to talk about 00:35:10.330 --> 00:35:14.290 computing and computer science and data analytics. 00:35:14.290 --> 00:35:16.660 This is another program that I had the privilege 00:35:16.660 --> 00:35:18.745 of working on, the ENGAGE program. 00:35:18.745 --> 00:35:21.745 This is actually not an acronym. 00:35:21.745 --> 00:35:23.710 It just stands for engagement, 00:35:23.710 --> 00:35:26.725 the opportunity to engage our users. 00:35:26.725 --> 00:35:29.110 So got a lot of questions about that and we just 00:35:29.110 --> 00:35:31.795 tend to write a capitalize it is what it is. 00:35:31.795 --> 00:35:33.460 But the idea is to develop 00:35:33.460 --> 00:35:36.415 these data-intensive training methods, that scale. 00:35:36.415 --> 00:35:38.110 Some of you might be familiar 00:35:38.110 --> 00:35:39.895 with a similar term called big data, 00:35:39.895 --> 00:35:41.890 where you take lots and lots of information. 00:35:41.890 --> 00:35:43.480 You look for patterns, you look 00:35:43.480 --> 00:35:45.235 for very interesting stories, 00:35:45.235 --> 00:35:48.370 and you allow that data to inform your design, 00:35:48.370 --> 00:35:50.590 your problem set, your solution, 00:35:50.590 --> 00:35:53.210 whatever you're moving toward accomplishing. 00:35:53.250 --> 00:35:55.765 For this particular project, 00:35:55.765 --> 00:35:57.670 we were working on K through 00:35:57.670 --> 00:36:00.220 12 as our user group, that is, 00:36:00.220 --> 00:36:02.710 how could we impact training and 00:36:02.710 --> 00:36:06.160 instruction for students at the K-12 level, 00:36:06.160 --> 00:36:08.050 especially in concepts as they 00:36:08.050 --> 00:36:09.790 relate it to development of 00:36:09.790 --> 00:36:12.565 problem-solving and conceptual reasoning 00:36:12.565 --> 00:36:15.265 that might help them as they become adults, 00:36:15.265 --> 00:36:17.260 and maybe they'll pursue 00:36:17.260 --> 00:36:20.330 technical careers as a by-product. 00:36:21.450 --> 00:36:23.950 Because we're talking about scaling, 00:36:23.950 --> 00:36:25.945 we're talking about using data 00:36:25.945 --> 00:36:29.020 for thousands and tens of thousands of users, 00:36:29.020 --> 00:36:31.690 we're talking about kids, another vulnerable population, 00:36:31.690 --> 00:36:35.110 so again, strictest IRB protocols are followed. 00:36:35.110 --> 00:36:37.390 In fact, for this project 00:36:37.390 --> 00:36:39.235 and the last project that I discussed, 00:36:39.235 --> 00:36:42.325 the Navy and the Army were 00:36:42.325 --> 00:36:45.490 IRB gateways to ensure 00:36:45.490 --> 00:36:47.800 that any work done within the military, 00:36:47.800 --> 00:36:51.170 within DoD, was appropriately managed. 00:36:53.820 --> 00:36:56.710 The inspiration for the ENGAGE program 00:36:56.710 --> 00:36:58.690 was called the Fold-It program. 00:36:58.690 --> 00:37:00.850 I think one of your previous speakers talked about 00:37:00.850 --> 00:37:02.980 Fold-It from University of Washington 00:37:02.980 --> 00:37:05.305 as being a really amazing, 00:37:05.305 --> 00:37:08.050 effectively program that allowed 00:37:08.050 --> 00:37:09.940 novice users to work on 00:37:09.940 --> 00:37:12.670 problems that typically experts work on, 00:37:12.670 --> 00:37:15.175 especially if it's about biochemistry problems or 00:37:15.175 --> 00:37:17.110 other activities that typically are 00:37:17.110 --> 00:37:19.990 done by those who are well-versed in the subject matter. 00:37:19.990 --> 00:37:22.720 But the benefit and the beauty of using these kinds of 00:37:22.720 --> 00:37:25.960 systems is not just that the novice becomes the expert, 00:37:25.960 --> 00:37:28.435 but they can be expert without really realizing 00:37:28.435 --> 00:37:29.140 that they're doing a lot of 00:37:29.140 --> 00:37:31.060 work and not putting the work in, 00:37:31.060 --> 00:37:32.575 that they're actually training. 00:37:32.575 --> 00:37:35.020 That's the benefit of these kinds of systems. 00:37:35.020 --> 00:37:38.875 That's the model we want to follow in the ENGAGE program. 00:37:38.875 --> 00:37:42.430 Effectively crowd-sourcing from players to ensure that 00:37:42.430 --> 00:37:47.020 the game creates these super users, 00:37:47.020 --> 00:37:49.330 as it were, but not just super users, 00:37:49.330 --> 00:37:51.340 the super knowledge holders 00:37:51.340 --> 00:37:53.410 of information and capability. 00:37:53.410 --> 00:37:55.810 Imagine what we could do with kids by 00:37:55.810 --> 00:37:58.645 teaching them constructs like problem-solving, 00:37:58.645 --> 00:38:00.610 computational thinking, without them 00:38:00.610 --> 00:38:03.115 thinking that those are actually what they're learning. 00:38:03.115 --> 00:38:05.410 Make it fun. So another name 00:38:05.410 --> 00:38:08.840 for instructional technologies is? 00:38:09.960 --> 00:38:13.915 A game. I want to show you, 00:38:13.915 --> 00:38:16.000 again, a couple of these examples. 00:38:16.000 --> 00:38:29.220 [BACKGROUND] 00:38:29.220 --> 00:38:33.300 [NOISE] There we go. 00:38:33.300 --> 00:38:35.130 Thank you and I'm 00:38:35.130 --> 00:38:37.485 wearing glasses if you can believe it. All right. 00:38:37.485 --> 00:38:40.785 So here's refraction, the first game. 00:38:40.785 --> 00:38:47.430 Refraction is great because it uses what's 00:38:47.430 --> 00:38:50.700 called a Playtracer to 00:38:50.700 --> 00:38:53.460 actually trace a player's activities in a day. 00:38:53.460 --> 00:39:02.250 [MUSIC] Refraction tends to be really fun. 00:39:02.250 --> 00:39:07.545 But what it allows players to do is, 00:39:07.545 --> 00:39:10.290 progress through the game and it actually 00:39:10.290 --> 00:39:13.995 teaches fractions by splitting and dicing through. 00:39:13.995 --> 00:39:18.360 It actually captures the player's moves 00:39:18.360 --> 00:39:19.950 during the game and it logs 00:39:19.950 --> 00:39:25.650 it by two groups. 00:39:25.650 --> 00:39:29.130 The path that they follow to success, 00:39:29.130 --> 00:39:31.920 to reaching a level and conquering a level, 00:39:31.920 --> 00:39:34.140 that is figuring out how to solve the fractions 00:39:34.140 --> 00:39:36.810 correctly or the fails. 00:39:36.810 --> 00:39:38.685 How long did it take them to get there, 00:39:38.685 --> 00:39:40.230 if they got there at all? 00:39:40.230 --> 00:39:43.020 This effective database, this 00:39:43.020 --> 00:39:45.570 store of all this data across many, many students, 00:39:45.570 --> 00:39:46.920 across many, many players, 00:39:46.920 --> 00:39:48.555 allows us to determine 00:39:48.555 --> 00:39:51.345 the level at which actual learning happens. 00:39:51.345 --> 00:39:53.340 Where are the breakdowns, 00:39:53.340 --> 00:39:54.585 where are the pain points, 00:39:54.585 --> 00:39:55.890 and how can we fix this within 00:39:55.890 --> 00:39:58.980 the next players or the next similar kind of player? 00:39:58.980 --> 00:40:01.770 Does it have as much difficulty or at least get 00:40:01.770 --> 00:40:03.390 the proper support in 00:40:03.390 --> 00:40:06.340 game to actually learn the concept sooner? 00:40:06.560 --> 00:40:15.150 I'm going to just play a couple of levels 00:40:15.150 --> 00:40:18.780 here and I also want to say that this was 00:40:18.780 --> 00:40:21.180 developed at 00:40:21.180 --> 00:40:23.160 the University of Washington under Zoran Popović, 00:40:23.160 --> 00:40:25.320 again, who is the lead for 00:40:25.320 --> 00:40:27.990 the hold at work and at the center of game science. 00:40:27.990 --> 00:40:30.120 With these games, you can see you can play online 00:40:30.120 --> 00:40:33.870 and experience for yourself. 00:40:33.870 --> 00:40:40.170 I'm going to split this ray quite easily. 00:40:40.170 --> 00:40:42.825 So that was easy. No, they're not all that easy. 00:40:42.825 --> 00:40:45.825 But depending on how well you do, 00:40:45.825 --> 00:40:50.040 it presents to you the next set of problems. 00:40:50.040 --> 00:40:52.680 So obviously this is not going to work and you see that 00:40:52.680 --> 00:40:55.470 it just doesn't allow the user to move forward, 00:40:55.470 --> 00:40:59.280 but it logs that move so that it can determine 00:40:59.280 --> 00:41:01.365 how often this person to play and makes this 00:41:01.365 --> 00:41:03.765 move and whether or not the person might need help. 00:41:03.765 --> 00:41:06.590 You notice a little pop-up appears, 00:41:06.590 --> 00:41:08.150 which is typical for games, right? 00:41:08.150 --> 00:41:12.060 You want to provide that level of feedback. 00:41:15.140 --> 00:41:17.250 These are pretty easy and it's 00:41:17.250 --> 00:41:18.570 a fun game, but I wanted to, again, 00:41:18.570 --> 00:41:21.000 break the monotony here, 00:41:21.000 --> 00:41:23.250 but feel free to go online and play 00:41:23.250 --> 00:41:26.250 these Treefrog Treasure or similar game. 00:41:26.250 --> 00:41:30.330 It allows the capture of students' play, 00:41:30.330 --> 00:41:33.870 and it adapts to the users as necessary. 00:41:33.870 --> 00:41:35.910 Again, it becomes this big data problem, 00:41:35.910 --> 00:41:37.470 but we're capturing all this data, 00:41:37.470 --> 00:41:39.434 capturing all this information, 00:41:39.434 --> 00:41:41.460 and learning from it in the sense that 00:41:41.460 --> 00:41:43.740 we can improve the game and makes it even better. 00:41:43.740 --> 00:41:45.960 Let us take a closer look at 00:41:45.960 --> 00:41:48.975 how the Playtracer actually works. 00:41:48.975 --> 00:41:51.030 I'm going to look here so that I'm 00:41:51.030 --> 00:41:52.890 not twisted behind you. 00:41:52.890 --> 00:41:54.540 What you're looking at is 00:41:54.540 --> 00:41:57.555 the way the Playtracer actually works. 00:41:57.555 --> 00:42:01.125 Under this first box here, 00:42:01.125 --> 00:42:03.930 you're actually looking at the start state, 00:42:03.930 --> 00:42:05.595 which is shown in yellow, 00:42:05.595 --> 00:42:08.175 and the goal state, that is where you want to end up 00:42:08.175 --> 00:42:11.430 in that level, in green. 00:42:11.430 --> 00:42:13.620 The second box, the middle box, 00:42:13.620 --> 00:42:14.910 which is the second iteration, 00:42:14.910 --> 00:42:17.280 as a player progresses through the game, 00:42:17.280 --> 00:42:19.650 shows you the observed states. 00:42:19.650 --> 00:42:20.820 It is where those states that 00:42:20.820 --> 00:42:25.020 the player visits and those states are shown in black. 00:42:25.020 --> 00:42:29.190 The more states or the more ways, the more problems, 00:42:29.190 --> 00:42:33.615 the more content that a player visits, 00:42:33.615 --> 00:42:35.400 the larger the circle, 00:42:35.400 --> 00:42:37.695 the larger that particular state. 00:42:37.695 --> 00:42:39.540 It shows an example of how we're 00:42:39.540 --> 00:42:41.550 capturing this information player 00:42:41.550 --> 00:42:43.410 by player in aggregate 00:42:43.410 --> 00:42:46.170 so that we can go back and analyze it. 00:42:46.170 --> 00:42:48.480 The 3rd picture actually 00:42:48.480 --> 00:42:51.540 shows the state transitions as shown by the arrows. 00:42:51.540 --> 00:42:55.020 So if you're familiar with database modeling, 00:42:55.020 --> 00:42:57.630 Finite Automata that you 00:42:57.630 --> 00:43:00.015 might learn in theory, this is similar to that. 00:43:00.015 --> 00:43:03.435 We're talking about movement within a game across states. 00:43:03.435 --> 00:43:05.580 What this particular algorithm captures, 00:43:05.580 --> 00:43:07.740 the Playtracer, which is pretty novel, 00:43:07.740 --> 00:43:09.510 is an opportunity to 00:43:09.510 --> 00:43:12.435 transit between states within a game, 00:43:12.435 --> 00:43:14.985 within a level, and determine how often 00:43:14.985 --> 00:43:17.730 students or players are succeeding and failing. 00:43:17.730 --> 00:43:19.890 More to the point, here's output 00:43:19.890 --> 00:43:22.650 from about 30 people, 30 players. 00:43:22.650 --> 00:43:25.740 The blue arrows, which I hope they show up blue, 00:43:25.740 --> 00:43:28.395 show you the level of success. 00:43:28.395 --> 00:43:30.360 That is the amount 00:43:30.360 --> 00:43:32.310 of times that players actually succeeded. 00:43:32.310 --> 00:43:33.945 They've got the fractions [inaudible] correct, 00:43:33.945 --> 00:43:36.975 or they figured out the right answer to the fraction. 00:43:36.975 --> 00:43:39.825 The red arrows actually show level failure. 00:43:39.825 --> 00:43:41.175 So looking at this, 00:43:41.175 --> 00:43:45.070 it's a visualization of the problem space. 00:43:45.070 --> 00:43:46.460 How do we model this? 00:43:46.460 --> 00:43:48.920 Well, this is the way we do it at least in a visual way. 00:43:48.920 --> 00:43:51.965 It's an amazing opportunity to look 00:43:51.965 --> 00:43:55.010 at how our players play and learn, 00:43:55.010 --> 00:43:56.840 in this case, for infractions and 00:43:56.840 --> 00:43:58.805 apply it back to the game. 00:43:58.805 --> 00:44:02.405 In-depth versions of the game can adapt to the user, 00:44:02.405 --> 00:44:05.315 adapt to the player, and that's what we want. 00:44:05.315 --> 00:44:07.830 I don't want to get ahead of myself, but imagine if we 00:44:07.830 --> 00:44:10.650 can deploy these algorithms, 00:44:10.650 --> 00:44:12.750 this kind of structure that allows us 00:44:12.750 --> 00:44:14.910 to educate and instruct all kinds 00:44:14.910 --> 00:44:20.680 of learners how truly personal that experience would be. 00:44:22.340 --> 00:44:26.415 Again, the novelty of this approach, the true benefit, 00:44:26.415 --> 00:44:28.050 it gives us the opportunity to see how 00:44:28.050 --> 00:44:30.615 well or poorly players played. 00:44:30.615 --> 00:44:32.850 It gives us insight into those pain points, 00:44:32.850 --> 00:44:33.840 again, that I mentioned. 00:44:33.840 --> 00:44:35.160 Where are their breakdowns? 00:44:35.160 --> 00:44:38.040 Were users messing up? 00:44:38.040 --> 00:44:40.455 How are they missing the content? 00:44:40.455 --> 00:44:42.795 But also, where did they get back on track? 00:44:42.795 --> 00:44:44.475 How do we take that information, 00:44:44.475 --> 00:44:46.395 learn from it, and pipe it back into the game? 00:44:46.395 --> 00:44:50.230 This visualization, this algorithm allows us to do that. 00:44:50.960 --> 00:44:54.630 The two takeaways from this, especially lessons learned, 00:44:54.630 --> 00:44:56.865 as we think about the broader impact 00:44:56.865 --> 00:44:59.475 of big data for education, 00:44:59.475 --> 00:45:02.760 it's entirely possible to 00:45:02.760 --> 00:45:05.415 improve instruction through adaptation, 00:45:05.415 --> 00:45:07.395 by adapting the technology. 00:45:07.395 --> 00:45:10.770 That's not always easy, but it's also important. 00:45:10.770 --> 00:45:14.565 It also makes a very personal learning experience. 00:45:14.565 --> 00:45:17.970 Applying both individual steps 00:45:17.970 --> 00:45:19.920 to what's a success or what's a failure for 00:45:19.920 --> 00:45:21.795 a learner applied across 00:45:21.795 --> 00:45:24.975 hundreds of thousands of users is amazing information. 00:45:24.975 --> 00:45:27.120 Again, this idea of 00:45:27.120 --> 00:45:30.975 making what we might consider the individual, 00:45:30.975 --> 00:45:34.540 the big data problem becomes enormous for us. 00:45:36.680 --> 00:45:40.020 Thinking about education, thinking about computing, 00:45:40.020 --> 00:45:45.690 and how we might teach even kids how to think more 00:45:45.690 --> 00:45:47.040 critically about what it means to 00:45:47.040 --> 00:45:49.710 be a computational thinker 00:45:49.710 --> 00:45:51.360 without them knowing that they're doing it. 00:45:51.360 --> 00:45:53.040 It made me think a little bit more 00:45:53.040 --> 00:45:54.870 too about how do we even 00:45:54.870 --> 00:45:56.550 get more individuals interested in 00:45:56.550 --> 00:45:59.460 math or computer science? 00:45:59.460 --> 00:46:04.440 More broadly, what we've learned about being in 00:46:04.440 --> 00:46:06.240 any kind of technical situation or 00:46:06.240 --> 00:46:09.390 technical group is that the diversity of ideas, 00:46:09.390 --> 00:46:11.490 not just from the ideas that come from the individual, 00:46:11.490 --> 00:46:12.855 but from the individual themselves, 00:46:12.855 --> 00:46:14.805 will they make the most dynamic teams? 00:46:14.805 --> 00:46:16.935 So how can we make an impact? 00:46:16.935 --> 00:46:19.080 How can I make an impact on better 00:46:19.080 --> 00:46:21.840 understanding the landscape of what it 00:46:21.840 --> 00:46:24.000 means to learn computer science at 00:46:24.000 --> 00:46:25.770 whatever age and how can we make 00:46:25.770 --> 00:46:28.155 an impact and increasing those numbers? 00:46:28.155 --> 00:46:30.360 If any of you have done any outreach 00:46:30.360 --> 00:46:31.710 or any student outreach, 00:46:31.710 --> 00:46:33.555 or understanding of what it means 00:46:33.555 --> 00:46:36.285 to encourage inclusion and computing, 00:46:36.285 --> 00:46:39.135 the best time to really start is middle-school. 00:46:39.135 --> 00:46:42.045 If we started to recruit at college, 00:46:42.045 --> 00:46:44.940 then we're often too late and those individuals who we 00:46:44.940 --> 00:46:49.530 recruit decide to move into the major often leave. 00:46:49.530 --> 00:46:51.870 I wanted to think more broadly about what it 00:46:51.870 --> 00:46:53.940 meant to actually do 00:46:53.940 --> 00:46:56.820 something about increasing the pipeline 00:46:56.820 --> 00:47:00.420 or making an impact in the pipeline early on. 00:47:00.420 --> 00:47:02.400 Looking at these data, 00:47:02.400 --> 00:47:06.420 these actually were made 00:47:06.420 --> 00:47:09.495 available by NSF in January of this year, 00:47:09.495 --> 00:47:12.660 I apologize, it's not very clear, 00:47:12.660 --> 00:47:14.355 but it certainly shows 00:47:14.355 --> 00:47:19.034 the proportion of women, minorities, 00:47:19.034 --> 00:47:21.510 and persons with disabilities in science and engineering 00:47:21.510 --> 00:47:24.705 across different disciplines including psychology, 00:47:24.705 --> 00:47:27.510 social sciences, biological sciences, 00:47:27.510 --> 00:47:29.490 physical sciences, math and 00:47:29.490 --> 00:47:32.650 statistics, CS and engineering. 00:47:32.660 --> 00:47:35.070 The point I want to make here, 00:47:35.070 --> 00:47:40.620 if you look at the bottom two lines, 00:47:40.620 --> 00:47:44.475 the first is engineering, 00:47:44.475 --> 00:47:49.155 and the blue there, computer science, 4.8. 00:47:49.155 --> 00:47:51.690 The data show, from previous years, 00:47:51.690 --> 00:47:52.770 if those numbers are holding steady, 00:47:52.770 --> 00:47:54.100 but they're certainly not increasing. 00:47:54.100 --> 00:47:56.510 So the question becomes, how do we make an impact? 00:47:56.510 --> 00:47:57.935 How did we increase those numbers? 00:47:57.935 --> 00:47:59.450 Or at least attempt to 00:47:59.450 --> 00:48:00.950 better understand why we're not seeing 00:48:00.950 --> 00:48:02.420 a more diverse set of 00:48:02.420 --> 00:48:05.490 computer scientists later in the pipeline. 00:48:06.800 --> 00:48:10.815 It motivated a colleague and me to 00:48:10.815 --> 00:48:12.360 create a program called 00:48:12.360 --> 00:48:14.430 Motivate that's actually the name of the program, 00:48:14.430 --> 00:48:17.625 where it's a five-level process that 00:48:17.625 --> 00:48:21.945 incorporates mentorship, after-school programming, 00:48:21.945 --> 00:48:23.385 which falls under what we call 00:48:23.385 --> 00:48:25.095 informal education and training, 00:48:25.095 --> 00:48:27.630 as well as formal training where we provide 00:48:27.630 --> 00:48:31.545 in-course or course-related training for computing, 00:48:31.545 --> 00:48:33.750 as well as including parental involvement. 00:48:33.750 --> 00:48:35.220 How do we engage the parents? 00:48:35.220 --> 00:48:37.590 Because a very important part of what we've learned about 00:48:37.590 --> 00:48:39.285 minority groups is 00:48:39.285 --> 00:48:41.100 understanding what the family structure is, 00:48:41.100 --> 00:48:42.900 understanding what some of those impacts 00:48:42.900 --> 00:48:44.280 are to those barriers are to 00:48:44.280 --> 00:48:46.215 entry fall often with 00:48:46.215 --> 00:48:48.690 parents just not being aware of the opportunities, 00:48:48.690 --> 00:48:51.780 so including parents as a part of that process, 00:48:51.780 --> 00:48:56.310 as well as this idea of making Computer Science real, 00:48:56.310 --> 00:48:58.440 applying it in everyday life so that students 00:48:58.440 --> 00:49:00.150 understand how they can see 00:49:00.150 --> 00:49:02.820 Computer Science in anything that they do. 00:49:02.820 --> 00:49:05.580 We did that as an effort to 00:49:05.580 --> 00:49:08.295 remove what we call the mystique of computing. 00:49:08.295 --> 00:49:10.755 Often for under-represented students, 00:49:10.755 --> 00:49:14.040 there's this belief or there's not a belief or 00:49:14.040 --> 00:49:15.390 understanding about what Computer Science 00:49:15.390 --> 00:49:17.760 is or what a computer scientist looks like. 00:49:17.760 --> 00:49:19.500 What's more, the fact 00:49:19.500 --> 00:49:22.260 that computer scientist doesn't look like them. 00:49:22.260 --> 00:49:27.165 We wanted to measure whether or not attitude 00:49:27.165 --> 00:49:31.665 shifted pre and post-exposition to these program ideas. 00:49:31.665 --> 00:49:33.599 How do we learn, 00:49:33.599 --> 00:49:34.950 first of all, what their attitudes are? 00:49:34.950 --> 00:49:36.750 We talk about user research, remember? 00:49:36.750 --> 00:49:38.760 Attitudes, behaviors as it relates to what 00:49:38.760 --> 00:49:40.200 a user's propensity towards 00:49:40.200 --> 00:49:42.265 a certain activity or technology is. 00:49:42.265 --> 00:49:43.995 This is the same idea. 00:49:43.995 --> 00:49:45.900 We had a peer host survey of 00:49:45.900 --> 00:49:48.390 these students to better understand [BACKGROUND] 00:49:48.390 --> 00:49:49.770 what they know about computing 00:49:49.770 --> 00:49:51.210 and whether or not they would 00:49:51.210 --> 00:49:54.060 pursue it based on their experiences in our camp. 00:49:54.060 --> 00:49:55.815 We actually did this through a summer camp, 00:49:55.815 --> 00:49:57.090 which was our informal 00:49:57.090 --> 00:49:59.205 training session for these students. 00:49:59.205 --> 00:50:01.710 We did this through a couple of methods. 00:50:01.710 --> 00:50:05.940 We provided an e-textile workshop, 00:50:05.940 --> 00:50:11.790 which is a part of this wearable computing movement, 00:50:11.790 --> 00:50:13.080 especially for young people. 00:50:13.080 --> 00:50:14.535 They were able to sow 00:50:14.535 --> 00:50:17.640 these technical systems into 00:50:17.640 --> 00:50:19.770 bags and clothes and other kinds of things 00:50:19.770 --> 00:50:22.260 just to see how they work and also use 00:50:22.260 --> 00:50:24.660 logic switching to see how 00:50:24.660 --> 00:50:26.580 the logic and be at electronics 00:50:26.580 --> 00:50:30.145 of the system actually operated. 00:50:30.145 --> 00:50:32.570 There's also a 3D-printing part of 00:50:32.570 --> 00:50:36.410 the workshop where students were able to learn CAD, 00:50:36.410 --> 00:50:38.825 and then print their designs, 00:50:38.825 --> 00:50:41.050 which was a lot of fun for the students. 00:50:41.050 --> 00:50:43.260 This is a pilot program that we 00:50:43.260 --> 00:50:44.940 did a couple of years ago and it 00:50:44.940 --> 00:50:49.140 allowed us to see that there was more work to be done. 00:50:49.140 --> 00:50:52.320 Obviously, with a small sample of 15, 00:50:52.320 --> 00:50:54.195 there weren't a lot of people. 00:50:54.195 --> 00:50:55.950 But it was great because it allowed us to 00:50:55.950 --> 00:50:58.080 get a sense of where these students were. 00:50:58.080 --> 00:50:59.580 The really nice part about it, 00:50:59.580 --> 00:51:02.085 it seemed for this sample anyway, 00:51:02.085 --> 00:51:05.505 is that six have participated in prior STEM experiences, 00:51:05.505 --> 00:51:07.290 which is great because the minute they came 00:51:07.290 --> 00:51:09.705 in with a certain level of expectation, 00:51:09.705 --> 00:51:12.540 certain experience that hopefully matches, 00:51:12.540 --> 00:51:14.670 impacted their experience in the program, 00:51:14.670 --> 00:51:16.125 but others as well, 00:51:16.125 --> 00:51:20.070 and it might have been part of what I might call 00:51:20.070 --> 00:51:24.555 this invisible collective efficacy. 00:51:24.555 --> 00:51:26.550 If you're around people who can 00:51:26.550 --> 00:51:28.800 do things and maybe you think you can do it too. 00:51:28.800 --> 00:51:31.170 Because what we found is that 00:51:31.170 --> 00:51:33.165 no student actually said 00:51:33.165 --> 00:51:34.680 maybe there were a couple of maybe who said, 00:51:34.680 --> 00:51:36.495 that they didn't like Computer Science. 00:51:36.495 --> 00:51:38.670 But no one actually said, or computing, 00:51:38.670 --> 00:51:40.350 but no one actually said I 00:51:40.350 --> 00:51:42.765 can't do it. Which was really important. 00:51:42.765 --> 00:51:44.910 It helped us to see that maybe 00:51:44.910 --> 00:51:47.160 there's something here to removing this mystique, 00:51:47.160 --> 00:51:50.205 removing what these barriers to 00:51:50.205 --> 00:51:52.350 the idea of what it 00:51:52.350 --> 00:51:54.090 means to be a computer scientist 00:51:54.090 --> 00:51:55.410 is for a lot of these students. 00:51:55.410 --> 00:51:56.850 That was encouraging, though 00:51:56.850 --> 00:51:58.590 there's still more work to do there. 00:51:58.590 --> 00:52:01.335 Another important part was this idea that 00:52:01.335 --> 00:52:03.960 if we can train or help students to 00:52:03.960 --> 00:52:07.650 understand that if you wear your clothes or 00:52:07.650 --> 00:52:11.700 if you wear the watch or you create a 3D-printed device, 00:52:11.700 --> 00:52:13.830 that that can become part of your everyday life or 00:52:13.830 --> 00:52:16.020 whatever you're developing, whatever you're designing. 00:52:16.020 --> 00:52:18.150 There's no reason why if you didn't create it 00:52:18.150 --> 00:52:20.700 and you didn't divine some use for it, 00:52:20.700 --> 00:52:23.730 that it can't be applied to you for your everyday living. 00:52:23.730 --> 00:52:25.020 That was an important part. 00:52:25.020 --> 00:52:27.150 Students were actually very gratified that they 00:52:27.150 --> 00:52:29.280 were able to create these devices or 00:52:29.280 --> 00:52:31.560 these clothes and I'll show 00:52:31.560 --> 00:52:34.110 you actually here at the top left, 00:52:34.110 --> 00:52:37.480 this monster-looking thing is actually your purse. 00:52:37.870 --> 00:52:40.955 What we found is that the girls couldn't actually 00:52:40.955 --> 00:52:43.670 sew which is it is what it is. 00:52:43.670 --> 00:52:46.760 But more and more, a few of us can sew. 00:52:46.760 --> 00:52:48.530 I can sew on a button or whatnot, 00:52:48.530 --> 00:52:52.200 but I can't create clothes like my mom and her mom could. 00:52:52.200 --> 00:52:53.580 Part of it is generational, 00:52:53.580 --> 00:52:55.245 but that was a very interesting response. 00:52:55.245 --> 00:52:56.370 But at least they were able to 00:52:56.370 --> 00:52:59.460 create and at the very top next to 00:52:59.460 --> 00:53:01.920 the pilot text is an example of one of 00:53:01.920 --> 00:53:03.870 the CAD drawings that one of the students 00:53:03.870 --> 00:53:06.555 actually were able to create through a 3D-printer. 00:53:06.555 --> 00:53:09.510 What was really great about this, 00:53:09.510 --> 00:53:11.220 I think there was a session this morning 00:53:11.220 --> 00:53:12.690 that talked a little bit about this, 00:53:12.690 --> 00:53:14.010 was again, the girls could see 00:53:14.010 --> 00:53:15.150 how this worked in everyday life. 00:53:15.150 --> 00:53:16.260 If there is a design which you've 00:53:16.260 --> 00:53:19.335 divined for a 3D-printer, 00:53:19.335 --> 00:53:22.214 then you can create it and just yesterday, 00:53:22.214 --> 00:53:23.865 I saw on the news and again 00:53:23.865 --> 00:53:25.845 this morning that we can actually 00:53:25.845 --> 00:53:27.390 create these kinds of 00:53:27.390 --> 00:53:30.735 devices for everyday use for certain people. 00:53:30.735 --> 00:53:32.595 The girls were able to understand that 00:53:32.595 --> 00:53:34.890 and to see that there really is 00:53:34.890 --> 00:53:37.529 application for this combining computing 00:53:37.529 --> 00:53:40.035 with providing a real need 00:53:40.035 --> 00:53:43.090 for real users is really powerful. 00:53:44.150 --> 00:53:47.910 To finish off this idea of 00:53:47.910 --> 00:53:50.850 what it means to provide opportunities for 00:53:50.850 --> 00:53:55.500 computing to those who may not be inclined to 00:53:55.500 --> 00:53:58.680 pursue with this idea of what it means to 00:53:58.680 --> 00:54:00.840 persist in computing this 00:54:00.840 --> 00:54:01.950 is a great book if you've never read it, 00:54:01.950 --> 00:54:05.490 I highly recommended, How Children Succeed by Paul Tough, 00:54:05.490 --> 00:54:07.800 who was a reporter for the New York Times. 00:54:07.800 --> 00:54:09.330 He did quite a bit of research for 00:54:09.330 --> 00:54:12.345 understanding what it takes for a lot of kids, 00:54:12.345 --> 00:54:15.825 many of whom were born in poverty to persist and succeed. 00:54:15.825 --> 00:54:20.760 Some of those traits are grit, 00:54:20.760 --> 00:54:25.350 perseverance, discipline, even self-control. 00:54:25.350 --> 00:54:27.210 But in the context of an individual 00:54:27.210 --> 00:54:28.380 who's born in poverty, 00:54:28.380 --> 00:54:31.125 who often doesn't know a lot of other options, 00:54:31.125 --> 00:54:33.210 becomes magnanimous in helping 00:54:33.210 --> 00:54:35.415 them succeed when given the opportunity. 00:54:35.415 --> 00:54:37.845 One of the questions that I ask is, 00:54:37.845 --> 00:54:41.835 "Do these apply to instruction in computing, 00:54:41.835 --> 00:54:44.175 maybe even learning computing? 00:54:44.175 --> 00:54:50.355 If we encourage a try and try and try mentality, 00:54:50.355 --> 00:54:51.750 maybe that'll go a long way 00:54:51.750 --> 00:54:53.625 towards encouraging students again, 00:54:53.625 --> 00:54:56.190 who might not be inclined to computing to succeed." 00:54:56.190 --> 00:54:59.200 Fascinating book, I highly recommend it. 00:54:59.330 --> 00:55:03.000 I'm nearly done. I noticed some of you 00:55:03.000 --> 00:55:04.410 looked up because there was that excitement 00:55:04.410 --> 00:55:05.850 of almost being done, but that's okay. 00:55:05.850 --> 00:55:08.040 I want to take a little bit further. 00:55:08.040 --> 00:55:11.580 I want to finish with a little bit of what I'm doing now. 00:55:11.580 --> 00:55:15.750 I again, I'm at Smarter Balanced which 00:55:15.750 --> 00:55:20.759 is recently moved to UCLA. 00:55:20.759 --> 00:55:23.730 In fact, Smarter Balanced was 00:55:23.730 --> 00:55:26.610 an independent organization affiliated with 00:55:26.610 --> 00:55:29.595 the State of Washington up until last December. 00:55:29.595 --> 00:55:33.525 The benefit of moving to UCLA is that we're housed with 00:55:33.525 --> 00:55:35.790 some additional world-class researchers who can 00:55:35.790 --> 00:55:36.900 help us better understand 00:55:36.900 --> 00:55:38.250 the information that we're generating, 00:55:38.250 --> 00:55:39.600 the data that we're generating, 00:55:39.600 --> 00:55:42.240 and how to create the systems that again, 00:55:42.240 --> 00:55:44.970 we want to be able to provide great assessments. 00:55:44.970 --> 00:55:46.260 But what else can we do to 00:55:46.260 --> 00:55:48.180 improve this experience for students, 00:55:48.180 --> 00:55:49.995 teachers, administrators, 00:55:49.995 --> 00:55:52.480 and even in preparing the parents? 00:55:53.480 --> 00:55:57.120 What's really amazing, again in 00:55:57.120 --> 00:55:59.730 this multidisciplinary discussion is that 00:55:59.730 --> 00:56:01.020 I'm working with a lot of folks that 00:56:01.020 --> 00:56:02.595 I would not normally work with. 00:56:02.595 --> 00:56:05.040 I work with psychometricians, 00:56:05.040 --> 00:56:06.735 the individuals who create 00:56:06.735 --> 00:56:09.960 the statistics to determine how accurate the test 00:56:09.960 --> 00:56:15.510 is or how correct an item is in the exam itself. 00:56:15.510 --> 00:56:17.685 That help create the content, 00:56:17.685 --> 00:56:20.265 the math, and the language arts pieces. 00:56:20.265 --> 00:56:21.930 The folks who are actually 00:56:21.930 --> 00:56:24.135 the district lease the member lease. 00:56:24.135 --> 00:56:27.690 All of these folks are stakeholders and as the designer, 00:56:27.690 --> 00:56:30.000 it's important to me to ensure 00:56:30.000 --> 00:56:32.670 that the most important voices are heard, 00:56:32.670 --> 00:56:34.440 and we can't forget those students 00:56:34.440 --> 00:56:36.525 who are from those unrepresented groups, 00:56:36.525 --> 00:56:39.300 not necessarily those who have not used 00:56:39.300 --> 00:56:42.615 technology or those who suffer physical disabilities. 00:56:42.615 --> 00:56:43.620 They're are also important to 00:56:43.620 --> 00:56:44.730 this community of test-takers. 00:56:44.730 --> 00:56:47.190 How do we take into account all 00:56:47.190 --> 00:56:49.725 these user needs in creating 00:56:49.725 --> 00:56:52.875 a computer-adapted assessment and 00:56:52.875 --> 00:56:54.630 allow it to be effective and 00:56:54.630 --> 00:56:57.555 valid and that's the challenge. 00:56:57.555 --> 00:57:00.120 But I argue that this 00:57:00.120 --> 00:57:02.355 is a step towards the right direction. 00:57:02.355 --> 00:57:04.890 How awesome would it be if we could 00:57:04.890 --> 00:57:07.170 create assessments that allowed us to 00:57:07.170 --> 00:57:10.050 operationalize this process of 00:57:10.050 --> 00:57:13.005 what it means to learn through an assessment? 00:57:13.005 --> 00:57:14.640 You probably don't think about 00:57:14.640 --> 00:57:16.920 it when you take an exam, you take a test. 00:57:16.920 --> 00:57:19.395 But sometimes you learn something, 00:57:19.395 --> 00:57:20.895 maybe you are something you didn't know, 00:57:20.895 --> 00:57:22.470 you learn something you forgot. 00:57:22.470 --> 00:57:24.540 But we'd like to be able create an exam and 00:57:24.540 --> 00:57:26.850 assessment that allows you to learn in 00:57:26.850 --> 00:57:28.760 the process of taking it and how 00:57:28.760 --> 00:57:29.810 amazing would it be if we could 00:57:29.810 --> 00:57:31.325 operationalize what that meant, 00:57:31.325 --> 00:57:35.975 how that looked, and we could apply it to it to a range, 00:57:35.975 --> 00:57:39.549 an array of instruction technologies. 00:57:39.549 --> 00:57:43.650 Or maybe if we marry this idea of what it 00:57:43.650 --> 00:57:48.315 means to assess an individual, 00:57:48.315 --> 00:57:51.150 but also at the individual level. 00:57:51.150 --> 00:57:53.400 If we took this idea of 00:57:53.400 --> 00:57:56.535 this play tracer that we saw in our engaged games 00:57:56.535 --> 00:57:59.370 and ported it as an algorithm in 00:57:59.370 --> 00:58:00.990 the backend of a system 00:58:00.990 --> 00:58:02.220 that actually performed assessments. 00:58:02.220 --> 00:58:03.270 Now there are challenges there 00:58:03.270 --> 00:58:04.290 because obviously assessment 00:58:04.290 --> 00:58:06.630 is very personalized, right? 00:58:06.630 --> 00:58:08.175 We had to be very careful 00:58:08.175 --> 00:58:10.080 about how that process comes to bear. 00:58:10.080 --> 00:58:12.405 But what would it mean if we can make that happen? 00:58:12.405 --> 00:58:14.280 Could we be looking at a situation where 00:58:14.280 --> 00:58:17.340 an individual learner walks away from 00:58:17.340 --> 00:58:19.860 an exam having learned more than they thought that they 00:58:19.860 --> 00:58:23.595 would and not just learned it, they can apply it. 00:58:23.595 --> 00:58:25.575 How amazing would it be 00:58:25.575 --> 00:58:27.585 if we could teach computational thinking? 00:58:27.585 --> 00:58:30.810 This idea of what it means to learn computing, 00:58:30.810 --> 00:58:34.305 not computing actually computational thinking is not computing. 00:58:34.305 --> 00:58:36.630 But if we can teach these concepts as 00:58:36.630 --> 00:58:39.000 ways to problem-solving and conceptual thinking and 00:58:39.000 --> 00:58:41.385 understanding in a way that is separate 00:58:41.385 --> 00:58:44.220 from Computer Science but it still is a way of 00:58:44.220 --> 00:58:47.415 thinking that force these young learners 00:58:47.415 --> 00:58:50.010 to adopt the very important part of what it means to 00:58:50.010 --> 00:58:52.890 be a successful person in life and 00:58:52.890 --> 00:58:56.430 hopefully in some technical field. 00:58:56.430 --> 00:58:58.500 I [inaudible] on these questions because I 00:58:58.500 --> 00:59:00.270 think that they're motivating me 00:59:00.270 --> 00:59:01.980 in my current position and I think 00:59:01.980 --> 00:59:04.065 they provide for a lot of questions, 00:59:04.065 --> 00:59:07.425 a lot of difficulty maybe, but a lot of fun. 00:59:07.425 --> 00:59:12.840 I appreciate your time and before I let you go, 00:59:12.840 --> 00:59:16.935 I want to end with this idea of 00:59:16.935 --> 00:59:19.410 these are exactly the kinds of questions that make 00:59:19.410 --> 00:59:21.990 the work that we all wish to do or should be doing, 00:59:21.990 --> 00:59:24.480 most certain that I do multi-disciplinary. 00:59:24.480 --> 00:59:26.820 I've tried to pull out for you, 00:59:26.820 --> 00:59:28.965 for I think are the most salient part 00:59:28.965 --> 00:59:31.230 of this whole process of 00:59:31.230 --> 00:59:33.030 combining computing across a lot of 00:59:33.030 --> 00:59:34.530 different areas and I 00:59:34.530 --> 00:59:36.060 hope you've enjoyed your journey with me. 00:59:36.060 --> 00:59:38.070 I know it's been a long one, but I hope 00:59:38.070 --> 00:59:40.080 that I get the opportunity to answer 00:59:40.080 --> 00:59:42.060 any questions but talk with you more and 00:59:42.060 --> 00:59:44.400 also about your own thoughts and ideas moving forward. 00:59:44.400 --> 00:59:45.780 I thank you for your time. 00:59:45.780 --> 00:59:56.000 [APPLAUSE]