Okay. Yeah. Okay, everyone, we're gonna
go ahead and get started. Um Hi, I
am Jamie Williams and I'm an assistant director in the
department, Student centers activities here at Virginia Tech and
I'm also um one of the folks who have uh
been working to put this assessment showcase together that we've
been doing this week. Um This has been uh
this has been a week of assessment uh sponsored by
the Office of Assessment Evaluation, the Division of Student
Affairs um in University Libraries. And so this is
our fourth day and this is our final day of
the assessment briefings. Um So we featured a lot
of assessment evaluation in this initiatives from across the university
this week. Um And this was this event was
sort of designed to facilitate a community of practice and
form new knowledge and help everybody get a picture of
, you know, kind of what assessment practices are
happening across the university. Um So in addition to
the briefings that we've had um this week we're going
to have a love poster session tomorrow, this is
gonna be popcorn and posters on the main floor of
the library right outside of the circulation desk. And
those posters have been up all week, so if
you by chance can't come tomorrow, try to stop
by at least see them. There's a lot of
great work displayed up there. Um So without delay
, I'm gonna go ahead and interviews are speakers today
. Um Our plan is to have each of us
speak for about 15 minutes, we have a timer
. These are calmer. David's are calmer today.
Um And then if time allows the audience can ask
questions after um after all the presentations. So we're
going to uh stay true to our word as much
as possible and be brief with the briefing. Um
And I'm sure that all representative will be happy to
stick around and chat and have cookies and coffee in
the back. So um we are going to have
, there's three of us today, we're gonna have
kate McConnell and she is going to talk to us
about using semi structured group interviews, no PhD and
assessment required and meghan O neill and she's going to
talk to us about meaningful assessment through faculty leadership in
the first year experience this program and I am going
to start us off simply because I'm already up here
um with assessing students staff development um in our department
of students whose activities using our aspirations for student learning
. So um I've kind of already told you a
little bit about a little bit about me. I
am in the Department of student centers activities. I
advise our student government association and I also coordinate the
assessment activities for the department um and that you might
say, hey sda an assessment don't really go together
, but in my world they kind of do um
they provide a very good balance for one another.
Um And I do assessment for the department because I
just love doing assessment. Um This assessment that I'm
going to talk about was conducted in the spring of
2014 and we utilized one of the assessment consulting groups
from dr Martha Glasses Assessment class in the Higher Ed
program. And there were three master students that helped
us work on this project, David. Mhm.
It didn't advance. Yeah. Thank you for you
. Are you talking? There we go. Okay
, thank you. Uh huh. Um We've been
having a few technical difficulties today with the laptop.
Um So students, there's activities is um a department
in the division of student affairs and we um we're
responsible for our three student centers on campus as well
as the more memorial chapel. There are 11 different
functional areas in the department and they are very different
in scope and what the responsibilities of those areas are
. Um But basically we're an area that strives to
provide spaces, programs and services that cultivate community um
for our students and others. And um we try
to support our division aspirations for student learning. Um
You can see up here, I have our mission
of our department. We also have core values that
go along with our mission. And those are excellent
selfless service, innovation, community and inclusivity involvement and
student development. So we're trying to um last year
we started with um with the revival of our Students
staff development program. And um that's sort of what
spurred the assessment. I'll talk a little bit about
that program and his revival in a minute. We
believe that our students staff should grow holistically so personally
, professionally, um, through their employment with student
centers and activities, so they're not just there to
work. Maybe some of them are just there for
a job, but we, we want them to
be able to grow and develop through that experience so
they should have a meaningful college experience. Um,
and you know, we're led to believe, and
we're told that, you know, students who work
on campus are more engaged in the university community typically
than students who work off campus. So we believe
in that and we try to help facilitate that as
much as possible. So our department is part of
this newly created engagement area in the division of student
affairs and um, it's specifically an area dedicated to
student engagement. And so it's our department along with
the leadership education, collaborative BT engage and new student
family programs. So when we're thinking about um,
how we fit into that engagement area and thinking about
about how our students are engaged, of course we
think about the organization's our students are leading or the
programs they're participating in or the services they use,
but we also need to factor in um, their
engagement through employment as well, um, and making
that meaningful. And that also helps us paint that
picture of what student engagement looks like in her apartment
as well as on this campus. Yeah, so
our goal was um, to, to really kind
of see where we were with the students staff development
program. So there was an effort to get into
that a little bit. There was an effort to
formalize this program. Um, and through some department
changes over the past several years, quite a few
department changes over the past few years, this program
became much less formalized. It was still happening in
pockets. Um, and individual supervisors were reports developed
dedicated to the development of their student employees, but
it became less formal. And so we wanted to
kind of revive that. Um, so as the
program is getting started again, um, we wanted
to ensure that we were grounding our work and our
aspirations for student learning. Um, and for those
of you in the room who may not. Uh
, no. The aspirations for student learning in the
division are commit to unwavering curiosity, pursue self understanding
, entirety, practice civility, prepare for a life
of courageous leadership and embrace a press them as a
way of life. And these are kind of the
things that we want, um, ultimately, all
of our students to embrace and and developed through during
their time here, in addition to their academics.
So we shows outcomes specifically related to civility, self
understanding integrity and a person. Um We felt that
these made a good tie back to the mission of
the core values of our department. Um So you
can see that we really want to we want our
students to focus on being courteous um and whatever sort
of interaction they're having with customers, other co workers
, et cetera. Um We want them to Mhm
. Okay, sorry, We want them to be
able to confront each other um and resolve conflict respectfully
and hold each other accountable for their integrity while they're
at work. Um When we get into self understanding
integrity, all of our student employees take strength quest
. Um and we want them to just be able
to identify their strengths, we want them to know
them. Um We also want them to be able
to describe how they're using those strengths in their positions
. Um and we want their values to show as
as they're working in the department and um as they're
leading their peers, we get into that person.
We um we want we want our students to realize
that a person is more than simply just community service
. It's also more than customer service person is really
, it's a lot deeper than that. Um We
want them to acknowledge how services in a woman into
their positions and when they're leading their peers at work
um to lead with a service oriented approach. So
there are several ways that these outcomes are and can
be delivered. Um We have formal workshops and trainings
in the department. Um There's also um there's also
regular evaluations with their supervisors and then informally just conversations
, everyday conversations that supervisors are having um with their
students. We want all of those to be methods
in which these outcomes are delivered with our students.
So the original idea was to survey our students staff
and then hold individual interviews um to get uh to
be able to ask deeper questions. Um so we
send a survey link to our at the time,
180 student employees in the department. That number changes
based on semester based on the year. Um and
we got an 18 response rate. Um The interviews
were scheduled to take place over the course of one
day um from nine a.m. In the morning until late
in the evening, um to try to accommodate the
students, um working class schedules as much as possible
. Um And the graduate students from Martha's class that
were helping with the project were in that room all
day just waiting for students to drop in and talk
to them about their experience. Um And those interview
questions were intended to engage our students in a conversation
um about what a prism means to them, also
to for them to describe how they incorporate their values
in their work and really get into that and how
they held their peers accountable um for their integrity at
work. Um Unfortunately, we did not have any
takers for the interviews, so we were unable to
get more of that qualitative information that we wanted,
um but we did get good information from our survey
, um and we we learned a lot of lessons
from a lot of things with this project. Um
So we we had about half of our students who
responded. Um they did indicate that they knew their
top five streaks. Um Only six of them actually
listed their strengths um as requested. It wasn't a
required question, Um, but we ask that they
list their strengths. Only six of them did,
but yeah, for the six of them that did
that, um, and some actually indicated in an
open ended response that they understand how their strengths contribute
to their work and they actually gave some examples and
articulated that. Um, so that was also about
six students, but at least we have six students
that we reached in that way. Um, You
can see 89 of them indicated that they felt that
they incorporate their personal values into their work. Um
, Many of them agree that they hold their fellow
employees accountable for maintaining integrity. Um, all of
them felt that they are always courteous and their interactions
with co workers and with patrons and customers and that
they resolve conflict in a respectful manner. Um,
so I imagine that we will get into that one
a little deeper in future assessments And um, 93
felt that they do instill a value of selfless service
when they're leaving their peers at work. And we
asked and opened a new question about this one as
well. Um, and some common themes emerge of
client needs, customer service and leading their peers by
example. So for the future, um, these
are just a few things that we've kind of gotten
from this, um, from this project and hey
, our students get strength quest, um, and
they probably understand how to apply their strengths at work
, um, but we need to make sure that
we're maintaining some intentionality when we're talking to them about
strength. Um, and not saying that we don't
, but we, we need to make sure that
we're doing this regularly. We're doing it in a
formalized way as much as possible, And then also
having conversations about it as well so that it's part
of it's not a one and done kind of thing
. You know, it's part of a common language
, it's part of a regular conversation um that they're
having with each other and also with their supervisor.
Um customer service is a priority. Um I think
that further conversations um with our students staff could yield
some more concrete examples of um you know what they
think courteous and civil behavior is. Um and of
course we were unable to get that from a survey
, mm but hopefully we'll be able to get some
of that in the future. And also um it
seems that our students staff is committed to a service
oriented approach. Um I think their version of service
and what it means to embrace that person as a
way of life are two different things. Um So
I think that I think also we need to re
evaluate whether or not we're going to continue to use
this um with our students staff development um because there
are many different versions of service um and um and
it has many different meetings um and integrity does as
well, and that's a different conversation, but um
you know, we need to kind of reevaluate whether
or not we're going to continue to use that.
Um and if so how intentional we're gonna be,
how we're going to deliver that, how we're going
to have those conversations um so that we can all
arrive on the same page of what a person is
really meaning and how that's being instilled interesting employees.
So that is what I have. Thank you.
30 seconds. Okay, I'll take it. Okay
, you've learned some things from your students about how
they received strength strength class. Are you inviting some
of that and trains that? They understand what these
what their strengths are. How do you realize them
more specific and you're working there is an effort going
on to do that. So are um, we
have a committee that is kind of in charge of
sort of looking at this information, looking at the
outcomes, figuring out um, the best ways to
deliver these things. So um trying to make sure
that we incorporate some sort of formal workshop a couple
times a semester. And also it's really about,
we're still kind of building a culture in our department
in general of talking about strength, making it a
part of a regular thing that most of us in
the department and the vast majority of student boys have
taken it. It's just making it a part of
the culture and everyday conversation. So there are efforts
being taken to do that. So I think it's
going to take a couple years to revive it.
You know, do it to the point that it
can really be and just really be an exciting program
that all the students want to take part of and
you know, that we can, we can really
see what's happening with it. So my question is
about the questions you used in your servant. It
seems like the findings, a lot of talk about
reflecting on how they perceive themselves to be, I
think integrity or courageous leadership president. And since I
just wonder if you thought about angling the questions differently
or in addition, asking them things about what they've
learned, how they may be changed in those areas
based on their employment or what conversations are elements of
their employment have contributed to the way they view those
things. Yeah. Um Yeah, absolutely. I'm
hoping that it I have uh we're trying to sit
down actually next week with the co chairs of the
committee and then get with all of the supervisors and
trying to keep them in a conversation, you know
, as we go along to say, hey,
you know, this is the information we've gotten,
this is where, you know, I'm saying we
kind of need to move forward. Um And I
think that's an absolutely critical thing that we do and
I really think doing if we can get them in
for interviews or focus groups or something like that,
I think there were a lot of things that played
into the timing of that, that's how we're going
to be able to get that richer information like that
and have them reflect. Um What I would like
to do is is eventually have it so that we
can we can track this. So the students who
are employed with us for 456 semesters be able to
track kind of how they move along. Um And
that may be done best through their evaluations. We
have our rubric that's kind of facing all of this
that the supervisors can use with them actually started working
on one of those with eric over their hair.
Um And um so that we can actually track and
see how they're progressing through their employment. I think
so I think there are a lot of factors.
We were kind of rushed. It was the end
, it was like April 15 when this happens it's
April the whole one long day of April that happens
. And so I think that had something to do
with at the time of year. Um I think
that mistakenly the email notification may have gotten an hour
later than we wanted. You know it's you know
how could everything just happens and yeah so it's were
you know send it out ahead of time um Get
our supervisors to to make sure that they are pumping
it up. Like I know we had certain supervisors
that were really just hey make sure you do this
um you know do it on work time. Um
You know so making sure we get a lot of
encouragement that way. Don't do it in april.
Okay. Yeah I think that was a lot.
Thank you. Okay. Okay so next up we're
going to have kate McConnell. Are you going to
come up with this? I don't know. Let
me try. Oh yeah look at that coffee thank
you. You see if I can get my clicker
to work here because I tend to yeah move around
a little bit. I love that there are screens
coming at you from every direction in this room.
All right. So I we're going to switch gears
just a little bit. I'm actually not really reporting
on any assessment work that I've done in terms of
a project to give you results instead. What I'm
hoping I do is give you a what I call
a quick and dirty uh tool for your tool box
around assessment. Um It is something um that we
can call semi structured group interviews or structured group interviews
. Uh Say no PhD and assessment required. There
is absolutely no background as a methodology ist that you
need to do this other than to be detail oriented
and sure of the questions you wanna ask. So
jumping right in. I think there we go.
Um many of us have explained how many of you
have done surveys with your students, How many of
you have had excellent response rates in the past five
years? Oh, let's see what we've downgraded our
definition of excellence. When I first got into the
field, people like, you know, 50% Fair
enough, then it was well good enough for government
work, 32%. Now there's some of our national
surveys upon which we base a lot of our profiling
and decision making. You know, we're talking 18
20 average nationally, you know, so it's it's
getting to be challenging the other piece of it is
that sometimes the survey isn't actually what you want for
a variety of reasons. And so the other go
to is focus group. The problem with focus groups
. However, how many of you have ever done
a focus group are attempting are attempting to focus groups
. Okay. They can be wonderful, powerful narrative
tools for getting really detailed information. Um when structured
well, they tend to be pretty free flowing.
You generally need a smaller number of participants just to
make sure that everyone gets equal airtime. You really
need to have someone running those focus groups who is
skilled at facilitation, because if you're looking to get
information from your students who want to be careful that
you're not actually feeding a vocabulary or leading the conversations
to purposefully in one direction or the other where you
might skew the results that you're gay. Um what
it does allow for is really in depth responses to
the questions and if you keep things small, you
know what you're really looking for again, as I
said, was that equal airtime peace, giving everyone
the opportunity to give their own lived experience their own
feedback on whatever the topic is that you're trying to
explore? One of the hang ups with a focus
group, However is that it often requires something called
transcription. Okay. Where you have actually recorded digitally
video, old cassette tape recorded. And I still
remember doing that. And what you do is you
preserve the record because if you are facilitating and taking
notes, you know, you're not in a position
as we know the myth of multitasking to actually remember
everything that happens in great detail. Uh so what
you want are those transcriptions to go back to,
to look for accuracy, things you've missed, the
nuances that we're trying to get out of a focus
group. Unfortunately, transcription if you've ever done any
yourself, okay, you know how time intensive that
can be. I did a qualitative dissertation and I
think a one hour interview took me about four hours
personally to transcribe. And so the first thing you
figure out then is that you want someone else to
do it for you. And as you start looking
you realize that they're charging a really good hourly rate
because it is that kind of uh intensive deep work
that has to take place with that. So while
focus groups are an excellent option for many qualitative expirations
from an assessment perspective, I think that we need
to have something else that we can go to.
Um that can give us qualitative information and engage lots
of students and providing some open ended feedback without either
breaking the bank or exhausting you. So the one
time you do the focus group is the last time
you ever do it because it just was so time
intensive and uh stressful to manage the process around.
So what I am proposing is that we take a
peek at something called structured group interviews and I'll be
the first to admit that this is an idea I
have stolen from a woman named Mary Allen, who
is kind of one of the, one of the
, you know, all of our fields have famous
people. Mary Allen is one of the five or
six famous assessment people. She's done a lot on
assessing general education and assessing broadly speaking academic programs.
Um She's out of the California State University system retired
now. Um But she came up with this idea
of an S. G. I. For all
of the reasons that I mentioned about the focus group
and their limitations. She was getting a lot of
requests from faculty in particular for her to do a
more open ended exploration of different issues with students.
And she said I just can't she do one focus
group as a favor, and then all of a
sudden it's snowballing into a sideline business. Uh she
said I have to figure out something else to do
here. So she created this format um that she
refers to as a structure group interview and what it
does is kind of put a box around some focus
group techniques that will get you meaningful data, okay
, when you have some very specific questions that you
might be looking to answer, so first, it's
important to know this is less interactive than a focus
group in the sense that it's less freewheeling. Um
there are more boundaries on students or respondents in terms
of what kinds of questions are being asked to respond
to, the nature of their responses, so they're
being guided to put their and package their responses in
certain ways. Okay, um the beauty of them
is that they can be facilitated with by people who
have really no training in that focus group facilitation world
, because a lot of what you're doing is essentially
constructing some uh interview worksheets ahead of time. Uh
so you're not necessarily responding on the fly to comments
that you're hearing and trying to build on this,
you know, very nuanced conversation. You're taking snapshots
uh along the way uh in a qualitative manner.
So I said it's highly structured. Another thing that's
appealing, particularly in a place like Virginia Tech.
It can be done with much larger groups in a
focus group. I've done structured group interviews with classes
here up to about 120 students at the same time
. Um It takes a little logistical work, but
it can be done. And what it does is
it allows every student in that class to give me
feedback on very specific questions. So I'm not sampling
10 students out of 120 and then trying to extrapolate
from their experiences, some rule about um, some
generalization about that class instead of getting feedback from everyone
. And then finally, the way you structure,
it kind of lends itself to a quick and dirty
reporting framework that can give you some key or core
findings important to remember. This is not a case
study, not an in depth analysis, but it's
going to be a view. Um, I would
say indicators okay, that can be acted upon pretty
quickly if you've constructed the questions in a useful way
. Does that make sense? Okay, so what
I thought I would do is just show you a
couple that I've done and then we're actually gonna play
with one in here so you can see what it
looks like. So broadly speaking they have three parts
. The first is kind of a warm up activity
where you ask the question that's meant to get students
to pick up the pencil. Actually start engaging in
the process and do a quick what I call temperature
check. So you create a like hurt question or
a statement that they need to respond to. You
can use any number of like scales, strongly agree
to disagree um you know an impact scale, anything
that you would like okay and then ask them to
give you a word or phrase okay that they associate
with the topic that you're asking them about. Then
you dive into the deep end with some targeted open
ended questions at the individual level. Um generally you're
talking about something that will take them a good,
you know, 10 minutes or so to respond to
individually okay, giving them some time to think.
Um and with each of these things, you're literally
handing them a worksheet so that they have the prompt
in front of them, they're recording their ideas there
, you know, kind of doing a mind dump
for you at that moment. Okay. The last
thing you do is a group debriefing where you actually
have, you know, depending on the size of
your group, one piece of paper circulating or you've
broken, you know, 100 and 20 broken them
up into smaller groups where, you know, 10
students have to help um generate ideas, do some
brainstorming in a graffiti session and then prioritize their response
for you. I will say that throughout this you
can do temperature checks to give students the opportunity to
give voice to their ideas. So it doesn't have
to be, you know, that's not a high
stakes testing environment where they can't share their answers or
or give feedback. What it does allow for the
students who want to have their voices heard to contribute
and be heard by their other classmates or the other
people in the group gives them a chance for some
air time. But then it also doesn't put pressure
on those students for whom that is a stressful uh
, notion. Um, you know, because there's
a tendency sometimes again, if you're not well trained
with the focus group methodology, that there may be
one or two voices that dominate a conversation. And
that sort of format. What this allows for is
those who want to share publicly. Can those who
don't can choose not to, but you're still getting
your information down on the papers that they're sharing with
you? Okay. So I have taken to using
this a lot with my grant evaluation work, where
there are specific questions that come up about a program
, you know, the p I says, I
think this is working, but I have concerns about
so, again, taking a very strategic question.
Um one of the grants I work with is the
step grant that helps fund the living learning communities that's
known as curie these are quantitative and physical science undergraduate
majors who all live in the same space over in
the hall, They're part of a broader umbrella learning
community called Invents and this is important to know because
that's mushing together uh male and female engineers in there
to learning communities that have been around for a long
time. The Davinci Life sciences learning community that also
has been around a long time. And when we
started curie we were kind of putting this little tiny
beginning of a learning community amongst all these broader,
well established programs. So we were really concerned about
their identity as an individual community and what they thought
they were getting out of out of events as well
as curious specific. So the first thing we did
was just ask them to provide us with a word
or phrase describing their response to the curie community,
their impressions ovens, and then we asked them to
give us their overall satisfaction with the community. Okay
, so this took 3, 4 minutes. Okay
, you can do this on an index card.
Okay, how sheet of paper? So we got
also um ranges on the community satisfaction. We had
interesting words given to us, but at the very
beginning we actually have something interesting happened here where the
students stopped us and said, well do you mean
the living learning community, like my life in the
hall? Or do you mean the seminar? I
have to take this class or with this learning community
? I would never have known that those were two
different things in the minds of the students. And
if I had sent them a survey and said give
me your feedback on the curing community, I had
no idea if they were responding to their their notion
of life in the dorm, which they were incredibly
positive about, or if they were offering a critique
of the seminar associated with this, which had some
room for growth. Okay, so immediately I got
some useful data, even unintentionally. So the second
thing we wanted them to do, going into some
of the targeting questions, let's talk about the community
and how all the pieces fit together. Okay,
so I literally have them draw pictures. Where do
you feel connection? Where are your friends? So
just you know, put X. Is where your
closest friends are from? Okay. Who do you
feel closest to? Yeah. Okay. We asked
them to give you more information about the purpose of
their program versus the purpose of curing. Okay.
And then finally we have them do that roundtable exercise
, giving relative strengths and weaknesses. Okay, literally
passing the paper until they exhausted all options and then
the best charges that they have to go back and
rank order them. So they came up with 15
strengths and a list of 25 things they wanted to
fix. That's a lot of information to navigate if
you're trying to make some strategic decisions. So we
said give us your top three and they did and
we were able to make some really good changes.
I have another example here that I can provide these
slides to and share with you as well. But
what I would like to do is give you a
chance to take a look at some activities and just
test out the flavor yourself. So pretend I'm doing
structured group interview on your perceptions of assessment. Okay
dangerously. To ask them in the system. Show
cake. The first thing I want you to do
is created with this statement assessment. Nice circle of
half one time. Uh huh. This is a
comment on a blog about Assessment, the Chronicle of
Higher Ed about six years ago. And at the
bottom I want you to tell me a word or
phrase that describes your response to assessment work within your
unit. You can do a broad questions. Broad
statement. Hippies are going and then drive it home
to a context. That makes sense to your responded
. It's you. Yeah. Again, did you
just want this part kind of quickly to get people
rolling? But also not a huge. These are
meant to be instinctual kind of response. And so
you split it up so you do have different pieces
of paper for each part. Now with the paper
. Okay, what you do is you'll see that
there's a five character code holding place. I keep
these things always very enough. What I ask people
to do is make up your own club so that
I can link the papers and once I've collected everything
back again. Okay, So, you know,
I have no idea. If you're giving me your
ATM your email, you know, I tell them
you do make it up, do whatever you want
. The second piece is getting you to think about
your home unit in department, one of the most
persuasive arguments within your unit for engaging in assessment.
Okay. What is the rationale for assessment that resonates
with your people in your home world? And then
the second question asks you to list those that don't
work in your world. Okay. Mhm. Yeah
. Yeah. And then the last piece of this
would be to circulate that debriefing sheet. Okay.
Unless you all come up with strengths and weaknesses about
assessment in your home world. Okay. Um what
we could do with that list, is that still
the most common strength? This is that we're seeing
across departmental cultures, There may be one that we
identify this really unique to a specific area and working
at attention. Um So again, this is really
just a snapshot of a way to get some information
from those students or team members, whomever you might
need to get some feedback from a very targeted specific
questions That potentially are arriving out of some of the
other assessment data. I could see this being a
response to a uh 60 black. You might get
on your student perceptions of teaching forms if you want
to see if they say that if everyone says,
you know this part of the project, we weren't
quite clear about it. We didn't like you could
do an investigation into one project in one class.
Um you could do an investigation into some feedback from
a survey this way without having to do the full
blown focus group methodology and you know, concern yourself
with the logistics and time and cost of that all
entails. I have up here and now I'm closing
in and David's gonna get the hook out. I
have a six page kind of description of how to
that. I can hand out to anyone who's interested
here. I have three copies of that, but
I'm also completely able to share that over email.
If you wanted to shoot me in line, I
can send it to you. It's, you know
, very basic, you know, step one,
step two kind of information um, up might be
now is K M Do you think David, R
E Z Like the breath E K. My maiden
name? I was the only drastic, I think
in the entire state of Virginia, K M graphic
at VT dot E D U. And I will
say that I also have come and done these four
folks with their organizations or their classes. Um,
as you know, my my role is primarily with
undergraduate education. So, um, that's one way
that I've done, but we can always talk if
you want, if you kind of want to do
that modeling of train the trainer where you have one
done for you and then you pick up the ball
and run with it. I'm happy to talk to
stop doing that kind of thing as well. Thanks
. 15 minutes is fast. All right, just
close this. So I've prepared for like 10 because
I know I talk a lot. Thank you very
do you want my pointer? Oh no. Okay
, force let's Yes. Mhm. Yes. Uh
huh. Yeah. Thanks God was asking. Mhm
. It's good here, shops and stuff. So
I'm gonna get started while David works on our technology
. Uh My name is Megan O neill and I'm
the associate director of first year experiences for Virginia Tech
. And in my role I'm responsible for helping our
faculty plan implement and assess their first year experience courses
. So if you so I just wanted to start
, I like kate. I'm not going to share
any results of our assessment. Our results are for
our faculty and for occasional reports I have to write
. But what I'm going to talk about today is
how the process of revising our assessment plan um really
increased, I don't like the term buy in because
it still implies a hierarchical and um linear relationship of
assessment. But how increasing our faculties participation made our
assessment practices considerably more meaningful. So it's more of
a thinking about strategies for getting other people invested in
assessment. Uh huh. So so if you know
anything about our first year experience program, you know
that it was part of our it was our quality
enhancement plan for our accrediting body sacks coc. So
what that meant was about five years ago we had
to implement a plan for two and kate says do
nothing less than transform teaching and learning at Virginia Tech
or the teaching and learning environment. So that was
our transformed. So that was our, what we
were asked to do and what we decided was to
implement a first year experience program. Now, other
universities have a first year experience program that, that
are usually more homogeneous. They are university one oh
one type courses. They introduce students to career major
exploration, um, to student success skills through transitional
information of navigating what college life is like after being
a high school student. Um, and we found
that that wasn't particularly going to fly at Virginia Tech
. We have a different student population than a lot
of other universities who do University one. And so
what we decided to do was to incorporate some broad
learning outcomes into a variety of different FY programs.
So we started with four pilot programs and now we
have 25 separate first year experience programs happening simultaneously at
the university. And they are located at the college
and department level, so some of them are college
wide. We won for the entire College of Natural
resources and the environment, but then we have different
ones for all the different colleges in the college of
Liberal arts and Human Sciences. So it was really
up to the leadership of a particular college or department
. They wanted to structure their programs, but again
, what we needed to have some overarching learning outcomes
that we could make sure we were transforming teaching and
learning at Virginia Tech. Uh so our particular goals
are to engage students through first year experiences that provide
curricular curricular learning opportunities, intellectual and practical skills that
are foundational students development into lifelong learners. So uh
not super easy to assess whether or not we're transforming
someone into a lifelong learner, but we know that
down into three particular skills we were looking at,
which were problem solving inquiry and integration skills. So
again very broad. So I think this is a
little weirdly smush because I cut and pasted it.
But so we use the definitions of inquiry, problem
solving immigration from the EU, the American Association of
Colleges and Universities provide some really nice definitions for us
and they also provide assessment rubrics. So we used
the value rubrics, um, and I don't have
time today to go into how we define this morning
apples or all that information is nicely provided for you
on our website. That's why you got VT dot
e D U. Uh and it will be in
our five Year Impact Report that will probably be up
on the university's website in March. So, uh
, but we decided to have both qualitative and quantitative
assessment measures. So this is what we told Sachs
we're going to do this before the program is actually
started, we're going to have qualitative assessment measures where
students would respond to reflective prompts and then they would
be qualitatively evaluated by a team of Raiders that were
faculty and led by an assessment professional. And so
they would be qualitatively rated from novice practitioner or expert
skill level on all of those things. Then we
have different quantitative measurements that we're going to use for
problem solving. We were going to do motivational strategies
for learning questionnaire, the MSL Q. Which was
um I believe someone else, another institution developed it
and we were going to incorporate it. Yes.
Then we did uh for inquiry, another multiple choice
tests and information literacy test which was developed in house
by our librarians and our assessment professionals. And then
for integrated learning they were going to develop a measure
of e portfolio skills integration. because we were deciding
, we were determining um we defined integration and something
that would happen through portfolio, think you could learn
, we're partnering with the portfolio people on so you
can hear a lot of we were and we like
it was going to do this so um this is
a robust assessment plan and we had high hopes for
being able to measure both qualitatively and quantitatively student learning
on those three learning outcomes. What we found was
that um so all of this was done kind of
with the help of a few faculty who are on
our project team, but mostly it was done with
the FBI office, the leadership team from the assessment
office, the center for Teaching and Learning on campus
um what is now t lows and then people from
the provost office. So uh once we started working
with faculty we found that it was quite it wasn't
what they needed and they were very helpful at sharing
with us what they needed. So um we got
both formal feedback from faculty, we asked them how
the assessment process was working. We also got a
lot of emails and just conversations from faculty about how
it was working and what we found was that the
the way to get that assessment plan to work was
to really listen to their needs. So when we
thought about assessment, in terms of uh if we
took a step back from thinking about in terms of
what Sachs was asked because Sachs doesn't actually ask for
all that much, they say we need to have
an assessment plan and we need to do is that
so if we take a step back from what we
told Sachs we're going to do, but they're not
necessarily holding us do. We found that if we
listen to what faculty really needed and we involved faculty
in the revision process of our assessment plan, we
got the types of results that I feel comfortable saying
really did transform teaching and learning because it transformed the
way a faculty thought about what they were doing in
their classroom. I can't necessarily say that after five
years of our first year experience program, students have
moved from a position of X to a position of
Y. In terms of inquiry skills. I don't
feel comfortable saying that different faculty might feel comfortable saying
that for their 25 different programs. But what I
can say is that when we listen to faculty,
we created an assessment plan that allowed them to really
think about what was happening in their classrooms and change
their teaching strategies. And so what are revised assessment
plan now looks like the faculty feedback was our qualitative
prompts which were originally very structured, are now revised
every year basically by our by our faculty teams.
So in our, so we have one that's for
physics students and our physics faculties sit down and take
a look at the qualitative prompt and they change the
language. It turns out some of the wording that
is used in the A. L. C.
And you rubric does not match how people talk about
problem solving in a business class. So we revised
the language now all that's done under the it's done
with the assessment office so we can still talk about
reliability and validity of these types of measures. So
it's not willy nilly but it is done purposefully to
match what's actually happening in the classroom. So again
when I talk about I don't see um why I
don't like the term buy in because it does imply
still like here's the assessment measure and faculty below it
. Need to believe in it when really what we
have happening is a continuous revision process with the faculty
and assessment professionals. So our qualitative prompts are constantly
revised, our quantitative prompts. We don't have a
P. M. S. L. Q.
Anymore. The one year we did it, the
results we got back from faculty. The faculty didn't
have anything, they didn't know what to do with
those results. So they were basically meaningless. The
only thing we were doing with them was checking off
the boxes that meself you done And if before my
just filled out for cates oh focused groups. What
was one of the roadblocks to assessment in my office
? It's time. And so I can't ask faculty
to do an M. S. L. Q
. Just to check off a box so we don't
do it anymore. We found that how we were
talking about integration or how the administration wanted us to
talk about integration in terms of e portfolios. Also
not really relevant to our faculty. It was going
to take an incredible amount of time to implement that
kind of thinking in practice. And so creating an
assessment measure for it was going to also not be
the best use of our time. We start with
the I. L. T. Because the data
we were getting allowed faculty to make real changes in
how they taught information literacy. They looked at the
data, they sat down with librarians and they changed
what was happening in the classroom. So again can
I say that faculty moved from being or can I
say that students moved from being novices to practitioners?
I don't know. But the classroom focus and what
was happening got considerably better. So again no quantitative
measures of integration and problem solving. And we also
worked, we found the people on campus who could
help us make those revisions. So faculty in particular
disciplines who know the language of their discipline. Uh
And we found that working with the university librarians to
continuously revise the I. L. T. Um
the nature of information literacy changes pretty fast. And
so if we keep it if we keep the information
literacy tests the same every year, we won't actually
be creating tests that help students. Uh huh.
So lessons learned what to ask if you are in
charge of creating an assessment plan and having worked for
faculty or how to get meaningful faculty involvement in your
assessment plan is what are you scared of? Uh
For me, I was scared when someone told me
to assess, I was scared of someone telling me
how to evaluate my students. And so the first
thing you have to do when you work with faculty
on assessment um in my experience is to help them
really understand the difference between formative and summit of assessment
and that this is about um improving teaching and learning
not evaluating the successful nous or lack of success of
a particular class. So kind of breaking through that
fear barrier is particularly helpful. Who defines the learning
outcomes and assessments personally? I found that if someone
else's defining one of those for you, your faculty
should be defining the other. And so if we
define the learning outcomes for our faculty where problem solving
, inquiry integration and what they need and so talking
with them about then how to measure those learning outcomes
was particularly helpful and I think you could also do
it vice versa. So there has to be a
conversation in both the learning outcomes and the measurement tools
. And then what can I do with the data
? If the data just sit somewhere then don't do
the assessment? Uh You I'm sure that there's other
reasons for like it's nice at the university. I
can tell the university like we did it but in
terms of getting faculty involvement and faculty leadership, if
the data is just gonna sit there and it's really
hard to get them involved. And then again how
will this assessment helped change teaching and learning? So
we're not using it to evaluate students, We don't
want it to prove a program success or programs failure
. We want to think about our assessment data in
terms of what can I tweak, how can I
reframe something I talked about in class. Um Should
I drop one assignment and increase the student time we
spend on another side? So that type of real
formative focus is what gets faculty um involvement and we
got all that because we allowed faculty to actively participate
in revising our assessment. So it looks very different
than it did five years ago or seven years ago
when we when we proposed it. But but I
believe what we got from it was a group of
individuals who are invested in continuously reflecting on their teaching
practices and making really quality changes for students. So
. Okay. Okay. Yeah, I know.
We're at one. Okay. Thanks. Thank you
all. Please take the leaves and coffee back and
also um if you can make sure to come to
the top corner post recession tomorrow, that'll be taking
place from noon to two. I realized I forgot
the time earlier, and that's on the main floor
of the library kind of circulation desk. Thank you 955 00:54:39.150 --> A:middle L:90% . Okay.