The Vocational Rehabilitation Return on Investment Project

ANN OUTLAW: Thank you all for viewing our
webinar featuring The Vocational Rehabilitation Return on Investment Project. I’m Ann Outlaw from the Knowledge Translation
for Employment Research Center, housed at the American Institutes for Research, and
funded by the National Institute on Disability, Independent Living and Rehabilitation Research,
or NIDILRR. The KTER Center works with an employment research
outreach team to identify the focus of our webinars. Recently, this team requested information
about fellow NIDILRR projects who are making an impact with real-life outcomes. We found the VR Return on Investment Project
was highlighted in NIDILRR’s report to Congress. This research project developed a rigorous
model to determine the employment impacts and return on investment for the VR program. Today, we’ll hear from the team about the
importance of gaining commitments from researchers and VR agency staff to create an economic
model reflective of the VR program, how they provide multiple avenues for stakeholder engagement
with feedback opportunities, and their focus on knowledge translation from the beginning
of the project to increase their impact and sustainability. But I’ll pass it on to Rick Sizemore who will
moderate today’s webcast. Rick? RICK SIZEMORE: Thank you so much. Welcome to today’s webinar on the Vocational
Rehabilitation Return on Investment Project. I’m Rick Sizemore. Currently, I’m the director of the Wilson
Workforce and Rehabilitation Center in Fishersville, Virginia. And I can say that our program has certainly
benefited from the VR-ROI Project turning data into what we refer to as actionable information. Let me take just a moment, if I could, to
look at the objectives for today’s webinar. We’re certainly going to provide an overview
of the Vocational Rehabilitation Return on Investment, or, as we refer to it, the VR-ROI
Project. We’ll also describe the project’s approach
to ongoing engagement of the state VR agencies and other stakeholders. And we’ll provide examples of successful knowledge
translation activities resulting from this ongoing stakeholder engagement. So that’s what we have in the plans for today. I’d like to get started. We have an all-star panel of top experts on
standby to go over this compelling information, including Dr. Bob Schmidt, who is the principal
investigator of the VR-ROI Project from the University of Richmond. Welcome, Bob. ROBERT SCHMIDT: Hi, Rick, happy to be here. RICK SIZEMORE: I thought we might have lost
you there for a minute. Good to have you with us today, Bob. Dr. Joe Ashley is the co-principal investigator,
also from the Virginia Department for Aging and Rehabilitation Services, where we also
are hearing from Dr. Kirsten Rowe, who is the project coordinator today. Welcome, Joe and Kirsten. JOSEPH ASHLEY: Thanks, Rick. KIRSTEN ROWE: Thanks, Rick, glad to be here. RICK SIZEMORE: Good to have you both with
us. And from George Washington University, Dr.
Maureen McGuire-Kuletz, who is a project liaison. Welcome, Dr. Kuletz. MAUREEN MCGUIRE-KULETZ: Thank you, so glad
to be with you all today. RICK SIZEMORE: Awesome. Well, today we’ll be discussing the approach
used by VR-ROI Project to develop this usable information we’ve been talking about. And here is what we hope to accomplish on
today’s webinar. First, we’ll give you a brief overview of
the VR-ROI Project and describe the project’s approach to engaging state agencies and stakeholders,
and then describe some examples of the project’s knowledge translation activities. Once again, those all important objectives. First off, we’ll begin with Dr. Bob Schmidt. Bob, our first question is that there have
been lots of other key professionals involved in this project in addition to those that
we’ll hear from on the webinar. Could you give us a sense of who all has been
involved in this exciting project? ROBERT SCHMIDT: Thanks, Rich. I’d be happy to. First of all, I should say, before I talk
about everyone, that this is a fairly complicated project. And we have separated our team into really
two components. But before I talk about those, I should pay
respect to a fellow who worked with us and has really been the inspiration for this work
for a long time, and that’s the late David Dean. David was active in the economics of disability
for I think close to 40 years, starting with graduate school. When he was a graduate assistant to Monroe
Berkowitz at Rutgers University. And we owe a debt of gratitude to him. He died too young. So we have really two major components of
people were active in it. The first we call the project management team,
and we made just about weekly to make sure that things move along smoothly. Now every member, every speaker today that
Rick introduced is part of the project management team. The person who was not able to meet with us
today is Dr. Rob Froehlich. He serves as a project manager, and he’s an
adjunct professor at the George Washington University. Secondly, we have a research team. Now everyone on this team is a PhD economist,
and they serve as consultants. I’m part of that team as well. The first member alphabetically is Chris Clapp. He’s a senior research associate at the Department
of Economics at the University of Chicago. The second member is John Pepper, who’s the
Merrill Bankard professor of economics and department chair at the University of Virginia. And the third member is Steven Stern, who’s
a professor of economics at Stony Brook University. I believe that you also asked me to talk about
the overarching theme of what we’re talking about today. And that is, I think we’ve hinted at that
already. But basically, the overarching theme of what
we’ve been doing in this project is nurturing an ongoing engagement and collaboration among,
first of all, the economists. Secondly, staff at our partner VR agencies. We’ll talk about those agencies in a minute. And third, the broader VR community. And what we’re trying to engage and collaborate
with everyone is is in developing and implementing and refining the project’s approach to VR-ROI. So we’re very interested in rigor, very interested
in interpretability, and very interested in making it useful to the entire VR community. So with that, I think I’ll turn it back to
Rick. RICK SIZEMORE: Well, thank you so much. And as we focus on all of this, we’d like
to get a sense from you of the project’s overview. And maybe you could get into some of that. ROBERT SCHMIDT: And I’d be happy to. So what we’re looking at here, and I think
we need the next slide, there we go. This is actually the second grant we received
from NIDILRR– and Ann spelled out exactly what that stands for. The second grant we received from NIDILRR. This research grant is designed to build on
that grant. The first one was I think developing an econometric
model. And it was the proof of concept. Now that that worked out fairly well, we’re
trying to expand our focus to a more diverse set of VR agencies and more recent cohorts. So we’re partnering with agencies from seven
states. And there are nine agencies. Now on the screen, they’re listed alphabetically. Now I’m going to look at them a little bit
differently. The first state mentioned is Delaware. And with them, we’re working with their general
agency. They have a blind agency as well, but we’re
working with the general agency. Additionally, we’re working with two agencies
from each of North Carolina and Virginia. So they have a separate general agency and
a separate blind agency. And we’re working with both of those. And then the other four, Kentucky, Maryland,
Oklahoma, and Texas have combined agencies. So we’re getting data for virtually all clients
in those. Now what we do is we collect cohorts of folks
who applied in a particular state fiscal year. The ones we have been looking at are applicants
in the year 2000, a separate cohort for 2007, and 2012. You’ll get a sense that we look at individual
disability, broad disability categories. And some of these disabilities have a relatively
small sample. Now our model requires a fairly large sample. So if we deal with a single agency, and that
might be if we’re looking at folks in the autism spectrum disorder and/or blind– or
separately, blind and visual impairments– those are relatively low incidence disabilities. And a single year of applicants is insufficient
for us to estimate the model accurately. And as a result, we are collecting over six
years where we have separate groups for that. Now one of the hallmarks of what we do is
we use readily available administrative data. So we do not ask agencies. We’re trying to keep this relatively convenient
for agencies to participate and hopefully benefit from what we’re working with them
on. And in doing that, we collect readily available
administrative data. So we do not ask agencies to go out and collect
surveys or collect any other data that they do not already have access to administratively. In particular, most of the data that we work
with are already in their database because they’re parts of reports to the federal government. They’re part of the standard reporting [INAUDIBLE]. So both things are in there, and we focus
on those. Additionally, we ask that they might go out
to whoever the Unemployment Insurance agency is, and we realize that sometimes that agency
charges the VR agency for records. So part of our early planning when we set
the proposal up for this project was to budget some money to help reimburse the agencies
because we realized there’s some programming costs, even though we hope to minimize them
and maybe reimburse UI agencies for that. So that’s part of our budgeting process. I think we’re ready for the next slide. Now in doing this, I want to say that when
we’re looking at the impact of service provision on the chance of getting a job or employment,
as well as earnings per folks who are employed once you gain employment, for a long, long
time the focus on doing this type of research– impact research, outcomes research– was on
what we call a generic client. So research looked at all individuals in the
program in VR, and just looked at a generic service. So individuals who receive significant or
substantive services, VR services, versus those who did not, what some people call a
treatment group and a comparison group or a control group of non-recipients. And so now what you’re really talking about
is a generic service type that receives some significant or substance services. For a generic client who receives services,
no demarcation by what type of disability they had or anything of that sort. And then the third part of that was that most
of this research looked at the report to the federal agencies, the RSA 911 report, to be
precise. And what was collected for a long time was
employment and earnings at closure. So essentially the week of closure. Now what we decided is we’d like to attack
all three of those a little differently. So on the first bullet point, you’ll see that
we generate estimates of the impact of specific types of VR services. So first of all, we don’t look at a generic
service– we do what we call crack the black box of VR services. We broadly try to aggregate service provided
into seven to nine service types that seem to make sense to folks that we have worked
with at VR agencies. Talk was about that. So we find the impact of each of these. And then so that’s one thing. The second thing is that rather than just
looking at how they did at the time they exited the VR program, we’d like to look at a longer
term on that. So that’s why we go out to UI or unemployment
insurance agencies to collect that information. So we look at a period of two to three years
before folks got into the program, collect information for that. That provides a baseline prior to service
received to see what the impact– so that’s a baseline for that. And then we try to collect at least five years
of earnings after application to VR. So we think that’s important because if you
simply look at what happened upon exit, or now with the more recent reporting requirements
a year after exit, you might miss part of the picture, might miss part of the impact. In particular, as an example, if you think
about transitioning youth who upon exiting the program might actually be in program during
high school now, during part of their schooling, and after exiting they’re just going out in
the labor force. They’re looking for a new job, they’re getting
their feet on the ground, they’re working their way up. And the service they receive, either training
or higher education services or GED or those things, it might take a little while for those
to show up, longer than certainly at exit, maybe even longer than a year. So we’d like see what happens two, three,
four, five, or more years out. Secondly, we estimate separately by disabling
condition. Now primarily when we looked at a lot our
folks who have been diagnosed with mental illness or cognitive impairment, including
both learning disabilities and intellectual disabilities, physical disabilities, including
both internal, musculoskeletal, and blindness of visual impairment. And then from those we can work up to an overall
ROI agency. So when we do our estimates, what we do is
we do separate estimates of service impacts by type of service for each disability from
each agency. So we do a lot of estimation, and it gives
us a pretty rich look at these things, what kind of commonalities there are across, as
well as what kind of differences are across the agencies. There’s more detail in our Resource section
on our website, which is Next slide. And we emphasize ongoing engagement with our
agencies in the broader VR community. And my colleagues will talk about this a little
bit more. I just wanted to talk to this, this slide,
and tell you how we go about that. So when we first begin working with an agency,
we do a little prep work. And then we make a trip, about three, four,
or five of us. Well, some people on conference call. We actually make a trip to the agency. And the purpose of this trip is several fold. One is to give folks there an idea what we
do and some past results to get an idea of what they might expect and how to interpret
it, and why this part is informative from their standpoint. As important, we try to get more information
about their particular agency, the institutional constraints, institutional structure. And one of the key things that we do in this
project is we try to capitalize on one of the very important features of the VR program. And that is a partnership, a real collaboration,
a cooperative agreement between a client and their counselor. That is, this is an individualized program. And those two folks work together in, first
of all, laying out goals for the VR stint. And then secondly, how to achieve those goals. That is, a service regiment. We take advantage of that in our modeling
to figure out who gets what kind of services and why. So that’s an important part of what we do. That’s the first thing we do. So we try to get an idea of how to group their
particular service. They might have 300 or 400 categories of services
down towards seven to nine. That’s the first thing. Then once we then start working with folks
in data and actually collect the data sets, we then do some early early checking on it. Then we put together some descriptive analysis,
like for example what percentage of folks receive this type of service, for this type
of disability, average age, is it percent for a male, and by race, and all those things
that are collected. And then we try to have a conference call
with folks there who know that. And what we’re asking is, does this stuff
pass the smell test. Does this look right? Or are we misunderstanding the data? Or did we somehow deal with it wrong? So we consistently have integrity checks built
in to the thing. That’s the second thing. Then we start estimating things, estimating
employment and earnings impacts, as well as ROI. And first we get those things. Then we probably would make another trip to
the agency to present those and really interrogate those results. Again, do these make sense? Does this look valid? What did we miss? What do we have wrong? And based on that, we then refine the model. That is, we customize it or incorporate more
of the institutional aspects that maybe we hadn’t seen in other agencies. So each one is a customized thing. What we try to do then is we try to do a methodologically
rigorous approach that is valid, understandable, and purchases a little bit more about things. Now what we’ve done is we have tested our
approach in the economic academic literature. And we have to date I think five articles
published that are reviewed by well-regarded economic academic journals. That’d be our blind review, so it’s blind
referee, double-blind referee process. So we think we’ve passed the rigor test. Next slide– I think I’m done. RICK SIZEMORE: You are indeed. And what a wonderful overview of VR-ROI. We appreciate that so much. Dr. Bob Schmidt from the University of Richmond. And he comes to us from his office in Richmond,
Virginia. Again, thank you, Bob. As we continue our conversation on the Vocational
Rehabilitation Return on Investment, we welcome Dr. Kirsten Rowe from the Virginia Department
for Aging and Rehabilitative Services. Welcome, Kirsten. KIRSTEN ROWE: Hi, Rick. RICK SIZEMORE: So, Kirsten, I’d like to get
your perspectives on the significance of the collaboration between the voc rehab professionals
and the research economists that were involved in this project. KIRSTEN ROWE: Well, as Bob has said, the VR
agencies are active partners, and the research team engages with them at multiple points
in the process. But we’ve also been engaged. VR has also been engaged and involved as an
equal partner from the very first grant applications. We were at Virginia DARS substantially involved
in helping to develop the successful grant application that led to the first NIDILRR
grant, as well as to the current one. And Dr. Joe Ashley, who has recently retired
from the Department for Aging and Rehab Services and is now a private consultant, is co-PI
on the current project and had that role while he was still working with our agency. I think all of us recognized it was really
important to have VR integrally involved in identifying the research questions and understanding
the research design and making sure it made sense in a VR context, and was going to produce
information that was relevant to VR. That level of involvement helped from the
very beginning to make sure that it was focused on the needs of the people. That is, the agencies and the people helped
by VR, so that was going to be meaningful and relevant for them. The other thing that Bob mentioned that I
think was really critical and that’s been an important reflection of the partnership
and how VR has been engaged as an equal partner has been in recognizing that participating
in this research project, while desirable and interesting and exciting, doesn’t come
without cost. And as Bob said, grant funds have been allocated,
made available to all the agencies to support some of their costs, particularly around acquisition
of data and providing of the information that researchers need to do the analysis. We also helped in the development of the current
Disability and Rehabilitation Research project grant to identify who were some of the stakeholder
groups that needed to be involved, how to engage with them, made sure that grant funds
were allocated towards those stakeholder engagement activities. And then also in helping to think about, so
what knowledge translation strategies were going to be useful for us. That is, for VR agencies and VR professionals,
so that we’ve got the academic economists audience, making sure that the research makes
sense from an academic economist research perspective. But we’ve also got the VR people, making sure
that those rigorous results make sense and are useful for these stakeholder groups, including
us. And I think that one of the things that has
made this so unique in our experience is the fact that this project by its very nature
is about understanding VR as it actually operates. It’s understanding what VR business as usual
is, rather than, as is the case with a number of other interesting research projects we’ve
been involved in, rather than laying something else on top of VR, like a new promising practice
and testing out its effectiveness in addition to the VR. So this is all about VR and what impact it
has for the people it serves. And that’s why it’s been so successful for
us and so important for us to be an equal partner. RICK SIZEMORE: And that’s, as you say, why
it has been so interesting and desirable to participate in that. Kirsten, it sounds like there’s been a lot
of discussion with vocational rehabilitation professionals as this project has evolved. So did you have any aha moments in your discussions
with the folks that you’ve talked to? KIRSTEN ROWE: Oh, yeah, multiple ahas, including
enhanced understanding of what economics is and the things that economists talk about
all the time that used to be like foreign language to me. But really in terms of engaging our other
VR stakeholders and getting VR professionals involved in the conversation, early on in
the project we were at a meeting of the Council of State Administrators of Vocational Rehabilitation–
they have national meetings– and we were explaining what the project was about and
what we hoped to get out of it and how the project was going to be implemented. And one of the participants in that meeting
commented that what we were describing, the fact that it was going to look at the specific
services specific people got, the fact that it was looking at the impact for each individual
who got services in terms of their long-term employment and earnings, his comment was,
wow, it sounds like this would be really valuable information to VR program participants to
help them make informed choices about the services that might be appropriate for them. And honestly, we hadn’t even thought about
the project having that use. But it was like, oh, yeah, this is not only
useful for us VR professionals in understanding what kinds of services are more effective
for which populations and what the employment and earnings outcomes are for the people we
serve, but, yeah, we can use this more broadly to help the people we serve understand why
it might make sense for them to get some assistive technology, how that might help them be better
employed and earn more in the long run, or how participating in post-secondary education
or training might ultimately lead to better long-term outcomes for them. That was one of the many ways in which ongoing
consultation and stakeholder engagement has been really important for the project. RICK SIZEMORE: Leading to those aha moments,
that Council of State Administrators for Vocational Rehabilitation, or, as many of us say, CSAVR,
the synergy that you find at those conferences really is a perfect backdrop to those aha
moments. And Kirsten, I just love that phrase, learning
to talk with economists because I’ve certainly over the course of this project really enjoyed
learning at a deeper level, like many of us have. So thank you so much, Kirsten. Dr. Kirsten Rowe, project coordinator, Virginia
Department for Aging and Rehabilitative Services in Richmond, Virginia where we also will connect
with Dr. Joe Ashley who just concluded a celebrated career in voc rehab at Virginia DARS. And he now is a consultant and works on a
variety of projects in the VR community. Welcome, Joe. JOSEPH ASHLEY: Well, thanks, Rick. RICK SIZEMORE: Joe, I’ve heard you mention
creating stakeholder involvement. You have a phrase that I absolutely love,
early and often. So how has this been part of the project? And what is the significance? JOSEPH ASHLEY: Well, thanks for the opportunity
today, Rick, and to all the audience. I think the early and often is a phrase that
we’ve used from the beginning from the field initiated project that Bob talked about where
we went out early on and talked with a variety of folks, including RSA and NIDILRR. We hadn’t had the grant a month before we
were up talking to leadership there about our project. And what we learned from that discussion is
that we needed to be a little clearer on one of the points Bob made– the readily available
administrative data, because a lot of people had ideas on things they’d like us to research,
but everything they were suggesting would have required a lot of data gathering. And what we are trying to do is put together
something about readily available data that can be used to turn into this long-term employment
outcome research. So our concept from that is that we learned
that we needed to be clearer on our purposes and the things that our model could do. We also learned some things that people wanted
us to be thinking about. So that became important as we began to look
down the road at the model that we have currently created. But that’s an example of some of the early
opportunities we had. And Bob mentioned some that we do going directly
out to agencies to talk to all of their staff. I would call it addressing their enlightened
self-interest, why is it important for them to be able to participate in our project. And I think that’s listening to or, if you
will, going out and actively listening to folks and providing them information so that
we have a good honest discussion about the project, what the expectations are, and the
data needs that we have. We’ve also taken time to reach out and have
a really robust advisory committee that we use where we put them to work a lot in this
project. And they were all the agency heads of any
agency involved was invited to be a part of it. They often would bring key staff too they
thought needed to hear the information. We had a state rehabilitation counselor, VR
council. And we had a group of community rehab programs,
some advisory committee, they had representation on the group. We had another economist involved from Mathematica. We had CSAVR representation. And we later added the National Council for
Administrators of the Blind agencies involved. And with that robust group, they would see
the early data, they would give us feedback. It was honest discussions about, well, does
this make sense with that smell test Bob talked about. And we would go back and forth with them on
what we were seeing and what issues might be in the data that we’re observing. This would often lead to follow-up questions
with agency staff at the administrative level, such as supervisors and directors, who then
could follow up with, well, this is what we’re observing, why would this happen. How does one determine what client goes to
which counselor? Those pieces of information to better develop
the instrumental variables that were being used. So with all of this feedback early in the
model development and ongoing in this process, it becomes important to us to be able to do
that. The other thing that we did with this was
have some focus groups that were organized. We went to CSAVR again and invited individuals
who were not necessarily agency heads in our agency, but agency heads and their key staff
who were interested. We had representatives from various state
rehabilitation councils. Maureen was good enough to get us some involvement
from the [INAUDIBLE] Center directors at that time, and a variety of other stakeholders
who might be at a CSAVR meeting. And with that group we had a professionally
facilitated discussion after we presented the model and said, how would people use this
model, what kinds of information would be needed for them to understand the model. That became important. How do you take this economics information
and turn it into something that’s more understandable and actionable? And those were very helpful to us to understand
how we needed to be thinking about putting the information out for others to consume. We also went to the National Rehabilitation
Association to get closer to people who were direct service providers and asked them similar
questions in a similar format. And they had slightly different takes on the
information they would like to see. Again, we had to put that in to the process
where we were building a model that could be understood and the information be used. As you’ll hear from Maureen more about another
piece we built in, was getting to some of these individuals that are a little closer
to the action in the VR agencies through a learning collaborative. Getting to people who understand how to translate
information to be VR audiences was critical, we thought, to being able to make sure the
information was actionable before it was over with. So I would say that is the variety of the
types of outreach efforts we’ve done. They all involve us presenting information
and getting feedback, taking the time to listen, and then take action on this information that’s
provided. I think those would be the main points, Rick. RICK SIZEMORE: Having worked with you for
over three decades, you are the master of gathering large groups of people together,
getting synergy, and learning more about what the true needs are. And I can only imagine just how exciting those
sessions were. But what were some of the takeaways that came
from your stakeholder meetings and focus groups? JOSEPH ASHLEY: Rick, I would say that it’s
the listening tour-type of option, being sure that we were listening. And then I think active learning. It became clear to these agency heads and
other participants that our economists were working very hard to learn how VR works and
better understand it so they could build a better model that represented them. We were asking these agency heads to trust
us, if you think about it, because their action data was going to be put out there to the
world. So we felt it was important to respect that
commitment and spend time learning and listening with them and then taking action. And if we couldn’t do something, they’d understand
why before it’s over with. And when you get into results, sometimes they’re
great. But if they’re not and you’ve taken the time
to do this, I think that’s where the trust comes in. And they understand that this is just where
the data led us. And we’ve had some great discussions on what
that means in certain agencies and things they might have done differently or they might
be ready to do differently. So that’s been a great part of this process. RICK SIZEMORE: And that’s wonderful. Well, if you’d like to learn more about Joe,
you can hear a podcast interview at VR Workforce Studio– A Blind Man with a Vision, which
has certainly been something many people have enjoyed. You can find that at
and iTunes, or on the Global Impact Today Radio Network. Thank you, Dr. Ashley. JOSEPH ASHLEY: Thank you, Rick. It was my pleasure. RICK SIZEMORE: Next up is Dr. Maureen McGuire-Kuletz,
who heads up the Rehabilitation Counseling Program at George Washington University. Welcome, Maureen. MAUREEN MCGUIRE-KULETZ: Rick, I’m so pleased
to be with everybody today. Thank you for including me. RICK SIZEMORE: Oh, absolutely. So Maureen, what do you see as the critical
points of knowledge translation. MAUREEN MCGUIRE-KULETZ: Well, I think that
knowledge translation is more than simply the distribution of practical scientific information,
relying on getting that information out with academic publications as the primary dissemination
plan. This project from the very beginning, through
the wonderful vision of Dr. Joe Ashley, engaged VR training experts as major project partners
from the very beginning. As you heard, resources were allocated for
stakeholder involvement throughout the process, including VR leadership, administrators, counselors,
legislators, advocacy groups, and, as you’ve also heard, people who are consumers of vocational
rehabilitation services. We have made extensive use and continue to
make extensive use of our advisory committees and expert panels. And we use learning communities, people who
are concerned and using this data, to have used that throughout the project. Anyone listening to the podcast who thinks
they’d like to learn more or become more involved in one of these learning community, please
go to our website,, and you can easily sign up for our newsletter and any
new information coming out, training opportunities, and a number of other things like that. One of the things we realized right away was
that we needed this feedback to better inform to make sure that staff knew what was going
on and to do a better job of introducing return on investment concepts to folks with the expression
I like to use, who are boots on the ground. They’re out serving in their community. How can I take this information and how can
I use it to better serve my customer or clients in the field? Then we also use these groups to help us look
at how we could develop project-related training and provide technical assistance products,
once again, for legislators, administrators, program developers, program evaluators, counselors,
advocacy groups, and consumers. RICK SIZEMORE: You mentioned the VR-ROI Newsletter. That is a fabulous resource. And for those on Twitter, you can certainly
follow a lot of what happens up-to-date and in-the-minute at @VR_ROI on Twitter. Maureen, can you tell us about a product that
came out of these stakeholder discussions and why that’s important to knowledge translation? MAUREEN MCGUIRE-KULETZ: Oh sure, Rick. Actually, there’s a number of products. Once again, when you go to the website, in
addition to the rigorous academic journal articles, there’s training, hand outs, and
a number of other things. But I’d say that one of the big things that
grew out of this was the development of the ethical use of data presentation and a Return
on Investment 101 modules. It was very interesting. At the very beginning, the National Council
of State Rehab Council Advisory Committee became very interested in the VR-ROI project,
but admitted that they weren’t terribly familiar with even what return on investment was and
what questions they need to be asking. And from their feedback, and these are consumers,
we learned and realized that we needed to create some VR, as we call them, ROI 101 modules
that they could reference and go back to and help get themselves up to speed so they knew
what they needed to be looking for as they work closely with their state agencies. Also, we heard the same thing from counselors
and some consumers. So we have found these modules, which are
available on the website at no cost. Or CRC credits are available, so please come
and check it out. We also, as I mentioned before, about the
ethical use of data. That we saw that the ethical use of data is
very important, not only by economists, but by the people who are taking the information
so that good decision-making is being made to help enhance the program and we’re not
misrepresenting the information that we find. As Dr. Ashley and Dr. Schmidt pointed out,
the data is the data. It’s making sure that it passed that quote,
unquote, “smell test” and it makes sense. But some things work well, some things don’t. The intent is not to eliminate but rather
to improve on services and find out how we can do many more good things to serve people
with disabilities. RICK SIZEMORE: It’s been so powerful to bring
groups of people who have not traditionally worked together in the past– governmental
staff and entities with economists and to have these types of discussions. So thank you for all that you’ve done. Dr. Maureen McGuire-Kuletz is the director
of Rehabilitation Counseling at George Washington University and has been the project liaison
on the Vocational Rehabilitation Return on Investment project. Thank you, Maureen. MAUREEN MCGUIRE-KULETZ: Thank you, Rick. So now it’s time for some discussion about
how we bring all of this together. And so we have some questions on the elevator
speech. So I’ll open it up. Let’s explore how we actually bring that into
focus into some short discussions that people can really grasp and carry forward. So, Joe? JOSEPH ASHLEY: I think Bob is going to start
by reviewing the elevator speech for us. ROBERT SCHMIDT: All right, I’ll be happy to,
I’ll be happy to. So the elevator speech is pretty concise. What it says, I’m just going to read it and
then I’ll talk about it a little bit more. So for those VR applicants in 2000, it says
is our earliest cohort, received VR services, 80% enjoyed earnings gains that exceeded the
costs of their VR, vocation rehab services. For every $1,000 spent by the Virginia Department
for Aging and Rehabilitative Services, the average– and by that, we mean the median,
half the folks were below, half folks were above– the median consumer earned $7,100
more over 10 years than they would have earned without VR services. And the top 10% earned $45,100 or even more
over the same period. Now there are three numbers in here, and I’d
like to explain each briefly. The first of those is that 80% enjoyed earnings
gains exceeding the costs of their VR. When we first presented material like this
in larger audiences, some agency heads, this person actually– he liked that figure. And what he liked about it– but first of
all, it’s pretty strong– but what he liked about is is he said that if you had said that
100% of applicants, 100% of customers exceeded their gains, exceeded the cost, then I wouldn’t
believe it. I’d know that you guys had rigged the data
somehow. It’s good to realize we all know that not
everyone benefits from this. Now having said that, let me say one more
thing. That these are 80% enjoyed earnings gains
that exceeded the cost of their VR. There are even more who had earnings gains
but they were not covered by the costs of VR. So that that’s the first number. The second number is that $7,100 number. Now I already said that’s the median, so half
above, half below. And basically what that says is is it’s a
seven to one ratio of earnings gains to dollars spent. So $2,000, the median we’d get $14,200. And it’s over a 10-year period. That’s at the point we make about how useful
it is to have longer run income and employment data, earnings and employment data. The third figure, and that’s really the seven
to one is quite strong. The third figure is the top 10%. So that’s a 45 to one ratio. For every $1,000, their earnings increased
by over $45,000. And that’s incredibly strong. That’s almost off the charts. So this points out one other thing we did,
which is getting away from the succinct nature of this, which is that when we make these
estimates, we actually make those earnings gains versus cost, we make them on an individual
basis. So that in this case, this is rounded up from
three different disability types– mental illness, cognitive impairments, physical impairment. And then we could round those up or scale
those up to cover the whole agency because that covers over 80% of the individuals who
applied to DARS. And doing that, we can then do an agency-wide,
or we can do it by disability type. This is simple nature. It is pretty strong results. RICK SIZEMORE: That is just excellent. I was so excited about it I jumped the gun
and wanted to go to where did this thing come from. And Joe, maybe it’s time now for you to talk
a little bit about the elevator speech and how it came about. JOSEPH ASHLEY: Thanks, Rick. Well, what happened, it started with the advisory
committee in particular. And we had some folks from the CSAVR that
are on that committee, and particularly the chair of the committee, which happened to
be my boss. Really wanted something that was more concise,
that was understandable to other people than economists, and would get people’s attention. So the concept was having something that could
be delivered relatively quickly, that met the standard of it’s accurate to what the
data shows, but was concise enough to be understood by others. And the intended audience for this was the
agency heads, the people to whom they reported. So in our case, it was the Secretary of Health
and Human Resources in Virginia. Other states have different folks. Was also used and wanted for to share with
members of Congress. And as I noted in some of the slides, when
we started this conversation, we weren’t sure how to do it. And then some of the first versions it took
a 23 story building to be able to have the elevator speech. It took a long time. But we began to narrow that down, so now it’s
what two, maybe three floors, depending on how fast your elevator is. And again, it is designed to get somebody’s
attention. In the meeting that Bob referenced earlier,
I had a five-minute opportunity during a larger meeting with the Secretary of Health and Human
Resources in Virginia, and went through this information with the secretary. And what he requested is that the whole team
come back, and we spent two hours with about 20 people that he wanted included from other
agencies and legislative staffers, those kinds of folks, to hear what we had with this information. So that’s the action that we needed hope for
to be able to do, grab attention, and then lead into a longer conversation. That’s really the pressure with running the
agency heads to we need something we can use that’s very quick. But we weren’t well the quick and dirty, and
so we went back and forth a little bit to get to this concise paragraph. RICK SIZEMORE: Now, very, very, very nice. I think we’re ready for our next slide. As the director of Wilson Workforce and Rehabilitation
Center, I know we have benefited from this ROI Project because it’s presented us information
that helped us demonstrate the program’s effectiveness, particularly in one of our programs, the PERT
Program that works with youth and helps them plan their careers. Can you describe the efforts to develop the
informational statement and how this became a joint project really between the researchers
and the PERT staff? KIRSTEN ROWE: Rick, this is Kirsten. I’ll be happy to take that question. So the researchers as well as the PERT staff,
the Post-secondary Education Rehabilitation Transition Program staff at the Wilson Center,
were interested in using the ROI model to look at that program. And because of the strength and flexibility
of the model, it was possible to use it for a specific program. And it produced some really positive and encouraging
results, which the staff of that program were thrilled to hear about, given that they work
at the front end of the rehabilitation process in conducting career and independent living
skills assessments with high school students who aren’t yet at the point of transitioning
from school to work, but who are starting that process. And then who work with community-based teams
that include school staff to implement those individual assessment findings with the students. And folks who are served by the program, as
you know, of course often get additional vocational rehabilitation services. And while the staff were excited, they weren’t
quite sure how to frame these results in a way that both made sense and was understandable
by the people they work with and still accurate in terms of maintaining the integrity of the
results. So the Wilson Center is across the Blue Ridge
Mountains from the Richmond area, so the members of the research team got in the car and drove
over the mountain and sat down with the program staff to talk through– so what do these results
mean and how can we frame them in ways that are accurate but also communicate what’s important
for us and for the people we serve. And so what they finally came up with is on
the slide– the graphic that’s on the slide comes directly out of PERT program material–
and it describes the impact of PERT on finding a job and later income. So anybody who participates in PERT has a
12% greater likelihood of finding and keeping a job. And if they go back to school for at least
one more year of education, the chance of getting and keeping a job increases by another
38%. And after finding a job, on average those
participating in PERT will have double the amount of earnings of similar people who don’t
participate in PERT. And that’s really powerful information that’s
framed in a way that respects the limits on the data results and communicates it in a
way that makes it make sense and helps provide ammunition, if you will, for staff to encourage
kids to stay in school and to continue with their vocational rehabilitation so that they
will indeed in the long run have much greater earnings than they might otherwise have. MAUREEN MCGUIRE-KULETZ: Rick, this is Maureen. If I could pop in here, I just would say this
is an excellent example of the ethical use of data– researchers working with the staff
to make sure information they’re presenting is accurate. And then the staff making sure that parents
and students understand the information and, more importantly, know how to use it to better
inform their decision-making. RICK SIZEMORE: What an excellent point. And I know here at Wilson as the staff became
aware of this information, it was a tremendous boost to their morale. And I don’t think really anyone anticipated
that really a user of this information might be the actual staff who were talking with
parents and students and offering information about the effects of what they’re getting
into at the rehab center here. And that led to a brochure that is still in
use today with that powerful information to inform, inspire, and sometimes motivate students
toward an end goal. We’re reaching the end of our webinar. Bob, do you have any closing comments? ROBERT SCHMIDT: Yes, I do. Thanks, Rick. First of all, I’m very happy to have been
part of this webinar today. For me, it’s been very useful, very interesting. But also I’d like to express my gratitude
for this whole project– it’s been wonderful. The interactive nature between the economists,
the VR agency staff, everybody on this thing, has been rewarding for all of us. The one thing you learn when you move out
of your discipline, out of your area of expertise, you’ll learn, first of all, that there’s a
lot to learn from a lot of people. Secondly, of course, and this isn’t just specific
economists, you find out that as economists or anyone in any particular area of expertise,
they often mostly deal with other people with that same area of expertise. You start talking in shorthand or what you
generally call jargon. And then you leave that jargon out on other
people and you find out that it’s not quite so clear as you thought. And as you try to explain it better and better,
you understand what’s going on better and better. And maybe you find some things where you can
prove your thought process on how to do that. And that’s certainly been the case here. As I said, it’s been iterative. We’ve learned a lot from agencies and our
trips to the agencies, not just about VR programs, but also about VR approach to the economics
of disability. For me it’s been wonderful. RICK SIZEMORE: It has been a great experience. I’d like to thank our panelists, Bob Schmidt,
Kirsten Rowe, Joe Ashley, and Maureen McGuire-Kuletz for the webinar today. And I’d like to thank you for joining us. Contact information is available through the
webinar for all of today’s speakers, in case you’d like to get in contact with them. I’d also like to invite you to join us for
inspiring success stories of vocational rehabilitation clients, as well as reflections from the champions
of business and industry that hire individuals with disabilities, and vocational rehabilitation
professionals who’ve dedicated their lives and careers to helping people with disabilities
become employed. You can join me, Rick Sizemore, for another
exciting episode of the VR Workforce Studio podcast at We have an inspiring and informative episode
that posted on May 1, featuring Shane Padgett, who became a long distance truck driver through
the E3 Targeted Communities Grant. So I hope you’ll join us for Shane’s story,
“VR at the Crossroads of Disability and Poverty.” Again, I’d like to thank you for joining us
for today’s webinar. Until next time, let’s all be the spark that
ignites vocational rehabilitation. We now turn things back over to Ann Outlaw. ANN OUTLAW: Well, thank you very much, Rick. And thank you for the entire panel for preparing
this presentation today. It was eye opening to hear the thoughtful
steps that your project takes to working together to produce those useful actionable and understandable
products for the diverse set of stakeholders you have at the table. Before we close here, I’d like to invite all
of our registrants to fill out a brief evaluation. It should just take about two or three minutes
to fill this out. If you registered for the podcast or for the
webcast today, I’ll email it to you. If you have not yet registered, you can find
it on our website at And without further ado, I’d like to thank
NIDILRR for supporting our products at the KTER Center, including this webcast and various
other activities. So thank you all and have a good day

Leave a Reply

Your email address will not be published. Required fields are marked *