Discussion: Characteristics, Challenges, and Opportunit

 

 

 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

In recent years, there has been increased pressure from funding agencies and 

federal, state and local governments for greater effectiveness and accountability 

of  prevention  and  intervention  programs.  This  rising  demand  for  program 

quality, and evidence of that quality, has fueled a growing interest in evidence‐

based  programs  (EBPs).  However,  there  remains  some  confusion  about  what 

constitutes an EBP, whether some EBPs are better than others, and the advantages 

and  disadvantages  of  implementing  EBPs.  In  this  Research  to  Practice  brief,  we 

provide an overview of what it means for a program to be evidence‐based, discuss 

the advantages and disadvantages of  implementing EBPs, and point readers  in  the 

direction of resources to help locate these programs and learn more about them. 

 

 

What are evidence‐based programs? 
A growing body of research in the social and behavioral sciences has demonstrated that 

certain approaches and strategies for working with youth and their families can positively 

impact important social problems such as delinquency, teen pregnancy, substance abuse and 

family violence. Many of these effective approaches and strategies have been packaged into 

programs targeting outcomes specific to individuals, schools, families, and communities.  

 

Those programs that have been found to be effective based on the results of rigorous evaluations 

are often called “evidence‐based.”  

 

WHAT WORKS, WISCONSIN – RESEARCH TO PRACTICE SERIES 

Evidence‐based programs:  

An overview

ISSUE #6, OCTOBER 2007 

BY SIOBHAN M. COONEY, MARY HUSER,  

STEPHEN SMALL, AND CAILIN O’CONNOR 

University of Wisconsin–Madison and University of Wisconsin–Extension 

Evidence‐based programs: An overview  2 

What Works, Wisconsin – Research to Practice Series, #6 

The importance of rigorous evaluation 
A  rigorous  evaluation  typically  involves  either  an 

experimental  design  (like  that  used  in  randomized 

controlled  trials)  or  a  quasi‐experimental  design.  In  an 

experimental design, people are randomly assigned to either 

the treatment group, which participates in the program, or 

the  control  group,  which  does  not.  After  the  program  is 

completed,  the  outcomes  of  these  two  groups  are 

compared. This type of research design helps ensure that 

any  observed  differences  in  outcomes  between  the  two 

groups are the result of the program and not other factors.  

 

Given  that randomization  is not always possible, a quasi‐

experimental design is sometimes used. In evaluations using 

this  design,  the  program  participants  are  compared  to  a 

group  of  people  similar  in  many  ways  to  the  program 

participants.  However,  because  a  quasi‐experimental 

design does not randomly assign participants to program 

and non‐program groups, it is not as strong a design as the 

experimental approach. Because there may be unobserved 

differences  between  the  two  groups  of  people  who  are 

being  compared,  this  design  does  not  allow  program 

evaluators  to  conclude  with  the  same  certainty  that  the 

program itself was responsible for the impacts observed. 

 

Most  programs  have  evaluation  evidence  from  less 

rigorous studies. Evaluations that do not include any type 

of  comparison  group,  for  example,  do  not  allow  for  any 

conclusions to be made about whether the changes seen in 

program  participants  are  related  to  or  caused  by  the 

program.  These  studies  sometimes  show  the  promise  of 

positive results, but they do not allow the program to be 

classified as evidence‐based. Programs with evidence from 

less rigorous studies are often referred  to as “promising” 

programs. 

An important element of EBPs is that they have 

been  evaluated  rigorously  in  experimental  or 

quasi‐experimental  studies  (see  box  on  this 

page).  

 

Not  only  are  the  results  of  these  evaluations 

important,  but  it  is  also  essential  that  the 

evaluations  themselves have been subjected  to 

critical peer review. That is, experts in the field 

–  not  just  the  people  who  developed  and 

evaluated  the  program  –  have  examined  the 

evaluation’s  methods  and  agreed  with  its 

conclusions about  the program’s effects. Thus, 

EBPs often have evaluation findings published 

in peer‐reviewed scientific journals. 

 

When a program has sufficient peer‐

reviewed,  empirical  evidence  for  its 

effectiveness,  its  developer  will  typi‐

cally  submit  it  to  certain  federal 

agencies  and  respected  research 

organizations for consideration. These 

organizations  “certify”  or  “endorse” 

programs by  including  them  in  their 

official  lists  of  effective  programs. 

This lets others in the field know the 

program  meets  certain  standards  of 

effectiveness.  (See  Appendix  A  for 

examples of these organizations.) 

 

Simply put, a program is judged to be 

evidence‐based  if  (a)  evaluation  re‐

search  shows  that  the  program  pro‐

duces  the  expected  positive  results; 

(b) the results can be attributed to the 

program  itself,  rather  than  to  other 

extraneous  factors  or  events;  (c)  the 

evaluation  is  peer‐reviewed  by 

experts  in  the  field;  and  (d)  the 

program  is  “endorsed”  by  a  federal 

agency  or  respected  research 

organization and included in their list 

of effective programs. 

 

Given  this definition of an EBP,  it  is 

important  to  distinguish  the  term 

“evidence‐based”  from  “research‐

based.”  Consider  our  earlier 

description  of  how  most,  if  not  all, 

EBPs were developed based on years 

of scientific research on what program 

components,  such  as  content  and 

activities, are likely to work for youth 

Evidence‐based programs: An overview  3 

What Works, Wisconsin – Research to Practice Series, #6 

and  families.  Because  EBPs  contain  program 

components with solid empirical bases, they can 

safely  be  called  “research‐based”  programs. 

However,  the  reverse  is  not  true.  Not  all,  or 

even  the majority, of research‐based programs 

fit  the  definition  of  an  EBP.  Just  because  a 

program contains research‐based content or was 

guided  by  research‐based  information,  doesn’t 

mean it has been proven effective. Unless it also 

has  scientific  evidence  that  it  works,  it  is 

incorrect to call it “evidence‐based.”
 

 

Are some evidence‐based 

programs better than others? 
Programs that meet the definition of evidence‐

based are not all similarly effective or equally 

likely to work in a given community. 

 

For  example,  some  EBPs  have  been  evaluated 

rigorously in several large‐scale evaluations that 

follow  participants  for  a  long  period  of  time. 

Others  have  only  undergone  one  or  two  less 

rigorous evaluations  (for example,  those using 

the  quasi‐experimental  design  described  on 

page 2). Those programs that are shown to be 

effective multiple times in experimental studies 

are  generally  considered  to  be  of  a  higher 

standard.  

 

Furthermore,  many  EBPs  have  been 

successfully  replicated  and  evaluated  in  a 

variety  of  settings  with  a  range  of  different 

audiences.  Others  have  only  been  evaluated 

with  a  particular  audience  in  a  certain 

geographical  area,  for  example.  When  a 

program  has  been  shown  to  be  effective  in 

different settings and with different audiences, 

it  is more  likely  that  it  will be effective when 

implemented elsewhere.  

 

Finally, EBPs can vary  in  the strength of  their 

effects.  For  example,  one  program  may  have 

evidence  that  it  reduces  delinquent  acts  in  its 

participants by 10 percent over the subsequent 

year,  while  another  program  has  evidence  of 

reducing  delinquency  by  20  or  25  percent. 

Generally, those programs that consistently pro‐

duce  a  greater  effect  than  other  programs  are 

thought to be better programs.  

 

Thus,  the  level  of  evidence  for  effectiveness 

varies across programs, and practitioners must 

use  a  critical  eye  when  judging  where  on  the 

continuum of effectiveness a program lies.  

 

 

Advantages of evidence‐based 

programs 
There  are  numerous  merits  to  adopting  and 

implementing EBPs. First, utilizing an EBP  in‐

creases the odds that the program will work as 

intended  and  that  the  public  good  will  be 

enhanced.  There  is  also  greater  efficiency  in 

using limited resources on what has been proven 

to work as compared to what people think will 

work  or  what  has  traditionally  been  done. 

Instead  of  putting  resources  toward  program 

development, organizations can select from the 

growing  number  of  EBPs,  which  are  not  only 

known to be effective but also often offer well‐

packaged program materials, staff training, and 

technical  assistance.  Using  EBPs  where 

appropriate can thus be viewed as a responsible 

and thoughtful use of limited resources.  

 

The  proven  effectiveness  that  underlies  EBPs 

can  help  secure  resources  and  support  from 

funding agencies and other stakeholders, such 

as  policy  makers,  community  leaders,  and 

members  of  the  targeted  population. 

Increasingly,  funders  and  policy  makers  are 

recommending,  if  not  requiring,  that  EBPs  be 

used  to  qualify  for  their  financial  support. 

Additionally, the demonstrated effectiveness of 

these programs can facilitate community buy‐in 

Evidence‐based programs: An overview  4 

What Works, Wisconsin – Research to Practice Series, #6 

and  the recruitment and retention of program 

participants.  

 

A  final benefit of EBPs  is  that  they may have 

cost‐benefit  information available. This type of 

information helps  to convey  the potential eco‐

nomic savings that can accrue when funds are 

invested in a program. Cost‐benefit information 

can  be  very  influential  in  an  era  where 

accountability and economic factors often drive 

public policy and funding decisions. 

 

 

Disadvantages of evidence‐based 

programs 
Despite  the  numerous  advantages  of  EBPs, 

there are some limitations that are important to 

consider.  A  major  constraint  is  the  financial 

resources needed to adopt and implement them. 

Most  EBPs  are  developed,  copyrighted,  and 

sold  at  rather  substantial  costs.  Program 

designers  often  require  that  organizations 

purchase  curricula  and  other  specially 

developed program materials,  that staff attend 

specialized  training,  and  that  program 

facilitators hold certain degrees or certifications. 

Furthermore, EBPs are often intended to be im‐

plemented  exactly  as  designed,  allowing  little 

room for local adaptation.  

 

Finally, organizations sometimes find that there 

are few or no EBPs that are both well‐suited to 

meet  the  needs  of  targeted  audiences  and 

appropriate  for  their  organization  and  local 

community setting. This situation  is especially 

common  when  it  comes  to  the  promotion  of 

positive outcomes rather than the prevention of 

negative  ones.  Because  the  development  of 

many EBPs was sponsored by federal agencies 

concerned  with  addressing  specific  problems, 

such  as  substance  abuse,  mental  illness,  

 

violence,  or  delinquency,  there  currently  exist 

many  more  problem‐focused  EBPs  than  ones 

designed  specifically  to  promote  positive 

developmental outcomes like school success or 

social responsibility. 

 

 

Where to find evidence‐based 

programs 
Practitioners  looking  for an EBP  to  implement 

in  their community or  learn more about  these 

programs will find the Internet to be their most 

useful resource. As mentioned earlier, a number 

of  federal  agencies  and  respected  research 

organizations  “certify”  or  “endorse”  programs 

that meet the organizations’ specified standards 

for effectiveness. Many of  these agencies have 

established  on‐line  registries,  of  lists  of  EBPs 

that  they  have  identified  as  effective.  While 

there are some differences in the standards used 

by  various  organizations  to  assess  whether  a 

program should be endorsed and thus included 

on their registry, most share the primary criteria 

regarding  the  need  for  strong  empirical 

evidence of program effectiveness.  

 

Organizations that endorse EBPs typically limit 

such  endorsements,  and  thus  their  program 

registry, to those programs that have shown an 

impact  on  specific  outcomes  of  interest  to  the 

organization. For example, programs  listed on 

the Office of  Juvenile  Justice and Delinquency 

Prevention’s  Model  Programs  Guide  have  all 

been  shown  to  have  an  impact  on  juvenile 

delinquency  or  well‐known  precursors  to 

delinquency.  

 

As  previously  mentioned,  because  the 

development  of  many  EBPs  was  funded  by 

federal  agencies  focused  on  specific  problems, 

most  existing  registries  of  EBPs  are  problem‐

oriented.  Occasionally,  EBPs  are  categorized 

according  to a strengths‐based orientation and 

address  outcomes  related  to  positive  youth 

Evidence‐based programs: An overview  5 

What Works, Wisconsin – Research to Practice Series, #6 

WHAT WORKS, WISCONSIN: RESEARCH TO PRACTICE SERIES
 

This is one of a series of Research to Practice briefs prepared by the What Works, Wisconsin team at the 

University of Wisconsin–Madison, School of Human Ecology, and Cooperative Extension, University of 

Wisconsin–Extension. All of the briefs can be downloaded from http://whatworks.uwex.edu.  

 

This series expands upon ideas that are discussed in What Works, Wisconsin: What Science Tells Us about 

Cost‐Effective Programs for Juvenile Delinquency Prevention, which is also available for download at the 

web address above. 

 

This publication may be cited without permission provided the source is identified as: Cooney, S.M., 

Huser, M., Small, S., & O’Connor, C. (2007). Evidence‐based programs: An overview. What Works, 

Wisconsin Research to Practice Series, 6. Madison, WI: University of Wisconsin–Madison/Extension. 

 

This project was supported, in part, by Grant Award No. JF‐04‐PO‐0025 awarded by the Wisconsin 

Office of Justice Assistance through the Wisconsin Governor’s Juvenile Justice Commission with funds 

from the Office of Juvenile Justice and Delinquency Prevention. 

 

 
 

             

development,  academic  achievement,  school 

readiness and family strengthening. 

 

While registries of EBPs are usually organized 

around  the  particular  outcomes  the  programs 

have  been  found  to  impact,  many  programs, 

especially those focused on primary prevention, 

often  have  broader  effects  than  this  pattern 

would suggest. Many EBPs have been found to 

be effective for reducing multiple problems and 

promoting a number of positive outcomes. For 

example, a parenting program that successfully 

promotes effective parenting practices may not 

only  reduce  the  likelihood  of  particular 

problems such as drug abuse or aggression, but 

may  also  promote  a  variety  of  positive 

outcomes  like  academic  success  or  stronger 

parent‐child relationships. For this reason, you 

will  often  see  the  same  program  appear  on 

multiple registries that focus on different types 

of outcomes. 

 

 

 

 

 

Now,  more  than  ever,  practitioners  have 

available to them a wealth of EBPs that build on 

the  best  available  research  on  what  works. 

Unfortunately,  they  are  currently  underused 

and often not well‐understood. Although EBPs 

do have some limitations, they can contribute to 

a  comprehensive  approach  to  preventing  a 

range of social and health‐related problems and 

enhancing  the  well‐being  of  individuals, 

families and communities.  

 

Evidence‐based programs: An overview – Appendix A  6 

What Works, Wisconsin – Research to Practice Series, #6 

Appendix A 

Evidence‐based program registries 
 

 

The following websites contain registries, or lists of evidence‐based programs, that have met specific criteria 

for effectiveness. Program registries are typically sponsored by federal agencies or other research organiza‐

tions that endorse programs at different rating levels based on evidence of effectiveness for certain participant 

outcomes. The registries listed below cover a range of areas including substance abuse and violence preven‐

tion as well as the promotion of positive outcomes such as school success and emotional and social compe‐

tence.  Generally,  registries  are  designed  to  be  used  for  finding  programs  for  implementation.  However, 

registries can also be used to learn about evidence‐based programs that may serve as models as organizations 

modify aspects of their own programs.  

 

 

Best Practices Registry for Suicide Prevention  

…

Evidence-based intervention and services for high-risk
youth: a North American perspective on the challenges of
integration for policy, practice and research

James K. Whittaker
Charles O. Cressey Endowed Professor Emeritus, School of Social Work, University of Washington, Seattle,

Washington, USA

A B S T R AC T

This paper explores the cross-national challenges of integrating
evidence-based interventions into existing services for high-resource-
using children and youth. Using several North American model
programme exemplars that have demonstrated efficacy, the paper
explores multiple challenges confronting policy-makers, evaluation
researchers and practitioners who seek to enhance outcomes for
troubled children and youth and improve overall service effective-
ness. The paper concludes with practical implications for youth and
family professionals, researchers, service agencies and policy–makers,
with particular emphasis on possibilities for cross-national
collaboration.

Correspondence:
James K. Whittaker,
School of Social Work,
University of Washington,
4101 Fifteenth Avenue NE,
Seattle, WA 98105-6299,
USA
E-mail: [email protected]

Keywords: children in need (services
for), evidence-based practice,
research in practice, therapeutic
social work

Accepted for publication: January
2009

I N T R O D U C T I O N

Across many national boundaries and within multiple
service contexts – juvenile justice, child mental and
child welfare – there is a growing concern about a
proportionately small number of multiply challenged
children and youth who consume a disproportionate
share of service resources, professional time and public
attention. While accurate, empirically validated popu-
lation estimates and descriptions remain elusive. The
consensus of many international youth and family
researchers, including those reported by McAuley
and Davis (2009) (UK), Pecora et al. (2009a) (US)
and Egelund and Lausten (2009) (Denmark) in this
present volume seems to be that some combination of
externalizing, ‘acting-out’ behaviour, problems with
substance abuse, identified and often untreated mental

health problems, experience with trauma and challeng-
ing familial and neighbourhood factors are often, and
in various combinations, manifest in the population of
children and youth most challenging to serve. Many of
these find their way into intensive out-of-home care
services, and Thoburn (2007) provides a useful
window into the out-of-home care status of children in
14 countries and offers useful observations on
improvements in collecting administrative data for
child and family services to inform both policy and
practice. Others call for a critical re-examination of the
present status of ‘placement’ as a central fulcrum
in child and family services policy and practice
(Whittaker & Maluccio 2002).

A sense of urgency is conveyed by the fact that
many child and youth clients of ‘deep-end’, restrictive
(out-of-home) services disproportionately represent
underserved and often socially excluded families and
communities of colour, and pose additional challenges
in service planning around the cultural compatibility
of proffered interventions (Blasé & Fixsen 2003;
Barbarin et al. 2004; Miranda et al. 2005). Important
work in this area includes ethnic and cultural

Author note: Portions of this paper in earlier form were
presented by the author at the 8th and 10th annual EUSARF
International Conferences at the University of Leuven,
Belgium, 9–11 April 2003 and the University of Padova,
Italy, 26–29 March 2008.

doi:10.1111/j.1365-2206.2009.00621.x

166 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd

variations on known effective practices. Lau (2006),
for example, offers a nuanced and sensitive treatment
of actual and potential adaptations in existing parent
training models. A basic concern with questions of
equity and social justice, coupled with a growing scep-
ticism about the efficacy of traditional residential,
‘place-based’ services, has heightened the search for
more preventive, family- and community-based, cul-
turally congruent service alternatives. All of this is set
against a backdrop of concern about the state’s ability
to provide effective parenting oversight and support
for children in care, as well as those who remain with
their families (Bullock et al. 2006). Fortunately, this
search is occurring at a time when researchers in many
countries are shedding light on mechanisms of risk
and resilience (Sameroff & Gutman 2004), change
processes involved in effective interventions (Biehal
2008) and the challenges faced by parents in multiply
stressed environments (Ghate & Hazel 2002; Ghate
et al. 2008) that are rich in their potential for contri-
butions to intervention design and evidence-informed
practice.

The primary purpose of this paper is to examine
some of the challenges and opportunities in incorpo-
rating evidence-based strategies and interventions
into existing service systems to better meet the needs
of high-resource-using children and youth. The
growing corpus of empirical research on promising
treatment strategies offers, if not clear-cut prescrip-
tions, then rich implications for future policy initia-
tives and service experiments.

Indeed, the pursuit of evidence-based practice, in
its many forms, increasingly attracts the attention of
those who plan, deliver and evaluate critical treatment
and rehabilitative services for vulnerable children and
their families across national boundaries and regions.
While definitions of ‘evidence-based practice’ empha-
size different dimensions of that construct, the
common themes of bringing ‘science-to-service’, and
its reciprocal ‘service-to-science’, are increasingly
evident in the child, youth and family services systems
in many European countries and North America, as
well as elsewhere. Simultaneously, reform efforts in
the USA and many European countries press for
community-based, family-oriented, non-residential
alternatives to traditional residential care and treat-
ment programmes for acting-out children and youth
with identified mental health problems (Chamberlain
2003; Weisz & Gray 2008). However, the impulse for
service reform and the availability of at least some
empirically validated model interventions do not of
themselves constitute a sufficient basis for system

reform, but instead serve to illuminate some of the
many fault lines that exist in the child and family
services field:

• The continuing tensions between ‘front-end’, pre-
ventive services and ‘deep-end’ highly intensive
treatment services and the unhelpful dichotomies
these tend to create and perpetuate

• The tensions between a widely shared desire to
adopt more evidence-based practices and the genu-
inely felt resistances to these, particularly when they
are used in a rigid fashion that requires strict adher-
ence to established protocols with little opportunity
for experimentation, customization or practitioner
discretion. For example, as one family support
researcher recently observed, we need much more
fine-grained analyses of the actual lived experience
of client families with the services offered to them
(S. P. Kemp 2008, personal communication). Such
analyses will almost certainly involved a ‘mixed-
methods’ approach using qualitative measures and
methods to augment quantitative studies

• The tension, as manifested in North America and
elsewhere between evidence-based and culturally
competent practices, reflects, among other things,
antagonism towards certain practice strategies
based on perceptions of the under-representation of
ethnic minorities in the study samples on which
certain models have been validated
As model programmes proliferate and are increas-

ingly removed from the particular political and cul-
tural niches within which they were developed, we
would do well to heed the cautions offered by Munro
et al. (2005) that researchers, planners and youth and
family practitioners are at a moment in time when
cross-national perspectives are critical in helping iden-
tify new ways of both framing problems and shaping
service solutions. Cross-national dialogue can help in
identifying different formats for collecting, analysing
and utilizing routinely gathered client information,
analysing subtle local adaptations of internationally
recognized evidence-based services and examining
the effects of differing policy contexts on service
outcomes.

T H E Q U E S T F O R M O R E E F F E C T I V E
I N T E R V E N T I O N S

For the remainder of this paper, I wish to do three
things: (1) briefly identify where we are in our search
for effective (evidence-based) interventions; (2) assess
how we are doing in increasing their availability to
high-resource-using troubled youth and their families;

Evidence-based intervention for high-risk youth J K Whittaker

167 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd

and (3) identify some particular challenges faced by
the individual practitioner, the social agency and the
public policy context in furthering the shared goal of
improving outcomes, and thus life prospects for
troubled children. The author’s bias will soon be
readily apparent. First, as one who has spent a lifetime
trying to bring both the precision of research methods
and the richness of research findings to the ‘shop
floor’ of children’s agency practice, I am convinced
that the evidence-based practice movement will not
succeed until it is embraced by those closest to the
children: the child and youth care workers, the social
workers, teachers, family support workers and others
who, with parents, toil on the front lines of helping.
This is not in my view a one-way street – Science-to-
Service – but presumes a vital feedback loop from
Service-to-Science where the insights and hypotheses
of those most directly involved in interventions
(including parent and child consumers) inform and
improve successive generations of applied research
studies. Second, I readily acknowledge the North
American bias apparent in many of my examples – I
write of what I know best – while recognizing a deeply
felt need in my country for European and other cross-
national perspectives if we are ever to achieve success
with our internal efforts at improving outcomes.

The search for evidence-based practices with chil-
dren and families is now well underway on both sides
of the Atlantic. Kazdin and Weisz (2003), Weisz
(2004), Burns and Hoagwood (2002), Macdonald
(2001), Pecora et al. (2009b) and McAuley et al.
(2006) survey effective interventions in child welfare
and child mental health services, as well as review
current research on service populations that will
inform the creation of novel interventions.

The simple, nominal definition of evidence-based
practice offered by Professor Geraldine MacDonald of
Queen’s University in Belfast provides a useful start-
ing point:

Evidence-based practice indicates an approach to decision-

making which is transparent, accountable and based on careful

consideration of the most compelling evidence we have about

the effects of particular interventions on the welfare of indi-

viduals, groups and communities. (MacDonald 2001, p. xviii)

It is clear that debates about what constitutes the
sufficiency and quality of evidence – where to set the
bar for rigour, how to distinguish evidence-based vs.
evidence-informed practice – continue apace both in
academic and practitioner discourse even as the
evidence-based practice movement as a whole contin-
ues to raise its profile in policy and services. These

competing definitions and nuances are, in toto, a sign
of health as they simply serve to underscore one or
another aspect of what is emerging as a more fulsome
understanding of what evidence-based practice con-
sists of. These aspects include, but are not limited to:

• a dual focus on aetiology and outcomes
• the incorporation of ethics and values as key com-

ponents

• the development of a collaborative process with
affected client groups

• a commitment to transparency in processes and
accountability
Many practitioners and practice researchers have

participated in the work of international groups such as
the Campbell and Cochrane Collaborations (Littell
2008) – originating in the health field – that attempt to
sift, sort and categorize the state of the evidence around
particular illnesses, socio-behavioural problems or
social welfare concerns. Many have also experienced –
closer to home – the increasing impact of national, state
and regional initiatives designed to increase the content
of proven, efficacious practices into child, youth and
family service systems. Such initiatives typically use
two strategies, often in combination:

Positive Reinforcement: e.g. ‘Laying Flowers Along Certain

Pathways’ by encouraging adoption of selected efficacious

model interventions. (One notes in passing that ‘efficacy’ of a

given intervention often increases in proportion to the dis-

tance from its country of origin!)

Coercion: e.g. Penalizing a programme, agency or practitioner

whose interventions do not reflect a sufficient quantity of

evidence-based practice according to an agreed-upon time

schedule. In the USA, this typically means that a practitioner

or service agency follows a prescribed protocol for interven-

tion or risks losing reimbursement for services rendered.

M OV I N G F R O M
‘ E F F I C AC Y- T O – E F F E C T I V E N E S S ’

In the USA at the moment, there is growing respect
for the complexities involved in moving from pilot
demonstrations of effective child, youth and family
interventions to broad-scale application: i.e. moving
from ‘efficacy’ to ‘effectiveness’ (Jensen et al. 2005;
Weisz & Gray 2008). What these terms signify are:
1. That individual investigators can demonstrate sig-
nificant results for novel treatments over standard (or
traditional) services through carefully controlled, rig-
orously conducted studies often including random-
ized controlled trials: the ‘gold standard’ of clinical
research. That is, they can demonstrate efficacy.

Evidence-based intervention for high-risk youth J K Whittaker

168 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd

2. Yet, these impressive results do not, on close
examination, appear to influence what might be
thought of as routine, day-to-day practice as con-
ducted in more familiar agency settings. Thus, the
evidence-based practice movement, while demon-
strating efficacy, cannot as yet demonstrate overall
effectiveness.

What explains this disconnect? Lisbeth Schorr, an
astute analyst of child and family services innovation,
sums it up succinctly: ‘Successful programs’, she says,
‘do not contain the seeds of their own replication’
(Schorr 1993, quoted in Fixsen et al. 2005).

Thus, if we are truly interested in effectiveness – i.e.
achieving wide-scale adoption of proven efficacious
interventions, we need to look beyond efficacy studies:
(1) to those contextual elements that influence prac-
tice decisions and client outcomes (Kemp et al. 1997);
and (2) to a different kind of research undertaking
that focuses directly on the processes involved in suc-
cessful adoption of proven efficacious interventions
(Weisz & Gray 2008).

John Weisz, one of the nation’s leading research
analysts in child mental health and a professor of
psychology at Harvard University as well as President
of the Judge Baker Children’s Center in Boston,
points the way forward on what is needed to ultimately
resolve the efficacy/effectiveness challenge:

A very important focus for the next stage of research on

interventions for children will be the effective implementation

of evidence-based practices by practitioners in service settings.

This will require an active collaboration between the research-

ers who develop and test interventions and the clinical, child

welfare, and education professionals who serve children and

families. (J.R. Weisz 2008, personal communication)

E X P L O R I N G T H E L A N D S C A P E O F
E V I D E N C E – B A S E D S E R V I C E S F O R
H I G H – R I S K YO U T H

Let us proceed, then, by exploring the context within
which evidence-based services are nested. Here, we
find some common and proximate elements familiar
to all who labour in the child and family services field,
as well as a few more distal forces that, nonetheless,
have a potential for considerable impact on the
identification, validation and eventual integration of
evidence-based practices. I will refer, briefly, to more
or less typical examples from within the US context.

Model intervention programmes

For purposes of illustration, I offer three interventions
that have received considerable attention in children’s

mental health services in the USA, and which have
been the objects of numerous community replications
and research study both in North America and else-
where (Whittaker 2005). These include:

• Multisystemic Therapy (MST), developed principally
by Dr Scott Henggeler, a psychologist now at the
Department of Psychiatry and Behavioral Sciences,
Medical University of South Carolina (Henggeler
et al. 1998; Schoenwald & Rowland 2002;
Henggeler & Lee 2003). http://www.mstservices.com

• Treatment Foster Care (MTFC), developed in several
clinical/research teams in the USA and represented
here by the model (Multi-dimensional Treatment
Foster Care) principally developed by Dr Patricia
Chamberlain and colleagues at the Oregon Social
Learning Center – a highly influential applied
behaviour analysis developmental research centre –
one of whose founding members is Dr Gerald
Patterson (Chamberlain & Reid 1998; Chamberlain
2002, 2003). http://www.MTFC.com

• Wraparound Treatment, a novel, team-oriented,
community-centred intervention developed by a
variety of individuals including the late Dr
John Burchard, formerly Professor of Clinical
Psychology at the University of Vermont, John Van
Den Berg, Carl Dennis and others beginning
in the early 1980s (Burns & Goldman 1999;
Burchard et al. 2002). http://www.rtc.pdx.edu/
PDF/PhaseActivWAProcess.pdf
[While space does not permit in depth analysis here,

the interested reader is directed to the previously cited
references, as well as to the web sites for each of these
three models that include multiple references to com-
pleted and in-progress research and demonstration
efforts, as well as specifics on programme principles
and components. A variation of the of the MTFC
model designed for younger children in regular foster
care is described in this present volume by Price et al.
(2009)].

These three interventions are specifically designed
to provide alternative pathways for children who
otherwise would be headed into more costly and
restrictive residential provision. Dr Barbara Burns,
Professor of Psychology at Duke University in North
Carolina and a principal author of the children’s
mental health section of our latest Surgeon General’s
Report on Mental Health (US Department of Health
and Human Services 1999) provides a succinct ratio-
nale for why this is warranted:

The most critical question for the future is, what will it take

to convince payers, public and private, to support the

Evidence-based intervention for high-risk youth J K Whittaker

169 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd

interventions that are backed up by evidence about improved

outcomes? Assuming that the pool of dollars available for

mental health treatment will not increase, it will be necessary

to shift resources away from institutional care (which lacks

evidence of effectiveness) toward community alternatives.

This will require a reduction in funds allocated to institu-

tional care, where a significant portion of the child mental

health money is still being spent. (Burns & Hoagwood 2002,

p. 13)

While reviews of residential care in both the UK
(Sinclair 2006) and the USA (Whittaker 2006)
confirm a move away from residential services,
recent comparative international contributions have
urged critical re-examination of the multiple varieties
of residential service (Courtney & Ivaniec 2009) to
meet the needs of at least some high-resource-
using youth. In part, this sentiment reflects the fact
that theory and model development, particularly in
the arena of intensive residential services has lan-
guished as development of comparable family-
centred services has flourished. Some have urged the
development of a conceptual schema for intensive
services – e.g. the ‘prosthetic environment’ – which
transects more traditional residential, family and
community boundaries is strengths-oriented and
incorporates educational, socialization and family
support services along with intensive treatment
(Whittaker 2005).

In focusing here on a few programme models spe-
cifically designed to serve as alternatives to residen-
tial care and treatment, and other forms of intensive
out-of-home service, one must acknowledge omis-
sion of a great deal of promising, empirically based
work that is presently being done with a wide range
of family-, school- and community-centred interven-
tions that is both more preventive in its focus and
appropriate for a much wider population of children
and families than space allows us to examine here.
See, for example, Carolyn Webster Stratton’s Incred-
ible Years Program (Beauchaine et al. 2005) and the
work of many others whose contributions in such
areas as family support illuminates a segment of ser-
vices more preventive in focus (Kemp et al. 2005;
Lightburn & Sessions 2006) and the contribution of
Jackson et al. (2009).

What, then, are the similarities and differences
of these three promising interventions? A recent
review (Burns & Hoagwood 2002) yields the
following:
1. All three interventions adhere to ‘systems of care’
values: The ‘systems of care’ framework derives from
both our National Institute of Mental Health and

private foundation initiatives in the 1980s, and is
defined as:

A comprehensive spectrum of mental health and other neces-

sary services which are organized into a coordinated network

to meet the multiple and changing needs of children and

adolescents with severe emotional disturbances and their

families. (Stroul & Friedman 1986, p. xx)

The system of care thus defined is based on three main ele-

ments. First, the mental health service system efforts are

driven by the needs and preferences of the child & family and

are addressed by a strengths-based approach. Second, the

locus and management of services occur within a multi-

agency collaborative environment grounded in a strong com-

munity base. Third, the services offered, the agencies

participating and programs generated are responsive to cul-

tural context and characteristics. [Though, as noted, this

remains a contested area with respect to some communities of

color.] (Burns & Hoagwood 2002, p. 19)

2. All three interventions are delivered in a commu-
nity – home, school, neighbourhood – context as
opposed to an office
3. All have operated in multiple service sectors:
mental health, juvenile justice, child welfare
4. All were developed and evaluated in ‘real world’
community settings, thus enhancing external validity
5. All show preference for the model treatment con-
dition in multiple randomized controlled trials
6. All lay claim to being less expensive to provide than
institutional care (Burns & Hoagwood 2002, p. 7).

Differences of course exist. For example, both MST
and MTFC possess a higher degree of specificity with
respect to intervention components than does wrap-
around. As of this writing, MST has perhaps the
strongest evidentiary base, particularly in clinical trials
showing positive effects, though some recent reviews,
including one by Prof. Julia Littell of Bryn Mawr
University in Pennsylvania conducted for the Camp-
bell Collaboration, have raised critical questions about
the evidence base offered in support of MST (Littell
2005, 2008). Finally, from a staffing perspective, MST
appears to make higher use of master’s-level-trained
professionals in service delivery than either MTFC or
wraparound.

To these three model programmes, we must of
course add numerous other evidence-based treatment
techniques targeted to specific conditions and prob-
lems, as reflected in recent reviews by Kazdin and
Weisz (2003), Weisz (2004) and Chorpita et al.
(2007). These model intervention programmes do not
of course exist in a vacuum, but both influence and are
influenced by a host of other elements in a typical state
or regional context in the USA.

Evidence-based intervention for high-risk youth J K Whittaker

170 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd

P U B L I C , V O L U N TA R Y A N D P R O P R I E TA R Y
S E R V I C E P R OV I D E R S

Model programmes such as MST, MTFC and wrap-
around are typically adopted by some segment of the
mixed system of service agencies (Public/Voluntary/
Proprietary) that make up the delivery system in a
given state, county or municipality. Public service pro-
viders are typically service funders as well, creating in
the view of some voluntary agencies an unequal influ-
ence in terms of what particular models are selected
for adoption, as well as on the masking of true admin-
istrative costs of programme implementation, given
the public sector’s economies of scale and presumed
ability to mask start-up costs. Given the wide varia-
tions in state and county service systems within the
USA, there are some anecdotal reports of the ten-
dency of certain model programmes to bend and
shape themselves into a widely varying array of
funding arrangements (referred to as ‘pretzelling’) in
order to gain a foothold and a leverage in a given
public system (K. Blasé 2007, personal communica-
tion) with the result that local service providers may
be held to similar outcome and process standards
while enjoying widely varying reimbursements to
support their efforts.

N AT I O N A L , R E G I O N A L A N D L O C A L
R E S E A R C H C E N T E R S A N D R E S O U R C E
N E T W O R K S

In addition to evidence-based programme models
that typically have their own internal capacity for pro-
gramme development, marketing, training, evaluation
and dissemination, a wide variety of university and
institute-based resource networks and research centres
play an increasingly important role in the promotion
of evidence-based programmes and practices. For
example, the National Implementation Research
Network (NIRN) was begun at the University of South
Florida as part of a larger effort to bring science-based
information to the forefront of child mental health
practice. Recently relocated to the University of
North Carolina, NIRN has done significant work in
documenting national, state and regional capacity to
support model programme development, and has
provided consultation to individual states and organi-
zations on effective strategies for integrating evidence-
based practices into the fabric of existing services
(Fixsen et al. 2005). For more information, see: http://
www.fpg.unc.edu/~NIRN/. The California Evidence-
Based Clearinghouse for Child Welfare Practice is

funded by the California Department of Social Ser-
vices, Office of Child Abuse Prevention and guided by
a state advisory committee and a National Scientific
Panel. The Clearinghouse provides guidance on
selected evidence-based practices in simple straightfor-
ward formats, reducing the consumer’s need to
conduct literature searches, review extensive literature
or understand and critique research methodology
(http://www.cachildwelfareclearinghouse.org/). The
Clearinghouse has developed a six-tiered schema for
sorting out promising programmes ranging from
‘Well-Supported – Effective Practice’ to ‘Concerning
Practice’ (e.g. shows negative effects on clients and/or
potential for harm).

A legislatively generated state institute, the
Washington State Institute on Public Policy (WSIP)
was created by the Washington state legislature to
conduct cost/benefit and a range of other studies on a
variety of classes of intervention, including child
welfare and early intervention (http://www.wsipp.
wa.gov/board.asp). Its generally thorough and well-
executed analyses have achieved wide dissemination
beyond the region and are frequently cited by model
programme developers as confirmation of their effec-
tiveness. Methodological concerns have recently been
raised about the general quality of intervention
research reviews (Littell 2005, 2008), including those
generated by WSIP, and within local practice commu-
nities, one hears anecdotally some concerns about the
potential for overly concrete inferences by legislative
bodies and funding sources whose attention may
extend only to the executive summary section of
detailed reviews of model programmes and not to the
caveats and nuances contained in their appendices
and footnotes.

Beyond these particular …

elines for Selecting an Evidence‐Based Program   

What Works, Wisconsin – Research to Practice Series, #3 

 

 
 

 

 

 

 

 

 

 

 

 

In recent years there has been a significant increase in the number of evidence‐

based  programs  designed  to  reduce  individual  and  family  problems  and 

promote healthy development. Because each program has undergone rigorous 

testing  and  evaluation,  program  practitioners  can  reassure  potential  program 

sponsors that the program is likely to be effective under the right conditions, with 

the appropriate audience and with the proper implementation. However, knowing 

which program is the “right” one for a particular setting and audience is not always 

easy to determine. When selecting a program, it is important to move beyond current 

fads or what the latest salesperson is selling and consider whether a program fits with 

the  local  agency’s  goals  and  values,  the  community  setting  and  the  needs  of  the 

targeted audience. The long‐term success of a program depends on the program being 

not only a good one, but also the right one.  

 

Unfortunately,  there  is  currently  little  research  on  how  to  best  go about  the  process  of 

selecting an evidence‐based program. Consequently, the guidelines we present in this brief 

are based primarily on our experiences working with community‐based organizations, the 

experiences of practitioners, and common sense. We have identified a number of factors that 

we believe should be considered when deciding which program is the most appropriate for a 

particular  audience  and  sponsoring  organization.  These  factors  can  be  grouped  into  three 

general categories: program match, program quality and organizational resources. In order to 

assist with the process of program selection, we have developed a set of questions to consider 

when selecting an evidence‐based program for your particular agency and audience.  

WHAT WORKS, WISCONSIN – RESEARCH TO PRACTICE SERIES 

Guidelines for selecting an evidence‐based program: 

Balancing community needs, program quality,  

and organizational resources 

ISSUE #3, MARCH 2007 

BY STEPHEN A. SMALL, SIOBHAN M. COONEY,  

GAY EASTMAN, AND CAILIN O’CONNOR  

University of Wisconsin–Madison and University of Wisconsin–Extension 

Guidelines for Selecting an Evidence‐Based Program  2 

What Works, Wisconsin – Research to Practice Series, #3 

Program match: Questions to ask 

 How well do the program’s goals and objectives 
reflect what your organization hopes to achieve? 

 How well do the program’s goals match those of 
your intended participants? 

 Is the program of sufficient length and intensity (i.e., 
“strong enough”) to be effective with this particular 

group of participants? 

 Are potential participants willing and able to make 
the time commitment required by the program? 

 Has the program demonstrated effectiveness with a 
target population similar to yours?   

 To what extent might you need to adapt this 
program to fit the needs of your community? How 

might such adaptations affect the effectiveness of the 

program? 

 Does the program allow for adaptation? 

 How well does the program complement current 
programming both in your organization and in the 

community? 

The  issues  raised  by  program  match,  program 

quality and organizational resources are overlap‐

ping.  Selecting  a  program  usually  requires 

balancing different priorities, so  it’s  important  to 

have  a  good  understanding  of  all  three  of  these 

before  determining  the  usefulness  of  a  program 

for a particular situation. 

 

PROGRAM MATCH 
A first set of factors to consider is related to how 

well the program will fit with your purposes, your 

organization,  the  target  audience,  and  the  com‐

munity where it will be implemented.  

 

Perhaps  the  most  obvious  factor  to  consider  is 

whether the goals and objectives of a program are 

consistent  with  the  goals  and  objectives  that  the 

sponsoring organization hopes  to achieve. While 

this may seem apparent,  it  is not uncommon for 

sponsors to select a program because there is grant 

money available to support it or everyone else is 

doing it. Just because a program is the latest fad or 

there’s  funding  to  support  it  doesn’t  necessarily 

mean  it  is  going  to  accomplish  the  goals  of  the 

sponsoring organization or meet the needs of the 

targeted audience. 

 

A  second  aspect  of  program  match  involves 

whether  a  program  is  strong  enough  to  address 

the level and complexity of risk factors or current 

problems  among  participants.  This  refers  to  the 

issue of adequate program duration and intensity. 

Changing existing problem behaviors or counter‐

acting  a  large  number  of  risk  factors  in  partici‐

pants’  lives  requires  many  hours  of  engaging 

programming over a period of time. For example, 

a short primary prevention program designed for 

families facing few problems or risks may not be 

effective  for  an  audience  already  experiencing 

more severe problems.  

 

Another  facet  of  program  match  concerns  the 

length of the program and whether your intended 

audience  will  be  willing  and  able  to  attend  the 

required  number  of  sessions.  Many  evidence‐

based  programs  are  of  fairly  long  duration, 

involving  multiple  sessions  over  weeks  or 

months.  A  common  concern  of  program  pro‐

viders  is  whether  potential  participants  will 

make  such  a  long‐term  commitment.  Because 

this is a realistic concern, program sponsors need 

to assess the targeted audience’s availability for 

and interest in a program of a particular length.1 

The reality  is,  if people don’t attend,  then  they 

can’t reap the program’s benefits. However, it is 

also important to keep in mind that programs of 

longer  duration  are  more  likely  to  produce 

lasting behavior change in participants. Program 

sponsors sometimes need to find a compromise 

between the most effective program and one that 

will be a realistic commitment for participants.  

 

Matching a program with the values and culture 

of the intended audience is also critically import-
ant.  Some  programs  are  intentionally  designed 

for  particular  populations  or  cultural  groups. 

Most are more culturally generic and designed 

1 Issue #2 in this series addresses strategies for 
recruiting and retaining participants.

Guidelines for Selecting an Evidence‐Based Program  3 

What Works, Wisconsin – Research to Practice Series, #3 

Program quality: Questions to ask 

 Has this program been shown to be effective? 
What is the quality of this evidence? 

 Is the level of evidence sufficient for your 
organization? 

 Is the program listed on any respected evidence‐
based program registries? What rating has it 

received on those registries? 

 For what audiences has the program been found 
to work? 

 Is there information available about what 
adaptations are acceptable if you do not 

implement this program exactly as designed? Is 

adaptation assistance available from the program 

developer? 

 What is the extent and quality of training offered 
by the program developers? 

 Do the program’s designers offer technical 
assistance? Is there a charge for this assistance? 

 What is the opinion and experience of others who 
have used the program? 

for general audiences.2  It’s  important  to consider 

whether  the  targeted  audience  will  find  the 

program acceptable and will want  to participate. 

The ideal situation would be finding evidence that 

a  program  is  effective  for  the  specific  pop-
ulation(s) you  intend to use it with. In that case, 
you  could  reasonably  expect  the  program  to  be 

effective when it is implemented well.  
 

Unfortunately,  many  evidence‐based  programs 

have only been evaluated with a limited number 

of  populations  and  under  a  relatively  narrow 

range of conditions. While many evidence‐based 

programs are effective and appropriate for a range 

of  audiences  and  situations,  it  is  rare  to  find  a 

program  that  is  suitable  or  effective  for  every 

audience  or  situation. In  many  cases,  you  will 
need to carefully read program materials or talk to 

the program’s designers to see whether adapting a 

program or using it with an audience for which it 

hasn’t been evaluated is reasonable.  

 

Depending on the design, programs may or may 

not  be  amenable  to  adaptation.  If  adapting  a 

program  to  a  particular  cultural  group  is 

important, then program sponsors should serious-
ly  consider  whether  such  changes  are  possible. 

Some program designers are willing  to help you 

with  program  adaptation  so  that  the  program’s 

effectiveness  will  not  be  undermined  by  these 

changes.3 

 

Finally,  when  considering  which  program  to 

select, sponsors should consider whether the pro-
gram complements other programs being offered 

by  the  sponsoring  organization  and  by  other 

organizations  in  the  community.  The  most 

effective  approaches  to  prevention  and  inter-
vention  involve  addressing  multiple  risk  and 

2 Issue #1 in this series addresses the issue of culture 
and evidence‐based programs.
3 Issue #4 in this series will address issues of program
fidelity and adaptation.

protective  factors,  developmental  processes  and 

settings.  Any  new  program  implemented  in  a 

community  should  address  needs  that  other 

community programs  fail  to address, which will 

help to create the kind of multi‐pronged approach 

that leads to greater overall effectiveness.  

 

PROGRAM QUALITY 
A second set of factors to consider when selecting 

a  program  are  related  to  the  quality  of  the  pro‐

gram itself and the evidence for its effectiveness.  

 

The  program  should  have  solid,  research‐based 

evidence  showing  that  it  is  effective.  For  a  pro‐

gram  to  be  deemed  evidence‐based,  it  must  go 

through  a  series  of  rigorous  evaluations.  Such 

evaluations  have  experimental  or  quasi‐experi‐

mental designs – meaning they compare a group 

of  program  participants  to  a  similar  group  of 

people who did not participate in the program to 

determine whether program participation is assoc‐

iated with positive changes. These kinds of eval‐

Guidelines for Selecting an Evidence‐Based Program  4 

What Works, Wisconsin – Research to Practice Series, #3 

TABLE 1: Selected evidence‐based program registries 

Blueprints for Violence Prevention 

http://www.colorado.edu/cspv/blueprints/index.html 

This registry is one of the most stringent in terms of endorsing programs as Model or Promising. Programs are 

reviewed by an expert panel and staff at the University of Colorado, and endorsements are updated regularly. 

Programs are added and excluded from the registry based on new evaluation findings. 

Helping America’s Youth 

http://guide.helpingamericasyouth.gov/programtool.cfm 

This registry was developed with the help of several federal agencies. Programs focus on a range of youth 

outcomes and are categorized as Level 1, Level 2, or Level 3 according to their demonstrated effectiveness. The 

registry is updated regularly to incorporate new evidence‐based programs. 

Office of Juvenile Justice and Delinquency Prevention Model Program Guide  

http://www.dsgonline.com/mpg2.5/mpg_index.htm 

This registry is one of the largest currently available and is continuously updated to include new programs. 

Programs found on this registry are designated as Exemplary, Effective, or Promising. 

Promising Practices Network 

http://www.promisingpractices.net/ 

A project of the RAND Corporation, this registry regularly updates its listings of Effective and Promising 

programs. Programs are reviewed and endorsed by project staff. 

Strengthening Americaʹs Families  

http://www.strengtheningfamilies.org/html/  

Although this registry was last revised in 1999, it is the only registry with a focus specifically on family‐based 

programs. Programs were reviewed by expert panels and staff at the University of Utah and the Center for 

Substance Abuse Prevention. They were then designated as Exemplary I, Exemplary II, Model, or Promising.  

Substance Abuse and Mental Health Services Administration (SAMHSA) National Registry of Evidence‐

Based Programs and Practices  

http://www.nrepp.samhsa.gov 

This recently re‐launched site no longer categorizes programs as Model, Effective, or Promising. Instead, 

programs are summarized and the quality of the research findings is rated separately for each outcome that has 

been evaluated. SAMHSA has also introduced a “Readiness for Dissemination” rating for each reviewed program. 

Nominations are accepted each year for programs to be reviewed; SAMHSA funds independent consultants to 

review nominated programs and update the registry. 

uations allow for a reasonable assumption that it 

was  the  program  itself  that  changed  people’s 

knowledge, attitudes or behavior.  

 

As  funders and program sponsors become more 

committed  to  implementing  evidence‐based  pro‐

grams, program developers are increasingly likely 

to  promote  their  programs  as  evidence‐based. 

However,  just  because  a  program  developer  ad‐

vertises  a  program  as  evidence‐based  doesn’t 

mean that it meets the standards discussed above. 

For  example,  a  program  might  be  “research‐

based,”  but  not  “evidence‐based.”  A  research‐

based  program  has  been  developed  based  on 

research about the outcomes or processes it add‐

resses.  However,  it  has  probably  not  been 

subjected  to  the  rigorous  evaluations  and  real‐

world  testing  that  are  needed  to  designate  a 

program as evidence‐based. The simplest way to 

determine evidence of a program’s effectiveness is 

Guidelines for Selecting an Evidence‐Based Program  5 

What Works, Wisconsin – Research to Practice Series, #3 

Organizational resources:  

Questions to ask 

 What are the training, curriculum, and 
implementation costs of the program? 

 Can your organization afford to implement this 
program now and in the long‐term? 

 Do you have staff capable of implementing this 
program? Do they have the qualifications 

recommended or required to facilitate the 

program? 

 Would your staff be enthusiastic about a program 
of this kind and are they willing to make the 

necessary time commitment? 

 Can this program be implemented in the time 
available? 

 What’s the likelihood that this program will be 
sustained in the future? 

 Are your community partners supportive of your 
implementation of this program? 

to examine  the designations given by well‐estab‐

lished  and  respected  evidence‐based  program 

registries. Program registries classify programs at 

different levels of endorsement based on evidence 

of  effectiveness  for  certain  participant  outcomes. 

See  Table  1  for  an  annotated  listing  of  program 

registries.  

 

If a program is not listed on a respected registry, 

then it is important to seek out scientific evidence 

of the program’s effectiveness. At a minimum, you 

should  review  any  evaluation  studies  that  have 

been  conducted  by  the  program  developer  and 

external evaluators. Ideally, these evaluations use 

an  experimental  or  quasi‐experimental  research 

design. Another sign of a high‐quality evaluation 

is  that  its results have been published  in a well‐

respected, peer‐reviewed, scientific journal. 

 

An  additional  indicator  of  program  quality  to 

consider  is  the  level  of  training  and  follow‐up 

support  available  from  the  program  designers. 

Some  programs  have  a  great  deal  of  resources 

available  to  help  program  implementers.  These 

resources  can  be  especially  important  if  you’re 

working  with  a  unique  audience  and  need  to 

make adaptations or if program implementation is 

particularly complex. As a general rule, more in‐

tensive training and more follow‐up support from 

the program developer will increase the effective‐

ness  and  sustainability  of  a  program  over  time. 

Some programs provide excellent  technical assis‐

tance; staff members are accessible and willing to 

address questions that arise while the program is 

being implemented. Often this technical assistance 

is  free, but sometimes program designers charge 

an additional fee for it. Therefore, the benefits and 

costs  of  technical  assistance  should  be  kept  in 

mind when selecting an evidence‐based program. 

 

Finally,  while  the  scientific  literature  and  infor‐

mation from the program developer provide key 

information  about  program  quality,  don’t  over‐

look the experience of practitioners who have imp‐

lemented  the  program.  …