Summary of the Wraparound Evidence Base

Summary of the Wraparound Evidence Base Summary of the Wraparound Evidence Base: April 2010 Eric J. Bruns, Ph.D. Associate Professor, University of Washington School of Medicine Co-Director, National Wraparound Initiative Jesse C. Suter, Ph.D. Research Assistant Professor, University of Vermont Wraparound is a team-based planning process intended to provide coordinated, holistic, family-driven care to meet the complex needs of youth who are involved with multiple systems (e.g. mental health, child welfare, juvenile justice, special education), at risk of placement in institutional settings, and/or experiencing serious emotional or behavioral difficulties (Walker & Bruns, 2008). Wraparound provides an “on the ground” mechanism for ensuring that core system of care values will guide planning and produce individualized, family-driven and youth-guided support that is community based and culturally competent (Stroul & Friedman, 1996). In the children’s services field, there is broad consensus that for youth and families with multiple and complex needs, the wraparound paradigm is an improvement over more traditional service delivery methods that are uncoordinated, professional-driven, deficit-based, and overly reliant on out of home placement. This is reflected in wraparound’s widespread adoption nationally and worldwide. A 2007 survey shows that 91% of U.S. states have some type of wraparound initiative, with 62% implementing some type of statewide initiative. Over 100,000 youth nationally are estimated to be engaged in a well-defined wraparound process (Bruns, Sather, & Stambaugh, 2008). Regardless of how popular an intervention is with providers or families, or how well it conforms to current values of care, such criteria can not be used as the sole basis for policy making or treatment decision making. In the current era of “evidence-based practice,” decisions regarding how we invest our scarce health care resources – as well as decisions about what treatment approaches will be used with a given youth or family – must also be based on evidence derived from properly designed evaluations. After all, youth with complex needs may be served via a range of alternative approaches, such as via traditional case management or through uncoordinated “services as usual” (in which families negotiate services and supports by themselves or with help of a more specialized provider such as a therapist). Other communities may choose to invest in an array of more specialized office- or community-based evidence-based practices that address specific problem areas, in the absence of wraparound care coordination. And of course, many communities continue to allocate significant behavioral health resources to out-of-community options such as residential treatment, group homes, and inpatient hospitalization. The range of options in which states and localities may invest, combined with resource limitations, demands that we develop evidence for what models work for which youth under which conditions. Increasingly, investment in wraparound is backed by controlled research. As of 2003, when the first meeting of the National Wraparound Initiative was held, there were only three controlled (i.e., experimental or quasi-experimental) studies of wraparound effects published in peer-reviewed journals. As of 2010, there are now nine controlled, published studies. Several of these newer studies include fidelity data as well as cost data, increasing our understanding of wraparound’s potential for impact and what is required to achieve that impact. In addition, the first meta-analysis of wraparound has now been published (Suter & Bruns, 2009). As a result of this expansion in controlled research, as well as the greater availability of dissemination materials, Wraparound is currently being reviewed for inclusion in the Substance Abuse and Mental Health Services Administration (SAMHSA) National Registry of Evidence-based Programs and Practices (NREPP). Kazdin (1999) says there are four criteria for assessing the status of an intervention’s evidence base: (1) A theory to relate a hypothesized mechanism to a clinical problem; (2) Basic research to assess the validity of the mechanism; (3) Outcome evidence to show that a therapeutic approach changes the relevant outcomes; and (4) Process-outcome connections, which display the relationships between process change and clinical outcomes. With respect to criteria 1 and 2, for youth and families with complex and overlapping needs, the theory of change for wraparound (Walker, 2008) provides rationale (with supportive basic research) for why wraparound treatment planning is likely to be more effective than services provided in the absence of this process. Some of the specific mechanisms of change include better treatment acceptability and youth/family engagement; better teamwork; an emphasis on problem solving; and an emphasis on increasing optimism, hope, self-efficacy, and social support. For condition 4, research is increasingly showing associations between system-, organizational-, and team-level fidelity and child and family outcomes. Bruns et al. (2005; 2006; 2008) as well as other authors (e.g., Walton & Effland, 2010) have shown that communities that adhere more closely to the wraparound principles as assessed via measures such as the Wraparound Fidelity Index tend to show more positive outcomes. On the flip side, communities with better developed system supports for wraparound tend to demonstrate higher fidelity scores. (You can see an entire section in the Resource Guide to Wraparound on this evidence). Ultimately, however, it is outcomes evidence from rigorous studies (criterion no.3) that is most relevant to evaluating an intervention’s evidence base. As described in our review of wraparound research, as of 2008, we found 36 published outcomes studies of wraparound. However, only a small number of these (n=7) were controlled studies that used random assignment or some type of comparison group design. In 2009, we published a meta-analytic review of these seven studies (Suter & Bruns, 2009). This analysis found that, on average across these studies, significant effects of wraparound were found for all four outcome domains we examined, including living situation, youth behavior, youth functioning, and youth community adjustment. Mean effect sizes across these domains (calculated as the difference between wraparound and control group means at posttest divided by the pooled standard deviation, or Cohen’s d) ranged from .25 to .59, with the largest effects found for living situation outcomes (e.g., youth residing in less restrictive, community placements and/or greater stability of placement). The mean effect size across all outcomes was .33–.40, depending on whether studies for which effect sizes were imputed were included (d=.33) or excluded (d=.40). These effect sizes are quite similar to effects found for established EBPs implemented under “real world” conditions and compared to some type of alternative treatment condition (Weisz, Jensen-Doss, & Hawley, 2006). As of 2010, there have been nine controlled studies of wraparound that have been published in peer reviewed publications. In the rest of this document, we present a summary of each of these studies (Table 1), followed by a summary of all significant behavioral outcomes found across the controlled studies 1(Table 2). Though many of these studies have significant methodological weaknesses, the “weight of the evidence” of these studies indicates superior outcomes for youth who receive wraparound compared to similar youth who receive some alternative service. On the strength of these studies, as well as others currently being completed, it is likely that wraparound will increasingly be referred to as an “evidence-based” process in the future. At the same time, much more wraparound research is needed. The diversity of contexts in which wraparound is implemented (e.g., for youths from birth to transition age as well as adults, and in contexts as varied as mental health, juvenile justice, child welfare, and schools) demands more effectiveness studies, so that we can better understand for which individuals and in what contexts wraparound is most likely to be effective. The many ways in which wraparound can be implemented also demand an expansion of the implementation research base on wraparound. For example, what are outcomes and costs of achieving different levels of fidelity? What modifications to the practice model achieve the best results? What training, coaching, and supervision yield the best fidelity, staff, and youth and family outcomes? What is needed at the organizational and system level to support high-quality wraparound implementation? Though the wraparound research base continues to grow, so does the list of questions for which we seek answers. 1 Two notes on the studies included in Tables 1 and 2 and the Suter & Bruns (2009) meta-analysis are worth making. First, one study included in Table 1 (Myaard et al., 2000) studied outcomes for N=4 youths participating in wraparound with outcomes assessed using a multiple baseline experimental design. Given this research design, this study is worthy of inclusion in a review of rigorous wraparound studies; however, due to its unique multiple baseline design, this study was not included in the 2009 meta-analysis nor are its outcomes included in Table 2. Second, one of the studies included in the meta-analysis (Bickman et al., 2003) presented evidence indicating that the “wraparound” condition that was evaluated did not conform to the principles or practice model of wraparound and was not meaningfully different from the comparison condition. Thus, while this study was included in the meta-analysis to be conservative, it is not included in Table 1 or 2. Table 1. Summary of nine published experimental and quasi-experimental outcomes research studies of wraparound. NOTE: The research selected for inclusion in this Table includes the nine experimental and quasi- experimental outcomes research studies published in peer-reviewed journals relevant to the wraparound process (8 controlled studies and 1 multiple-baseline study). Studies are organized by the population studied. These include four studies of youths served through the child welfare system, two studies of youths served because of their involvement in (or risk of involvement in) juvenile justice, and four studies of youths served because of their intensive mental health needs. Study Citations Outcome(s) Child Welfare Randomized control Clark, Lee, Significantly fewer placement changes for youths in study (18 months) of Prange, & the wraparound program, fewer days on runaway, youth in child welfare McDonald, fewer days incarcerated (for subset of incarcerated custody in Florida: 54 in 1996; youths), and older youths were significantly more wraparound vs. 78 in Clark et al., likely to be in a permanency plan at follow-up. No standard practice foster 1998. group differences were found on rate of placement care. changes, days absent, or days suspended. No differences on internalizing problems, but boys in wraparound showed significantly greater improvement on externalizing problems than the comparison group. Taken together, the findings provided moderate evidence for better outcomes for the wraparound program; however, differences appear somewhat limited to boys and externalizing problems. Matched comparison Bruns, Rast, After 18 months, 27 of the 33 youth (approximately study (18 months) of Walker, 82%) who received wraparound moved to less youth in child welfare Bosworth, & restrictive environments, compared to only 12 of the custody in Nevada: 33 in Peterson, 32 comparison group youth (approximately 38%), and wraparound vs. 32 2006; Rast, family members were identified to provide care for 11 receiving MH services as Bruns, Brown, of the 33 youth in the wraparound group compared to usual Peterson, & only six in the comparison group. Mean CAFAS scores Mears, 2007 for youth in wraparound decreased significantly across all waves of data collection (6, 12, 18 months) in comparison to the traditional services group. More positive outcomes were also found for the wraparound cohort on school attendance, school disciplinary actions, and grade point averages. No significant differences were found in favor of the comparison group. Matched comparison Rauso, Ly, Initial analyses for a larger matched sample of youth study (12 months) of Lee, & Jarosz, (n=102 wraparound vs. n=210 for group care) found N=210 youth in child 2009 that 58% (n=59) of youth discharged from welfare custody in Los wraparound had their case closed to child welfare Angeles County: 43 within 12 months, compared to only 16% (n=33) of discharged from youth discharged from group care. Wraparound vs. 177 Of those youth who remained in the care of child discharged from group welfare for the full 12 months follow-up period (n=43 care. for wraparound vs. n=177 for group care), youth in the wraparound group experienced significantly fewer out of home placements (mean = 0.91 compared to 2.15 for the comparison group). Youth in the wraparound group also had significantly fewer total mean days in out of home placements (193 days compared to 290). During the 12-months follow-up, 77% of the Wraparound graduates were placed in less restrictive settings while 70% of children who were discharged from RCL 12-14 were placed in more restrictive environments. Mean post-graduation cost for the wraparound group was found to be $10,737 compared to $27,383 for the group care group. Matched comparison Mears, Yaffe, Youth in the wraparound group approach showed study (6 months) of & Harris, significantly greater improvement in functioning N=126 youths involved 2009 (d=.50) as assessed by the Child and Adolescent in the child welfare Functional Assessment Scale (CAFAS) compared to system in Clark County, youth receiving traditional child welfare services. NV: 96 in wraparound Youth in the wraparound group also showed vs. 30 in traditional child significantly greater movement toward less restrictive welfare case residential placements (d=.71) as assessed by the management. Restrictiveness of Living Environment Scale (ROLES). More wraparound youth experienced a placement change during the 6 month follow up (23% vs. 49%); however, this was due to youth in the wraparound group being more likely to move to less restrictive placements during the study period. No differences were found for child behavior as assessed by the CBCL, school, or juvenile justice outcomes. Juvenile Justice Randomized control Carney & Study supported the hypothesis that youth who study (18 months) of “at Buttell, 2003 received wraparound services were less likely to risk” and juvenile justice engage in subsequent at-risk and delinquent behavior. involved (adjudicated) The youth who received wraparound services were youth in Ohio: 73 in less likely to miss school unexcused, get expelled or wraparound vs. 68 in suspended from school, run away from home, or get conventional services picked up by the police as frequently as the youth who received the juvenile court conventional services. There were, however, no significant differences, in formal criminal offenses. Matched comparison Pullmann, Youths in the comparison group were three times study (>2 years) of Kerbs, more likely to commit a felony offense than youths in youth involved in Koroloff, the wraparound group. Among youth in the juvenile justice and Veach-White, wraparound program, 72% served detention “at some receiving MH services: Gaylor, & point in the 790 day post identification window” (p. 110 youth in wraparound Sieler, 2006 388), while all youth in the comparison group were vs. 98 in conventional subsequently served in detention. Of youth in the MH services Connections program who did serve detention, they did so significantly less often than their peers. Connections youth also took three times longer to recidivate than those in the comparison group. According to the authors, a previous study by Pullman and colleagues also showed “significant improvement on standardized measures of behavioral and emotional problems, increases in behavioral and emotional strengths, and improved functioning at home at school, and in the community” (p. 388) among Connections youth. Mental Health Randomized control Evans, Significant group differences were found in favor of the study (12 months) of Armstrong, & case management/ wraparound program for behavioral youths referred to out-Kuppinger, and mood functioning. No differences were found, of-home placements for 1996; however, with respect to behavior problems serious mental health Evans, (internalizing and externalizing), family cohesiveness, problems in New York Armstrong, or self-esteem. No differences found in favor of the State: 27 to family Kuppinger, TFC group. Overall, small sample size plus loss of data centered intensive case Huz, & on many of the outcome measures resulted in the management McNulty,1998 study having very low power to detect differences (wraparound) vs. 15 to between groups. treatment foster care. Quasi-experimental (24 Hyde, Primary outcome was a single rating that combined months) study of youths Burchard, & several indicators: restrictiveness of youth living with serious mental Woodworth, situation, school attendance, job/job training health issues in urban 1996 attendance, and serious problem behaviors. Youths Baltimore: 45 returned received ratings of “good” if they were living in regular or diverted from community placements, attending school and/or residential care to working for the majority of the week, and had fewer wraparound vs. 24 than three days of serious behavior problems during comparison youths. the course of previous month. At 2-year follow-up, 47% of the wraparound groups received a rating of “good,” compared to 8% of youths in traditional MH services. Limitations of the study include substantial study attrition and group non-equivalence at baseline. Experimental (multiple-Myaard, The multiple baseline case study design was used to baseline case study) Crawford, evaluate the impact of wraparound by assessing study of four youths Jackson, & whether outcome change occurred with (and only referred to wraparound Alessi (2000). with) the introduction of wraparound at different because of serious points in time. The authors tracked occurrence of five mental health issues in behaviors (compliance, peer interactions, physical rural Michigan. aggression, alcohol and drug use, and extreme verbal abuse) for each of the youths. Participants began receiving wraparound after 12, 15, 19, and 22 weeks. For all four participants, on all five behaviors, dramatic improvements occurred immediately following the introduction of wraparound. Table 2. Summary of all behavioral outcomes for the wraparound process with supporting citations from eight controlled studies Section 1: Statistically significant (p<.05) behavioral outcomes Effect Outcome Citation Size Carney & Buttell, 2003, p. Less assaultive 0.30 561 Carney & Buttell, 2003, p. Ran away less 0.45 561 Carney & Buttell, 2003, p. Suspended from school less 0.47 561 Carney & Buttell, 2003, p. Missed less school 0.47 561 Carney & Buttell, 2003, p. Less likely to be picked up by police 0.49 561 Less likely to be suspended from school 0.22 Clark et al., 1998, p. 530 Less likely to spend more time incarcerated 0.31 Clark et al., 1998, p. 529 Fewer days on runaway 0.34 Clark et al., 1998, p. 528 Residing in more permanency-type settings 0.17 Clark et al., 1998, p. 526 Less likely to spend time on runaway 0.22 Clark et al., 1998, p. 529 Less likely to experience a high number of 0.25 Clark et al., 1998, p. 529 placement changes Improved behavioral functioning on CAFAS 0.61 Evans et al., 1998, p. 566 Improved moods / emotions on CAFAS 0.61 Evans et al., 1998, p. 566 Improved overall functioning on CAFAS 0.50 Mears et al., 2009, p. 682 Residing in less restrictive placements 0.71 Mears et al., 2009, p. 682 Reduced recidivism for any offense 0.25 Pullman et al., 2006, p. 386 Reduced recidivism for felony 0.26 Pullman et al., 2006, p. 388 Fewer days served in detention 0.66 Pullman et al., 2006, p. 388 Fewer episodes in detention 0.75 Pullman et al., 2006, p. 388 Less likely to serve in detention 0.85 Pullman et al., 2006, p. 388 Improved school GPA 0.69 Rast et al., 2007, p. 22 Improved overall functioning on CAFAS 0.69 Rast et al., 2007, p. 20 Fewer disciplinary actions 0.95 Rast et al., 2007, p. 22 Moved to less restrictive living environments 1.09 Rast et al., 2007, p. 21 Fewer emotional and behavioral problems on 0.86 Rast et al., 2007, p. 19 CBCL Fewer out-of-home placements 0.84 Rauso et al., 2009, p. 65 More stable living environment 0.57 Rauso et al., 2009, p. 66-67 Residing in less restrictive placements 0.98 Rauso et al., 2009, p. 66 Section 2: Behavioral outcomes that were not statistically significant, but with positive effect sizes Outcome Effect Size Citation Carney & Buttell, 2003, p. Less likely to be arrested 0.23 561 Less likely to be in clinical range on CBCL or YSR 0.23 Clark et al., 1998, p. 532 Fewer unexcused absences 0.50 Rast et al., 2007, p. 22 Combined rating indicating lower restrictiveness of placement, improved school attendance, and 0.68 Hyde et al., 1996, p. 78 fewer negative behaviors. Note on effect sizes: The effect size reported for these outcomes is the standardized mean difference, typically referred to as Cohen’s d (1988). Effect sizes were calculated as the difference between wraparound and control group means at posttest divided by the pooled standard deviation. Effect sizes were generated using an effect size program created by Wilson (2004) and presented such that positive values always indicated positive results for youth receiving wraparound relative to youth in control groups. All effect sizes were adjusted using Hedges’ small sample size correction to create unbiased estimates (Hedges & Olkin, 1985). The magnitude of effects is typically interpreted using Cohen’s (1988) guides for small (0.20), medium (0.50), and large (0.80) effects. References for Outcomes Review Bruns, E.J., Rast, J., Walker, J.S., Peterson, C.R., & Bosworth, J. (2006). Spreadsheets, service providers, and the statehouse: Using data and the wraparound process to reform systems for children and families. American Journal of Community Psychology, 38, 201-212. Carney, M. M., & Buttell, F. (2003). Reducing juvenile recidivism: Evaluating the wraparound services model. Research on Social Work Practice, 13, 551-568. Clark, H.B., Lee, B., Prange, M.E. & McDonald, B.A. (1996).Children lost within the foster care system: Can wraparound service strategies improve placement outcomes? Journal of Child and Family Studies, 5, 39-54. Clark, H. B., Prange, M. E., Lee, B., Stewart, E. S., McDonald, B. B., & Boyd, L. A. (1998). An individualized wraparound process for children in foster care with emotional/behavioral disturbances: Follow-up findings and implications from a controlled study. In M. H. Epstein, K. Kutash & A. Duchnowski (Eds.), Outcomes for children and youth with emotional and behavioral disorders and their families: Programs and evaluation best practices (pp. 513-542). Austin, TX: Pro-ED, Inc. Evans, M.E., Armstrong, M.I., Kuppinger, A.D. (1996). Family-Centered Intensive Case Management: A Step Toward Understanding Individualized Care. Journal of Child and Family Studies, 5, 55-65. Evans, M. E., Armstrong, M. I., Kuppinger, A. D., Huz, S., & McNulty, T. L. (1998). Preliminary outcomes of an experimental study comparing treatment foster care and family-centered intensive case management. In Epstein, M.H. (Ed); Kutash, K (Ed); et al. (1998). Outcomes for children and youth with emotional and behavioral disorders and their families: Programs and evaluation best practices. (pp. 543-580). Xviii, 738 pp. Hyde, K. L., Burchard, J. D., & Woodworth, K. (1996). Wrapping services in an urban setting. Journal of Child & Family Studies, 5, 67-82. Mears, S. L., Yaffe, J., & Harris, N. J. (2009). Evaluation of wraparound services for severely emotionally disturbed youths. Research on social work practice, 19, 678-685. Myaard, M. J., Crawford, C., Jackson, M., & Alessi, G. (2000). Applying behavior analysis within the wraparound process: A multiple baseline study. Journal of Emotional & Behavioral Disorders, 8, 216-229. Pullmann, M. D., Kerbs, J., Koroloff , N., Veach-White, E., Gaylor, R., & Sieler, D. (2006). Juvenile offenders with mental health needs: Reducing recidivism using wraparound. Crime and Delinquency, 52, 375-397. Rast, J., Bruns, E.J., Brown, E.C., Peterson, C.R., & Mears, S.L. (2007). Impact of the wraparound process in a child welfare system: Results of a matched comparison study. Unpublished program evaluation. Rauso, M., Ly, T. M., Lee, M. H., & Jarosz, C. J. (2009). Improving outcomes for foster care youth with complex emotional and behavioral needs: A comparison of outcomes for wraparound vs. residential care in Los Angeles County. Emotional & Behavioral Disorders in Youth, 9, 63-68, 74-75. Other References Bickman, L., Smith, C., Lambert, E. W., & Andrade, A. R. (2003). Evaluation of a congressionally mandated wraparound demonstration. Journal of Child & Family Studies, 12, 135-156. Bruns, E.J., Leverentz-Brady, K.M., & Suter, J.C. (2008). Is it wraparound yet? Setting fidelity standards for the wraparound process. Journal of Behavioral Health Services and Research, 35, 240-252. Bruns, E.J., Sather, A., & Stambaugh, L. (2008). National trends in implementing wraparound: Results from the state wraparound survey, 2007. In E.J. Bruns & J.S. Walker (Eds.), Resource guide to wraparound. Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health. Bruns, E.J., Suter, J.S., Force, M.D., & Burchard, J.D. (2005). Adherence to wraparound principles and association with outcomes. Journal of Child and Family Studies, 14, 521-534. Bruns, E.J., Suter, J.S, & Leverentz-Brady, K. (2006). Relations between program and system variables and fidelity to the wraparound process for children and families. Psychiatric Services, 57, 1586-1593. Kazdin AE: Current (lack of) status of theory in child and adolescent psychotherapy research. Journal of Clinical Child Psychology 28:533–543, 1999 Stroul, B.A., & Friedman, R.M. (1996). A system of care for children and youth with severe emotional disturbances. Washington, DC: CASSP Technical Assistance Center, Center for Child Health and Mental Health Policy, Georgetown University Child Development Center. Suter, J.C. & Bruns, E.J. (2009). Effectiveness of the Wraparound Process for Children with Emotional and Behavioral Disorders: A Meta-Analysis. Clinical Child and Family Psychology Review, 12, 336-351. Walker, J. S. (2008a). How, and why, does wraparound work: A theory of change. In E. J. Bruns & J. S. Walker (Eds.), The resource guide to wraparound. Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health. Weisz, J.R., Jensen-Doss, A., & Hawley, K.M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. The American Psychologist, 61, 671-689.