Unclaimed Children Revisited: Focusing on Outcomes – A Case Study of the Michigan Level of Functioning Project

Download PDF

Learn about our Unclaimed Children Revisited project.

This is an excerpt from the full report.

Introduction

Unclaimed Children Revisited Background

Unclaimed Children Revisited (UCR) updates Unclaimed Children: The Failure of Public Responsibility to Children in Need of Mental Health Services (1982), a seminal report authored by NCCP’s former director, Dr. Jane Knitzer. The initial report served to rally the child and adolescent mental health field to take policy action towards reform.

The current study is a multi-pronged initiative that generates new knowledge about policies across the United States that promote or inhibit the delivery of high-quality mental health services and supports to children, youth, and families. UCR places a strong emphasis on identifying services that are culturally competent, developmentally appropriate, and research-informed. The initiative encompasses four main projects:

• national survey of state-level children’s mental health directors and advocates;
• statewide case study of California, with a focus on 11 counties;
• case study of outcomes-based management in children’s mental health service delivery in Michigan; and,
• working paper series that explore the state of the field on family and youth engagement, financing, trauma, school-based mental health and cross-systems support of effective practices.

The Michigan Case Study

The major aim of Unclaimed Children Revisited is to identify policy-supported state efforts to promote quality of care for children and youth with mental health conditions in the public mental health system. The Michigan Case Study examines one such initiative – Michigan’s Level of Functioning Project (LOF). It is a 14 year-old effort to monitor and improve outcomes for children and youth with severe emotional disturbance (SED), through the use of the Child and Adolescent Functional Assessment Scale (CAFAS), a research-based tool that measures children’s daily functioning (See Box 1). The LOF aims to gather data on outcomes for children and youth with SED served through Michigan’s public mental health system. This data can then be used to inform decision-making at the policy and service delivery levels.

In 1998 Michigan mandated contracted providers in the public mental health system to use a functional assessment tool. The Child and Adolescent Functional Assessment Scale (CAFAS) was mandated for every child with SED enrolled in county mental health services. Michigan leads a county-run, state-supervised public mental health system through the Michigan Department of Community Health. It requires, through its Medicaid Provider Manual, that all children and youth accessing public mental health services receive an assessment using the CAFAS. Data from the CAFAS is then used to inform both clinical and administrative decision-making. Michigan also uses the CAFAS as a tool to assist in determining service entry. While different programs have different standards for entry, in general, home based services require a total CAFAS score of 80 (and one caregiver subscale score of 20 or 30) and the state’s Medicaid home-and community-based waiver program [1915(c)] requires a score of 90 or higher, if the child is age 12 or under, and a CAFAS score of 120 or higher, if age 13 or older (see Box 1). These state guidelines for program entry were established based on data analysis and recommendations of the LOF.

Michigan’s Level of Functioning Project (LOF) couples its’ CAFAS mandate with a voluntary program of technical assistance, data analysis and support, and a learning collaborative to assist counties and their providers in collecting and analyzing CAFAS data and monitoring quality. The state contracts with Eastern Michigan University for Kay Hodges, a Michigan-based researcher and the CAFAS’ author, to direct the project. Along with Dr. Hodges, two graduate research assistants collect data from participating community mental health centers (CMHCs). These CMHCs encompass 163 participating programs. According to Dr. Hodges, between 85-90 percent of the state is represented in the LOF.

Each provider that uses the CAFAS must undertake a competency-based training and receive biennial booster trainings. CAFAS coordinators from CMHCs who participate in LOF have quarterly meetings and receive monthly, quarterly, and semiannual reports as well as a report card on children’s progress in their center. CAFAS coordinators help providers review individual-level data and meet monthly with supervisors to compare their progress with overall state progress and benchmarks. Additionally, the LOF data collection and analysis has been used by state administrators. State officials reported that they base their selection of evidence-based practices on CAFAS data. For instance, CAFAS data showed a large number of youth with a behavioral disorder. As a result, state officials implemented provider training for an evidence-based practice: Parent Management Training – Oregon model (PMTO). One state administrator explained, “It came out of a meeting about the ones we can’t improve, the severe kids. [That] there’s no EBP that (can help) isn’t true, so let’s look at the data, and that made the introduction of EBPs the easiest system change in my career.”

The study’s aims were to:

• describe Michigan’s efforts to infuse a culture of quality and a focus on outcomes management in its child mental health service delivery system; and
• highlight ways in which outcomes-based management has facilitated:
• – improved service quality as measured by the implementation of evidence-based practices;– infrastructure-related support for access and quality;– service redesign;

  – application of System of Care principles such as, family and youth empowerment, and cross-systems collaboration;

  – use of a public health, age-appropriate services delivery approach, and

  – concrete examples of the lessons learned in Michigan relevant for states and localities interested in leading a quality movement.

Our Research Questions

• What are the strengths of the LOF’s use of a measurement tool (in this case the CAFAS) by Michigan counties to enhance service delivery?
• What major barriers did counties encounter?
• How did the use of a measurement tool like the CAFAS promote and assist in improving services?
• How did the use of a measurement tool like the CAFAS promote and support other system goals?

Our Research Approach

NCCP investigators interviewed 103 stakeholders at the state and local level across the following six Michigan community mental health service delivery regions: Wayne, St. Joseph, Saginaw, Livingston, Ingham, and Hiawatha, covering 10 counties in total (see figure 1). Stakeholders in these communities included county and state system leaders, service providers, and families and youth (see figure 2). Interviews were conducted between December 2007 and May 2008. While the majority of system leaders and service providers interviewed were part of the public mental health sector (70 percent), respondents from juvenile justice (11 percent), education (eight percent), child welfare (five percent), and public health (three percent) sectors were also interviewed to assess their knowledge of the CAFAS and the degree of collaboration across systems. This research was overseen by the Columbia University Medical Center’s Institutional Review Board.

Due to the large number of respondents, MaxQDA,© a qualitative analysis software tool, was used to organize and synthesize the large amount of interview and focus group data. We used the software to organize data by a coding scheme related to the research questions. Once responses were coded we analyzed data by respondent type and county to identify common themes. Responses were also quantified using STATA statistical software. Our research approach provided insight into how the use of a functional assessment tool across the state is working on the ground level. As with any research design however, this study has several limitations.

• While efforts were made to address all of the protocol questions with each respondent, response rates vary for questions, either due to time limitations where there was not enough time to ask all of the questions on the protocol, or due to the respondent’s inability to answer certain questions (such as respondents outside of mental health who were not using functional assessments). Response rates for questions are listed throughout the findings.
• System leaders and providers across mental health and non-mental health agencies were questioned about their knowledge and use of the CAFAS – even though only mental health providers are mandated to use the CAFAS – to see if the functional assessment tool was used beyond mental health agencies in the state. Given that there is no expectation that providers outside of mental health know about the CAFAS, we present when relevant the findings for mental health and non-mental health providers and system leaders.
• While not a major focus of the study, youth and parents or caregivers were interviewed to capture their knowledge of and perspective on the CAFAS. County agencies identified the families for us to interview, however, due to resource limitations, we were only able to interview a small sample of youth and family members.
• While the interview data collected provides practical insights from providers into the benefits and challenges of using a functional assessment tool, this study is not an assessment of the CAFAS tool and its psychometric properties.

About the Michigan Community Mental Health Service Regions (see Appendix 2)

• Wayne County had the highest total population in 2008 of 1,981,654 persons.
• Wayne County had the highest percentage of persons in poverty (21 percent).
• Hiawatha region had the lowest median household income of $39,782 and Livingston County had the highest median household income of $72,700.
• Wayne County also had the largest number of children 695,035 while the Hiawatha region had the smallest number of children 31,090.
• Wayne County had the highest percentage of African-American children (45 percent), Saginaw County had the largest percentage of Latino children (nine percent), Ingham had the largest percentage of Asian children (five percent), and the Hiawatha region had the largest percentage of American-Indian/Alaska Native American children (10 percent).

About the Respondents (see Figure 1)

• The majority of the respondents were service providers (49 percent or N=50).
• System leaders at the state and county levels were the next largest group (40 percent or N=42).
• Other respondents included family members or caregivers and youth (11 percent or N=11).

The experiences of the system leaders, providers, and families involved in data monitoring efforts, such as the LOF project, provide important guidance in helping to understand how such efforts can improve individual and system level outcomes. These perspectives collectively can inform others interested in replicating Michigan’s approach. In this report, we highlight the findings and discuss their implications in order to help guide others interested in implementing an outcome measurement system.

Download PDF