Research and Evaluation
AHP has extensive experience carrying out research and evaluation activities for Substance Abuse and Mental Health Services Administration (SAMHSA), the National Institute for Justice (NIJ), the Department of Housing and Urban Development (HUD), state and community agencies, and stakeholders for these agencies.
Our work has included:
- » Designing and implementing cross-site, multi-method evaluation studies; Collaborating with grantee sites
- » Analyzing qualitative and quantitative data, including large secondary data sets
- » Producing reports targeted to multiple audiences
Adolescent Substance Abuse Treatment
Program Evaluation for Assertive Adolescent & Family Treatment Program (AAFT). AHP leads a multi-site, process and outcome evaluation funded by the Center for Substance Abuse Treatment (CSAT) to assess the 2009 Family-Centered Substance Abuse Treatment Grants for Adolescents and their Families. AAFT grantees serve adolescents and/or transition-age youth who have a substance use disorder and may also have co-occurring mental health disorders. They are required to implement two evidence-based practices: the Adolescent Community Reinforcement Approach and Assertive Continuing Care.
The overarching objective of the AAFT process and outcome evaluation is to assess and document the process of implementation and to explore the role that implementation supports play in how these programs evolve. To achieve these goals, the evaluation approach includes collecting both quantitative and qualitative data using Web surveys, key informant interviews, case study visits, observation, document review, and secondary data analysis.
Strengthening Adolescents and Family Empowerment (SAFE) Project. AHP conducted an evaluation of a program implemented at the Dimock Community Health Center and North Suffolk Mental Health Center. The project assessed adolescents for treatment using the Global Appraisal of Individual Needs and utilized the Assertive Community Reinforcement Approach coupled with Assertive Continuing Care, all evidence-based interventions with proven effectiveness in building community capacity for family-centered treatment.
AHP assessed program performance by documenting and measuring client and program outcomes. We also performed a process evaluation that assessed fidelity for the overall intervention. The project included submitting data to a coordinating center and following rigorous data submission guidelines.
The Ladder Project: Building Recovery One Step at a Time. AHP is evaluating the Ladder project, an Assertive Adolescent and Family Treatment (AAFT) Program funded by the Center for Substance Abuse Treatment (CSAT). The Institute for Health and Recovery, Inc. will implement the Assertive Community Reinforcement Approach coupled with Assertive Continuing Care to provide services to civilly-committed transition-age youth (ages 18-24) and their families/primary caregivers within Norfolk and Plymouth counties in southeastern Massachusetts. The project will serve an average of 35 youth and their families per year, reaching a total of 105 youth and their families over the three-year grant period.
IHR YARD ROSC Grant. AHP serves as the evaluator for a Center for Substance Abuse Treatment (CSAT) Recovery-Oriented Systems of Care Program, the YARD Project (Young Adult Recovery Destination). The goal of this grant is to expand local recovery-oriented systems of care (ROSC) to address gaps in treatment capacity for substance abuse treatment and recovery services. The YARD Project is a collaborative initiative to develop a recovery-oriented substance use and co-occurring disorders treatment and recovery center for transition-age youth, ages 18-24, residing in South Boston. The project will serve approximately 129 youth through evidence-based practices and an array of recovery support services.
AHP is assessing program performance by documenting and measuring client outcomes and by conducting a process evaluation. AHP collaborated with program staff to develop and use age-appropriate and trauma-sensitive materials and procedures. Client outcomes of interest include reduced substance use, improved mental health status, improved social connection, increased housing stability, enhanced employment/education, improved recovery status, decreased legal problems, and improved functioning. AHP’s role also includes providing technical assistance for complying with the reporting requirements of the Government Performance and Results Act of 1993.
CSAT TCE Effective Adolescent Substance Abuse Treatment. AHP served as the evaluator for a CSAT-sponsored effective adolescent substance abuse treatment program that was part of a multisite initiative (38 sites nationwide). Over a three-year period, the Rhode Island site used evidence-based practices—Motivational Enhancement Therapy and Cognitive Behavioral Therapy—to serve 240 adolescents identified as needing substance abuse treatment. In addition to substance abuse problems, the majority of participating youth had significant trauma and criminal histories. Expected outcomes included reduced substance use; increased residential stability; improved physical, mental, and emotional health; reduced involvement with the criminal justice system; improved educational and vocational involvement; and improved family functioning.
Co-Occurring Disorders and Behavioral Health
National Evaluation of the Co-Occurring State Incentive Grant (COSIG) Program. AHP, in collaboration with the Human Services Research Institute and the University of Massachusetts Medical School, lead an evaluation of the COSIG program, a SAMHSA initiative. COSIG is designed to support states’ efforts to develop and enhance their infrastructures and treatment service systems in order to increase capacity to provide services to persons with co-occurring substance abuse and mental health disorders and their families. This four-year evaluation was designed to assess the accomplishments of the 15 COSIG states in meeting the national goals and objectives of the program, and to determine the factors associated with success as well as barriers experienced.
AHP developed an evaluation plan that incorporated the perspectives of relevant stakeholders including grantees, project staff, evaluators, and consumers and their families. The team implemented the evaluation through site visits and consultations with grantees, analysis of state-produced documents, secondary analysis of data provided to SAMHSA, development and piloting of a self-report measure of Dual Diagnosis Capability, collaboration in outcome studies, and assessment of the COSIG program design. AHP identified the generalized lessons for the field that emerged from the states’ experience.
Alaska Co-Occurring State Incentive Grant (COSIG) Evaluation. AHP evaluated the State of Alaska’s five-year redesign of its behavioral health systems. This infrastructure development project was a state-wide multi-level effort to transform Alaska’s set of fragmented, primarily rural, substance abuse and mental health services into an integrated behavioral health system, using the Continuous, Comprehensive, Integrated Systems of Care (CCISC) best practice model.
The evaluation included case studies of four Alaskan communities, extensive key informant interviews, observation of participants in key planning groups and training efforts, and secondary data analysis. The Year One evaluation focused on activities and outcomes at the state system level. Years Two and Three focused on programs and clients respectively. Years Four and Five tracked client outcomes over time. Annual reports provided yearly feedback to project, implementers, Federal funders, and other key stakeholders on project progress and challenges.
CSAT Co-Occurring Disorders Study. AHP conducted a primary research study comparing two methadone maintenance programs for persons with co-occurring substance abuse and mental health disorders. One program represented a parallel approach to treatment in which the substance abuse agency referred clients to a second agency for mental health treatment. The other program represented an integrated approach to treatment in which substance abuse and mental health treatment were incorporated within the same program.
This longitudinal, intent-to-treat design followed admission and in-treatment samples for 12 months. The stratified, in-treatment sample included persons who had been in treatment for up to four years at the time of study enrollment. To help staff in both programs understand and adhere to the model being tested, AHP staff developed program manuals for both parallel and integrated treatment approaches. AHP staff has given presentations on this study at professional conferences such as the annual meeting of American Public Health Association (APHA), grantee meetings, and local site advisory committee meetings.
Healthcare and Behavioral Health
HRSA Grants for Policy-Oriented Rural Health Services Research – Preventive Care: Supports and Barriers to Best Practices for a National Sample of Rural Medicare Beneficiaries. Using an integrative approach, including secondary data analysis of a large nationally representative dataset (the Medicare Current Beneficiary Survey, n=14,000), and complementary qualitative interviews with rural elders, this HRSA-funded research study examined the factors influencing rural elders’ receipt of “gold standard” health care screenings and vaccinations.
The study’s primary purposes were to
- » describe utilization of best practice preventive health care screenings and vaccinations among a national rural sample of Medicare beneficiaries, as compared to urban elders
- » identify individual level factors associated with utilization of these practices
- » explore community and health system factors which impede or facilitate the utilization of best practice screenings and vaccinations among rural elders
Statistical analyses included propensity scoring, Logistic Regression, and Ordinary Least Squares Regression, with weighted data.
Health Choice Network. AHP provided technical assistance and consultation to two behavioral health care management programs implemented by Community Health Center members of the Health Choice Network (HCN) in order to increase their evaluation capacity. The first project, DiabetikSmart, was a replication of a model diabetes care management program developed in California and being implemented in eight HCN-member Community Health Centers across South and Central Florida. The second project, the Behavioral Wellness Program, was a pilot depression care management project being implemented in two program sites.
AHP helped HCN evaluation staff design and implement a cross-site process evaluation for both programs, including development of research questions and methods, review of existing instrumentation and development of new protocols, help with establishing field procedures and manuals, review of computer-based patient management and tracking systems, development of data entry programs, and help with data analysis and report writing.
Elder Community Care. AHP leads a three-year evaluation of a Targeted Capacity Expansion program, Elder Community Care (ECC), funded by the Center for Mental Health Services (CMHS) of the Substance Abuse and Mental Health Services Administration (SAMHSA).
The program delivers evidence-based mental health and/or substance abuse treatment targeted to serve 300 older adults living in 14 communities in Metrowest Boston, the grantee provides community-based training and education modeled on The Gatekeeper Model of Case-Finding and Response; offers a single point of entry for accessing home-based mental health and substance abuse services as well as a full range of other services; offers a graduate-level social work internship program by partnering with a local university; and provides a telephone support service for ECC elders, modeled after the evidence-based Tele-help/Tele-check program.
The evaluation includes both process and outcome components; uses qualitative and quantitative methods in an integrated approach; and involves the target population in the design, collection, and interpretation of data. It includes outcomes developed by the project team, including those related to emotional health, suicidality, physical health, substance use, social connectedness, quality of life, and service use. The evaluation will assess treatment outcomes, as well as process and fidelity measures.
The evaluation team will provide regular feedback to service providers and staff on program performance in order to help the program improve treatment services. The approach incorporates the mandated NOMS measures and employs procedures shown to achieve six-month follow-up rates exceeding 80 percent.
Homelessness and Housing
Targeted Capacity Expansion Program, Families Living Together (FLT). AHP leads a five-year evaluation of a CSAT/CMHS-funded program that provides Integrated Clinical Case Management for 260 homeless mothers, most with histories of abuse and trauma, who are living in emergency shelters in greater Boston. The evaluation includes process and outcome components, using qualitative and quantitative methods in an integrated approach. Program participants participate in designing, collecting, and interpreting data.
Outcomes tracked relate to residential stability, physical health, mental health, substance abuse, legal involvement, trauma, social support, and parenting practices. AHP is using a qualitative approach to understanding how the process is working and what might be improved. Consumer staff administer interviews, track and retain evaluation participants, and review all evaluation materials to provide a consumer perspective.
CMHS/CSAT Homeless Families Initiative. AHP conducted primary research to describe the population of homeless women with dependent children admitted to a family or domestic violence shelters and evaluate services provided to these families. It included assessing the impact on a range of health-related outcomes, such as mental health and substance abuse and residential stability.
In Phase I, AHP conducted a pilot study involving interviews with mothers to describe the clinical and demographic characteristics of the population of women who are living with their dependent children, have a mental health or substance abuse diagnosis or both, and have been admitted to a family/domestic violence shelter.
In Phase II, AHP used a quasi-experimental design to evaluate the impact of receiving case management services using the Family Critical Time Intervention (FCTI) model on outcomes for families, including housing stability, mental health functioning and trauma recovery, alcohol and substance abuse, family income, and family preservation. The outcomes of individuals receiving the services in one county were compared to the usual post-shelter, outreach/resettlement programs available in two additional counties. AHP also examined the predictive value of a new measure of family risk of future homelessness.
Multi-Site Study of Supported Housing. AHP conducted research in both Phase I and Phase II of this national multi-site study of supported housing for people with serious mental illness. During Phase I, AHP worked with three of ten study sites (New York City, Massachusetts, and upstate New York) to develop a Fidelity Framework to help providers determine how well they adhere to recognized supported housing principles, as well as to collaborate on a design for Phase II.
During the second phase of the study, AHP conducted research in two of six study sites (Massachusetts and upstate New York) to compare housing models, including parameters such as who is served, how they are served, what outcomes are achieved, and at what program cost outcomes are achieved. The research sought to determine whether supported housing is more effective than a linear residential continuum to help people with serious mental illnesses achieve residential stability, higher functioning, improved quality of life, and independence.
Linking Treatment to Housing. AHP serves as the evaluator for Linking Treatment to Housing (LTH), a five-year project funded by the CMHS Services in Supportive Housing Initiative. The grantee is the Boston Public Health Commission. LTH uses the Housing First model of permanent supportive housing. Its goals are to improve residential stability and reduce psychiatric symptoms and substance use for chronically homeless adults with co-occurring mental illness and substance use.
The program recruits long-term shelter dwellers from two Boston-based shelters, connects them to subsidized housing in the community using housing vouchers and housing search provided by project partners, and provides case management and integrated treatment services using the Critical Time Intervention (CTI) model. The project also aims to develop a Service Consortium to provide integrated services and support long-term stability in the community for newly housed homeless individuals.
AHP is assessing program performance by conducting both an outcome and process evaluation.
WorkFirst. AHP is evaluating WorkFirst, a three-year demonstration project that aims to promote housing retention and increase incomes for 140 formerly homeless adults by linking specialized employment services within the Housing First model. WorkFirst helps participants find and retain employment by providing coaching, skill building, job placement, job retention, and housing retention services. The evaluation addresses the following questions
- » Do intensive, integrated employment services help improve employment and housing outcomes?
- » Do people who receive employment services at different points in time relative to their housing placement have different housing outcomes?
- » Does provision of intensive, immediate work services lead to cost offsets as participants are able to fund a greater share of their housing?
To address these questions, AHP designed a two-group longitudinal evaluation. The first group consists of WorkFirst participants, while the comparison group consists of men and women who meet the same criteria as the first group, but are not WorkFirst participants. Longitudinal data is collected by program staff from both the WorkFirst group and the comparison group.
Clearview/Sherman Street Program. The five-year grant to support the ClearView/Sherman Street Program was awarded by the Center for Substance Abuse Treatment (CSAT) as part of its Homeless Families Program. The ClearView/Sherman Street Program is implemented by a consortium of three organizations: ClearView Center, St. Catherine’s Marillac Family Shelter, and AHP. This effort was developed to ensure that families leaving shelters in Albany County have access to trauma counseling, as well as to other mental health services, substance use treatment, and aftercare case management. ClearView Center and St.Catherine’s Center provide the service components of this capacity expansion and access improvement project, while AHP conducts the evaluation.
Interpersonal Violence and Trauma
National Rural Faith-Based Domestic Violence Program Evaluation. In 2005, the Office on Violence Against Women (OVW) of the U.S. Department of Justice (DOJ) launched a national initiative to create and expand domestic violence programming in rural areas through small faith-based and community organizations. To administer the initiative, OVW selected three agencies as “intermediaries” to identify and recruit local sub-grantees in designated geographical areas, provide them with pre- and post-award technical assistance to enhance the delivery of domestic violence services, and build their capacities to continue to offer such services after grant expiration. AHP evaluated the roles and effectiveness of these intermediaries and the faith-based organizations themselves.
The evaluation utilized both qualitative and quantitative data collection methods to examine whether the use of intermediary agencies provides an effective mechanism to recruit and expand the number of rural service providers among local faith-based and community organizations and whether these local organizations could offer effective domestic violence services.
A Statewide Profile of Abuse of Older Women and the Criminal Justice Response. This study, which was funded by the National Institute of Justice (NIJ), examined all police-reported cases of elder abuse by an intimate or family member across the entire state of Rhode Island. This groundbreaking, first-in-the-nation study revealed significant findings, documenting the dramatic difference between abuse of older women, 50 to 59 years, and those 60 years and older. While the majority of abusers of the former were current or former intimates, the majority of the latter were adult children and grandchildren. The study found that the latter were not overstressed caregivers, but predators, many with criminal histories outside the family.
The study also revealed a disturbing gap between law enforcement and adult protective services, with little exchange of information between the two agencies that share legal responsibility for the protection of vulnerable elders. The study was featured in a joint NIJ-National Institute of Aging conference on elder abuse research.
A Statewide Study of Stalking and its Criminal Justice Response. For this researcher-initiated study of intimate stalking and its criminal justice response, AHP reviewed the population of intimate stalkers as reported to Rhode Island police between 2001 and 2005 (inclusive) and compared them with a representative sample of other intimate abusers reported to Rhode Island police during the same period who not only engaged in stalking, but also were cited for a variety of other domestic violence offenses. AHP considered the suspects, the victims, the incidents, the criminal justice response, as well as reabuse that occurred up to six years after the study stalking incident. We interviewed focus groups of key criminal justice actors and advocates across the state to analyze the quantitative findings.
CMHS Evaluation of Jail Diversion and Trauma Recovery. AHP has been contracted by CMHS to conduct a cross-site evaluation of the 12 states funded to implement the Jail Diversion and Trauma Recovery Program, with Priority to Veterans. The purpose of this 5-year contract is to coordinate the collection of multi-site data and to aggregate, analyze, and synthesize the data for dissemination.
The evaluation will determine the extent to which trauma-integrated treatment and supports implemented through the jail diversion program result in improved client outcomes, particularly for veterans. It will also document grantee implementation of screening and treatment strategies in pilot projects statewide. The evaluation examines client-level outcomes across pilot program sites using data collected through in-person interviews as well as secondary sources on arrests and services. A process evaluation is being conducted to document the strategies and practices used in the pilot programs, as well as to determine how these strategies and practices have been integrated in the state infrastructure.
CSAT Targeted Capacity Expansion Grant for Rural Barnstable County. Under a three-year CSAT targeted capacity expansion (TCE) grant, AHP is evaluating the Women Achieving Vital Empowerment (WAVE) project, which is designed to increase substance abuse services to victims of domestic violence in rural Barnstable County on Cape Cod, Massachusetts. Evaluation methods and procedures are informed by the rural context for the study, and include frequent and extensive outreach to update contact information, to keep the women in these isolated areas engaged in the evaluation, and to ensure their safety needs are addressed.
The WAVE project includes training for staff of both substance abuse and domestic violence agencies on how to respond to these issues in an integrated fashion. The project is also providing evidenced-based program enhancements delivered at the domestic violence/sexual assault program sites.
AHP is working closely with project developers to create logic models and fidelity instruments for the new intervention. AHP then will measure outcomes at the system, program, and individual levels.
The National Child Traumatic Stress Initiative (NCTSI). AHP conducted an evaluation of The Parsons Center, an NCTSI community practice site in upstate New York,. The site focused on improving the quality of services and support for traumatized children.
AHP activities included conducting needs assessment interviews with key managers and staff of Parsons Center programs. In addition, we selected and trained 17 clinicians from the Parsons departments of Prevention, Therapeutic Foster Care, Residential Services and Outpatient Treatment. This included use of the Real Life Heroes Practitioner Manual, structured session outlines, and fidelity checklists. Data were collected from children, their primary caregivers, and clinicians at four time points over 12 months.
AHP also evaluated the effectiveness of the Systematic Training to Assist in the Recovery of Trauma (START) curriculum to determine if the training was effective in increasing participants’ knowledge of trauma, trauma-related disorders within children, and appropriate interventions.
Mental Health Policy
CMHS-funded New Freedom Initiative: State Coalitions to Promote Community-Based Care (Olmstead). For the national evaluation, AHP collected, analyzed, and reported on Web-based progress report and interview data from all 54 participant states and territories. Sites were integrated into design decisions and survey questions through a series of conference calls. We produced materials prior to each call to facilitate discussion and decision-making.
The project created a customized database to track project activities and collect evaluation data. Individual state profiles and trend analysis reports were analyzed to measure progress. To support the project’s data collection activities, AHP offered data collection tools in multiple formats and provided ongoing consultation to all grantees.
New York State MH Financing Paper. Commissioned by the New York State Conference of Mental Hygiene Directors, AHP examined changes in mental health program financing in New York State. AHP staff:
- » Conducted a comprehensive literature review
- » Examined changes in mental health services and public policy in New York State and nationally
- » Conducted a secondary analysis of published and unpublished financial data from the New York State Office of Mental Health.
We then drafted a report that identified changes in expenditures and linked these to programming and policy. The report was presented by AHP staff at the annual NYS State Conference of Mental Hygiene Directors.
Oklahoma Department of Mental Health and Substance Abuse Services (ODMHSAS). Oklahoma is one of seven states to receive a five-year Mental Health Transformation State Incentive Grant (TSIG) from the federal Substance Abuse and Mental Health Services Administration (SAMHSA). The goal of these grants is to transform state mental health service systems from systems dictated by bureaucratic and financial incentives to recovery-facilitating systems driven by consumer and family needs.
ODMHSAS contracted with AHP to conduct needs assessment activities during the first year of the grant. AHP completed more than 100 focus groups and key informant interviews with a wide range of constituency groups from diverse geographic areas, as well as with the leadership of state agencies working in partnership with ODMHSAS. AHP also worked with ODMHSAS Decision Support Services staff to conduct secondary analyses of existing data from ODMHSAS and other relevant state agencies.
AHP is leading two projects related to the Comprehensive Mental Health Plan developed by ODMHSAS as part of this grant. These include:
- » A study describing state agency policy changes across the TSIG grant period, based on in-person interviews with state agency representatives and other key policy makers
- » A behavioral healthcare workforce study, designed and implemented under the guidance of a local advisory team. AHP worked with the local advisory team to identify local workforce-related concerns and interests, then planned evaluation activities that would respond to them.