This report summarizes key findings regarding SSS processes and client outcomes among a cohort of 1107 clients served predominantly in 2022.
Stop Soldier Suicide (SSS) provides personalized, confidential suicide-specific care and intervention services via telehealth to veterans and service members from all branches, regardless of discharge status and at no cost to them. Founded in 2010 by Army veterans Brian Kinsella, Nick Black and Craig Gridelli, SSS is the only national nonprofit organization focused solely on reducing the military suicide rate. The organization has collaborated with some of the nation’s foremost suicide prevention experts to adapt evidence-based intervention methodologies, such as the Chronological Assessment of Suicide Events (CASE) and Collaborative Assessment and Management of Suicidality (CAMS), for telehealth. As clients receive our suicide-specific services, SSS staff document their engagements with clients, record their clinical impressions, and collect self-reported outcome measures from the clients. The SSS Research and Evaluation Team regularly analyzes this information to evaluate the impact SSS is having and to identify ways to improve.
This report summarizes key findings regarding SSS processes and client outcomes among a cohort of 1107 clients served predominantly in 2022. All clients received a full intake assessment and had an initial risk level assigned by a Wellness Coordinator (WC) between July 1, 2021 and June 30, 2022. We then analyzed their follow-up records and assessments through December 31, 2022, thus tracking processes and outcomes for a minimum of 6 months for every client in the cohort. Below we outline the key findings from our process measures and client outcomes, provide details about our methods for gathering and analyzing the data, we discuss future work and conclusions, and then provide an appendix with supplemental figures and findings.
The majority of clients who fill out an online help request form receive a call within 20 minutes by our 24/7 call center to confirm the information they have submitted. They then receive a second call to schedule their intake.
On average, clients in our evaluation cohort had their full intake assessment within one week after first contacting SSS.
Based on their initial intake assessment, 31% of clients were identified as high risk clients (HRCs), 28% as moderate risk clients (MRCs), and 41% as low risk clients (LRCs).
3.6 months for HRCs
2.5 months for MRCs
1.0 month for LRCs
Once every 7 days for HRCs
Once every 8 days for MRCs
Once every 10 days for LRCs
95% of HRCs and 88% of MRCs had a recorded and complete crisis response plan (CRP)¹.
40% of HRCs and 24% of MRCs received treatment using the Collaborative Assessment & Management of Suicidality (CAMS), an evidence-based modality specifically tailored to individuals at risk for suicide²
Those who received CAMS began treatment on average 33 days after the intake assessment. There was no substantial difference by risk level (33 days for HRCs and 34 days for MRCs)
Of clients receiving CAMS, HRCs received an average of 6 sessions and MRCs received on average 4 sessions of CAMS
16 hours of care per HRC
11 hours of care per MRC
5 hours of care per LRC
At the conclusion of treatment, 79% of HRCs and 77% of MRCs stated they were able to manage their thoughts and feelings related to suicide
49% of clients had a decrease in psychological pain
49% decrease in stress
48% decrease in agitation
47% decrease in hopelessness
41% decrease in self hate
30% of HRCs and 21% of MRCs self-rated their overall suicide risk as having improved during treatment
25% of HRCs and 8% of MRCs experienced a reduction in their suicide risk level or acuity as determined by further assessments during treatment
On average, 40% of clients experience improvements in well-being and resilience; and overall scores improved by 18% for well-being (34% for HRCs, 5% for MRCs, and 13% for LRCs) and 23% for resilience (33% for HRCs, 8% for MRCs, and 18% for LRCs).
Clients rated their experience with SSS very favorably, with an overall net promoter score of 84 (benchmarks in the healthcare industry range from only 38³-58⁴).
All of our process and outcome measures were recorded throughout each client’s journey in our Salesforce customer relationship management software. WCs tracked each activity they performed while helping clients, and marked which of the activities involved interacting with the client (e.g., having a phone call) and which did not (e.g., leaving a phone message or sending an email with resources). During calls with the client, WCs also filled out assessments within Salesforce to record information gathered from the client and monitor the client’s progress.
Some of the outcome measures were based on the WC’s perspective and others were based on the client’s perspective. For example, overall suicide risk was reported by the WCs on the following scale during each risk assessment: high acute, high chronic, moderate acute, moderate chronic, low acute, low chronic. A reduction in acuity, as reported in the findings above, was defined as the proportion of clients whose last risk assessment was at a lower level (e.g., high to moderate) or diminished severity (i.e., acute to chronic) than their initial risk assessment. Overall suicide risk was also self-reported by the clients on a scale of 1 (will not kill self) to 5 (will kill self). All clients receiving CAMS and some clients receiving other clinical support, such as collaboration and coordination with the clients’ systems of formal and informal care (i.e., systems collaboration), provided this overall self-assessment of suicide risk. For those who responded two times or more to this question, SSS took the difference between their first and last response to measure the proportion of clients with reductions in self-reported risk.
Several of the outcome measures are based on the CAMS suicide status form-4 (SSF-4⁵), and were asked both in CAMS and systems collaboration sessions. The ability to manage thoughts and feelings is one of these measures. It is not asked at baseline, only at follow-up sessions. The outcome reported in the findings section is based on the last session in which the client reported this metric. From the SSF-4, we have also adopted the use of the self-rated scales of psychological pain, stress, agitation, hopelessness, and self-hate. These all range from 1 (low) to 5 (high). For those who responded to these questions two more times, we calculated the percentage of clients who reported a lower level at their last session compared to their first.
Client well-being and resilience are assessed using self-reported measures every 30 days. The Wellness Coordinator asks the items from the Personal Wellbeing Index (PWI)⁶ and the Brief Resilience Scale (BRS)⁷ to the clients over the phone. The PWI consists of seven well-being questions that the client rates on a scale of 0 to 10, and the overall score is an average of these scores multiplied by 10 to provide a value that ranges from 0-100. These questions cover the following topics: Standard of Life, Personal Health, Achieving in Life, Personal Relationships, Personal Safety, Community-Connectedness, & Future Security. The BRS consists of three positively framed statements and three negatively framed statements regarding resilience that the client rates on a scale of strongly disagree, disagree, neutral, agree, and strongly agree. These are: I tend to bounce back quickly after hard times, I have a hard time making it through stressful events, It does not take me long to recover from a stressful event, It is hard for me to snap back when something bad happens, I usually come through difficult times with little trouble, and I tend to take a long time to get over set-backs in my life. Responses are scored from 1 (low resilience) to 5 (high resilience) and the overall score is the average across the six statements. For those responding to these questionnaires two or more times, we took the difference between the last and the first score and divided it by the first score to get the percent change in score reported in the findings section above.
Net Promoter Score (NPS) is a nationally used client satisfaction metric used by tens of thousands of companies across multiple industries⁸ ⁹. The central question asked across multiple iterations of the NPS is: “On a scale from 0 to 10, how likely are you to refer us to a friend or colleague.” Based on the score (0-10), the customers fall into one of the three categories: Promoters (9 or 10); Passives (7-8); and Detractors (0-6). After each CAMS session, clients were sent a link to a feedback survey with the following NPS question: “Based on the totality of your experiences with Stop Soldier Suicide, how likely is it that you would recommend us to your friends, family, or colleagues who have similar needs?” The NPS was then calculated as the percentage of individuals who were Promoters, minus the percentage who were Detractors, multiplied by 100. The overall score ranges from -100 to 100, and it was the only outcome measure in our evaluation that was collected without the WC being present.
Response rate was low for certain outcome measures, especially when we limited to those who responded twice or more for purposes of examining changes over time. We created inverse probability weights to adjust for these lower response rates ¹⁰ ¹¹. This was done using logistic regression. We created separate regression models to calculate the probability that anyone in the evaluation cohort responded to two or more PWI measures, two or more BRS measures, two or more of the different CAMS scales (overall self-rated risk, and the five risk factor scales), and one or more NPS measure. Each of the models adjusted for age, sex, time engaged with SSS, initial risk level determined by the WC, any CAMS session, any systems collaboration session, any crisis responses plan, any crisis event while with SSS, any missed appointment while with SSS, and whether over the course of SSS services they had reduced acuity of WC-determined suicide risk. The models for predicting clients who had two or more of the different outcome measures also included a covariate to adjust for those who had at least one response to the same outcome measure, and the baseline score of that measure. The inverse of the predicted probabilities from these logistic regression models were used to calculate weighted averages and percentages for each of the outcome measures.
The analyses conducted were all descriptive in nature of the total population served. The appendix of supplemental figures and findings includes greater details on the characteristics of the clients in the evaluation cohort, the distribution of process measures among clients, and some of the outcome measures split out into their sub questions and amount of change over time.
This upcoming year, there will be a significant shift in how we measure outcomes among clients. We are incorporating several new outcome measures into a redesigned Salesforce interface with considerably more breadth and depth of data capture and management. These measures have been carefully selected after conducting a comprehensive review of questionnaires related to suicide and suicide risk factors. A core set of outcome measures that will allow us to better quantify lives saved and suicide attempts averted will be asked immediately prior to intake and every 30 days thereafter. After intake and at the middle and end of treatment, we will also collect a set of measures related to underlying mental, emotional, and physical conditions. These questionnaires will be sent to clients as links in an email and text message. Clients will also receive follow-up reminders in their patient portal and by their WCs to fill out these questionnaires. Next year, we will start to report on these additional outcome measures.
SSS has had a significant impact among its clients. In this evaluation, clients over time showed that they were able to manage their thoughts and feelings, they experienced substantial reductions in overall suicide risk and suicide-related risk factors, and they increased in their overall well-being and resilience. Clients were very satisfied with the care they received. They consistently received evidence-based services and interventions from SSS. These services were offered weekly over several months. Over the coming year, SSS will continue refining its approach to quantifying and evaluating its life-saving efforts.
¹ Bryan, Craig & Mintz, Jim & Clemans, Tracy & Leeson, Bruce & Burch, T. & Williams, Sean & Maney, Emily & Rudd, Michael. (2017). Effect of crisis response planning vs. contracts for safety on suicide risk in U.S. Army Soldiers: A randomized clinical trial. Journal of Affective Disorders. 212. 10.1016/j.jad.2017.01.028.
² Swift, J.K., Trusty, W.T. and Penix, E.A. (2021), The effectiveness of the Collaborative Assessment and Management of Suicidality (CAMS) compared to alternative treatment conditions: A meta-analysis. Suicide Life Threat Behav, 51: 882-896. https://doi.org/10.1111/sltb.1...
³ See: https://www.questionpro.com/blog/nps-benchmarks/. Accessed July 11, 2023.
⁴ See: https://customergauge.com/benchmarks/blog/nps-healthcare-net-promoter-score-benchmarks#Average-NPS. Accessed July 11, 2023
⁵ Managing Suicidal Risk: A Collaborative Approach, Second Edition, by David A. Jobes. Copyright © 2016.
⁶ International Wellbeing Group (2013). Personal Wellbeing Index: 5th Edition. Melbourne: Australian Centre on Quality of Life, Deakin University
⁷ Smith, B. W., Dalen, J., Wiggins, K., Tooley, E., Christopher, P., & Bernard, J. (2008). The Brief Resilience Scale: Assessing the Ability to Bounce Back. International Journal of Behavioral Medicine, 15(3), 194–200
⁸ Colvin, Geoff (2020). "The simple metric that's taking over big business". Fortune. Retrieved 3 June 2020.
⁹ Reichheld, Frederick F. (December 2003). "One Number You Need to Grow". Harvard Business Review. 81 (12): 46–54, 124. PMID 14712543.
¹⁰ Härkänen, T., Kaikkonen, R., Virtala, E. et al. Inverse probability weighting and doubly robust methods in correcting the effects of non-response in the reimbursed medication and self-reported turnout estimates in the ATH survey. BMC Public Health 14, 1150 (2014). https://doi.org/10.1186/1471-2...
¹¹ SKINNER, C. J., & D’ARRIGO, J. (2011). Inverse probability weighting for clustered nonresponse. Biometrika, 98(4), 953–966. http://www.jstor.org/stable/23...
Suggested citation: Richardson, J.S., Beatty, A.E., DePalma, S., Batten, S.V., Hotle, K. (2023). 2022 Program Evaluation of Stop Soldier Suicide. Retrieved Month Date, Year from www.stopsoldiersuicide.org/research.