By Liesel Ritchie, Carolyn Kousky, Kathleen Tierney, and Simone Domingue
The recent Natural Hazards Mitigation Saves: 2017 Interim Report, found that disaster mitigation projects save $6 for every federal dollar spent. In 2016, FEMA’s Individual and Community Preparedness Division funded researchers at the Natural Hazards Center and Resources for the Future to develop methods to assess the return on investment from FEMA’s Whole Community concept and resilience-building activities.
Community Can be Beautiful by Alan Levine. Public domain, 2017.
Unlike the 2017 study, which focuses on the benefit of physical mitigation investments, our recent research concentrates on the returns to investment in community-based efforts such as disaster preparedness, outreach, partnership building, and public education—the whole community activities that are often more difficult to quantify. The objective of the present study was to develop methods and approaches that could be used around the country.
Return on investment (ROI)—the net earnings of a project divided by costs—is an economic calculation traditionally used as a decision aid in the private sector for evaluating investments. ROI provides an easy metric for comparing projects by their expected returns.
In the past several decades, there has been growing interest in using ROI to evaluate a range of public programs. This expansion has been accompanied by a broadening of what returns and costs are considered. When social benefits are included, the analysis is often termed social return on investment, or SROI. SROI is closely linked with other forms of program evaluation because it draws heavily on stakeholder engagement and uses logic models or impact maps to guide the analysis. To date, however, applying SROI in the preparedness context has been fraught with challenges.
One of the first tasks in this study was to associate the typical components of a logic model—inputs, outputs, and outcomes—with the costs and returns in examining SROI (See Figure 1). The costs roughly correspond to the inputs in a traditional logic model. The outputs produce a variety of outcomes. These outcomes are at times the “returns” in an ROI model, and at times they produce the “returns.”
Figure 1.
The study team selected Tulsa, Oklahoma, as a case study site for the research. With a long record of community-based disaster risk reduction activities, Tulsa was an ideal place to develop prototype approaches to measuring returns on whole community investments—approaches that other communities might want to use to better assess the costs and benefits of resilience-related activities. The study team conducted 43 interviews in Tulsa with key actors who had long-term experience in the city’s whole-community preparedness and resilience activities.
The study informed the development of a 14-step process for conducting SROI analyses for whole community activities (See Figure 2). These steps are:
- Ensure that the community is supportive of SROI analysis
- Establish project budget and secure funding
- Determine study scope
- Identify stakeholders
- Identify stakeholder inputs, outputs, and outcomes
- Create a logic map
- Determine data collection procedures for inputs/outputs
- Implement data collection
- Determine attribution
- Identify necessary non-market valuation studies to be undertaken
- Conduct valuation studies
- Value other inputs and outcomes
- Calculate SROI metric
- Conduct sensitivity analysis
Figure 2.
The study also identified two primary sets of concerns for those wishing to carry out SROI studies on preparedness activities.
The first concern is the appropriateness of using SROI methods to assess these specific types of activities. Although demonstrating that positive returns on preparedness investments can be beneficial to organizations and agencies, it can be extremely challenging to value many of the benefits of these programs in dollar terms. It can also feel inappropriate to some stakeholders to express changes to life, safety, and well-being in dollar terms. Neglecting these difficult-to-monetize impacts, however could show insufficient returns at a given point of time, which could jeopardize funding for preparedness programs. Relatedly, preparedness investments can have many spillover benefits and there is often not a clear boundary to the analysis—in other words, it is difficult to decide when to stop counting.
The second set of concerns involves the practical feasibility of conducting such analyses. SROI studies can be time consuming and timelines must take into account stakeholder schedules and commitment to the research. Special expertise and training are also required. Finally, SROI studies must be able to deal with counterfactuals—that is, to show that observed returns are actually the result of specific preparedness programs and not attributable to other factors. There will be times when it will be prudent to ask whether the benefits of a credible SROI study justify the costs.
We do not mean to imply that it is not feasible to assess whole-community activities using SROI and ROI methods. It is feasible, but those wishing to conduct such studies should be aware of the following caveats:
First, SROI studies in areas similar to disaster preparedness are limited. Many of the publications we reviewed for our study that claimed to have used SROI methods did not actually quantify or value any outcomes. Instead, they might simply use the framework SROI provides to look qualitatively at investment impacts.
Second, methods for evaluating benefits from preparedness activities are less well developed than those for assessing brick-and-mortar mitigation activities and will require new market valuation studies.
In our research, we identified a number of difficulties with implementing fully-quantitative SROI studies for whole community activities. For example, the interviews conducted for this study indicate that program personnel lack time and resources to collect and track the kinds of data needed for such analyses and that by and large they lack the expertise to conduct analyses on their own. Future studies will need to come with adequate additional funding and be carried out by qualified researchers.
At the same time, we also found a strong community-level interest in pursuing approaches that could help bolster the case for the benefits of these investments. Given the challenges and costs associated with conducting rigorous and credible SROI analyses of whole community activities, it might be prudent to select a few cases for thorough examination and use the results of those studies to guide broader research.
Liesel Ritchie is associate professor at Oklahoma State University and former associate director of the Natural Hazards Center at the University of Colorado Boulder. Ritchie has more than 20 years of experience in evaluation and research and has studied a range of disasters, including the Exxon Valdez and BP Deepwater Horizon oil spills; the Tennessee Valley Authority coal ash release; Hurricane Katrina; and earthquakes in Haiti and New Zealand. Since 2000, her focus has been on the social impacts of disasters and community resilience.