Modeling Evacuation Behavior in the 2019 Kincade Fire, Sonoma County, California

Publication Date: 2021

Abstract

Wildfire is a growing global concern for rural and urban areas. Statistics show that the intensity and negative consequences of wildfire have increased in recent decades, creating serious challenges for fire and emergency services, as well as communities in the wildland-urban interface (WUI). To reduce the risk of wildfire and to enhance the safety of WUI communities threatened by wildfires, it is imperative to enhance our understanding of human behavior, especially evacuation behavior, in wildfires. To this end, we developed a new survey instrument and distributed it online among households impacted by the 2019 Kincade Fire in Sonoma County, California. Based on the survey data, we then developed a machine learning model (random forest) to predict people’s evacuation decision (evacuate/stay) and applied interpretable machine learning methods to explain the model. The results show that the random forest model can not only better predict the outcome when compared with traditional logistic regression modeling techniques, but it can also generate richer insights to improve our understanding and facilitate emergency officials’ decision-making regarding when to warn/evacuate a community. For example, we found that pre-fire perception of safety and risk perception (at the time of evacuation decision) were the two most important variables identified by the random forest model when making evacuation decision predictions. These two variables also show strong nonlinear impacts on the predicted probability to evacuate.

Introduction and Literature Review

Largely due to climate change, California wildfires have increased in size by a factor of eight, and the annual burned area has grown by nearly 500% in recent years (Williams et al., 2019¹). In 2019, the Kincade Fire burned 77,758 acres in Sonoma County, California within two weeks’ time. During the fire, 374 structures were destroyed, 40 structures were damaged, and nearly 200,000 people were forced to evacuate from their homes (La Ganga, 2019²).

The Kincade Fire has revealed many evacuation-related challenges, such as power outages negatively impacting information dissemination (della Cava & Lam, 2019³) and households engaging in multiple protective actions, including saving pets and livestock and assisting others. As wildfires become more prevalent and frequent, it is important to learn from previous fires. More specifically, it is essential to understand the household wildfire evacuation processes, e.g., what influenced households’ decisions to evacuate and how these decisions were made during evacuation (Kuligowski, 2021⁴; Kuligowski et al., 2020⁵; Wong et al., 2020⁶; Lovreglio et al., 2020⁷; McCaffrey et al., 2018⁸; Toledo et al., 2018⁹). Therefore, our team investigated, via online surveys, how households exposed to the Kincade Fire perceived and responded to the fire threat. The findings and insights gained from the Kincade Fire may be used to improve preparedness and evacuation procedures for future wildfire events in California and throughout the entire United States.

Quantitative methods were used to better understand household evacuation processes. We adapted a wildfire evacuation survey originally developed to study public response to the 2016 Chimney Tops 2 Fire in Tennessee (Kuligowski et al., 2020) to the Kincade Fire context. The survey explored how different factors (e.g., evacuation orders and other information, previous experience, preparedness activities, demographics, etc.) affected people’s risk perception and evacuation decision, as well as their evacuation behavior, including timing and pre-evacuation actions. The collected data were pre-processed and applied to model evacuation decision-making processes using both traditional logistic regression modeling and emerging machine learning techniques.

Recently, machine learning has been applied to model people’s choices in normal conditions (e.g., Zhao et al., 2020a¹⁰; Wang & Ross, 2018¹¹; Lhéritier et al., 2019¹²; Hagenauer & Helbich, 2017¹³; Xie et al., 2003¹⁴) as well as emergencies (e.g., Zhao et al., 2020b¹⁵; Wang et al., 2019¹⁶; Liu & Lo, 2011¹⁷; and Lo et al., 2009¹⁸). This is mainly because machine learning techniques generally have a flexible modeling structure and can be used to capture the complex underlying relationships between the input variables and the target (Zhao et al., 2020b). However, it is largely unknown how machine learning results compare with results from traditional logistic regression when applied to wildfire evacuation behavior modeling, especially when the size of the dataset is relatively small. Therefore, in this report, we aim to address this research gap by conducting a preliminary comparison between machine learning methods and logistic regression for modeling wildfire evacuation decisions (i.e., whether people chose to evacuate or stay in place). These two types of approaches will be evaluated in terms of prediction accuracy and model interpretability. The key results and knowledge generated from this report can provide critical insights for decision-makers responsible for protecting wildland-urban interface (WUI) communities in California via improved wildfire evacuation planning and management.

Methods

Research Questions

We developed a set of research questions (RQ) to guide our research design. They are summarized as follows:

• RQ1: What was the wildfire evacuation process during the 2019 Kincade Fire?

• RQ2: How can machine learning be used to model wildfire evacuation decisions?

• RQ3: What factors affected people’s evacuation decisions in the 2019 Kincade fire?

To answer these three research questions, we conducted a survey to understand household evacuation processes during the Kincade Fire in Sonoma County, California. Based on the survey data, we first conducted an exploratory analysis to understand household evacuation processes (RQ1). Then, we built a random forest model, a popular machine learning method, to model wildfire evacuation decisions (RQ2). Lastly, we applied interpretable machine learning to assess the relationships between different factors and the evacuation decision (RQ3).

Study Site Description

The Kincade Fire started on October 23, 2019 and was fully contained by November 6, 2019 (13 days later) in Sonoma County, California. On the day before the fire started, high winds led to a large-scale power shut-off, leaving around three million people without power. About 200,000 residents were ordered to evacuate due to the wildfire, the largest fire in the 2019 California wildfire season. Starting in the northeast area of Geyersville in Sonoma County, the fire burned 77,758 acres, destroyed 374 structures (including both residential and commercial), damaged 60 buildings, and injured a total of five people (four fire personnel and one civilian). The resulting cost of the fire totaled $620 million, accounting for both economic losses and property damage. Sonoma County is divided into several evacuation zones. Evacuation orders were issued to a total of 10 evacuation zones (186,651 people) between October 26 and November 2, 2019.

Data Collection Procedures and Final Sample

An online survey was developed to collect data from local residents of Sonoma County between October 22, 2020 and January 10, 2021. This survey was distributed through the Qualtrics platform via Facebook, personal networks, local news outlets, and local community groups. A screening question asked potential respondents to input the zip code of their location when the Kincade Fire occurred. This screen question was used to ensure that all participants were impacted by the Kincade Fire in 2019. In total, 270 respondents completed the survey and data from all 270 surveys were used in this study. The study area and the heatmap of the survey responses are illustrated in Figure 1.

Figure 1. The Study Area and the Survey Response Distribution

The final sample is made up of a majority of female respondents (77%) with a median age of 55 to 64 years. For the education level, 30% of respondents had an associate's degree; 34% had a bachelor’s degree, and 34% had a graduate degree, which shows that the respondents are highly educated. The distribution of the income level is relatively even among income categories, with the $75,000 to $99,999 category having the highest frequency (29%). In addition, most of the participants (56%) were employed when they took the survey (including self-employed). Among the 270 respondents, 29% of them had a medical condition. Compared to 2019 Census data for Sonoma County, our sample population has a similar racial makeup and slightly lower median household income.

Data Analysis

After data collection was completed, the first step was to conduct a quick exploratory analysis to understand the data and to investigate the evacuation process during the Kincade Fire (RQ1).

Based on the survey data, we constructed a set of variables for further analysis (see Table 1). Note that some variables that have multiple categories, such as residence duration, structure of residence, education, income, and age, were recoded into binary variables before training the models.

Table 1. Descriptive Statistics of Variables Used in the Models

By using the variables listed in Table 1, we built a random forest model, a well-established machine learning method, to model the binary evacuate/stay decision (RQ2). Random forest was first introduced by Ho in 1995 and later extended by Breiman and Cutler (Breiman, 2001¹⁹). It is an ensemble learning method for classification and regression that constructs a number of decision trees in training and predicting the majority class (for classification) or mean prediction (for regression) of the individual decision trees (Liaw & Wiener, 2002²⁰). Random forest is among the most accurate general-purpose classifiers with the capability of handling high-dimensional data (Biau, 2012²¹). This is because it can not only reduce the overfitting problem of the individual decision tree by using bootstrapping, but also automatically capture complicated relationships (i.e., nonlinearities and interactions) between variables (Zhao et al., 2020b). Plus, random forest is insensitive to missing values, noises, and irregularities in the data.

Recently, random forest has been applied in several studies to model and interpret people's choices (e.g., Lhéritier et al., 2019; Hagenauer & Helbich, 2017; Zhao et al., 2020a&b). However, few studies have attempted to model and explain people's emergency behavior using interpretable machine learning (Zhao et al., 2020b). In this study, we used the random forest model as well as machine-learning interpretation tools (i.e., variable importance and partial dependence plots) to investigate the factors affecting the householder’s decision to evacuate during the Kincade Fire (RQ3). This allows us to investigate which factors significantly affected evacuation decisions and the possible nonlinear trends of these factors.

Researcher Positionality, Reciprocity, and Other Ethical Considerations

Obtaining ethics approval is essential for disaster research in order to ensure the protection of both the survey respondents and their data. This research was approved as exempt by the University of Florida Institutional Review Board (#IRB201903432), as no personal identifiable information was collected by the survey. Note that all participants provided informed consent before taking the survey.

Dissemination of Findings

The research team worked closely with local stakeholders in Sonoma County to disseminate the research instrument and analyze the data. We frequently shared research progress and findings with Dr. Nancy Brown, Community Preparedness Program Manager of the Sonoma County Department of Emergency Management (also a co-author of this report).

Findings

Exploratory Analysis

As a first step, we conducted an exploratory analysis of the data. The data shows that approximately 96% of respondents realized that wildfires could be a problem in the community prior to the occurrence of the Kincade Fire. About half of respondents (46%) believed that their property could be threatened, and only 17% of respondents believed that their houses were well-built and safe. Additionally, 76% of respondents indicated that they took different measures and developed household emergency plans before the Kincade fire to protect their physical safety and property. Among the residents who made an emergency plan, 67% of them planned to evacuate before the wildfire spread to their residence. All the respondents have prior evacuation experience with 43% of them stating that they had evacuated at least one time before the Kincade Fire.

Among the 270 respondents, about 79% decided to evacuate in response to the Kincade Fire. Also, 73% of respondents indicated that one or more officials notified them that they were in the mandatory evacuation area and/or needed to evacuate. In total, 41% of respondents reported they first received the warning on October 26 and 27, 2019. The first recorded evacuation order was issued at 10 a.m. on October 26, 2019. Based on the survey results, the most common official channels by which people were made aware of evacuation orders in their area were text messages (64%) and social media such as Twitter and Facebook (36%) as well as telephone (28%). Moreover, 44% of the residents were notified that they are in the mandatory evacuation area and/or to evacuate by someone they knew. The most common informal channels were text messages (25%), telephone (21%), and face-to-face (16%). They also indicated they were mainly informed by someone they knew on October 26 and 27, 2019. In total, 23% of respondents were informed about the need to evacuate by both officials and someone they knew.

Random Forest Model

We trained a random forest model by using the variables shown in Table 1. The random forest model was fully tuned using grid search (a widely-adopted method for hyperparameter tuning) and the best random forest model was identified with 100 decision trees and 9 randomly selected variables considered for each split at the nodes of each decision tree.

We also compared the prediction accuracy of the random forest model with that of the traditional logistic regression model (Kuligowski et al., 2021²²) and found that the random forest model significantly outperformed the logistic regression model in terms of prediction accuracy and F1 score (two commonly-used model performance metrics). In particular, the average prediction accuracy of the random forest model is 0.81 while that of the logistic regression model is 0.77, by using a 5 times 20-fold cross-validation.

The random forest model was implemented in R by using packages randomForest (Liaw & Wiener, 2002).

Model Interpretation

In this subsection, we present the explanation results for the random forest model by using variable importance and partial dependence plots (two popular techniques in interpretable machine learning). These two methods were implemented in R by using packages caret (Kuhn et al., 2020²³) and pdp (Greenwell, 2017²⁴).

Variable Importance

Variable importance is a well-established interpretable machine learning technique that measures the importance of each variable regarding its impact on predicting the outcome (in our case, the evacuation decision). With higher dependence of a variable to make predictions, this variable is more important for the trained model. In this study, the variable importance is computed by permuting the out-of-bag samples in order to measure the prediction strength of each variable (Breiman, 2001).

The variable importance plot for the 10 most important variables of the final random forest model is illustrated in Figure 2. The results show that pre-fire perception of safety is the most important variable, with around 31% importance. The second, third, and fourth most important variables are risk perception, prior awareness of threat, and received order from official/unofficial source, respectively, with 21%, 18%, and 16% importance. This result is consistent with prior research findings (e.g., Kuligowski et al., 2020). Some socio-demographic variables (i.e., gender, number of adults in the household, median household income, age, and college degree) and one built environment variable (mobile or manufactured homes) are also relatively important for the random forest model. However, we also found some variables have negative variable importance, indicating that these variables are not contributing to improving the prediction of the model. We will further investigate why these variables had negative importance and how to deal with it in future work.

Figure 2. Top Ten Most Important Variables from the Random Forest Model

Partial Dependence Plots

Partial dependence plots are one of the most applied model-agnostic techniques for interpretable machine learning. They can graphically display the relationship between the selected input variable(s) and the predicted probabilities for classification problems (Friedman, 2001²⁵). In particular, this tool is often used to reveal nonlinearities and interactions among variables. Recently, it has been shown that partial dependence plots may be used to draw causal interpretations for black-box models (Zhao & Hastie, 2021²⁶).

So far, partial dependence plots have been applied in the travel behavior literature to study how various factors, such as socio-demographics and built environment, influence people’s travel behavior (e.g., Zhao et al., 2020a; Zhang et al., 2020²⁷; Xu et al., 2021²⁸; Ding et al., 2021²⁹). However, nearly no previous work has applied partial dependence plots to study how different factors affect householders’ evacuation decisions in any hazards. In the preliminary results of this study, we show the partial dependence plots for the two most important variables, i.e., pre-fire perception of safety (Figure 3) and risk perception at the time of evacuation decision (Figure 4).

Figure 3. Partial Dependence Plot for Pre-fire Perception of Safety

According to Figure 3, it is clear that the relationship between pre-fire perception of safety and the probability of choosing to evacuate is highly nonlinear. When people’s pre-fire perception of safety levels are below 2.5, they tend to have a high chance of evacuating, but the evacuation probability decreases significantly when levels range from 2.5 to 4. The likelihood of choosing to evacuate is equally likely between pre-fire perception of safety levels from 4 to 5. This finding is intuitive, but has never been empirically identified before.

Figure 4. Partial Dependence Plot for Risk Perception

Figure 4 also illustrates the highly nonlinear relationship between risk perception and the probability of choosing to evacuate. Note that for risk perception, we conducted a maximum likelihood factor analysis to combine the four items, including 1) I might become injured, 2) Other people/pets/livestock might become injured, 3) I might die, and 4) Other people/pets/livestock might die. The factor scores were then estimated to characterize respondents’ risk perception levels (which can take on positive or negative values). Figure 4 shows that the respondents’ evacuation probability increases rapidly when their risk perception levels are less than -0.9 and stays largely the same when their risk perception levels are more than -0.9.

The findings we have gained from this study offer a much deeper understanding of how these environmental, social, and individual-based factors influence people’s evacuation decisions. In previous work, logistic regression was used (Kuligowski et al., 2021), but using logistic regression, we can only interpret the significance level, magnitude, and direction of the beta coefficient estimates for different variables, which cannot automatically capture the potential nonlinear relationships between the dependent and independent variables. With powerful machine learning techniques, we can directly gain such insights to advance our understanding of wildfire evacuation behavior.

Conclusions

In this study, we conducted a web-based survey in Sonoma County, California to collect data on people’s evacuation behavior during the 2019 Kincade Fire. Using the survey data, we developed a machine learning model (random forest) and applied interpretable machine learning methods to explain the model.

An important finding is that the random forest model outperforms the traditional logistic regression model in terms of prediction. Additionally, the random forest model also offers richer insights for understanding people’s evacuation decisions (i.e., evacuate/stay). In particular, we found that pre-fire perception of safety was the most important variable when predicting the decision to evacuate. This variable showed a strong nonlinear relationship with the probability to evacuate. In addition, risk perception was the second most important variable for prediction, which also had a nonlinear relationship with the evacuation probability. The next step will be to use machine learning techniques to predict risk perception based on individual, environmental, and social factors present in representative fire scenarios. With this information, we can eventually provide essential information to officials responsible for evacuating communities on the segments of the population that will require targeted and personalized warnings, for example, to increase their risk perception levels in future wildfire events.

There are some limitations of this study. First, since the survey was disseminated online only, sampling bias may have been introduced. The respondents were required to have access to computers/smartphones and the internet to fill out the survey. Therefore, some population segments may be under-represented in our sample. Future studies will be required to test these findings in other fires and populations to increase their generalizability.

Second, we only applied random forest to model the evacuation decision. Although random forest is the most accurate general-purpose model (Biau, 2012) and has achieved good performance in many previous studies (e.g., Zhao et al., 2020b; Hagenauer & Helbich, 2017), future studies will include multiple machine learning techniques, such as artificial neural networks, support vector machine, and gradient boosting trees, in order to conduct a comprehensive model comparison.

References