Evaluating the Reproducibility of Tree Risk Assessment Ratings Across Commonly Used Methods

Introduction

Trees are a critical aspect of urban green infrastructure which provide many benefits to humans, including removing harmful air pollutants, reducing stormwater runoff, mitigating the urban heat island effect, and adding to the visual aesthetics of the urban landscape (Hauer and Johnson 2003; Nowak and Dwyer 2007; Vogt et al. 2015). Trees can also pose risks to both people and property (Ellison 2005; Fay 2007; Koeser 2009; Vogt et al. 2015; Dunster et al. 2017; Smiley et al. 2017). As such, the balance of maximizing tree-related benefits and mitigating the associated risk is relevant to tree care professionals who maintain the urban forest (Ball and Watt 2013; Koeser et al. 2016a; Smiley et al. 2017; Carmichael and McDonough 2019; Coelho-Duarte et al. 2021; Judice et al. 2021; Klein et al. 2021b; Klein et al. 2022b). By identifying trees that have an unacceptable level of risk and knowing what mitigation measures will lower that risk, tree care professionals can reduce the likelihood of personal injury, property damage, and potentially costly litigation (Hauer and Johnson 2003; Ellison 2005; Fay 2007; Hauer and Peterson 2016).

Tree risk assessment is a systematic evaluation of trees to identify defects and other conditions that could lead to failure within a specified timeframe (likelihood of failure), analyze their potential to impact a target (likelihood of impact), and estimate the associated consequences (consequences of failure) (Hauer and Johnson 2003; Ellison 2005; Norris 2007; Dunster et al. 2017; Smiley et al. 2017; Norris and Moore 2020). Tree risk assessment methods should be replicable, specific, and verifiable—working within the bounds of a defined acceptable level of risk and knowledge of the costs and benefits associated with mitigation measures (Norris 2007; Koeser et al. 2016a; Klein et al. 2022b). However, previous research has shown that tree risk assessments are inherently subjective, and that the assessor has just as much influence as the method on the overall risk rating (Norris 2007; Koeser and Smiley 2017; Koeser et al. 2017). While experience and professional credentialing often influence how risk is rated (Smiley et al. 2017; Klein et al. 2021a; Klein et al. 2022b), different experiences and training may result in divergent views on risk, the need for mitigation, and appropriate mitigation measures (Ball and Watt 2013; Klein et al. 2016; Koeser and Smiley 2017; Norris and Moore 2020).

In a study involving nearly 300 arborists conducting basic (Level 2) assessments of 3 different trees, Koeser and Smiley (2017) found that likelihood of impact and consequence of failure ratings were variable among all participants, and that the assessor was 4 times more likely to impact the overall risk rating than characteristics of the tree being assessed. Likewise, arborists with experience and professional credentialing were 4 times more likely to recommend retaining and monitoring rather than tree removal compared to those with no such training. Additionally, arborists with risk assessment training were significantly more likely to assign lower likelihood of impact and likelihood of failure ratings, as well as lower overall risk ratings.

Klein et al. (2016) compared the perceived occupancy rates with actual occupancy rates as measured by traffic counters (Klein et al. 2022a). In this study, likelihood of impact ratings among participants with experience and credentialing were less variable than those without. However, in a study capturing consequences of failure ratings, Klein et al. (2021a) found no correlation between an assessor’s professional experience and estimates of branch size. Specifically, participants with no risk assessment experience more accurately estimated branch size than participants with previous experience. However, participants who had the International Society of Arboriculture (ISA) Tree Risk Assessment Qualification (TRAQ) and Board Certified Master Arborist^® (BCMA^™) credential were more likely to give a higher consequence of failure rating than other participants based on estimations of branch size and branch size class (Klein et al. 2021a). The results suggest that something other than part size may be responsible for variability in consequence of failure ratings and may contribute to a larger body of data indicating that variability in tree risk assessments is impacted by the assessor (Norris 2007; Koeser et al. 2015; Klein et al. 2016; Koeser et al. 2017; Koeser and Smiley 2017; Coelho-Duarte et al. 2021). The current study compares 2 systems that have relatively balanced risk inputs (Norris and Moore 2020) and are being increasingly used by tree managers (Koeser et al. 2016c), the method taught as part of the ISA TRAQ and the Quantified Tree Risk Assessment (QTRA) method.

The TRAQ method is based on ISA’s Best Management Practices (BMP): Tree Risk Assessment (Smiley et al. 2017). This system uses 2 matrices to determine the overall risk of a tree or tree part: the first to determine a rating for the likelihood of failure and impact, and a second that considers that rating in conjunction with a rating for consequences of failure. Each of these risk factors is expressed with qualitative terms. Likelihood of failure is categorized as improbable, possible, probable, or imminent. Likelihood of impact is described as very low, low, moderate, or high. Consequences of failure are qualified as negligible, minor, significant, or severe. The TRAQ system directs users to conduct assessments at 1 of 3 levels of detail (Dunster et al. 2017): limited visual (Level 1), basic (Level 2), or advanced (Level 3) assessments. Level 2 assessments are 360° inspections of a tree from ground-level, used to identify the species and notable defects, evaluate condition, and determine potential targets that could be impacted, for which the assessor evaluates the 3 components of risk (i.e., likelihood of failure, likelihood of impact, and consequences of failure)(Dunster et al. 2017; Smiley et al. 2017).

The QTRA system proposes a formula-based approach to maintaining an acceptable level of risk, defined as a 1/10,000 chance of significant harm occurring (Ellison 2005; Ellison 2019). QTRA’s risk factors of a target’s impact potential, the size of the tree or tree part, and the probability of failure are expressed in broad ranges of numerical ratios that are categorized with ordinal numbers. As with TRAQ, the assessor performs a 360° inspection of the tree from ground-level to assess the species and condition of the tree, identifies defects that could lead to failure as well as the size of the affected tree part, identifies potential targets of impact, and gauges the consequences of such a failure. Target ranges are categorized according to either occupation rates of people or vehicles, ranging from 1:1 to 1:1,000,000, or the financial value of the property (Ellison 2019). Tree part size ranges of greater than 18 in (450 mm) to 1 in (25 mm) are grouped into categories 1 through 4 with impact potential ranges from 1:1 to 1:2,500. Probability of failure ranges from 1:1 to 1:10,000,000 are grouped into categories 1 through 7 with an associated timeframe of 1 year. Once each risk factor has been quantified, the assessor inputs them into a QTRA manual calculator or software program to calculate the overall risk of harm (Ellison 2019).

Comparing these systems has the potential to gauge their reproducibility and illuminate assessor biases that affect assessment outcomes and mitigation recommendations. Verifying the validity and analyzing the effect of common assumptions, mental shortcuts, and standard operating procedures of tree risk assessors is crucial for decision-making based on actual rather than perceived risk, preventing unrealistic recommendations and undesirable mitigation such as unwarranted tree removals or the retention of compromised trees (Ellison 2005; Koeser 2009; Hauer and Peterson 2016; Smiley et al. 2017; Coelho-Duarte et al. 2021; Judice et al. 2021).

As such, this research had 3 main objectives. First, trees were rated with both the TRAQ and QTRA methodologies to evaluate differences in ratings and statistically compare variability among assessors. Next, as there is no requirement to take the TRAQ training to use the TRAQ system as informed in the ISA BMP, we compared ratings derived from the TRAQ approach for respondents possessing or lacking the associated credential. Beyond these 2 primary objectives, we evaluated the impact of timeframe on rating consistency. As such, our final objective was to compare variability for likelihood of failure ratings using both a 1-year and 3-year timeframe. The insight gained from the study will contribute to furthering our understanding of tree risk assessment, risk perception, and the decision-making process. This has the potential to benefit not only tree management but also overall public health and safety.

Methods

Results

Of 450 individuals selected for this study, 60 (13.3% response rate) submitted risk assessments. Of these, 21 were QTRA users, 28 were TRAQ credentialed, and 11 were neither QTRA users nor TRAQ credentialed (Table 1). The vast majority (> 90%) of respondents were male with all groups having an average age of over 50 years. Similar gender demographics have been found in several past studies related to arboricultural professionals. For example, Kuhns et al. (2002), Klein et al. (2021a), and Klein et al. (2022b) all found that males comprised approximately 90% of industry professionals while females made up about 10%. Likewise, O’Herrin et al. (2020) surveyed urban foresters throughout the United States and found that 79% of respondents were male. Similarly, as part of a recent needs assessment, ISA surveyed its membership (n = 4278) and found that approximately 75% of certified arborists and 76% of TRAQ arborists were male (ISA, unpublished data). Most respondents (85%) conducted risk assessments as part of their work routines (Table 1). Similarly, Koeser and Smiley (2017) evaluated the impact that assessors have on tree risk assessment ratings and recommended mitigation and found that 79.1% (n = 296) of participants conduct risk assessments as part of their job.

When response rate is less than 100%, and especially less than 70%, nonresponse error can be a limitation and should be examined (Bose 2001). One way to potentially limit this type of error is to assess whether the sample is representative of the target population (Bose 2001). Demographic data for the specific target audience (e.g., specific to the geographical focus of the study) was not available. However, our TRAQ respondents seem to align with the population they represent. According to a recent survey conducted by the ISA, 54% of TRAQ certified arborists worldwide are 45 years of age or older (ISA, unpublished data). While our ability to conduct demographic comparisons is limited, there is no indication the age of TRAQ participants does not align with the population.

Another approach to controlling for nonresponse error is to group respondents into “early” and “late” respondent groups and compare them on key study variables (Bose 2001; Lindner et al. 2001; Lindner and Wingenbach 2002). This approach considers nonrespondents a “linear extension of the latest respondents” (Lindner et al. 2001), recognizing late respondents as more like nonrespondents than early respondents (Bose 2001). Thus, if trends are identified, they could signify potential differences between respondents and nonrespondents and suggest a need to be especially concerned about nonresponse error. For the TRAQ and QTRA respondents for whom we had a survey postmark date, we split each group by the earliest half and latest half and compared them on key study variables (Lindner et al. 2001). For the TRAQ respondents, we compared one-year LOF for trees #1, #2, and #3 between early and late respondents. For QTRA respondents, we compared POF for trees #1, #2, and #3 between early and late respondents. Given the ordinal nature of LOF and POF, we employed 6 chi-square tests of independence to test the relationship between early/late respondents and LOF or POF, respectively. None of the chi-square analyses revealed significant relationships at p = 0.05 (LOF1, X2 [3, 44] = 2.464, p = 0.482; LOF2, X2 [3, 44] = 3.365, p = 0.339; LOF3, X2 [3, 44] = 2.607, p = 0.456; POF1, X2 [4, 21] = 2.100, p = 0.717; POF2, X2 [4, 21] = 9.259, p = 0.055; POF3, X2 [4, 21] = 6.300, p = 0.178), and thus we concluded that there is no indication nonrespondents are different from respondents.

In comparing self-reported financial and health risk tolerances from our DOSPERT questions, we failed to detect any differences between those in the USA and in the UK or Australia. Participants from the USA assessed the risk with our health scenarios slightly higher than the UK and Australian group (mean rating 7.2 versus 6.9), though this difference was not enough to pass our threshold for significance (P-value = 0.071). Financial risk ratings were even more closely aligned for the 2 demographics (7.3 for the USA and 7.2 for the UK and Australia) and not significant (P-value = 0.444).

When assessing final risk ratings, we were not able to make direct comparisons between the assessments derived using the QTRA method and those derived from the TRAQ/ISA BMP method given differences in terminology. However, we have summarized these responses here for completeness. In assessing the 3 tree videos, the majority of assessments (n = 62) conducted by QTRA-trained arborists classified risk associated with the trees as “broadly acceptable” or having a risk of harm of 1:1,000,000 or less. The remaining assessments were nearly evenly split between “tolerable” (i.e., RoH between 1:1,000,000 and 10,000; 24%) or “unacceptable” (i.e., RoH greater than 1:10:000; 23%) ratings.

Of the 82 risk assessments conducted by the TRAQ group, 14% were assessed as having “low” risk, 38% were assessed as having “moderate” risk, 43% were assessed as having “high” risk, and 5% were assessed as having “extreme” risk. In contrast, ISA BMP participants (n = 32) were more likely to rate risk as “extreme” (13%), less likely to rate risk as “high” (25%), less likely to rate risk as moderate (31%), and more likely to rate risk as low (31%). Figure 1 shows a comparison of the risk ratings between both TRAQ and ISA BMP participants.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Comparison of ratings submitted by ISA Certified Arborist^® (CA) participants with the International Society of Arboriculture Tree Risk Assessment Qualification (TRAQ) and an ISA BMP group. Each respondent assessed 3 trees (n = 82 assessments for TRAQ and n = 32 assessments for ISA BMP).

In addition to looking at patterns of risk ratings, we compared the variability of the decisions made across the 3 methods tested. In running simple tests of homogeneity of variances, we found that none of the methods produced more reproducible (or conversely, more variable; min. P-value = 0.251) results.

While the risk-rating terminology for the 2 methods differed, all respondents were given the same options for mitigation based on their assessment, allowing for direct comparison between the 3 groups. As Figure 2 shows, arborists lacking any formal risk assessment training (i.e., ISA BMP) were more likely to recommend removal than their peers with either QTRA training or the TRAQ credential. In fact, they recommended removal half of the time. This finding is consistent with the work of Koeser and Smiley (2017), who found that as experience increases, the risk rating decreases, and the recommended mitigation is more likely to be retain and monitor the tree rather than remove it. This increased likelihood to recommend full tree removal among the ISA BMP respondents was significantly greater than what was observed with the QTRA respondents (P-value = 0.040). The TRAQ group was less likely to recommend removal than the ISA BMP participants but more likely to recommend removal than the QTRA group, though neither difference was statistically significant (P-values = 0.390 and 0.126; respectively). Similarly, QTRA participants were more likely to recommend monitoring than their TRAQ (P-value = 0.037) and ISA BMP (P-value = 0.064) peers (Figure 2).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Differences in mitigation strategies recommended by respondents with Quantified Tree Risk Assessment training (QTRA; n = 21), respondents with International Society of Arboriculture Tree Risk Assessment Qualification training (TRAQ; n = 28), and respondents lacking either training (ISA BMP, n = 11). Each participant assessed 3 trees and was permitted to recommend more than 1 strategy per tree. As such, percentages represent a total of 74, 119, and 38 mitigation recommendations for QTRA, TRAQ, and ISA BMP, respectively.

Finally, in looking at the impact of timeframe on risk ratings, we found that likelihood of failure ratings increased when the TRAQ and ISA BMP participants switched from a 1-year to a 3-year timeframe (Figure 3). In contrast, we were unable to detect any differences in the variability of likelihood of failure ratings when comparing the 2 timeframes (min P-value = 0.161; Figure 3).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Box and whisker plots depicting the variability of likelihood of failure ratings made using 1-year and 3-year timeframes. A likelihood of failure rating of 1 = improbable, 2 = possible, 3 = probable, and 4 = imminent. The bolded numbers represent median values. The green boxes represent the interquartile range of the ratings. The whiskers represent the minimum and maximum recorded values. Based on 117 assessments from 39 participants.

Discussion

Currently, the most commonly used tree risk assessment methods are qualitative (Koeser et al. 2016b) and rely on a certain degree of subjectivity on the part of the assessor (Norris 2007). Since tree risk assessment is subjective, predicting tree failure at or close to 100% accuracy is unlikely given the current methods, approaches, and systems. Tree risk assessment and the outcome of decisions with evaluated trees can ultimately be influenced by many outside factors (personal bias, media, extreme weather, politics, policy, etc.) that have nothing to do with the health of the tree and the safety of the targets within the fall zone.

Our findings build on previous work by Norris (2007), who compared 8 risk assessment methods by 12 experienced arborists who evaluated 8 different trees for a variety of targets, defects, and potential consequences. This was the first study to shed light on some of the variability associated with the most common and currently used tree risk assessment methods, as well as touch on the idea of how the assessor’s perception of risk and personal biases can ultimately affect the outcome of an assessment.

Similar to the work of Norris (2007), Reyes de la Barra et al. (2018) evaluated 30 trees against 4 different risk assessment methods: ISA (Matheny and Clark 1994), ISA BMP (Smiley et al. 2017), United States Department of Agriculture Forest Service (Pokorny 2003), and Sampaio et al. (2010). The risk outputs for each method were then standardized using a range of 0 to 100% (0 to 25% = low risk; 25 to 50% = medium risk; 50 to 75% = high risk; 75 to 100% = extreme risk), then compared against one another. The authors found considerable variation across the 4 methods, with the Sampaio method rating the associated risk to be typically the highest and the ISA BMP method rating the lowest when evaluating the same trees.

One of the challenges of comparing risk assessment methods is gathering a sample of trained arborists who are knowledgeable in the methods of interest and having them evaluate the same group of trees. While some risk assessment methods are used concurrently within a region or country (Koeser et al. 2016c), others are very much tied to specific locations around the world. To overcome this challenge, we mailed videos of trees to our survey group. While this approach allowed us to access a greater number of fully trained arborists than has previously been seen in the literature (Norris 2007; Reyes de la Barra et al. 2018; Coelho-Duarte et al. 2021), the use of videos has its own limitations as it is somewhat of an abstraction compared to a true site visit. Moreover, 3 of the respondents (5%) stated that they were unfamiliar with the species selected.

This study found the perceptions of financial or health risk were relatively static across our respondents from the USA, UK, and Australia. While the countries have many similarities with regard to political origin and language, their legal systems do vary, which may influence tree risk ratings and perceptions of risk.

Concerning the methods assessed and their influence of risk mitigation, the pattern observed in Figure 2 was as expected. Our past work (Koeser and Smiley 2017) showed that arborists with risk assessment training were more apt to recommend more passive mitigation measures like monitoring and less likely to recommend tree removal. Though not a significant difference, the tendency for the QTRA group was to recommend removal less often than the TRAQ group. This result is consistent with the authors’ experiences evaluating trees with both systems (unpublished data). When noting differences between the 3 groups, it should be noted that the actual risk posed by the trees is unknown. As such, there is no way to identify the “most accurate” group. Additionally, we did look at the variability among the 3 inputs (i.e., likelihood of impact, likelihood of failure, and consequences of failure) for the TRAQ and ISA BMP groups. Contrary to Koeser and Smiley (2017), none of the inputs were any more or less variable than the others (min P-value = 0.091).

Finally, it was interesting to see that increasing the timeframe of risk assessments from 1 year to 3 years did not impact reproducibility. A timeframe for the likelihood for failure is a core concept of tree risk assessment, and willingness to accept risk is a key component with developing the time interval. In addition to risk, this interval might include the resources available to periodically evaluate trees, the structural condition of the tree (especially in the case of trees with an imminent likelihood of failure), and the local climate, as the latter can influence tree growth and decay rates. Our findings indicate there is no need to add another consideration (i.e., minimizing variability) to this list. As one would expect, likelihood of failure ratings increased as one increased the timeframe.

Although we believe that the results of the study are in line with past work on the topic of tree risk assessment ratings and the influence of the assessor, it’s worth noting several potential limitations to such studies. These may include accounting for nonresponse error and ensuring the target population is represented in the sample; a relatively small sample size; the use of videos rather than assessing trees in person; differences in age, knowledge, and experience among respondents; and an unfamiliarity of the species assessed in the study. Such limitations have the potential to lead to variability of results.

Conclusion

As in other past works, this study shows that risk assessment training as indicated through earning a credential can have a significant impact on prescribed risk mitigation activities, leading to suggested management decisions that favor continued monitoring over tree removal. Past research has documented this with the ISA TRAQ training program. This effort shows the same pattern with risk assessments derived using the QTRA approach. Moreover, while risk assessments can vary from evaluator to evaluator given a whole host of factors, the timeframe selected (1 versus 3 years) for the inspection does not appear to influence overall reproducibility of assessments. Future work should continue to look at both the reproducibility of assessments and accuracy of all commonly used tree risk assessment methods.

Conflicts of Interest

The authors reported no conflicts of interest.