On September 9, 2009, Towers Perrin released its report of the firm’s 2008 Survey of Directors and Officers Liability Insurance Purchasing Trends, which can be accessed here. Towers Perrin’s anticipated annual report again this year will undoubtedly be widely read throughout the D&O insurance industry. The report is a good resource and it is full of useful and interesting information.


Because the Report is so widely read, I think it is very important to highlight some specific issues about  the report. As I noted in connection with the 2007 report (here), the survey report is subject to some very important limitations that may not always be fully appreciated or understood.


In my view, the most significant limitation is one that is duly noted in the final two sentences of the Report section headed "Statistical Terms Used in This Report." As the Report states, the Report is the product of a survey, which means that the data in the Report are drawn from a "non-probability sample." That is, participants "choose – or are selected" to participate, and therefore the sample "is not random." Most importantly, because "not all potential respondents are likely to participate, survey biases must be considered when interpreting results."


It is the danger that this last point – the possibility that the reported results reflect "survey biases" – that most concerns me. In particular, the reference to the possibility that the survey respondents were "selected" is particularly relevant.


Specifically, the broker rankings section of the Report reveals that fully 95.2% of all survey responses came from the clients of just four brokerage firms. The same four firms also dominated the 2007 survey results, but the 2008 results reflect an even greater concentration, as the four firm’s clients represented "only " 88% of the survey respondents in the 2007 survey. In the 2008 survey, only 4.8% of all respondents are clients of firms other than the four brokerages. Indeed, clients of the three global insurance carriers represented just 1.8% of the respondents.


These observations should not be taken as a criticism of these four survey-predominating brokerages. I will stipulate that they are in fact strong and significant industry participants. But no informed person actually thinks they are the four largest D&O brokers in the country. They are undeniably the leading firms in getting their clients to complete the Towers Perrin survey. Again, no criticism here; I salute their enterprising spirit in achieving this result. However, no one should confuse the survey "ranking" with an actual market share ranking, nor could anyone fairly attempt to use the survey results to try to create that impression.


I emphasize this aspect of the Report because the survey bias in the broker participation population has pervasive effects throughout the entire report. Indeed, given that the pool of actual survey respondents for all practical purposes represents the clientele of those four brokerage firms, the Report fairly might be characterized as a description of the purchasing patterns of the clients of those four firms, rather than of the marketplace as a whole.


However, the Report itself does not address whether or not this rather categorical "skew" in the survey response population affects the other reported results, although it pretty obviously could significantly affect many of the other observations in the report. For example, the Report’s attempt to rank carriers by policy count and premium could simply be a reflection of the predilections of the four firms. The same is true with respect to such issues as respondents’ decision whether to purchase Side A insurance or IDL insurance.


There are other limitations arising from the characteristics of the respondents. Many of the respondents are very small.—nearly 40% reported assets under $6 million. Nearly 70% of the respondents had under 100 employees.


Another perhaps more significant concern with the 2008 Report is that the survey participants completed the survey during the third quarter of 2008. Not only does that mean the data are a year old, but also the survey results may fail to reflect the enormous changes in both the global economy and in the insurance marketplace during the last twelve months. Thus, there is some risk that the survey results, to whatever extent they fully and accurately reflect marketplace conditions of a year ago, may not reflect current conditions, given the enormous changes since the survey was conducted.


In addition, the Report also makes numerous year over year comparisons, noting changes between the results of the 2007 survey and the results of the 2008 survey. The difficulty with these comparisons is that there is no way of knowing whether or not the differences in the survey results are simply the result of a different mix in survey respondents, rather than a change in the underlying circumstances. To be sure, the Report does several times work hard to provide comparisons showing the results reported by repeat survey respondents. But there are numerous comparisons throughout the Report that are not so limited.


With respect to the concern noted above about the concentration of survey respondents in the portfolios of just four brokerage firms, it is a fair observation that the survey is open to all. If survey participation were more widespread, many of the concerns noted above might be alleviated. However, the opposite appears be happening, as participation by other brokerage firms is clearly declining, for reasons that might well be surmised.


None of this is meant as a criticism of Towers Perrin, which should be saluted for performing the survey and distributing the survey report without charge. Moreover, Towers Perrin itself acknowledges that there may be biases arising from the survey population distribution. So I don’t mean to criticize Towers Perrin, or anyone else for that matter. Rather, my analysis here is presented as a petition to all industry participants that in using the survey data, they should explicitly recognize and acknowledge the sample bias limitations inherent in the report. In particular, no one should try to make the survey results represent anything more than they actually do, particularly with respect to the concentrations noted above.