Last week, Towers Perrin released its report of the firm’s 2007 Survey of Directors and Officers Liability Insurance Purchasing Trends, which can be accessed here. The firm’s annual survey report is widely read throughout the D&O insurance industry, and is generally viewed as an important information resource. Every year, the survey report is full of interesting observations, and this year’s version is no exception. The report merits reading at length and in full.
But while the survey report is widely read, I don’t know if the survey’s limitations are always fully understood or appreciated.
The report itself expressly acknowledges that it is based on a “self-selecting non-probability sample” The significance of this fact is briefly explained in the final two sentences at the bottom of the report’s preface page, where the report states that:
A non-probability sample is one in which respondents choose – or are selected – to participate. Such a sample is therefore not random. Because not all potential respondents are equally likely to participate, survey biases must be considered when interpreting results.
It is this latter point – that is, that “survey biases” must be considered when interpreting the survey’s results – that is all too often overlooked when the survey’s results are cited.
Let me just say that in referring to “bias” here, I am not in any way criticizing the report or its authors. The word “bias” as commonly understood has a negative connotation, but in this context, the word bias simply represents a mathematical property. But while the word bias should not suggest any negative connotations here, it should also be understood that, as stated in Wikipedia (here), “a biased sample causes problems because any statistic computed from that sample has the potential to be consistently erroneous.”
The survey results that most clearly reflect the sample bias are in the report’s discussion of what it calls “broker rankings.” As a footnote makes clear, the table relates solely to retail brokers, and does not contain any information about wholesale brokers. But even with respect to retail brokers, the table on which the “ranking” is based shows that over 88% of the survey responses relate to just four brokerages. Nor are these four survey-dominant brokerages the nationwide industry giants – to the contrary, these four participants would more accurately be described as strong regional players with an important presence in their respective geographic regions. The three largest nationwide industry giants meanwhile are represented collectively in only about 1.2% of responses.
My observations here should not in any way be taken as a criticism of these four survey-predominating brokerages. I will stipulate that they are in fact strong and significant industry participants. But no informed person actually thinks they are the four largest D&O brokers in the country. They are undeniably the leading firms in getting their clients to complete the Towers Perrin survey. Again, no criticism here; I salute their enterprising spirit in achieving this result. However, no one should confuse the survey “ranking” with an actual market share ranking.
I emphasize this aspect of the survey report because the bias in the broker participation population has pervasive effects throughout the entire report. For example, the four survey-predominant brokerages all have portfolios that are heavily weighted toward the technology and life sciences industries. Not too surprisingly, therefore, the two industry groups most heavily represented among both public and private company survey participants are “Technology” and “Biotechnology & Pharmaceuticals.” These two industry groups together represent about half of both public and private company survey participants.
Obviously, this heavy concentration of survey participants in just these two industry groups does not correspond to the economy as a whole. But this industry concentration – which is a direct result of the concentration of the survey population in the portfolios of a small handful of brokerages – has very significant ramifications for the report’s other findings. The report itself expressly recognizes this in the portion where it discusses the distribution of survey respondents’ primary insurance among the various leading carriers. The report’s analysis recognizes that the distribution of primary insurers is directly affected by the industry distribution, and the report examines this effect in detail.
But while the report examines in the impact of the survey population industry distribution on the distribution of business among primary insurers, the report does not elsewhere make this analysis. For example, the report does not similarly consider whether or not the industry concentration is relevant to the distribution of business among excess carriers, nor does it consider the possible impact of the concentration of the survey population on the other findings in the report.
I emphasize these points because I think they show a couple of important things. First, not only is the survey population concentrated into the portfolios of just a small handful of brokers, but this concentration has important implications for the rest of the report. It clearly affects, for example, the industry concentration of the survey population, which in turn affects the reported distribution of primary insurance among the various carriers.
These apparent effects raise the question whether the concentration of the survey population has similar effects on the other areas examined in the survey report. While the impact of the population concentration is most self-evident in the industry distribution, it is more difficult to tell from the report whether the other components of the report’s findings are similarly affected by the survey population’s concentration in the portfolios of just a very small handful of brokers.
It is a fair observation that Towers Perrin makes survey involvement available to all industry participants, not just the four survey-predominant firms. It is also a fair observation that if survey involvement were more widespread, many of the concerns noted above might be alleviated. But what has happened is that a few brokerage firms have clearly made their clients’ participation in the survey a top priority, while other brokerage firms have obviously decided to take a different approach, for reasons that one might speculate are related.
None of this is meant as a criticism of Towers Perrin, which should be saluted for performing the survey and distributing the survey report without charge. Moreover, Towers Perrin itself acknowledges that there may be biases arising from the survey population distribution. So I don’t mean to criticize Towers Perrin, or anyone else for that matter. Rather, my analysis here is presented as a petition to all industry participants that in using the survey data, they should explicitly recognize and acknowledge the sample bias limitations inherent in the report. In particular, no one should try to make the survey results represent anything more than they actually do, particularly with respect to the concentrations noted above.
The Option Backdating Case Resolution Scorecard: Over at the Securities Litigation Watch, Adam Savett has prepared an updated options backdating case resolution scorecard, which can be accessed here. Savett has a number of interesting observations about case dismissals and the speed of case resolution. The D&O Diary’s own scorecard of options backdating lawsuit dismissals, denials and settlement can be accessed here.