Methodology
An Overview
The 2008 State Liability Systems Ranking Studywas conducted for the U.S. Chamber Institute for Legal Reform by Harris Interactive, Inc. The final results are based on interviews with a nationally representative sample of 957 in-house general counsel or other senior litigators at companies with annual revenues of at least $100 million. Interviews averaging 23 minutes in length were conducted by telephone and took place between December 18, 2007 and March 19, 2008.
Sample Design
A representative sample of companies with annual revenues of at least $100 million annually was drawn using idExec, Dun & Bradstreet, AMI, and Aggressive List. An alert letter was sent to the general counsel at each company. In the cases where the general counsel at a particular company could not be identified, the alert letter was sent to another senior person at the company such as the Chief Executive Officer or Senior Vice President. This letter provided general information about the study, notified them that an interviewer from Harris Interactive would be contacting them and requested their participation. It also included a fact sheet about the study, the 2007 press release and an article about the 2007 results that was published by Bloomberg LLP. A copy of the letter and enclosed materials appears in Appendix C. This year, in addition to receiving an alert letter, some contacts were told that a $50 or $100 donation would be made to a charity in exchange for agreement to participate in the study. This initiative was implemented toward the end of the interview period in order to increase cooperation and assure an adequate sample.
The sample was segmented into two main groups. Of the 957 respondents, 57 were from insurance companies, with the remaining 900 interviews being conducted among public and private corporations from other industries. The proportion of interviews with insurance companies represents 6% of the total sample which is the typical representation of insurance companies in the universe of companies with $100 million or more in revenues. Since property casualty insurance companies have extensive experience with state liability systems, for the purposes of this study we worked to ensure that our proportion of insurance companies matched the overall population.
Respondents had an average of 19.3 years of relevant legal experience (including their current position), had been with their company an average of 11.6 years, and had been in their current position an average of 9.1 years.
Telephone Interviewing Procedures
The 2008 State Liability Systems Ranking Study utilized Harris' computer-assisted telephone interviewing (CATI) system, whereby trained interviewers call and immediately input responses into the computer. This system greatly enhances reporting reliability. It reduces clerical error by eliminating the need for keypunching, since interviewers enter respondent answers directly into a computer terminal during the interview itself. This data entry program does not permit interviewers to inadvertently skip questions, since each question must be answered before the computer moves on to the next question. The data entry program also ensures that all skip patterns are correctly followed. The on-line data editing system refuses to accept punches that are out-of-range, it demands confirmation of responses that exceed expected ranges, and asks for explanations for inconsistencies between certain key responses.
In order to achieve high respondent participation, in addition to the alert letters, numerous telephone callbacks were made in order to reach the respondent and conduct the interview at a convenient time for the respondent.
Once a qualified respondent was identified, the respondent was first asked about his/her familiarity with several states. First, 24 states out of the list of 50 possible states were presented to the respondent. Within these 24 states, the 17 states presented were the following: Alaska, Hawaii, Idaho, Iowa, Kansas, Maine, Montana, Nebraska, New Hampshire, North Dakota, Rhode Island, South Dakota, Utah, Vermont, West Virginia, Washington and Wyoming. These states were prioritized in order to get a sufficient number of evaluations, since in the past years of this study, data for these states were based on fewer evaluations. The remaining 7 states were randomly selected from the remaining states not mentioned above.
Respondents were then given the opportunity to name any other state, aside from the states already presented, and specify if they are very or somewhat familiar with that state.
If the respondent was very or somewhat familiar with a given state, the respondent was then given the opportunity to evaluate that state's liability system. Similar to 2007, the maximum number of states a respondent had the opportunity to evaluate was 10. On average, each respondent evaluated 4 states, up from an average of 3 states in 2007. In previous years (2002 – 2006), respondents were given an opportunity to evaluate a maximum of 15 states, evaluating an average of 6 states. This was changed in 2007 in order to reduce the burden on respondents and increase the likelihood that they were familiar with the states they were rating.
Significance Testing
Reliability of Survey Percentages
It is important to bear in mind that the results from any sample survey are subject to sampling variation. The magnitude of this variation (or error) is affected both by the number of interviews—the base size—and by the level of the percentages expressed in the results.
Table A-1 shows the possible sample variation that applies to percentage results for this survey. The chances are 95 in 100 that a survey result does not vary, plus or minus, by more than the indicated number of percentage points from the result that would have been obtained if interviews were conducted with all persons in the universe represented by the sample. For example, if the response for a sample size of 300 is 30%, then in 95 cases out of 100, the response in the total population would have been between 25% and 35% (+/-5%). Note that survey results based on subgroups of small size can be subject to large sampling error.
Recommended Allowance for Sampling Error of Proportions (Plus or Minus) |
|||||
| Survey Percentage Result | |||||
| Sample Size | 10% or 90% | 20% or 80% | 30% or 70% | 40% or 60% | 50% |
| 1600 | 2 | 2 | 2 | 2 | 3 |
| 1500 | 2 | 2 | 2 | 3 | 3 |
| 1400 | 2 | 2 | 2 | 3 | 3 |
| 900 | 2 | 3 | 3 | 3 | 3 |
| 800 | 2 | 3 | 3 | 3 | 3 |
| 700 | 2 | 3 | 3 | 4 | 4 |
| 600 | 2 | 3 | 4 | 4 | 4 |
| 500 | 3 | 4 | 4 | 4 | 4 |
| 400 | 3 | 4 | 4 | 5 | 5 |
Significance of Differences Between Proportions
Sampling tolerances are also involved in the comparison of results from different surveys or from different parts of a sample from the same survey (subgroup analysis). Table A-2 shows the percentage difference that must be obtained before a difference can be considered statistically significant. These figures, too, represent the 95% confidence level.
To illustrate, suppose the two percentages in question are 34% and 25%. More specifically, suppose that one group of 300 has a response of 34% "yes" to a question, and an independent group has a response of 25% to the same question, for an observed difference of 9 percentage points. According to the table, this difference is subject to a potential sampling error of 6-7 percentage points. Since the observed difference is greater than the sampling error, the observed difference is significant.
Sampling Error of Difference between Proportions Approximate Sampling Tolerances (at 95% Confidence Level) To Use in Evaluating Differences between Two Percentage Results |
|||||
| Survey Percentage Result | |||||
| Sample Size | 10% or 90% | 20% or 80% | 30% or 70% | 40% or 60% | 50% |
| 900 v. 900 | 3 | 4 | 4 | 5 | 5 |
| 500 | 3 | 4 | 5 | 5 | 6 |
| 300 | 4 | 5 | 6 | 7 | 7 |
| 200 | 5 | 6 | 7 | 8 | 8 |
| 100 | 6 | 8 | 10 | 10 | 10 |
| 50 | 9 | 11 | 13 | 14 | 14 |
| 500 v. 500 | 4 | 4 | 6 | 6 | 6 |
| 300 | 4 | 6 | 7 | 7 | 7 |
| 200 | 6 | 7 | 8 | 8 | 8 |
| 100 | 7 | 9 | 10 | 11 | 11 |
| 50 | 9 | 12 | 13 | 14 | 15 |
Sampling error of the type so far discussed is only one type of error. Survey research is also susceptible to other types of error, such as refusals to be interviewed (non-response error), question wording and question order, interviewer error, and weighting by demographic control data. Although it is difficult or impossible to quantify these types of error, the procedures followed by Harris Interactive, Inc. keep errors of these types to a minimum.