White Paper
US Public Opinion toward Voting Technologies
M. Glenn Newkirk
1 March 2004
InfoSENTRY Services, Inc.
www.infosentry.com
919.838.8570
© 2004. InfoSENTRY
Services, Inc. All Rights Reserved. Reproduction and distribution without prior
written permission from InfoSENTRY Services, Inc. is prohibited.
TABLE OF CONTENTS
Introduction
and Research Methods
Overall
Opinions About Voting Technology Security
Detailed
Attitudes Toward Electronic Voting Systems (DREs)
Detailed
Attitudes Toward Optical Scan (Precinct Count) Voting Systems
Detailed
Attitudes Toward Optical Scan (Vote By Mail) Voting Systems
Detailed
Attitudes Toward Internet Voting Systems
The
Effect of Internet Use of Voting Technology Trust
Trust
in Voting Systems: Partisan Views
Appendix
1: Survey Technical Details
Introduction and Research Methods
InfoSENTRY is involved throughout the country with election information system technologies in their various forms. We work with county and statewide voter registration systems. We assist clients with migration from punchcard voting systems, implementation of optical scan systems, use of Vote-By-Mail (VBM) systems, and conversions to Direct Record Electronic (DRE) voting systems.
The chaos of the 2000 Presidential Election, its aftermath in the 2002 “by elections,” and the passage of the Help America Vote Act (HAVA) sent jolts through the election community. Critics and partisans questioned election processes and election technology. They questioned the ethics and competence of election officials, without having much experience in elections administration prior to 2000. Elections topics moved from occasional blips in the back pages and filler segments on cable news shows to very frequent items on the front page and featured segments in major news stories.
No issue has generated more heated rhetoric following HAVA’s passage than the replacement of older voting systems with electronic voting technologies. The move to implement all-electronic DREs and plans to expand Internet voting brought new voices to question the integrity of the new technologies. As one State Elections Director commented, “Before HAVA we very rarely heard the phrase ‘election security.’ Now that’s all anyone talks about every time we start to conduct an election.”
In these debates, critics of the newer technologies often
say—and have their concerns readily repeated in the media—that
there is great public concern over the security of these electronic systems. Is
the general public as technophobic as the critics claimed it is? Would
introduction of new technologies further depress
InfoSENTRY decided to find out how much the American public
trusted—or mistrusted—the major election technologies that are
competing to replace the vast number of punchcard, lever, and paper-based
voting systems in the
ORC completed the interviewing during the period February 6 – 9, 2004 among a national probability sample of 1026 adults comprising 512 men and 514 women, 18 years of age and older, living in private households in the continental United States. The margin of error in this survey is plus or minus three percentage points. In instances in which the total of responses varies from 100% on a particular question, the variation is due to mathematical rounding. Appendix 1 contains a substantial discussion of the survey’s methodology provided to us by ORC.
ORC’s professional cadre of interviewers asked respondents the following question:
Now I am going to read
to you some methods people use to vote in elections for public officials and
ballot issues throughout the
The interviewers then read four different descriptions of voting technology that ORC assisted in devising to avoid esoteric and biased language. One description was of an all-electronic, computerized voting system that is commonly known among elections practitioners through the shorthand terms of Direct Record Electronic (DRE) and “touch-screen” systems. Regardless of the specific vendor’s implementation and procedures, this technology involves going to a polling place and making choices directly on a computer screen--and having the computer count the results.
The second description was of a voting technology known as “in-precinct” or “precinct-count” optical scan technology. Use of this voting technology involves going to a polling place, marking choices on a paper ballot, and having the ballot counted by a computer scanner.
The third description also involved optical scan computing
with a twist in that it involved what is referred to in the
The final description involved using a computer at home, office, overseas, or some other place of the voter’s choice to cast a ballot over the Internet. This technology is by far the newest and least used in public elections of the choices presented to the survey’s respondents.
ORC’s computer-assisted telephone interviewing software randomly rotated the order in which the interviewers read the descriptions to each respondent. This procedure prevented an inadvertent bias arising from a simple, consistent placement of a description before or after another description.
ORC added a question to provide us with one more interesting look at a possible dimension that we suspected might be on the minds of election administrators. That dimension was a determination of the respondents’ political party orientations. Respondents could self-identify themselves as a Republican, Independent leaning toward Republican, Independent, Independent leaning toward Democrat, or Democrat. A person declining identification went into the “other” category.
InfoSENTRY Services, Inc. is solely responsible for the interpretations and analyses in this White Paper.
Overall Opinions About Voting Technology Security
Confidentiality and accuracy are two key pillars of information systems security. They were the focus of the survey’s main question asking respondents to rate their trust in the confidentiality and accuracy of election results.
Finding 1:
Significantly more
The data in Table 1 indicate that 40% of the respondents expressed “very high trust” in the security of DREs and another 28% expressed a general level of trust (that is a “4”) in electronic voting technology. Twenty-nine percent of the respondents gave optical scan (precinct count) the highest trust rating and the next highest rating came from another 30% of the respondents. Respondents gave both optical scan (Vote By Mail) and Internet voting lower trust ratings.
The survey found that both DRE and Optical Scan (precinct count) voting technologies received positive trust ratings for confidentiality and accuracy by majorities of American adults. DREs received higher ratings than paper-based technologies and the other “all electronic” technology, the Internet.
Table 1:
Summary of Public Trust in Voting
Technologies’
Confidentiality and Accuracy
(Note: The “mean
score” is the mathematical average of the responses[1].)
It is interesting to note that the nationwide sample of respondents placed the highest trust levels in the voting technologies that involve “in precinct” voting. Proximity to the vote tabulation process appears to increase the level of trust in the technologies’ ballot security.
Finding 2:
Internet voting, the newest and least used of the voting technologies, is also
least trusted to produce confidential and accurate election results.
Table 2 and the graphic representation in Figure 1 show what political candidates refer to as the “net rating” of the respondents’ attitudes toward the voting systems. We obtain the “net trust scores” by adding all of the positive ratings (4 and 5), adding all of the negative ratings (1 and 2), and subtracting the negative ratings from the positive ratings.
Overall, the adult public nationwide has a significantly higher level of net trust in all- electronic voting systems (DREs) than it does in precinct-count optical scan systems (by an 11 point spread in the net trust score), Internet voting (by a whopping 68 point spread), and Vote-By-Mail (by a 63 point spread).
Table 2:
Net Trust Scores in Voting
Technologies’
Confidentiality and Accuracy
Figure 1:
Detailed
Attitudes Toward Electronic Voting Systems (DREs)
A variety of computer technologies make up what are known as Direct Record Electronic (DRE) voting systems. Some use “touch screens” similar to many bank automated teller machines. Others use push-buttons or dials by which voters indicate their choices on notebook-size computer workstations. Some use standalone voting devices. Others use small networks of cable-connected voting stations. Others use wireless voting connections. However, almost all have in common the condition that the system is electronic, relying at the current time on paper only to produce an “audit tape” of contest results.
Apart from the general ratings, there are interesting variations in the survey’s responses about DRE security revealed by a closer look at the respondents’ demographic characteristics.
Finding 3: Trust
in the confidentiality and accuracy of DRE voting technology is consistently
positive across gender and age groups, except for persons 65+ years old, among
whom the technology still receives positive trust ratings.
Table 3 indicates that men (66% total positive[2]) and women (69% total positive) generally gave consistently positive trust ratings to DREs’ confidentiality and accuracy. Women were slightly more likely to give a “very high trust” rating (43%) than were men, of whom 36% gave the highest rating. On the other end of scale, 11% of the men indicated a “very low trust” in DREs. Only 6% of the women respondents gave the lowest trust rating to DREs.
Table 3:
Trust in DRE Voting
Systems:
By Gender and Age
Group
The results in Table 3 also indicate a very consistent response pattern of trust across most of the age groups. Typically in the age groups, between 39% and 44% of the respondents gave the “very high trust” rating to DREs’ confidentiality and accuracy. However, in the 65+ age group, only 31% gave the highest trust rating to DREs.
At the other end of the rating scale, 25% of the 65+ age group gave a low trust rating (either a 4 or a 5) to DREs for producing confidential and accurate results. This combined “low trust rating” among the 65+ age group was substantially lower than among any other age group.
Supporters of DRE voting systems will quickly point out that
post-election surveys, particularly in
Finding 4: Whites,
Blacks, and Hispanics have overall positive trust levels of DRE voting
technology, although Blacks’ trust is not as strongly held as is the trust
among Whites and Hispanics.
Table 4 indicates that Whites, Blacks, and Hispanics have overall positive trust levels of DRE voting technology’s security. Sixty-eight percent (68%) of Whites, 67% of Blacks, and 60% of Hispanics responded with either a 4 or 5 trust rating to DREs’ security of election results. It is only at the highest level of “very high trust” that Blacks’ trust (30%) in DREs is lower than the trust expressed by Whites (41%) and Hispanics (39%).
Table 4:
Trust in DRE Voting Systems:
Among White, Black, and Hispanic
Adults
Finding 5: Overall
trust in DRE voting systems is positive in all regions with highest ratings in
the South and West.
Table 5 indicates relatively little regional variation in the public trust levels for DREs. The combined trust (adding results for ratings 4 and 5) are 67% in the Northeast, 64% in the North Central Region, 68% in the Southern Region, and 71% in the Western Region.
Table 5:
Trust in DRE Voting Systems:
By Region of the Country
Detailed Attitudes Toward Optical Scan (Precinct Count) Voting Systems
Another major voting technology is the “in-precinct” or “precinct count” optical scan system. The voter fills in small ovals or connects ends of arrows on a paper ballot. Before leaving the polling place the voter feeds the ballot into a computerized optical scan device that “reads” the ballot choices. Upon closing the polls, election officials relay the results by carrying data packs or transmitting results over modem-connected phone lines.
Finding 6: Trust in optical scan (precinct count)
voting systems is consistently positive among men and women and across most age
groups.
Table 6 indicates that almost identical percentages of men (58%) and women (60%) express trust in the security of optical scan (precinct count) voting results.
The table also indicates that majorities in each age group also have consistently positive views of optical scan’s ability to deliver confidential and accurate results.
Table 6:
Trust in Optical Scan (Precinct
Count) Voting Systems:
By Gender and Age Groups
Finding 7: White adults and self-identified
Hispanic adults express higher trust levels in optical scan (precinct count)
voting devices than do Black adults.
Table 7 indicates that majorities of White and Hispanic adults trust precinct-optical scan systems to produce secure election results. Thirty percent (30%) of White respondents and 29% of Hispanic respondents express “very high trust” in optical scan systems. For both groups, majorities (61% of Whites and 54% of Hispanics) rated their trust of optical scan voting systems with positive ratings, either a 4 or a 5 on the 5-point scale.
However, African-Americans have less trust in the paper-based optical scan technology. Only 46% of the Black respondents indicated trust in optical scan technology’s confidentiality and accuracy. Among this group of adults, 27% indicated either low (4) or very low (5) trust in optical scan—even with its built-in paper trail and its characteristic of being an in-precinct activity.
A later analysis of trust in Vote By Mail voting will show a similar lack of African-American trust in that form of paper balloting.
Table 7:
Trust in Optical Scan (Precinct
Count) Voting Systems:
Among White, Black, and Hispanic
Adults
Finding 8:
Respondents in Western states expressed substantially higher trust in
precinct-count optical scan voting systems than did respondents in other
regions.
Table 8 demonstrates that 39% of the adults in the West expressed very high trust (5) in in-precinct optical scan voting systems. Another 27% of Western adults indicated general trust (4) for these systems, for a total of 66% trust in optical scan (precinct count) systems. For a point of comparison, reference back to Table 5 shows that 71% of Western respondents gave a positive trust rating (either 4 or 5) to DREs’ confidentiality and accuracy.
Table 8:
Trust in Optical Scan (Precinct
Count) Voting Systems:
By Region of the Country
However,
the Northeast region respondents expressed the lowest level of trust for
optical scan in all the regions, with only 51% responding with a 4 or 5 on the
5-point scale. It is possible that this lower level of trust in the Northeast
correlates with a generally lower level of use of precinct-count optical scan
voting systems in that region. Respondents in the Southern states (at 60%
positive trust) and North Central states (at 57% positive trust) produced trust
scores for in-precinct optical scan voting devices that were in between the
overall trust ratings in the Northeast and the West.
Detailed Attitudes Toward Optical Scan (Vote By Mail) Voting Systems
The other use of optical scan technology is in the so-called “central count” systems. Election administrators use these systems heavily in areas where “Vote By Mail” or very heavy absentee voting is common. Voters receive their ballots in the mail, mark the same kinds of ovals or arrows described in the previous section, and return the ballots either in the mail or in person to the local election office.
This system is prevalent in
Finding 9:
Respondents in both genders and most age groups have consistent, net negative
views of optical scan (Vote By Mail) technology.
The survey found that only 33% of the males and 30% of the females in the sample gave positive trust ratings (either a 4 or 5 on the 5-point scale) to the Vote By Mail optical scan technology’s confidentiality and accuracy. However, 42% of the men and 40% of the women gave low trust ratings (either a 1 or a 2 on the scale) to the voting method.
Table 9:
Trust in Optical Scan (Vote By
Mail) Voting Systems:
By Gender and Age Groups
While most of the age groups gave slightly negative trust ratings to the Vote By Mail (central-count optical scan) voting method, the 65+ age group provided a nearly balanced response. In this group 39% gave an overall positive trust rating (4 or 5) while 40% gave and overall negative rating (1 or 2).
Finding 10:
African-Americans have a more negative view of Vote By Mail than do Hispanics
or Whites.
Table 10 indicates that while no racial or ethnic group identified in the study had overall positive trust ratings for Vote By Mail, African-Americans had a majority (55%) of negative ratings for the Vote By Mail optical scan technology. That number compared with 40% of Hispanics and 39% of Whites who expressed similar negative ratings.
These findings bolster earlier conclusions in this White Paper regarding African-Americans’ concerns about paper-based voting systems. (Please see Finding 6 and Table 7.)
Table 10:
Trust in Optical Scan (Vote By
Mail) Voting Systems:
Among White, Black, and Hispanic
Adults
Finding 11: Adults
in Western States gave higher ratings to Vote By Mail and central count optical
scan technology than did respondents in any other region of the country.
Table 11 presents survey results that will surprise very few election officials in the West. Approximately half (51%) of the Western states’ respondents in the sample indicated a positive trust level in the ability of Vote by Mail and optical scan technology to produce secure election results. This level compared with 20% in the Northeastern region, 30% in the North Central region, and 26% in the South.
There are two important points to keep in mind with respect
to this finding. First, as noted earlier, Vote By Mail is largely a
Western-states phenomenon at this time. It is used exclusively in
The second point is that approximately one in eight voters
in the
California, not
Table 11:
Trust in Optical Scan (Vote By
Mail) Voting Systems:
By Region of the Country
Detailed Attitudes Toward Internet Voting Systems
Finding 12: Women are
slightly less negative than men toward Internet voting systems, although
majorities of both genders fail to place trust in the technology’s
ability to deliver confidential and accurate election results.
Table 12 shows that only 31% of men trust the Internet as a secure voting technology. A similar number of women (32%) in the national survey expressed trust in Internet voting systems. However, while 51% of the males expressed low trust in Internet voting, a smaller number of women (43%) gave the “low” or “very low” trust response on the survey.
Table 12:
Trust in Internet Voting Systems:
By Gender and Age Groups
Finding 13: There
is an inverse relationship between age and trust in the Internet’s
ability to deliver accurate and confidential election results.
For many, the data in Table 12 will come as no surprise.
Among the 65+ age category, the 46%-level expressing “very low trust” in Internet voting was over three times the 13% expressing “very low trust” in the other all-electronic voting technology, DREs. (Please see Table 3 for data on age groups’ attitudes toward DRE voting systems.)
On the other end of the age line, almost half (47%) of the 18-24 year olds expressed trust (either 4 or 5 on the rating scale) in the Internet’s ability to deliver accurate and confidential election results. It is also useful to keep in mind that the World Wide Web as an alternative technology for voting is slightly more than 10 years old. DREs have been available for slightly longer. Paper alternatives have been voting options for several centuries.
Note that according to an article in the February 9, 2004 issue of The Detroit News, 46,000 Michigan Democrats cast votes in that state’s presidential primary using the Internet. Reports indicate that the primary passed without a security incident more serious than a few voters who were unable to establish a connection to the voting site.
Finding 14: Hispanics gave substantially
higher ratings to Internet voting security than did either White or Black
respondents.
While respondents in all three racial and ethnic categories expressed concerns about Internet voting, Hispanics were the only one of these categories of respondents with a slightly net positive view of Internet voting’s confidentiality and accuracy. A total of 41% of Hispanics responded with a positive level of trust (either a 4 or 5 on the 5-point scale), while 38% of Hispanic respondents provided negative trust responses. This “net rating” of a +3 is in substantial contradistinction of a -16 net trust rating for Whites and -21 net trust rating for Blacks on this voting technology.
Table 13:
Trust in Internet
Voting Systems:
Among White, Black,
and Hispanic Adults
Finding 15: The
Western region is least negative to Internet voting security, while the South
is most negative to this electronic network voting technology.
Almost four in ten (38%) of the
survey’s Western Region respondents gave positive trust rating (either 4
or 5 on the 5-point scale) to Internet voting’s security, while a
slightly higher number (43%) gave a negative trust rating of either a 1 or 2.
While still a net negative rating of -5 points, this trust level was far better
than the net trust ratings in the other regions.
In particular, Table 14 shows that a slight majority (51%) of Southerners gave negative trust ratings to Internet voting. Only 28% gave positive ratings, resulting in a net negative rating of -23 points from the South. This was the largest negative spread for Internet voting.
Why is the West so much less negative on Internet voting that other
regions? Two possible reasons exist. First, it is an area with a significant
high-tech industrial base, with computer technology on the
the West has a significant Hispanic population as a percentage of its total population. As we have already seen, Hispanics constituted an ethnic group with a significantly higher trust rating in Internet voting than did other racial/ethnic groups in the study.
Table 14:
Trust in Internet Voting Systems:
By Region of the Country
The Effect of Internet Use of Voting Technology Trust
Does the use of the Internet at work or home appear to influence attitudes toward voting technology security? To probe this possibility, we cross-tabulated a question on Internet use with the net trust ratings the four voting technologies on our survey.
Finding 16:
Internet users are significantly more likely than non-Internet users to trust
the confidentiality and accuracy of DRE voting systems.
Table 15 and Figure 2 show that Internet users are significantly more likely than non-Internet users to trust DRE voting security. While non-Internet users have a mean trust score a slightly positive 3.48, Internet users responded with a mean trust score of 3.92.
However, while Internet users trust the voting security of this technology more than do non-Internet users by a mean trust score range of 2.72 to 2.30, Internet users still give an overall negative trust score to the technology for voting.
Table 15:
“Mean Trust Scores”
in Voting Systems:
By Status of Internet Use
Note from data in the table and Figure 2 that Internet users generally are more likely to trust the accuracy and confidentiality of all of the voting technologies in the survey. The closest gap between Internet users and non-Internet users was in the mean trust scores of Vote By Mail, which stood at 2.85 and 2.80 respectively.
Figure 2:
Trust in Voting Systems: Partisan Views
The general consensus among political pundits is that we are a sharply divided nation, particularly between Republicans and Democrats—and Independents who constitute the swing vote on many issues and candidates.
Does this bipolarization extend to attitudes toward voting systems security?
Finding 17:
Republicans and Independents who lean toward Republican candidates have a
slightly higher level of trust in the security of both electronic and
paper-based voting technologies than do Democrats and Independents who lean
toward Democrats.
Table 16 and its graphical representation in Figure 3 show that Republicans and Independents-Leaning-To-Republican have slightly higher “mean trust scores” in the security of all voting technologies than do their counterparts on the Democratic side.
Largest distinction is in attitudes toward in-precinct optical scan systems. For that technology, the spread between the mean trust scores of Republicans and Democrats is 3.87 to 3.55, respectively. That is a difference of 32 points. On the other hand, the spread in mean trust scores between Republicans and Democrats for DREs is only 17 points (3.99 to 3.82, respectively).
Instead of a high level of bipolarization on the issue of voting system security in the nationwide sample, we found a relatively small difference in degree in attitudes of Democrats and Republicans.
Table 16:
Trust in Voting Technology:
By Political Party Orientation
Figure 3:
Trust in Voting Technology:
By Political Party Orientation
Conclusion
We draw two basic conclusions from all the findings and interpretations of this survey’s numbers.
First, the trust levels are not as high as we, and we believe all election officials, would like to see them. We can only imagine that election officials and voting system vendors share that wish. Given the negative fallout attached to the Presidential election in 2000, the highly-publicized failures of vendors to perform in various elections around the country, and the constant drum-beat of conspiracy theorists to exacerbate the negative attention and the vendors’ failures, we are in some respects pleasantly surprised that the trust levels are not lower. Still, election officials, vendors, and the critics of specific voting technologies have a responsibility to advocate positions and take steps that will improve trust in the election system.
Second, a widely reported concern over all-electronic, in-precinct (DRE) voting, which might call for radically altering the design and function of these systems to provide “paper backup,” does not exist. That is not to say the people who have this concern do not express it passionately. Voting technology critics and conspiracy theorists have succeeded in getting the media to characterize fears of electronic voting systems (DREs) as “increasing public concern.”
However, this survey presents clear public opinion findings from a nationwide study that a “silent majority” against computerized voting systems does not exist at this time. A substantial majority of American adults trusts in the confidentiality and accuracy of DREs. A smaller majority also trusts in the confidentiality and accuracy of in-precinct optical scan voting systems. There is a lack of nationwide trust in both Vote By Mail and Internet voting. We suggest this lack of trust rests on a lack of familiarity with Vote By Mail and a general mistrust of the Internet. That suggestion is subject to empirical support or rejection by future studies.
This survey does not attempt to address the technical aspects or merits of any of the voting technologies included in the survey. It focuses solely on attitudes toward the technologies’ ability to deliver confidential and accurate election results.
While the technical arguments continue, a very substantial burden of proof rests on the shoulders of voting system critics to prove that their proposed solutions to real or hypothetical voting system “security problems” will not actually decrease public trust in voting technology across the board. That burden will be particularly difficult to carry if their solutions make those systems more complex, difficult to understand, and error-prone.
We have a personal note to add. We did not get to take this survey. Indeed, we would have been astounded if ORC called us! However, we would have given a “5” response to three of the technologies and a “4” to Internet voting. The reason is simple. We trust our county’s Board of Elections and its Director of Elections to implement any voting system in a responsible and secure manner. They have a history of fairness and accuracy. For us, that history is a better guide than histrionics.
We have the pleasure of working with elections customers who use DREs, Vote By Mail optical scan, and in-precinct optical scan. We have conducted independent, third-party security assessments, quality assurance reviews, disaster recovery tests, and user acceptance tests on all of the technologies. Sometimes the vendors failed those reviews, tests, and assessments initially. Most of the time, they made the necessary program modifications, took the required mitigation steps, and implemented the systems properly. The vendors took those steps because the Election Boards and Administrators refused to purchase and install the systems if the vendors did not meet Federal standards, state standards, and local requirements. That continued level of integrity and the strong involvement of the political parties in keeping watchful, respectful eyes on the process will make our next year’s survey on public attitudes toward voting technology all the more interesting.
Appendix 1: Survey Technical Details
This report presents the findings of a telephone survey conducted among a national probability sample of 1026 adults comprising 512 men and 514 women 18 years of age and older, living in private households in the continental United States.
Interviewing for this CARAVAN® Survey was completed
during the period February 6 - 9, 2004.
All data collection efforts took place at Opinion Research
Corporation’s Central Telephone Facility in
All CARAVAN interviews are conducted using Opinion Research Corporation's computer assisted telephone interviewing (CATI) system. The system is state-of-the-art and offers several distinct advantages such as: full-screen control which allows multi-question screens, fully-programmable help and objection screens to aid interviewing, an extremely flexible telephone number management system and powerful data checking facilities. CATI ensures that interviews are conducted in the most efficient manner and allows interviewers easy response recording. This interviewing method also allows for the most accurate form of data entry by guiding the interviewer through the programmed question flow and by providing on-screen interviewer instructions.
The most advanced probability sampling techniques are employed in the selection of households for telephone interviewing. Opinion Research Corporation utilizes an unrestricted random sampling procedure that controls the amount of serial bias found in systematic sampling to generate its random-digit-dial sample. The sample is fully replicated and stratified by region. Only one interview is conducted per household. All sample numbers selected are subject to up to four attempts to complete an interview.
Completed interviews are weighted by four variables: age, sex, geographic region, and race, to ensure reliable and accurate representation of the total population, 18 years of age and older. The raw data are weighted by a custom designed program which automatically develops a weighting factor for each respondent. Each respondent is assigned a single weight derived from the relationship between the actual proportion of the population with its specific combination of age, sex, geographic characteristics and race and the proportion in our CARAVAN sample that week. Tabular results show both weighted and unweighted bases.
The use of replicable sampling, standardized interviewing procedures and representative weighting provides that all CARAVAN studies are parallel to one another. Thus, CARAVAN usage is appropriate both for point-in-time analysis as well as tracking and trend comparisons.
Included in the Technical Information which follows are tables of sampling tolerances of survey results, and a copy of the question series as it appeared in the survey questionnaire.
As required by the Code of Standards of the Council of American Survey Research Organizations, we will maintain the anonymity of our respondents. No information will be released that in any way will reveal the identity of a respondent. Our authorization is required for any publication of the research findings or their implications.
Opinion Research Corporation's CARAVAN is a shared cost data collection vehicle. Opinion Research Corporation has exercised its best efforts in the preparation of this information. In any event, Opinion Research Corporation assumes no responsibility for any use which is made of this information or any decisions based upon it.
CARAVAN Telephone Sampling Methodology
Opinion Research Corporation's national probability telephone sample is an efficient form of random-digit-dialing. The sample is designed to be a simple random sample of telephone households. Unlike published directories, Opinion Research Corporation's national probability telephone sample includes both unlisted numbers and numbers issued after publication of the directories. The following procedure was used to create the sample:
• Opinion Research Corporation has an annual license for GENESYS, a custom RDD sample generation system developed by Marketing Systems Groups.
• The methodology for generating random digit dialing (RDD) telephone samples in the GENESYS system provides for a single stage, EPSEM (Equal Probability of Selection Method) sample of residential telephone numbers. It is updated twice a year.
• When a national probability sample is needed, a random selection is made from approximately 40,000 exchanges in two million working banks.
• Each telephone number is transferred to a separate call record. The record shows the computer generated telephone number to be called, as well as the county, state, MSA (if applicable), band and time zone into which the telephone number falls. Our computerized interviewing system (CATI) uses this information to keep track of regional quotas. The CATI interviewing program also keeps track of the disposition categories for each call attempt.
Reliability Of Survey Percentages
Results of any sample are subject to sampling variation. The magnitude of the variation is measurable and is affected by the number of interviews and the level of the percentages expressing the results.
The table below shows the possible sample variation that applies to percentage results reported from Opinion Research Corporation's CARAVAN sample. The chances are 95 in 100 that a CARAVAN survey result does not vary, plus or minus, by more than the indicated number of percentage points from the result that would be obtained if interviews had been conducted with all persons in the universe represented by the sample.
Size of Sample on Approximate Sampling Tolerances
Applicable
Which Survey Results to
Percentages At or Near These Levels
Are Based
10%
or 90% 20%
or 80% 30%
or 70% 40%
or 60% 50%
2,000 interviews 1% 2% 2% 2% 2%
1,000 interviews 2% 2% 3% 3% 3%
500 interviews 3% 4% 4% 4% 4%
250 interviews 4% 5% 6% 6% 6%
100 interviews 6% 8% 9% 10% 10%
Additional Sampling Tolerances
for Samples of 1,000 Interviews
9% or 91% 8%
or 92% 7%
or 93% 6%
or 94% 5%
or 95%
2% 2% 2% 1% 1%
4% or 96% 3%
or 97% 2%
or 98% 1%
or 99%
1% 1% 1% .2%
Sampling Tolerances When Comparing Two Samples
Tolerances are also involved in the comparison of results from independent parts of any one Opinion Research Corporation's CARAVAN sample and in the comparison of results between two independent CARAVAN samples. A difference, in other words, must be of at least a certain number of percentage points to be considered statistically significant. The table below is a guide to the sampling tolerances in percentage points applicable to such comparisons, based on a 95% confidence level.
Differences
Required for Significance At
Size of Samples or
Near These Percentage Levels
Compared
10% or 90% 20%
or 80% 30%
or 70% 40%
or 60% 50%
1,000 and 1,000 3% 4% 4% 4% 4%
1,000 and 500 3% 4% 5% 5% 5%
1,000 and 250 4% 6% 6% 7% 7%
1,000 and 100 6% 8% 9% 10% 10%
500 and 500 4% 5% 6% 6% 6%
500 and 250 5% 6% 7% 7% 8%
500 and 100 6% 9% 10% 11% 11%
250 and 250 5% 7% 8% 9% 9%
250 and 100 7% 9% 11% 11% 12%
100 and 100 8% 11% 13% 14% 14%
Definition Of Classification Terms
The following definitions are provided for some of the standard demographics by which the results are tabulated. Other demographics are self explanatory.
Geographic Region
The continental states are contained in four geographic regions as follows:
North East
New England:
Middle Atlantic:
North Central
East North Central:
West North Central:
South
South Atlantic: Delaware, Maryland, District of Columbia, Virginia, West Virginia, North Carolina, South Carolina, Georgia, Florida
East South Central:
West South Central:
West
Mountain:
Pacific:
Significance Testing
When results from sub-groups of a CARAVAN sample appear in the detailed tabulations, an indicator of statistically significant differences is added to the tables run on our standard demographic banners. The test is performed on percentages as well as mean values. Each sub-sample is assigned a letter. When the percentage of one sub-sample is significantly different from the percentage of another sub-sample, the letter representing one of the two samples appears next to the percentage (or mean) of the other sample.
For instance the percentage of males answering yes to a particular question may be compared to the percentage of females answering yes to the same question. In the example on the next page, the male sample is assigned the letter B, and the female sample is assigned the letter C. Here, respondents were asked whether a certain business practice is acceptable. 67% of women said that it was -- a proportion significantly greater than the 57% of males who believe that the practice is acceptable. To indicate that women are significantly more likely to find the practice acceptable than are men, the letter B -- the letter assigned to the male sub-sample -- appears next to the “67%” in the female column. Similarly, the 37% of men that find the practice unacceptable is significantly greater than the 29% of women who do so and, therefore, the letter C -- the letter assigned to the female sub-sample -- appears next to the “37%” in the male column.
Significance testing is done to the 95% confidence level.