Chapter 2 - Quantitative Research

At the end of Phase I, EPA, in consultation with CLI Partner and Task Force members, concurred with the recommendation that quantitative research in Phase II would be valuable to better understand consumers' preference for, comprehension of, and satisfaction with current product labels. A quantitative approach was favored because it was necessary to obtain statistically sound data to support the findings from the earlier qualitative research. Furthermore, unlike qualitative research data, quantitative research data are representative of the study population and projectable to the entire population. Quantitative research was also used to determine the prevalence of particular opinions on a given issue expressed in the qualitative interviews. Additionally, quantitative research was appropriate for measuring both attitudes and behavior of consumers to current and new product labels. Demonstrating their support for this concept, the CLI Partners volunteered to fund and direct this research, which they felt would be of use even beyond the CLI. Quantitative research also provides a baseline that can be surveyed periodically to determine changes in attitude and behavior.

The Phase II quantitative consumer research was designed to assess consumer comprehension, attitudes, behavior and satisfaction with labeling and to evaluate labeling alternatives (for both registered and non-registered products) in the outdoor pesticide, indoor insecticide, and hard surface cleaner categories. The quantitative survey was organized along the six learning objectives identified by the CLI Partner and Task Force members at the beginning of Phase II. These learning objectives are as follows:

Quantitative Learning Objectives

Determine the current situation relative to consumers' satisfaction with the format and content of existing labels;

Determine consumers' hierarchy of importance of basic label information;

Determine where on the label consumers expect to find particular information, such as First Aid and ingredients;

Determine consumers' current comprehension of label language;

Determine whether or not a preference exists for non-FIFRA over FIFRA labels (for household cleaner category only); and

Determine consumers' reaction to standardized safe use, environmental, health and safety information.

Each learning objective was intended to generate research findings that would enable the EPA and CLI Stakeholders to take immediate and short-term steps toward label improvements. Some changes, such as revised guidance and regulations, are almost entirely under the purview of the EPA. Other changes are entirely within the purview of the product marketers but may be subject to EPA label approval. Others, such as consumer education, involve many Stakeholders and would be implemented over a longer time period. The results of the quantitative research were expected to lead to certain actionable steps, such as:

quantify key learnings from the qualitative research in Phase I of CLI;
collect data that will serve as input into additional quantitative research, such as consumer evaluation of potential new label formats;
benchmark current consumer practices and preferences, so that changes in behavior/preference (based on label changes) can be assessed;
provide information that will allow the EPA and its Partners to consider policy implications and to take some immediate action steps;
guide the Consumer Education Subgroups's efforts;
guide the Storage and Disposal Subgroup in making recommendations; and
provide information for potential changes to label formats.

Strategy for the Quantitative Research

The design and implementation plan of the quantitative research was developed by the Research Core Group, consisting of EPA personnel, industry and trade association Partners, people from other federal and state agencies, and other interested CLI Stakeholders. The Core Group began, by addressing the learning objectives identified at the beginning of Phase II by CLI Partner and Task Force members, to develop the quantitative screening and survey questionnaires. Several of the members of the research group were market researchers in their own organizations and, therefore, had extensive experience with survey design. The quantitative research was voluntarily undertaken and funded by industry and trade association Partners of CLI including: AgrEvo Environmental Health; American Cyanamid (American Home Products); Bayer Corporation; the Chemical Specialties Manufacturers Association (CSMA); Dow AgroSciences; FMC; Reckitt & Colman; S.C. Johnson & Son, Inc.; The Procter and Gamble Company; The Clorox Company; Purcell Industries, Inc.; Riverdale Chemical Co.; SC Johnson; The Andersons, Inc.; The Scotts Co.; Solaris (Monsanto); United Industries Corporation; and the RISE (Responsible Industry for a Sound Environment). This group of companies hired an independent survey research firm, National Family Opinion (NFO) Research, Inc. to implement the study.

During Phase II, the Core Group met on a weekly basis via telephone conference calls, and occasionally in ad hoc face-to-face meetings, to discuss the development of the survey instruments, the implementation of the survey itself, and interpretation of the data once the results of the survey were available. In July 1998, a smaller subgroup of the Core Group met in Washington, D.C., to discuss the survey data in detail and establish some of the preliminary findings from the survey results. This smaller group consisted of EPA Task Force members, and market researchers from Amway Corporation; Bayer Corporation; S.C. Johnson and Son, Inc.; and the Procter and Gamble Company. In August, the subgroup finalized the preliminary findings and prepared data tables to illustrate these conclusions. In September 1998, the subgroup presented these results at the Partner and Task Force meeting in Alexandria, VA.

Quantitative Study Design

The quantitative study consisted of three parts: an initial screening (to identify potential study participants), followed by telephone interviews and a self-administered mail questionnaire among those selected to participate in the main portion of the quantitative study.

the three parts of quantitative study design: initial screening identifies potential study participants; telephone interviews conducted by NFO; participants complete self-administered mail questionaire

Screening to Identify Product Category Users for Use in the Study

In the first part of the quantitative phase of the study, a postcard with a very short screening questionnaire (screener) was mailed to members of the NFO Panel.⁽¹⁾

The screener contained questions to identify consumers eligible for participation in the main portion of the quantitative study (and to eliminate those consumers not eligible for participation). Screener questions asked respondents the following:

Whether any household member used a household cleaner in the past 12 months. For those who indicated usage of a household cleaner, the age and gender of the household member who is the primary user of household cleaners;

Whether any household member used an indoor insecticide in the past 12 months. For those who indicated usage of an indoor insecticide, the age and gender of the household member who is the primary user of indoor insecticides;

Whether any household member used an outdoor pesticide in the past 12 months. For those who indicated usage of an outdoor pesticide, the age and gender of the household member who is the primary user of outdoor pesticides; and

Whether the respondent had gone to the store to purchase each of the three types of products, but did not because of information contained on the label of the product.

In March 1998, the screening postcard was mailed out to a total of 10,000 NFO consumer panel households. The distribution of recipients who received this postcard was balanced to be representative of the U.S. population as a whole on age and gender of the head of household, geographic region, household size, market size, and household income. An additional 2,250 postcards were mailed out to households from three low incidence groups of interest (minority, lower formal educational level, lower income) on NFO's panel of 550,000, to ensure adequate representation in the final survey results. These low incidence groups were also balanced to be representative of their counterparts in the overall U.S. population. In April 1998, returns were closed out and the returns tabulated. A total of 8,447 households returned the postcard (69% of the number sent out). These results were then used to determine which households and which individuals to include in the main portion of the CLI quantitative study (i.e., phone and mail questionnaires) for appropriate demographic representation. Appendix 2-1 contains the screening questionnaire.

Non-User Results

As stated above, non-users (in the past 12 months) were excluded from the main portion of the quantitative study. It must be noted that among the group of consumers who said on the screener that they had not used the specific products in the past 12 months (and were thus ineligible for inclusion in the main portion of the study), a small number also indicated on the screener that they went to the store to buy such a product, but did not purchase it because of information on the package (6% of those who did not purchase household cleaners, 7% of those who did not purchase indoor insecticides, and 5% of those who did not purchase outdoor pesticides). The information on the package cited as the reason consumers did not buy the product was not specified. It cannot be determined, therefore, what biasing impact, if any, was created by excluding these consumers from the study. Based on the low number of consumers who were excluded (between 5% and 7% of non-users for each category), it is unlikely that any such biases would alter the survey findings in any meaningful way.

Sample for the Telephone Interviews and Mail Questionnaire

For each product category, a group was formed of participants who indicated that they had used that type of product in the past 12 months. Additionally, supplemental samples of low-income households (i.e., those making less than $10,000 per year), less educated heads of household (i.e., those with less than high school education), and minorities were drawn for all three categories, and a supplemental sample of fogger users was also drawn for the indoor insecticides category. These additional samples were needed because the overall incidence of these groups in the U.S. population is so low that there would not be enough members of these groups in the nationally representative sample to allow for meaningful quantitative analysis of these particular groups.

These supplemental groups (i.e., supplemental samples) were included only for analyses that looked specifically at the group for which the supplemental sample was pulled. For example, the respondents who were part of the supplemental group for low-income households were included only in the separate analysis of consumers from low-income households. Excluding these special supplemental groups of respondents from other groups (e.g., the nationally representative sample) prevented the creation of an unnatural skew toward over-representing consumers from those groups for which a supplemental sample was pulled. It is important to note that, due to random selection, there are still members among the nationally representative sample who fall into the demographic groups for which supplemental samples were pulled.

The samples for each product category were balanced to be representative of the portion of the U.S. population that uses that particular category (i.e., household cleaners, indoor insecticides, outdoor pesticides). The samples were balanced on the following demographic variables:

age of user,
gender of user,
household income,
household size,
market size,
and geographic region.

The self-administered mail questionnaires were mailed out to a total of 6,438 households, broken down as follows:

Nationally representative sample of category users:
All categories	1,775 per category

Supplemental Samples
	Low-education heads of household	Low-income households	Minority households	Fogger users
Indoor insecticides	102	122	77	144
Household cleaners	102	124	90	N/A
Outdoor pesticides	108	132	112	N/A

When survey returns were closed in early June 1998, a total of 3,234 consumers (50% of the total sent out) completed both the telephone and mail portions of the study, with approximately 850 to 900 being nationally representative users of each of the three product categories. As appropriate, the remainder of returns were used to supplement the various low incidence groups.

Telephone and Mail Questionnaires

The main portion of the CLI quantitative study was composed of 1) a telephone interview, followed by 2) a self-administered 8-page mail questionnaire. The telephone interview was used to collect information that would have been difficult to collect without direct interaction with an interviewer (e.g., having the respondent state where certain label sections were located). Telephone interviewers also allowed for clarifications and follow-up probing of responses regarding comprehension. Questions on the phone survey were rotated so that any order bias or "question fatigue" would be avoided. The mail questionnaire was used to collect a large amount of detailed information that could not be collected over the telephone due to time (i.e., length of interview) considerations. The telephone interview also asked consumers for "top of mind" responses to mimic actual consumer behavior (e.g., exercise of choices and capabilities) when they encounter the label both in the store and at home.

There were three different versions of the survey: one for household cleaning products, one for indoor insecticides, and one for outdoor pesticides, with the bulk of questions being identical on all three. In April 1998, participants were sent one version of the questionnaire booklet, along with a letter of instruction and a "mock" label (for use in both the phone and mail portions of the study). The mock label was representative of a typical product label for the product category for which respondents were selected. Participants were instructed to await a telephone call before completing the self-administered mail questionnaire. After allowing time for mail delivery, respondents were contacted by phone in early May and asked to complete a 10-minute telephone interview (average time), with responses collected using a computerized telephone questionnaire. After completion of the telephone survey, respondents were then instructed to complete the 8-page mail questionnaire and return it to NFO Research. After one month for completion and return of the self-administered mail questionnaire, returns were closed in early June 1998, and all results from the telephone and mail surveys were then tabulated. Only results from those completing both the telephone and the mail portions of the survey were included in the final results.

Procedure for telephone and mail questionaires: April 1998, respondents receive questionaire booklet; early May, respondents complete telephone interview; respondents complete questionaire and return it to NFO research; June 1998, results from telephone interviews and mail surveys tabulated.

Survey Questionnaires and Learning Objectives

The telephone and mail survey instruments were designed by the Core Group (quantitative research group) to address the learning objectives outlined at the beginning of this chapter. The learning objectives, questions from each questionnaire relating to that learning objective, and the potential action steps emerging from these questions are provided in Table 2-1.

In addition to the learning objectives, the Core Group developed the survey instruments to investigate consumer attitudes, behaviors, and understanding related to specific areas and issues, including:

Consumer Education -- What other sources of information, besides the product label, do consumers turn to for information about the product?
Product Ingredients -- Do consumers understand the ingredient listing on products and know how to use this information?
Signal Words -- Do consumers understand the signal word hierarchy for CAUTION, WARNING, and DANGER?
Storage and Disposal -- What are consumers' current storage and disposal practices?
Precautionary Statements -- What are consumers' understanding and use of precautionary statements?

Telephone Interview Outline

The telephone interview questionnaire used "mock" labels to ask questions related to consumers' comprehension of and ease of finding information on the labels. More specifically, the telephone questionnaire tested respondents' ability to locate key sections of the label, the accuracy with which respondents were able to locate these sections, and their opinions on the ease of finding these sections. Respondents also were asked what they thought certain language on the label meant, including specific key words and phrases. Finally, the telephone survey asked several demographic questions. (See Appendix 2-2 for copies of the phone questionnaires, and Appendix 2-3 for the mock labels.) Each interview was conducted by trained interviewers from NFO Research, Inc., and lasted approximately 10-12 minutes. At the conclusion of the telephone interview, the interviewer instructed the respondent to complete the written questionnaire in his or her own time and mail it back to NFO Research, Inc., once completed.

**Table 2-1: Learning Objectives, Survey Questions, and Potential Action Steps**
Learning Objective	Questions Relevant to the Learning Objectives Addressed the Following:	Potential Action Steps
1) Determine current satisfaction with the format and content of existing labels	Telephone: ease of locating key label sections Mail: overall satisfaction with the current label likes and dislikes of label sections	If current labels are not meeting consumers' needs, provide general input on which sections need further revisions. Level of consumer dissatisfaction indicates strength of motivation for change, thus determining focus and degree of difficulty for education effort.
2) Determine consumers' hierarchy of importance of basic label information 3) Determine where on the label consumers expect to find label information	Telephone: ease of locating key label sections Mail: where and how often consumers read sections of labels information on labels that are the most and least important where consumers expect to find information on labels, and which information they want to find most easily where consumers expect to find recycling icons	Make format recommendations, such as organizing information when needed in the store, before use, or in case of emergency.
4) Assess consumers' comprehension of current label language	Telephone: comprehension of language by label section Mail: meaning of the recycling icons likes and dislikes about label sections	Identify terminology that consumers find difficult to understand. Recommend additional qualitative work with consumers to understand what terminology should be used, as appropriate. Recommend word changes (limited) .
5) Determine preference of FIFRA versus non-FIFRA labels (for household cleaner category only)	Mail: like and dislikes about label sections consumers' preference for FIFRA and non-FIFRA labels paired preference statements	Quantify whether non-FIFRA label is preferred to FIFRA language. Make word changes where possible. Make format recommendations, such as organizing information when needed in the store, before use, or in case of an emergency .
6) Solicit consumers' reactions to standardized information on safe use, environmental, and health information	Mail: most and least important information to consumers where consumers expect to find information on a label, and which information they want to find most easily where and how often consumers read sections of the label	Provide direction on the types of information that could be standardized. Make format (location) recommendations.

Mail Questionnaire Outline

The mail questionnaires (see Appendix 2-4) were designed to address the following specific questions:

respondents' overall satisfaction with current labels;

when (i.e., in the store or right before use) and how often respondents read label sections;

if they do not read the label, why not;

most and least important information to respondents;

where respondents expect to find information on a label, and which information they want to find most easily;

respondent likes and dislikes about product label sections;

other sources (besides the label) for product information;

meaning of recycling icons, including what actions respondents think the icons are asking them to take, and where they expect to find these icons on the product packaging;

respondent preference for FIFRA versus non-FIFRA labels (for household cleaning product category only);⁽²⁾
respondent preference for FIFRA language and alternate wording;
respondent attitude toward reading product labels; and
respondent habits and practices, such as: products used; accident experience; current storage, disposal, and recycling practices; and the incidence of product category use and non-purchase due to confusion about the label.

Quantitative Research Data

National Family Opinion Research completed collection of the survey responses and data tabulation during the months of June and early July⁽³⁾. In the final count, the total number of responses received for the mail and the telephone surveys were as follows:

Household Cleaners -- 894 completes;
Outdoor Pesticides -- 846 completes; and
Indoor Pesticides -- 889 completes.

Statistical Testing of Data

When comparing different groups of data quantitatively, statistical tests are needed to help determine which data are meaningful and which are not. A two-tailed t-test, which compares the percentages or means of interest and the sample sizes, was used to determine whether differences existing among groups are significant on a statistical level.

This type of statistical testing is done based on the level of significance desired. Data are most frequently tested for significance at levels between 80% and 95%. The higher the level of statistical testing performed, the more likely it is that data differences detected in the study reliably reflect differences in the "real world." If a significant difference between two data points at the 95% confidence interval is found to exist, this means that the same study, if conducted 100 times, would show a significant difference reflected in its data at least 95 of those times. For the CLI study, data were tested at the 95% confidence level. In the raw data tables, significance was routinely tested. For each question asked, the mean, standard deviation, and standard error are also shown for each type of respondent.

Breakdown of CLI Data

The Core Group determined that it would be important to investigate whether significant differences existed among various groups of respondents. To this end, the raw data were broken down by various demographic categories and by ways in which respondents answered several key questions. These breakdowns were necessary so that analysis and comparisons could be made among different groups that responded to the questionnaire. For example, the gender category allowed the Core Group to determine if there is any significant difference between the numbers of males and females who read information on product labels. A total of seven demographic categories were made for the CLI study as follows:

gender (male, female);
household income (less than $10,000; $10,000-$24,999; $25,000-$49,999; and $50,000 or greater);

respondent education level (less than high school, high school graduate, and some college level education);
minority status (yes, no);
age of respondent (18-34, 35-54, and 55 or older);
presence of children in the household (yes, no);
dog/cat ownership (yes, no); and
overall satisfaction level expressed with the label for that category, as indicated on the mail questionnaire.

In addition, seven categories were made to compare the ways in which respondents answered key questions of interest for the Core Group's analysis, as follows:

frequency with which labels are read (respondents who read label section "occasionally or every time," or those who "do not read label sections occasionally or every time");
ability to correctly identify most sections (respondents who were able to correctly locate label sections and those that could not correctly locate label sections two or more times);
whether or not respondents looked for information about ingredients (respondents who said that they looked for ingredient information and those that said that they did not look for this information);
preferred ingredient format (respondents' preference for four different ingredient information presentation options (for details on these options, refer to question 4c on mail questionnaires in Appendix 2-4);
whether or not respondents looked for information about harmful effects of the product (respondents who said that they look for information on a label on the harmful effects of a label, and those that said that they did not);
preferred labeling format (respondents who answered that they would "make no change to the current label format," those that said they would like to see "headings to highlight key facts," and those that said that they preferred the suggested "box format"); and
geographic region (indication of where respondents were from for use by the Storage & Disposal Subgroup to see how respondents from states with strong household hazardous waste management programs ("strong HHW") answered questions in comparison to those respondents from other states ("other HHW")).

Data Precision

Based on a standard statistical measure for sample sizes of about 850 to 900 respondents, the data for the nationally representative sample of users for each of the three product categories are accurate to + 3½% at the 95% confidence interval. This means that if the study were conducted 100 times and 50% of respondents gave a certain response, 95 out of those 100 tests would yield a result for that response if given by between 46.5% and 53.5% of respondents. As percentages move towards the extremes (i.e., closer to 0% and 100%), the precision of these data points will actually be higher. It is important to note that these precision measures refer to specific data points, and not to differences between data points. Precision for groups with smaller sample sizes will be lower.

1. A complete set of the quantitative data may be found in the EPA's Public Docket, Administrative Record AR-139. The availability of the data for public review was announced in a Federal Register (FR) notice (63 FR 57298, October 27, 1998).

1. Consumers were screened from NFO Research's consumer panel of 550,000 households. The panel of 550,000 was randomly chosen from the population as a whole. The NFO panel consumers have agreed in advance to participate in marketing research studies. When households become members of the NFO panel, they provide a large amount of demographic information about their household (e.g., age and gender of household members, household income, household size, education and employment information on the male and female heads of household, and many other types of information). This large database of pre-recruited households allows NFO Research to:

easily find households which are willing to participate in marketing research studies, particularly those that are longer and more complex in nature;
design the sample (i.e., determine which households are chosen to participate in the study) in a way that ensures that the demographic make-up of participants (and thus the results) are representative of the U.S. population as a whole; and
eliminate the need to collect a series of demographic information from each respondent, since the panel database already contains a large amount of demographic information for each panel household.

2. Pesticides, disinfectants, and antimicrobial cleaners are subject to labeling requirements under FIFRA. Other products (i.e., in the case of products covered by CLI, non-disinfectant and/or antimicrobial household cleaners), are governed by other authorities. In the cleaner category, therefore, product labels are markedly different, depending on whether FIFRA or a different statute applies, even through the products in the bottle may be similar. For the CLI quantitative research, respondents in the household cleaners category were presented with a FIFRA and a non-FIFRA label to determine how each was perceived.

3. A complete set of the quantitative data may be found in the EPA's Public Docket, Administrative Record AR-139. The availability of the data for public review was announced in a Federal Register (FR) notice (63 FR 57298, October 27, 1998).

[Go to next section]

Publications | Glossary | A-Z Index | Jobs