Background
The use of qualitative and quantitative research with a large number
of consumers to determine consumer behavior and opinion is a relatively
new and unique approach for EPA. The Consumer Labeling Initiative
(CLI) had its Phase I Report peer reviewed in 1996 and found the
reviewers' comments to be quite informative and helpful. Given the
potential magnitude and impact of the recommendations deriving from
the CLI's Phase II research, the EPA and CLI participants wanted
to determine whether we had gone about our research appropriately
and whether independent researchers believed the recommendations
were supported by the research. With those goals in mind, a peer
review of the Phase II Draft Report was undertaken.
Document Reviewed
The document reviewed was the Consumer Labeling Initiative Phase
II Report - Draft, July 1, 1999. The Report contained the following
major sections: 1) Executive Summary; 2) Overview of Phase II of
the CLI; 3) Quantitative Research; 4) Qualitative Research; 5) Quantitative
and Qualitative Research Conclusions; 6) First Aid - Qualitative
Research; 7) Phase II Sub-groups; 8) Partner and Task Force Meetings;
9) Stakeholder Interactions and Comments; 10) CLI Phase II Recommendations;
and finally many appendices supporting the research efforts. Appendices
included: 1) Lists of participants; 2) Quantitative, qualitative,
and first aid research screening documents, discussion guides, questionnaires,
and mock labels; 3) notes of all major meetings; and, 4) a list
of stakeholders who had submitted comments. The stakeholder comments
were not included in the reviewed draft, and were included in the
subsequent revision.
Peer Reviewers
The peer review was conducted by four independent reviewers not
associated with either the United States Environmental Protection
Agency, or the Consumer Labeling Initiative project. Reviewers were
selected based on their expertise or experience in the fields of
consumer behavior, consumer opinion, risk and hazard communications,
consumer research and testing, and consumer education. Reviewers
included: Dr. J. Stanley Black, Community Response Analyst, Office
of Community Relations, Illinois Environmental Protection Agency;
Dr. Albert J. Ignatowski, Principal, HazCom Consulting, and Senior
Fellow, Wharton School, Risk Management and Decision Processes Center,
University of Pennsylvania; Dr. Sidney I. Lirtzman, Dean, Zicklin
School of Business, and Emanuel Saxe Professor of Management, Baruch
College, City University of New York; and Beth Resnick, Associate
Director, Division of Public Health Practice, National Association
of City and County Health Officials.
Reviewers were asked to respond to 26 questions in five categories:
Study Design; Study Implementation--quantitative, qualitative, and
research groups; Study Results and Recommendations; the Peer Review
Process; and any other comments not falling into those categories.
The questions, which are included later in this chapter, asked,
for example, about the appropriateness of the study methodologies
and statistical methods chosen, the adequacy of the screening and
survey instruments used, whether the key learning objectives were
represented sufficiently in the research; whether the findings and
recommendations were clearly supported by the research, etc.
Summary of Reviewers' Comments
Generally all of the reviewers' responses to the review questions
were quite positive. However, there were some specific criticisms
which are mentioned below. The most negative comments concerned
the length and complexity of the written questionnaire. Comments
which were submitted concerning specifics in the report itself have
been addressed in the final version and so are not addressed here.
Study Design
All of the reviewers agreed the methodologies used in the study
were appropriate and addressed the key learning objectives. One
reviewer suggested that presenting randomly selected householders
with a set of varied label formats might have provided more realistic
results than the mail or phone surveys, but he also said the cost
and logistical complications of that approach might not have been
warranted by the increased value of the information obtained. Another
reviewer believed that more valid information with respect to comprehension
would have been obtained using personal interviews. Another reviewer
didn't think enough focus was given to label alternatives for low-level
readers and non-English speakers.
Qualitative Study Implementation
Screeners
Generally the reviewers reported the recruitment procedures to
be adequate. However, one reviewer stated it would have been better
to focus on non-purchasers of products, and use the purchasers as
a control group. Another reviewer said the selection criteria for
the focus groups seemed quite inexact, but went on to say there
were no claims that the groups were representative but only aimed
for a reasonable variability.
Discussion Guides and Learning Objectives
All reviewers said the guides seemed sufficient and the objectives
were represented. One reviewer suggested he would have asked participants
for their preferences regarding label formats before showing any
mock samples.
Mock Label Adequacy
All the reviewers agreed the labels were appropriate. One reviewer
believed that there was too much emphasis on designing labels that
mimicked existing FIFRA label design requirements, and then asking
respondents if they liked them. The reviewer went on to say he would
have preferred even more emphasis been devoted to isolating some
key features of label design and presenting them in a manner to
elicit respondent preferences.
Another was concerned there appeared to have been too many
labels.
Quantitative Study Implementation
Screeners and Discussion Guides
All reviewers generally agreed the recruitment screeners and discussion
guides were appropriate and adequate.
Written Questionnaire
All of the reviewers felt the written questionnaire was entirely
too long. They had concerns about its complexity, smallness of type,
and dense format. They were concerned the length could have lead
to "question fatigue" and at least one reviewer expressed some concern
about projecting the results because of that fatigue. Another reviewer
suggested it would have been better if the items in the questionnaire
were divided among subgroups of the study population, with appropriate
redundancy for checking constancy.
Statistical Methods
Generally the reviewers were satisfied, but one reviewer said the
tables were primitive (only percentages are reported) and he couldn't
tell if tests of significance were performed routinely or not. He
went on to say the size of the quantitative sample is large enough
that some of the results have to be considered very important.
Learning Objectives
All agreed the learning objectives were adequately represented
in the mail and phone questionnaires.
One reviewer did not think it appropriate that industry funded
the quantitative research.
Research Groups
When asked "did the work of the groups appear to reflect what was
being learned in the qualitative and quantitative research" all
but one reviewer claimed they were unable to answer the question
because of its vagueness. One reviewer did say the work of the groups
was consistent with the gist of the results from the quantitative
research.
Study Results and Recommendations
Findings Supported by Research
All the reviewers agreed the findings were supported by the research.
One said, however, there was no attempt to qualify or moderate the
findings based on the quite divergent results of the subgroup of
respondents, namely the less-well educated, lower-income, and minority
populations. Another reviewer expressed concern about whether we
actually can determine consumers' current comprehension of the label
language; although, he goes on to say "if one looks at the results
of the preference data it is possible to draw the inference there
is significant lack of comprehension of the standard label language
because of preference for language which uses simpler words, phrases,
and is active and directive toward specific goals." He later states
it is only in the interviews on the first aid statements one is
able to find reports of consumer confusion as to the meaning of
words and phrases. This particular reviewer believed only personal
interviews should be used to determine comprehension.
Use of Quotes
While the reviewers said the discussion and recommendations seemed
relevant in relation to the quotes used, most said a wider sampling
of quotes would have given them more confidence in the quotes selected.
Enough Raw Data Presented
All the reviewers agreed there was enough data presented. One reviewer
said it should only be construed to represent consumer opinion and
not actual behavior. He went on to say that while demographic information
was obtained for all respondents, the tables are not broken down
by these groups so the impact, if any, can be directly assessed.
Another reviewer said "it is a very rich resource for evaluating
consumer responses in this area."
All the reviewers agreed the conclusions and recommendations were
supported by the findings and data.
Other Comments
Does the Report Adequately Explain the Project
All said yes, although one did say it was repetitive.
Are Stakeholder Concerns Adequately Represented/Addressed
Some reviewers felt stakeholder concerns were adequately represented,
while others expressed some confusion or dissatisfaction. Limited
stakeholder comments appeared in the version which was given to
the peer reviewers. Significant additional stakeholder comments
were included in the final version. One reviewer said more consumers
and state and local agency representatives should have been included
in the planning and steering groups and that increased retailer
participation would have been helpful as well.
Storage and Disposal
One reviewer believed the extensive information on waste and container
disposal was not well incorporated into consideration of the label
design. Recommendations for including this information on labels
seem "weak." The input for the various stakeholder groups was interesting
but not directly germane to the study purpose. Another reviewer
said it would have been more objective if both industry and the
state and local organizations had presented reports or papers, rather
than providing information differently.
Consumer Education
One reviewer suggested the education campaign should include references
to source reduction and other alternative products and that retailers
should be included since they will most likely play a large part
in this effort.
One reviewer commented the study could have been significantly
strengthened if more of the "interested parties" were professional
hazard communicators. The reviewer went on to say he did not wish
to diminish the value and import of much of what was learned; he
found many of the conclusions immediately useful.
Peer Review Process
The reviewers all agreed this type of review should be done for
similar efforts. One reviewer wrote the review procedure was commendable
and long overdue. All reviewers agreed allotting more time to do
the review would have been helpful. All agreed the materials provided
to do the review were sufficient, but could have been organized
better to facilitate the review; for example, the order of appendix
materials, clearer labeling of appendix materials, references to
the appropriate sections included in the questions, etc. One reviewer
said the materials were unwieldy and offered several suggestions
on how to improve the report.
Study Design
1. Were appropriate methodologies chosen to conduct
the study?
2. Were appropriate methodologies chosen to address
the key learning objectives?
Study Implementation
Qualitative:
3. Were the recruitment screeners appropriate
to acquire the type of consumers needed to conduct this study?
4. Were the questions asked in the discussion
guides appropriate and/or sufficient to acquire the necessary consumer
opinions about labels?
5. Were the key learning objectives represented
in the discussion guides?
6. Did the mock labels/samples appear to be adequate
for the participants?
Quantitative:
7. Were the recruitment screeners and practices
appropriate to acquire the type and quantity of consumers needed
to conduct the quantitative survey?
8. Was the telephone interview outline adequate
for its purpose?
9. Was the length, structure and content of the
written questionnaire appropriate?
10. Were appropriate statistical methods and processes
used to compile and evaluate the data from the surveys?
11. Were the key learning objectives adequately
represented by the questions on the mail and phone surveys?
Research Groups:
12. Did the work of the groups appear to reflect
what was being learned in the qualitative and quantitative research?
Study Results and Recommendations
13. Are the findings supported by the research?
14. Are the implications reasonable, based on
the findings?
15. Based on the quotes provided in the text from
the focus groups, do the discussion and recommendations seem relevant?
16. Is enough raw data presented to provide the
reader with a clear picture of consumer behavior/opinions regarding
labels?
17. Are the conclusions supported by the findings
and data?
18. Do the recommendations appear supported by
the research findings?
19. Do the report findings/recommendations concerning
the consumer education campaign, storage and disposal, standardized
information, etc. appear to be supported by the research?
Peer Review Process
20. Should the Agency consider this type of review
for similar research efforts? If not, why not?
21. Were the materials sufficient for your review?
If not, what additional materials would you like to have seen included
in the package.
22. Was the time allotment adequate for review
of the material and preparation of comments? If not, how much time
do you believe is reasonably required to perform this review?
23. What changes would you suggest to improve
the process?
Other
24. Does the report adequately explain the goals,
process, and accomplishments of the project?
25. Are stakeholder concerns adequately represented/addressed?
26. Are there any additional areas you would like
to address or comments you would like to include?
|
![[logo] US EPA](../gif/logo_epaseal.gif){kind=logo}