![[logo] US EPA](../gif/logo_epaseal.gif){kind=logo}
PPDC Performance Measures Workgroup Meeting Notes - January 18, 2006
PPDC Workgroup Members: Larry Elworth, Tom Beidler, Daniel Botts, John Schell, Nancy Golden, Jim Burnette, Bob Rosenberg, Jimmy Roberts, Michael Fry, Isi Siddiqui
EPA staff: Sherry Sterling, David Widawsky, Kathy Davis, Gabby Fekete, Richard Dumas, David Hrdy, Jerri Dorsey, Ed Brandt, Lindsay Moose
Public: Patrick McCain, Cindy Baker, Eric Maurer, Jonathan Young, Jim Thrift, Jim Kunstman,
- Initial discussion focused on what the Workgroup should, and could, do that would be most helpful to OPP.
- It was agreed that the meeting would consist of three parts: reviewing the proposed measures, suggesting where gaps might exist, and identifying and discussing areas that would be good to measure but where data doesn’t exist, is too costly, etc.
- It was also agreed that the Workgroup would provide input on the process that OPP used to develop its measures.
- The importance of balancing pesticide risks and benefits was mentioned, and OPP was given credit for attempting to recognize this in the proposed measures. OPP was also recognized for reaching out to stakeholders during the process and not simply instituting measures without input.
- It was pointed out that there are different approaches to measuring performance, both within EPA and even between the mission areas, and that the approaches needed to be compared and contrasted to get a full picture of the scope of work.
- The Workgroup agreed that implementation of these performance measures is key, and that they would be interested in reviewing any implementation plans developed by OPP.
- Sherry Sterling (OPP) provided an overview for the Workgroup that included (1) the goals of OPP’s project, (2) OPP’s current measures, (3) OPP’s daft measures, (4) the process and timeline for OPP’s measures work, and (5) how OPP will use the draft measures.
- POST-MEETING COMMENT: After reviewing the notes, Tom Beidler requested that the following be reflected in the notes –
“At least one member of the Workgroup felt that registrant data and expertise were being under-represented in the LOGIC model used to frame some of the measures.”
Human Health Measures and Related Issues
- Richard Dumas gave the Workgroup an overview of the development of the draft Human Health measures, with Kathy Davis providing information on the workers safety elements of the Human Health measures.
- In discussing human health measures the Workgroup suggested that OPP be cautious when using National Report on Human Exposure to Environmental Chemicals (NHANES) data because the data measured metabolites and there might be an increase in detection of metabolites but a corresponding decrease in risk. OPP stated that it was aware of this potential problem and considers it utilizing the NHANES data. It was also suggested that OPP might select a few sentinel pesticides where we had more confidence in NHANES data to provide an overall picture.
- The workgroup asked about the possibility of developing measures in a couple of areas not already included –
- tracking the amount of training being done for certified applicators; and
- tracking improvements in control of vector-borne illness
- The workgroup asked to what extent OPP considered measures outside of our statutory obligations. OPP responded that measures will continue to evolve as science evolves and as priorities change. The focus of the currently proposed measures are not counting the actions we do to meet statutory obligations but on finding ways to measure the impact our decisions are having in the real world.
- The schedule for measures development was reviewed and the workgroup stated its desire for the PPDC to submit a report, or provide some type of formal input, to OPP as OPP moves into implementation of the measures.
- Concern was raised that OPP not abandon all of its "old" output targets (e.g., number of registrations per year) in moving to these new measures. OPP responded that the old output targets would still be tracked and reported but that the new measures would use these output targets as the building blocks for outcome measures. Where in the past the output target was the end product, and the measure of program success, that is no longer good enough. OPP will now measure what impact its decisions make. It was also mentioned that as long as statutory obligations under statutes such as the Pesticide Registration Improvement Act (PRIA) and the Federal Insecticide, Fungicide and Rodenticide Act (FIFRA) exist, OPP will need to continue meeting output targets.
- The Workgroup pointed out the immensity of the entire pesticide program. It was suggested that instead of trying to institute immediate measures that cover the entire program it would be wise to conduct a few pilots that focus on a segment of the regulated community – for example, pesticide uses for a specific crop – to illustrate the way in which measures are developed, used, and have relevance to the program and the stakeholders. OPP responded that in fact it was taking that approach.
- The Workgroup asked to what extent OPP had focused on the costs, feasibility, difficulty, and effectiveness of proposed measures. In effect, did OPP look at a wish list of what it wanted to measure without considering if the proposed measures could actually be accomplished and if they would in fact meaningfully measure program performance? The Workgroup felt the key was to find ways to not just collect general measures, but to accurately determine OPP’s role in the observed trends, results, etc.
- In addition, the Workgroup wanted to know to what extent OPP is comfortable living with the proposed measures given that it will be held accountable for results. The Workgroup cautioned OPP to not immediately drop tried and true measures and completely move into new, untested measures. OPP responded that it fully intended to build on the existing measures, but that even those needed to be explained better so that results were obvious. Measures that OPP’s work groups have put forward as "ready to go" are certainly based on existing work and will be able to be implemented quickly.
- As for the data supporting the measures, the Workgroup asked if OPP was comfortable basing measures on data sources that it doesn’t in any way control, or which might not be designed to answer the questions OPP needs answered. In addition, is OPP sure that data are reliable or provide the necessary level of detail?
- The Workgroup asked what impact OPP has on data sources (e.g., NHANES) and whether internal sources (e.g., Science to Achieve Results (STAR) research grants) could provide more targeted data. The Workgroups emphasized and OPP agreed that it is critical to understand the context of a database before it is used for performance measurement purposes. OPP responded that it realized the limitations on some sources of data but that it believed that useful data are available.
- At the Workgroup’s request, OPP discussed the structure used to develop measures (work groups, Coordinating Committee, Steering Committee, etc.)
- Some workgroup members observed that the measures are couched in terms of "reducing risk," but it may be more accurate to say “reducing exposure.”
- For HH5*, the Workgroup asked how we intended to accomplish a reduction in acute poisonings. OPP responded that it would be accomplished through labeling and regulatory actions that focused on pesticides that are shown to be problematic. *HH5 = Reduce the number of acute poisoning incidents from pesticides in and around the home.
- The Workgroup asked if there could be a measure similar to HH5 that focused on a better recognition and diagnosis of pesticide poisonings by health care providers.
- The Workgroup questioned the absence of a measure promoting Integrated Pest Management (IPM) in schools. OPP responded that one reason was an inability to insure that the presence of an IPM plan has any real impact on real world exposure and risk. Workgroup members responded that there are extensive data available.
- The Workgroup raised concern over the use of Poison Control Center data. The consensus was that if the data are used, there is a need to have it validated and not just accept raw data which may result from nothing more than a phone call about a possible problem and not a confirmed poisoning incident.
- Concern was raised by the Workgroup that having measures indicating OPP’s desire to reduce exposure to pesticides could give the impression that the current levels are unacceptable. For example, why should lowering residue levels be a priority if current tolerance levels are protective? It was pointed out that as technology improves (e.g., better detection methodologies), the number of detects will increase. That is not necessarily a risk issue if the increased detects are below tolerance. [Clarification: OPP uses the raw data from PDP; it does not use the summary report which only shows the percentage of detects.]
- The Workgroup questioned why the measure related to reducing risk to family members of agricultural workers was not forwarded to the Steering Committee. OPP indicated that the reason was that it is not clear how we would obtain the data needed to determine success and whether that data would be reliable. OPP went on to explain that it might be possible to obtain data through the National Agricultural Workers Survey (NAWS) or by expanding NHANES, but this would entail great expense and would still most likely not result in reliable data.
- OPP provided an overview of the development of worker protection measures, including some discussion of the logic model and its implementation.
- The Workgroup expressed some reservations related to the use of NAWS data since it is collected by the Department of Labor, which is viewed with suspicion in the agricultural community.
- For WS3*, the Workgroup recommended caution and stated that OPP should be careful to track what types of violations are reported and whether it is a meaningful pesticide misuse or not. OPP explained that it planned to follow that advice. * WS3 = Percent change in number of Worker Protection Standards (WPS) violations for each category over time.
- For WS5*, the Workgroup expressed concern for adequate sample size. Additionally where problems exist, they are most likely related to record-keeping or other low risk problems that wouldn’t be impacted by OPP’s activities.* WS5 = Reduce the number of certified applicators with repeat violations.
- Lindsay Moose (OPP) gave an overview of the draft “protect the environment” measures, including the water quality measures and the endangered species measures.
- The Workgroup expressed a concern for ES1*, the time-limiting factor is in making the effects determinations, not in production of the Bulletins. In which case, the reduction in time to produce bulletins may not be an appropriate measure.*ES1 = Percent reduction each year in average cost and average time to produce Endangered Species Bulletin.
- The Workgroup asked why there were measures on endangered species but not other species. It gives the appearance that OPP only cares about species when they reach endangered status.
- The workgroup also raised the need to consider beneficial organisms, either promoting their health or limiting adverse impacts, in a measure.
- The Workgroup suggested that OPP make meeting its obligations under the Migratory Bird Act as a measure. There were several suggestions that birds would make a good species to use in developing a terrestrial measure.
- OPP was asked if it was looking at any data that linked species on the endangered species list with pesticides. OPP indicated that it did in fact have an existing measure that tracked the health of species for which pesticides are listed as having an adverse impact.
- The usefulness of ES3* was questioned since states are required to consider endangered species impacts in FIFRA Section 18 requests; it doesn’t seem to be needed.* ES3 = Cumulative percent of Section 18 requests with a credible effort to consider Endangered Species Act (ESA) implications.
- The Workgroup expressed concern that OPP does not have strong measures for protection of terrestrial ecosystems. The Workgroup expressed concern about the use of bees in a performance measure. When the Workgroup questioned why birds were not being looked at on a regional basis, OPP responded that they had been focused on finding a national measure.
- Sherry Sterling briefly reviewed the elements of the “other benefits” measures which had been covered more fully at the October 19 meeting of this Workgroup. Ed Brandt (OPP) provided an overview of the Strategic Ag Initiative measures.
- The Workgroup suggested a possible measure related to “quality of life” which would somehow measure the benefits to homeowners of a home that is free from insects (e.g., roaches). Results could possibly be determined through use of a consumer survey.
- The Workgroup mentioned the possibility of a measure related to vector control (e.g., benefits for limiting West Nile Virus through the control of mosquitoes). In a related comment, it was mentioned that the Food Quality Protection Act (FQPA) required OPP to develop a list of public health pests that could be useful in coming up with this measure.
- The Workgroup asked about the importance of measuring minor use benefits. One potential measure would be to measure speed in handling IR-4 applications.
- The Workgroup expressed concern that SA1* doesn’t have the same amount of data behind it that some of the other measures did. It was also noted that one way to encourage submission of data would be to increase grants to cover the additional cost to OPP’s partners. There was concern that this measure is more qualitative than quantitative.* SA1 = Reports in Strategic Ag Initiative (SAI) database show increase in use of whole farm practices on transition gradient.
- The Workgroup expressed interest in seeing measures related to PESP. OPP responded that work is continuing on those measures; however, they are not ready for review at this time.
- Each Workgroup member was invited to offer a summary of his/her impressions.
- The Workgroup commented that a multi-year strategy for phasing-in and validating new performance measures seems appropriate. OPP indicated that it planned to move on some measures shortly but that in fact others would be phased-in.
- There was also concern that the proposed measures don’t capture the entirety of OPP’s activities. OPP stated that it was not intended to, but that as the program implements the first set of measures it will continue developing measures that fill existing gaps.
- Various questions were raised by the Workgroup such as: Were they in a position to recommend which measures to proceed on? Are there a specific number of measures that OPP was looking to move forward with? Can some measures be short-term and others long-term? How does OPP prioritize? OPP indicated that there was no number in mind but that it was planning to move forward with a number of measures shortly. Several workgroup members expressed some concern that EPA not adopt measures which could easily reflect negatively on the program simply because OPP did not have sufficient control over data collection methods and interpretation. There is an opportunity to have some short-term and some long-term measures, and the program is intended to proceed that way.
- The Workgroup indicated a belief that Strategic Ag Initiative (SAI) and “other benefits” measures needed more work than the other areas reviewed.
- The Workgroup stated that the key is to base our decisions on data. If OPP determines that sufficient data exist, it should move forward with the measure.
- As for gaps, several Workgroup members pointed out the obvious gap related to measuring terrestrial health and their belief that it is important to address that gap.
- It was agreed that OPP would provide the notes from the meeting and that the workgroup would have a chance to review and revise those notes. Following that, a subsection of the workgroup will draft “recommendations” to be approved by the entire workgroup. It is the Workgroup’s goal to get this work done in 30-45 days. The workgroup plans to present its recommendations at the next meeting of the full PPDC. The workgroup will meet prior to the PPDC meeting to go over final preparations for their PPDC presentation.
- The Workgroup suggested that it meet with OPP’s Performance Measures Steering Committee (or some other appropriate group) to discuss the workgroup’s recommendations.
- The Workgroup expressed a desire to get feedback from OPP on how its recommendations are used in OPP’s performance measures decision-making process.
- Draft and circulate notes from the meeting. Members are urged to comment on the notes by February 1 to Sherry Sterling.
- A subgroup of the workgroup drafts “recommendations” and report on its work. If you are interested in working on this subgroup, contact Sherry Sterling by February 1.
- Draft report and recommendations circulated to Measures Workgroup for review and comment.
- Presentation developed for PPDC meeting by EPA staff and review by Workgroup (PowerPoint, handouts, etc.).
- Meeting of Workgroup in conjunction with next PPDC meeting to discuss Workgroup report and presentation; meeting with OPP Measures Steering Committee or other appropriate group of people.
- It is the Workgroup’s goal to get this work done in 30-45 days. The Workgroup plans to present its recommendation at the next meeting of the full PPDC.