![[logo] US EPA](../gif/logo_epaseal.gif){kind=logo}
PPDC Performance Measures Work Group Meeting Notes - September 26, 2005
PPDC PERFORMANCE MEASURES WORK GROUP
Meeting Notes
26 September 2005; Crystal Mall 2
EPA Attendees :
Sherry Sterling - Performance Measures Work Group Co-Chair
Mary Monell - OPP Deputy Office Director
Arty William - Associate Director Environmental Fate and Effects Division
Pat Cimino - staff support
Member Attendees:
Steve Balling - by conference call
Tom Beider - in person
Dan for Carolyn Brickey - by conference call
Amy Brown - in person
Jim Burnette - by conference call
Larry Elworth - by conference call
Michael Fry - in person
Caroline Kennedy - in person
Bob Rosenberg - by conference call
Isi Siddiqui - by conference call
Julie Spagnoli - in person
Public Attendees:
John Kepner - Beyond Pesticides
Introduction
Marty Monell : Welcomed the workgroup and provided opening remarks. Expect the workgroup to have 3 to 4 intense meetings on this subject; OPP will look at the recommendations of the group.
EPA/OPP has been working on measurement development since the Government Performance and Results Act (GPRA) was enacted. Over time, the shift has been from looking at outputs to looking at outcomes. The Office of Management and Budget’s (OMB) use of the Performance Assessment Rating Tool (PART) as a measure of effectiveness has resulted in all government agencies focus on measuring results. Measuring results makes for good government and is a good way to manage. If we can’t answer the question “so what” regarding the impact of our programs activities, then we need to reassess how we are managing the program.
The program is reviewing goals and what we are trying to achieve. In doing this, we are adding measures to our January 2006 work plan for achieving our goals. The measures that we develop will not be final, but will be a start. They should be useful for the next PART examination scheduled for next spring.
OPP needs help from the workgroup on thinking about what measures are appropriate. Even if we do not have data for the measure, we want to identify potential measures. We may not be able to follow-up on data less measures, but want to be as inclusive as possible.
OPP management would like the workgroup to focus on a framework for developing measures for Benefits from the Pesticides Program.
Sherry Sterling. We are looking at October 19, the day before the formal PPDC meeting, for the next work group meeting. We propose that the focus of that meeting would be the framework for developing measures for the “Benefits” mission area. In the future, we would plan to bring the measures for human health and environmental protection to this work group for discussion.
Larry Elworth - Is the Agency looking for direction as well as substance?
Marty Monell answered that OPP would like direction and substantive input, as well as anything else the workgroup thinks is relevant.
Larry Elworth - One of the measures is the number of acres using reduced-risk pesticides. Isn’t this USDA’s responsibility as well as others? What is USDA’s role and how does that fit in with EPA’s role for this measure?
Marty Monell - It is our responsibility for working with USDA as well as others to attain this goal. It is not good enough for EPA to say that this is USDA’s job alone.
Julie Spagnoli - This is a good example of a joint objective. For instance, CDC may be tracking what is happening regarding reductions in insect/rat vectored diseases, but it takes contributions from OPP in registering/making pesticides available for vector control.
Marty Monell - Agencies should have joint responsibilities. We are looking to OMB to put pressure on other Agencies to help with joint goals/measures. At this point, OMB has opted out of doing this.
Sherry Sterling - We are adopting measures in order to better manage our programs. This includes reaching out to other Agencies that have complementary/joint programs.
Isi Siddiqui - Are there any plans for inter-Agency meetings or is there an informal process for this?
Sherry Sterling - It is informal at this stage.
Background Presentations and Work Group Discussions
Sherry Sterling provided review of GPRA, PART and other background information for the workgroup members. Arty Williams presented on overview of the process used in developing performance measures for the Endangered Species task area.
Julie Spagnoli and Amy Brown - Asked what OMB was/is looking for re: program areas and connections to other Agency programs.
Sherry Sterling - OMB has not looked at connections yet.
Michael Fry - Has OMB given any indication of what they consider are measures?
Sherry Sterling - OMB has given us some indication of what they want us to do; they encourage the use of the logic model process in arriving at measures.
Larry Elworth - While OMB’s rationale is important, it is not the only rationale from OPP’s or others’ point of view. If there is no ability to evaluate what is being done – be it good, bad or indifferent -- then this would be an exercise of reporting numbers.
Sherry Sterling - OMB recognizes that outcomes may not occur within a single year. OPP focus is to develop outcome measures, and to move quickly to establishing baselines and target/goals. This work group will be helpful in the OPP process by building, adding, refining/adjusting measures as these measures undergo development. OPP has found that the logic model process is helpful in refining measures and action plans for measuring. Measures are different from indicators; the measures that we are currently developing will be used in strategic planning/managing. OMB has encouraged the use of logic models to build measurement activity.
Steve Balling - Regarding the National Pesticide Program Goal “Protect public health and the environment by ensuring pesticides are safe and available for a healthy America ,”shouldn’t ‘effective’ be added to this statement (i.e. effective pesticides) as well as ‘safe’ and ‘available’? We can register a whole lot of pesticides that are safe, but are not effective.
Julie Spagnoli - Pesticides also need to be effective for public health uses as well as agricultural uses.
Michael Fry - Some measures requires substantial cooperation with industry. Is there an accountability system for industry? For instance companies need to develop reduced risk pesticides and voluntarily cancel older, riskier pesticides.
Sherry Sterling - The goal is to measure OPP’s performance, not industry’s performance.
Caroline Kennedy - Are the registration and reregistration measures final?
Sherry Sterling - Everything is open for discussion at this point.
Larry Elworth - How is evaluation incorporated into the accountability measures?
Sherry Sterling - “Evaluation” is one of the considerations in the development of our measures.
Larry Elworth - It appears that measures will be quantitative. How will changes in thinking/attitudes be measured?
Sherry Sterling and Arty Williams - This aspect is being worked on by OPP. Difficult to do this quantitatively, so are considering qualitative measures. Consideration of the need for changes in thinking necessary for long-term outcome of better endangered species protection, for example.
Larry Elworth - How about an effectiveness measure? People/organizations outside of OPP are responsible for obeying label directions, enforcement, and education. An enormous indicator of success of OPP reaching intended outcomes is its ability to work with outside people/organizations.
Amy Brown - States have experience using logic models to quantify changes in attitudes. Logic models will often express progress in attaining a goal as have short-, mid- and long-term outcomes. Short-term outcomes often include attitude changes, mid-term would be changes in practices and long-term outcomes would be improving/maintaining/restoring the environment and human health. Need to get these bits into the right part of the logic model.
Julie Spagnoli - Can we incorporate what this group does into the PRIA PPDC process improvement workgroup? Measuring PRIA process improvements.
Sherry Sterling - We can certainly discuss this..
Arty Williams - The endangered species logic model is different than other logic models that the Agency is developing. The approach we have taken is a ‘wait and see attitude’ on how it can be used for measures. The logic model process was a good way to identify what needed to be accomplished and it’s a good way to array resources and customers. Long-term outcomes were determined before the logic model process started.
Isi Siddiqui - OPP should list registrants and growers as resources to attain goals.
Michael Fry - Also add to resources knowledge base (OPP databases and knowledge of pesticides).
Steve Balling - Is there any consideration of limitations or barriers (such as lack of funding or resources)?
Arty Williams - Resources are a concern. Once decent measures have been developed, then a determination needs to be made on whether they are high value and low burden measures. Are the measures costly or inexpensive to obtain? Are they readily available or are funds needed to obtain them? The funding issue is not addressed until the end of the logic model process in order to prevent stifling a measure because it is too costly.
The OPP endangered species logic model work group identified over 30 measures, but winnowed them down to less than fifteen after burden, costs, and value had been considered. The logic model process goes through much iteration. The OPP endangered species logic model work group also looked at what questions having a measure would answer and whether or not questions were valuable to answer. For instance, would an OPP endangered species measure be able to focus on reducing pesticide risk in an endangered species focus area? How would we measure reduced risk or levels of pesticides in a focus area? What baseline do you use? Would the baseline be developed using levels of pesticides found in water, air other resources in the focus area? After going through this process the OPP endangered species logic model work group has narrowed the field down to six potential measures of success. These should be available at the PPDC meeting in October.
Michael Fry - Will the list be provided to the workgroup? Will cost be a determining factor?
Arty Williams - The OPP endangered species logic model workgroup did not exclude measures due to cost. A targeted monitoring program would probably be needed to manage costs because monitoring can be quite expensive. The OPP work group left monitoring in their measures proposal although it is costly because they feel it is important and needed.
Michael Fry - Is seems that one of OMB’s punitive measures is to cut the budget. If a program is not found to be effective, it will be cut. Will OMB add funding to very effective program?
Sherry Sterling - OMB does not add funding to governmental agency budgets. The agency is responsible for requesting increases in budget.
Arty Williams - The major measures for an endangered species protection goal would be: 1) compliance with OPP Endangered Species program; and 2) protection of endangered species.
Steve Balling - It will be tough to quantify protecting endangered species from pesticides.
Arty Williams - That is the ultimate outcome; we want steps along the way to get to this goal.
Caroline Kennedy - How does the logic model take into account the cumulative effect of pesticides on endangered species.
Arty Williams - Our risk assessments don’t look at this currently.
Caroline Kennedy - What if an endangered species had a recovery plan that listed pesticides as part of the problem. Is any attempt made to get a baseline on the species?
Arty Williams - Reregistration will not be looking at impact on species.
Caroline Kennedy - OPP is still doing risk assessments chemical by chemical. Why can’t we look at a baseline and go from there? Who else will do this work if OPP does not do it?
Tom Beider - OPP does not have control over all the factors. Recovery rates for endangered species have been dismal over the past 30 years due to multiple complexities and stressors on the environment.
Larry Elworth - Have other parts of OPP used the logic model process? Would like to be able to talk to other OPP logic model workgroups.
Julie Spagnoli - In addition to the ‘Task Areas’ that OPP will be developing measures (Water quality, worker protection, endangered species, strategic agriculture initiative & PESP, food safety /aggregate risk), could we add “public heath’ to the ‘Task Areas’?
Next Steps
Sherry Sterling - The current draft of the PPDC’s agenda has 2 hours devoted to a performance measure discussion. That is time for the work group to address the full PPDC. This could be part of the discussion at an October 19 work group meeting.
Bob Rosenberg - Will the workgroup be suggesting alternatives, endorse what OPP has done, other?
Sherry Sterling - The work group could discuss any observations that they have as of October 19, and plans for the future.
Julie Spagnoli - We will need more information and clarification. We only have looked at one ‘Task Area’ so far – endangered species.
Larry Elworth - Can the workgroup request what they would like to look at (re ‘Task Areas’)?
Amy Brown - We have not had an opportunity to discuss benefits yet. No advice available yet.
Julie Spagnoli - We can say that we have just started (getting background) and will continue discussions (areas to explore).
Amy Brown - Can we limit OPP presentations to 10 minutes for future meetings? Send us background information ahead of time and we will read it and be prepared for the meetings.
Sherry Sterling - Yes, that would be fine.
Isi Siddiqui - Concern about meeting on October 19; need more time to get ready for a PPDC presentation on October 20. Not enough information to fill 2 hours of time on the PPDC schedule at this point. Workgroup will need more face-to-face discussions to have substantive report for the PPDC.
Amy Brown - Agree that there is not enough information to report to use 2 hours of PPDC meeting time. Suggested one hour instead of 2 hours.
Sherry Sterling - Understand clearly – the workgroup wants more time to talk and less time for presentations from OPP. Any background information will be sent to workgroup ahead of time for future meetings.
Larry Elworth - Workgroup needs to be clear about what we are trying to accomplish. Would be helpful to have: 1) a list of things needed to accomplish; 2) list of things that workgroup needs to do, or else we will just end up wandering around.
Sherry Sterling - Will add these to the agenda for the October 19 meeting.