Joint NRSWG/STSWG Issue Paper on Modeling Issues

Thursday, April 24, 1997

1:30 - 2:30 pm Conference Call

Conference Call Participants:

Science and Technical Support Work Group

Tom Helms	U.S. EPA (Co-Chair, STSWG)

Harvey Jeffries	UNC (Co-Chair, STSWG)

Tom Moore	Arizona (STSWG)

David Sanders	U.S. EPA (STSWG)

John Silvasi	U.S. EPA (STSWG)

Steve Ziman	Chevron (STSWG)

Charlene Spells	U.S. EPA (STSWG)

Tom Braverman	U.S. EPA

National and Regional Strategies Work Group

Ted Russell	Georgia Tech (NRSWG)

Robin Dunkins	U.S. EPA (NRSWG)

Contractor Support:

Bob Wayland, SAIC; Rebecca Battye, EC/R

Summary of the Meeting

David Sanders opened the meeting with a discussion of the objectives for this call. Mr. Sanders indicated that this was primarily a brainstorming meeting to develop ideas. He cautioned the participants not to rely totally on Ned Meyer's straw man paper to develop this Issue Paper. Mr. Sanders stated that Ted Russell should take the initiative in the development of this particular paper.

Ted Russell responded that he intended to use Mr. Meyer's straw man document, Steve Ziman's response material and his own "bizarre" thoughts to develop the initial draft of this paper.

Steve Ziman asked who the client was for this writing effort?

Ted Russell stated that this effort was motivated by the Integrated Strategies subgroup of the National and Regional Strategies Work Group (NRSWG). Mr. Russell stated that this subgroup was interested in gathering information and guidance on what sort of models/tools were available to address Integrated Implementation. Additionally, he stated that the subgroup was interested in the limitations of these tools, and what happens if these tools were not available within the necessary time frame. He concluded by saying that these were the three basic issues: (1) What models/tools were available?; (2) What were the limitations of these models/tool?; and, (3) In what time frame would these models/tools be available? Mr. Russell also expressed some interest in discussing the underlying dependence of these models and their ability to treat integrated processes (e.g., what if models-3 is not available?).

Robin Dunkins added that this was an accurate characterization of the charge given to Mr. Russell by the NRSWG membership.

Tom Moore stated that this scope covered Integrated Implementation Strategies, and was thus much larger than the original definition of the Control Strategies paper.

Ted Russell responded that this particular Issue Paper had to address the models, the input data sets, and the tools that were available to evaluate the model results. He indicated that Donna Lamb had brought these issues up many times in recent weeks. He encouraged the group to grasp the full array issues that were currently on the table.

Steve Ziman stated that Ms. Lamb was motivated by the comments she had received on her initial draft. Mr. Ziman stated that these issues were revealed to her very vividly at that time. He continued saying that Ms. Lamb was making some points in that initial draft document that were simply not valid. He reminded the participants that these errors resulted in Tom Moore writing a "tools section" for the initial paper. Mr. Ziman concluded saying that he believed that Ms. Lamb was now asking for a more formal response than what himself, Tom Moore and S.T. Rao originally provided. He stated that this request was good, even if STSWG could not provide the end-all answers to her questions at this time.

Ted Russell reminded the group that there appears to be an assumption in FACA that the modeling tools exist; however, he stated that some members on this call would not adhere to this belief.

Harvey Jeffries questioned if Integrated Implementation had been defined yet?

Ted Russell responded that this was still under debate. However, he continued that this did not necessarily assume that there would be only one model. He stated that Integrated Implementation could require more than one model, or more than one tool to develop the necessary confidence levels. Mr. Russell emphasized that this group needed to understand what the limitations were for the available models/tools.

Tom Moore suggested that the name of the paper be changed from a "modeling" paper. Mr. Moore suggested that the paper be titled "Uses and Abuses of Evaluation Tools for Integrated Implementation".

Ted Russell indicated that he was not against changing the paper's title, just not to one that radical!

Steve Ziman suggested "Limitations in the Modeling Process"?

Ted Russell stated that he wanted the audience to begin thinking of modeling as a process, and that the title should reflect this.

Tom Moore again suggested that the title should get away from the inclusion of "modeling". He indicated that if we changed the title, then we could include more of S.T. Rao's material. He stated he agreed with the basic process, just not the insistence on using "modeling" in the title..

Steve Ziman interjected that one of the issues that was often neglected was that we develop large data bases to evaluate models, but we that we rarely exhaust the available information (i.e., a lot of good information is never analyzed).

Tom Moore reiterated that there were a number of things that could be done that had nothing to do modeling. Mr. Moore stated that this fact did not imply a weakness in the models, rather it simply showed that models cannot do everything.

Ted Russell attempted to re-focus the discussion by stating that there was consensus on the concept that modeling was a process. Mr. Russell suggested that he put his thoughts down on paper, and then distribute them via electronic mail, and allow everyone to comment on the basic concepts.

Steve Ziman responded that electronic mail was fine; however, the group did need to have discussions from time to time.

Ted Russell agreed with Mr. Ziman, but encouraged the group to commit as much as possible to electronic mail, such that there would be a written record of all the material.

Robin Dunkins reminded the participants electronic mail was fine; however, she emphasized that she would be enforcing the deadlines, and this effort needed to go onto the FACA website in time for the June meeting.

Tom Moore questioned if this meant an abstract two weeks prior to the June Meeting and final draft one week prior to the meeting?

Robin Dunkins stated that this was basically correct. She reminded the participants that the submittals could be as detailed as necessary, and that a final version would be required for either the August or October 1997 FACA meetings.

Harvey Jeffries suggested that the initial definition be a separate entity from the final paper. He suggested that the document for the June meeting be very simple, and only explain the mission statement. He suggested that the August/October document could be more detailed and contain the specifics. Mr. Jeffries indicated that he thought the June document would be no more than a couple of pages.

Robin Dunkins agreed that NRSWG did want this document to be short and simple.

Ted Russell suggested that the participants get things out via e-mail very quickly. He asked Mr. Jeffries for an electronic mail distribution list.

Harvey Jeffries indicated that this information would come from David Sanders.

Robin Dunkins interjected that she would the focal point for distribution, since this paper was a NRSWG effort, not a STSWG product.

Harvey Jeffries acknowledged that he was incorrect, and that Ms. Dunkins would be the focal point for this paper and not Mr. Sanders.

Steve Ziman requested that all electronic documents be in WordPerfect 5.1.

Robin Dunkins agreed that only WordPerfect 5.1 would be used.

Ted Russell stated that he would begin to funnel information to Robin Dunkins, and that phone calls would be scheduled as needed. However, Mr. Russell encourage the use of electronic mail as much as possible.

Ted Russell now began the discussion which attempted to frame the issues to be addressed by this paper. Mr. Russell stated that one of the important points of the paper was the issue of modeling being a process, rather than one of choosing a specific model or set of models. He stated that this process would iterative, and potentially lengthy, and would involve more than the standard "apply this model and evaluate it in this way". Secondly, Mr. Russell stated that there was a true need for better emissions and meteorological input information, and that the currently available data needed to be better utilized. He said this group needed to outline a process that would minimize "gaming" in the use of models. Mr. Russell concluded that the process needed to favor better science as opposed to stagnation, and that the group needed to address any uncertainties that existed in the process. He reminded the group that the process included model inception, model implementation and model evaluation.

Harvey Jeffries echoed Mr. Russell's statements, and reminded the group that quality control and quality assurance also need to be on the list. He explained a recent situation where thousands of CRAY hours were squandered because of the lack of rigorous QA/QC procedures.

Ted Russell agreed that QA/QC would be added to the list.

Harvey Jeffries questioned if there was some reasonable way of measuring QA/QC up- front, rather than waiting until the end of a modeling effort to find that you had errors all along?

Ted Russell responded that there were ways, but that perhaps a better way needed to be included in this process.

Steve Ziman suggested that methods to conduct adequate performance evaluations were necessary in the process Mr. Ziman stated that this was an extremely important issue, and that this could not be glossed over.

Harvey Jeffries reminded the participants that EPA had considered the methodology of "freezing" the code to ensure QA/QC; however, he stated that this was not the answer. Mr. Jeffries suggested that this group needed to understand how much of this process should be high- level technical expertise and what could be accomplished with off-the-shelf support from contractors.

Ted Russell questioned Mr. Jeffries if he was referring to a process that would ensure that all applications were developed by the proper, qualified individuals?

Harvey Jeffries responded that this was correct, and that Glenn Cass had raised this issue at the April FACA meeting.

Steve Ziman argued that the individual states did not have the computational resources in place to accomplish the goals outlined by Mr. Jeffries.

Ted Russell interjected that all of the QA/QC and uncertainty calculations would be done on the national level for the meteorology and emissions data bases.

Tom Moore disagreed with Mr. Russell. He stated that he believed that this group was discussing a level of complexity that has never been done before. He reminded the participants that the group was discussing fugitive estimates which in the past may have been done at only two sites, and now these estimates would be cranked-up to national level inventories.

Harvey Jeffries stated that no modeling study takes place without some level of research being done first. He said he could not comprehend dropping one of these models on a specific state without some level of guidance and research to support them; however, he replied that he was not sure how this guidance would fit into the process.

Tom Moore reminded that group that the emissions were the responsibility of the regulator.

Steve Ziman indicated that we needed to break the mold used in the past, where everything but the kitchen sink was included in a modeling process. Mr. Ziman suggested that group look at alternative situations like the Ross and Reynolds study. He suggested that the group needed to grasp the lay of the land with regard to spatial variation with different strategies. Otherwise, Mr. Ziman stated that the group would develop a very limited process, which would result in a very limited answer.

Ted Russell stressed that the process must evolve over time, and not become stagnant.

Harvey Jeffries questioned how much emissions information was going to be included? Mr. Jeffries indicated the U.S. EPA/ORD already understands that it would have to provide 12 km meteorology. He stated that Tom Moore had made it clear that emissions estimates would need to be a local process. Thus, Mr. Jeffries indicated that their would be a hierarchal relationship of responsibilities. He encouraged the group to discuss which pieces of the problems could be centralized, and which elements would most likely never be centralized.

Tom Moore voiced his agreement with Mr. Jeffries.

Harvey Jeffries stated this concept did bring to mind the volume of data involved, and how one would handle the flow of information.

Steve Ziman stated that there would have to be some baseline of information that would be provided at the national level. However, he said, that using meteorology as an example, there were some regions where 12 km meteorology simply would not work. Thus, how does one deal with this problem if the data was provided nationally?

Harvey Jeffries responded that generating 4 km meteorology for an area the size of a RAMP required the dedication of an enormous amount of computational resources.

Steve Ziman responded that the only way to challenge a model is with data; the more data supplied, the better the results (e.g., FDDA to improve met model outputs)! Mr. Ziman also stated that the group need to discuss time lines, and under time constraints this information would be provided.

Tom Moore suggested that what Mr. Ziman was referring to was really a reality check!

Harvey Jeffries responded that U.S. EPA/ORD was already looking into this issue. Mr. Jeffries reminded the group that people cannot think of this process as a straight linear system - it is a process, and it is very iterative, some of which could be very large, and require mid-course corrections.

Tom Moore stated that he believed this group would get two chances, and thus the time constraints were framed.

Ted Russell responded that was okay. He suggested that the group develop a process that could be accomplished in this time frame. He reminded the group that this would be a work in progress, and that the group did not want to limit itself to any specific process.

Tom Moore indicated that he meant the deadlines are legal in nature, and that this group needed to state clearly up-front what the intentions were.

Steve Ziman indicated that the Conceptual Model document never really discussed how one demonstrates attainment. He indicated that Mr. Moore's insistence on this group having only two shots implied that people want a yes or no answer. Mr. Ziman reminded the group that EPA has been moving away from that approach with its new weight of evidence methods. He said that what one should expect from one of these tools in this process, is that the tool will provide site specific attainment demonstrations, not a national answer.

Tom Moore responded that the group was at the margins. He indicated that the group could design a process that would provide a more realistic model; however, this model probably would not demonstrate attainment. Mr. Moore concluded that if we are required to demonstrate attainment, then perhaps the group would be in an impossible situation.

Steve Ziman responded that a lot of people are driven to expect a yes or no answer.

Harvey Jeffries stated that this is exactly what we keep coming back to: uncertainty issues in numerical modeling. Mr. Jeffries indicated that we needed to discuss the limitations of these tools.

Ted Russell agreed that the group would have to acknowledge where the uncertainties were located. In terms of the time frame issue, Mr. Russell suggested that there were two time frames: a regulatory time frame, and a time frame to get a suitable answer. He concluded that these two time frames often did not coincide with each other. Mr. Russell suggested the following for a "starter title": Modeling Process, Emission Inventories, Their Uses and Limitations.

Steve Ziman suggested that evaluations should be included in the title.

Tom Moore suggested that "estimates" be substituted for "inventories", because an inventory implies a finite calculation.

Ted Russell disagreed saying that EPA generates inventories based upon emission estimates.

Tom Moore agreed.

Ted Russell volunteered to get a quick draft out to the participants for their review. He indicated this would be accomplished prior to Monday, April 28, 1997.

Tom Moore indicated that there was a draft guidance document on PM-fine monitoring networks, which had been developed by John Watson and Neil Frank. He suggested that he would contact Neil Frank and distribute copies of the document. He stated that Judy Chow and Mark Pitchford were also involved in generating this document.

David Sanders ended the call.