The Modeling Process and Emissions Inventories:

Their Development, Availability, Evaluation, Use, and Limitations:

Executive Summary

A Joint Paper Between the NSWRG and the STWG

August 5, 1997

Progressive steps towards effective integrated air quality management involving consideration of multiple pollutants (e.g. ozone and PM), multiple effects (e.g., health, haze, welfare), will place new demands on emissions inventories and air quality models. This issue paper addresses their development, availability, use, evaluation and limitations. It is generally viewed that air quality models exist to develop ozone and PM SIPs. However, it is also generally agreed that there are significant uncertainties in emissions inventories and air quality modeling, primarily arising from uncertain emission estimates and inadequate air quality data, particularly for PM. It is recognized that there are resource limitations in overcoming the reduced confidence arising from these uncertainties. Air quality modeling and emissions inventory development should be viewed as evolving processes, one outcome of which can be identifying a strategy to lead an area into achieving specific goals (e.g., attainment), along with information on how to assess progress towards that goal (e.g., RFP). As the process evolves, uncertainties should decrease, with a concurrent increase in the confidence of strategies resulting from the processes.

A number of Statements about emissions inventories and air quality modeling are presented in the paper and its appendices, and are generally supported by the community. An overwhelming theme was the need to improve the process, and model evaluation in particular. Further, scientific peer-review of models, emissions inventory and air quality modeling practices should be a foundation of the processes. While there are resource constraints, it is viewed that the costs of controls being suggested, and mitigating health and welfare impacts of polluted air, suggest that such resources would be well spent: We need better emissions, air quality and meteorological data to improve confidence in the strategies being developed and tested. Public-private partnerships and stakeholder approaches are a mechanism to provide such information.

Four issues are also examined, with various options and recommendations. A first issue was the appropriate use of models in identifying controls and regulations needed to reach attainment. While there are concerns that models may have significant uncertainties, and some suggest that those uncertainties make their use for determining regulations inappropriate, no better method was found. However, those uncertainties are such that air quality analyses will not be unambiguous and uncontested, though more thorough model evaluation will improve our confidence in their use. Thus, we recommend that models still be a backbone air quality management, particularly with more rigorous model evaluation and the use of complementary data analysis. A second issue dealt with a related topic: uncertainty in modeling and emissions inventories. While it may be more resource intensive, it was felt that more rigorous quantitative uncertainty analysis is important to air quality management.

A third topic dealt with the flexibility of an agency, or other stakeholder, to choose an air quality model. It is viewed that the current EPA approach, which is fairly prescriptive, is suboptimal. Indeed, model choice flexibility is viewed as having a number of positive benefits, not the least of which is the ability to use a more sound and appropriate method. The air quality models used should be peer-reviewed. The major criticism of allowing model flexibility is that one could end up with dueling models. However, modern air quality models should give similar results if applied correctly, and if two models show distinctly different behavior it is reason for alarm, not for dismissing the use of one of the models. Associated with model choice was the concern over the use of proprietary/closed models. While flexibility is supported, this support does not extend to closed models, or those whose costs or licensing conditions would inhibit the full testing of the model.

While the concept of an AOI is still young, and thus what it, or a pre-AOI is, is still fuzzy, the workgroup feels that it is very important to use scientifically well-founded methods to determine pre-AOIs. Less advanced techniques, e.g., trajectory analysis, were viewed as being too likely to mis-identify the appropriate regions and hinder the process. Indeed, the use of a quick and dirty technique up front could slow the whole process.

The final issue examined was how to reduce emissions inventory uncertainties. It is strongly believed that this is a key uncertainty, and that with a concerted effort it can be reduced significantly. This would be resources well spent. It is suggested that greater use of bottom-up and top-down approaches be examined, and that they be reconciled.

Also included in the paper are two key appendices. One discusses the present EPA approach to modeling, with comments on the process. A second further examines the model evaluation and testing processes, and what is suggested for the appropriate use of models and emissions inventories for regulatory applications.

The workgroups were able to reach consensus (or near consensus) on the various issues presented. One reason for this is that the modeling community views the process as one that has been and will continue to be important, but also one that can improved. Many of the suggestions are based on improving the scientific integrity of the process.

This is a preliminary document prepared by a subgroup member of the National and Regional Strategies Work Group and the Science and Technical Support Work Group as part of the Federal Advisory Committee Act (FACA) Subcommittee process, with considerable input and help from other workgroup members. It is not an EPA document. This document has seen only preliminary review by the work groups and has been substantially changed since that review. Members of the NSWRG and STWG subgroups are still developing the issues and options for this topic. Interested people should forward their comments to Ted Russell of the National and Regional Strategies Work Group.

The Modeling Process and Emissions Inventories:

Their Development, Availability, Evaluation, Use, and Limitations

Date:	August 5, 1997

Contact:	Dr. Ted Russell <trussell@pollution.ce.gatech.edu>

Status:	Draft #6.1.2b

Note: Readers are encouraged to read related issue papers for discussions concerning Area of Influence (AOIs), Areas of Violation (AOVs), Regionally Integrated Plans (RIPs), and Regional Air Management Partnerships (RAMPs). These terms are defined in Appendix A, taken from the 2/13/97 RAMP/AOI paper. These papers are all available from TTN and WWW. The following provides an initial list of papers directly related to this issue paper:

×	Designation Issues for New NAAQS (7/25/96)

×	Institutional Mechanism for Development and Implementation of Regional Strategies (2/13/97)

×	Area of Violation Boundaries (2/20/97)

×	Regional Air Management Partnerships and Areas Of Influence (2/13/97)

×	AOI Tools Issue Paper: Science and Technology Workgroup (12/13/96)

Further, readers are encouraged to read other papers in process, in particular those dealing with RFP, monitoring and emissions.

Introduction

This paper deals with issues relating to the use of models and emissions inventories in response to the FACA discussions. The paper first discusses some of the reasons this issue paper was suggested, then identifies and addresses some findings, which appear as "Statements". Next, a number of issues upon which FACA can make recommendations are presented.

Central to air quality management, including various regulatory purposes such as attainment demonstration, is the use of air quality modeling. Further, such models are often part of air quality modeling systems which can include atmospheric data, meteorological and emissions models, and emissions inventories. The largest body of knowledge about the use of air quality modeling to identify strategies to reach attainment (or for showing continuing non-attainment) is the process of ozone modeling, including developing inventories of its precursors, meteorological modeling, and control strategy assessment. The process for modeling PM-fine and regional haze is not so well developed and will require significant basic data acquisition and analysis, with the concurrent development of appropriate analysis techniques. This paper is meant to address specific issues dealing with emission inventory development and air quality modeling, particularly looking at guidance for the future.

One specific issue which this paper was asked to address included the existence of models to conduct the type of analysis suggested by other issues raised in the FACA process, and their strength and weaknesses. Further issues included the confidence one might have in the use of current (and future) models, and the key uncertainties. A third was to identify weaknesses in current modeling practice. A further question posed was the use of models and emissions inventories to support trading programs or other management strategies.

It is useful to compare future recommended directions for modeling/inventory assessments with current practices. Current practices are summarized in Appendix B. Also contained in Appendix B are specific recommendations or comments in reply to the current practices and how they should be revised. Many of the issues raised below are in direct response to those practices, and thus a familiarity with the contents of that appendix is suggested.

The following eight Statements concerning modeling/inventories are strongly supported throughout the community, and rapid action by the agency and others is strongly encouraged to address items raised.

Statement 1: Models are available for use in designing ozone and PM SIP's, and for other regulatory modeling purposes. However, quality of data bases and lack of rigorous performance evaluation should make one cautious in interpreting their output. Further, continued model development is warranted for better dealing with integrated air quality management (e.g. addressing ozone and PM together, looking toward multimedia integration in the future).

Statement 2: Availability of adequate emissions and other inputs is probably the greatest limitation to the roles of models in regulatory applications. We need better emissions, meteorological, and air quality data to apply the models with confidence to develop effective controls. We can make the "motherhood and apple pie" Statement since it is very true. We need better emissions, meteorological and air quality data to apply the models with confidence. (Also, we do not make best use of all the data that is available.) A partial list of important improvements is contained in Table 1.

Statement 3: It is essential that emissions data be collected and reported consistently. This is necessary for strategies to be fair, to reduce biases in selection of control measures, and to evaluate/quality assure emissions information/collection procedures against independent data. One drawback of using only a consistent method is that there is the possibility of having consistently wrong estimates, and as noted above, emissions inventories need to be improved. Thus, special studies should be encouraged to improve emissions estimates, and the results of those studies should be diffused into the process. The EPA check-list approach is insufficient. More effort in rectifying bottom-up and top-down emissions assessments is needed.

Statement 4: Regulatory demands on model outputs need to be reconciled with limits on the resources needed to develop technically defensible analyses. The resource problem consists of 4 dimensions: capable, well-trained personnel; money to collect needed information and apply the models; time and desire. Computer/data base management limitations present a resource constraint that is being reduced constantly.

Statement 5: Appropriate use of models and emissions inventories in attainment planning should be considered processes consisting of several essential steps which are iterative. Air quality modeling and emissions inventory development should be viewed as processes. Emissions inventory development includes quantifying emissions factors, assessing process and activity rates, emissions speciation, spatial allocation of emissions, direct monitoring, etc. Air quality modeling includes (or should include) review of past modeling, data collection (air quality, emissions, source activities and meteorological), emissions inventory construction, quality assurance of the data and inventory, air quality model input preprocessing (including construction of meteorological fields and emissions modeling), model development, model application, evaluation (of both the inputs and outputs), review of existing air quality control programs likely to affect the air pollutant of interest, and then, finally, control strategy analysis. The process is not as linear as this may make it appear and is evolving. There will be iteration, likely at every step. For example, evaluation of the inputs could well lead to reanalysis of those inputs. Then, model evaluation can lead to identification of other questions in the model inputs and/or model formulation. This iteration should be encouraged, and documented, to lead to a better final product. The iteration should also be done at a larger scale. Every few years the process should be reviewed and repeated both to improve the prospective estimates of how air quality will improve and what level of controls are necessary, but also to assess how well prior modeling exercises were able to simulate the response of the atmosphere to the controls applied during the interim. Peer-review of model capabilities is part of the model development and evaluation aspects of the process. The process should be evolutionary in a way to bring the best tools to bear in air quality management.

Statement 6: Performance tests for air quality models should be significantly more rigorous than current common practice. Greater attention should be given to the ability to predict precursors, key indicator species and to simulate physical processes in a realistic manner. Further, the model should be stressed to provide greater confidence that the model is performing adequately for the right reasons. A more detailed discussion of model evaluation is provided in Appendix C. Model evaluation (which includes, but is not only model performance evaluation) is one of the greatest concerns in the community, especially the regulated community.

Statement 7: Model attainment demonstrations and RFP-related modeling requirements should be closely coordinated so as to provide a clear set of requirements to States in which a limited number of analyses meet several regulatory goals/requirements so as to limit resources demanded of the States. This implies that "mid-course reviews" required in the attainment demonstration process and retrospective analyses required for RFP should be timed so that they are one in the same. Further, scientific peer-review of air quality models and emissions inventory procedures should be conducted prior to the mid-course reviews and retrospective RFP such that the results of those reviews can be used in a timely fashion. This is discussed further in Appendix D.

Statement 8: In the above Statements, as well as the issue paper following, the term scientific peer-review is often used. The use of the term scientifically peer-reviewed refers to review and publication in the open literature, i.e., scientific journals. Other forms of peer-review are inferior, and preference should be given to those methods that have passed scientific peer-review. Scientific peer-review is considered valuable, if not crucial, to the integrity of the process.

The preceding discussion and concerns in the public and private sectors leave a number of issues needing further resolution. Important ones are addressed below. Options for resolving each issue are also identified along with tentative recommendations for consideration by the FACA subcommittee in making its recommendations to the EPA. Note that many of the issues are related closely with other issues, and the recommendations build upon each other with the results that execution of the recommendations will lead to much stronger emissions inventory and air quality modeling processes, and hence increased confidence that the strategies resulting from their use will be effective.

Table 1. Examples of Important Information Needed to Improve Emissions Inventory and Air

Quality Modeling Processes

Needs Reasons Emissions factors (e.g., AP-42) More accurate mass emissions rates Emissions speciation Improved assessment of source impacts on air quality and magnitude of source emissions Size and chemically resolved primary PM emissions Ability to assess source impacts on PM levels Improved Mobile source emissions model (e.g. Mobile X) Better characterization of the mass, timing, location and composition of automotive emissions Top-down/bottom up reconciliation of emissions in a number of areas Identification of emissions inventory gaps and needs Better understanding of aromatic chemistry Improved ability to model aromatics, an important VOC class Size and chemically resolved PM measurements Ability to rigorously evaluate model performance More meteorological measurements Provide better meteorological fields for air quality modeling Quantitative methods for source apportionment Improved source apportionment

Issue 1. What is the appropriate use of models in identifying strategies leading to attainment (or continuing non-attainment) and RFP assessment?

Currently, air quality models are the backbone for identifying strategies that lead to attainment and can be used more effectively in RFP demonstrations. A model is evaluated by its ability to simulate an historical episode. After acceptable performance is obtained, the model is used to simulate the impact of the application of emissions controls. (See Appendix B for a discussion of how the modeling process should be conducted before its use for control strategy identification and RFP assessment.) The current demonstration process requires model results which show that the set of controls lead to acceptable air quality. However, models are not perfect, nor are their inputs. In typical ozone model applications, the simulated ozone levels are generally within no better than 25% of the observations in the base case. The disagreement is likely due to a combination of the lack of spatial representativeness in the observations, model imperfections and uncertainties in the inputs, with most of the disagreement due to the latter. This level of disagreement can be much greater than the detail at which their results are used, e.g., at the ppb level.

Another issue is the application of a deterministic model to a stochastic process. In order to evaluate a model, it is applied to historical episodes. However, that meteorology will never be repeated. Controls may produce different results in future meteorologies. These questions suggest that the air quality modeling process is flawed when applied to showing that future air quality will meet an air quality goal specified at the part per billion (ppb) level. (The uncertainties can be as large as the change in concentrations desired, though this should not detract from model use if the user recognizes that the model is one of the tools that can be used to develop an integrated air management plan and process.) Another weakness, often cited, is that the changes in emissions expected due to growth and controls are also uncertain.

On the other hand, the models, particularly when applied correctly, are the scientifically most sound means of quantifying how air quality will respond to future emissions changes during similar meteorological episodes. Those episodes are chosen to represent meteorologies that lead to conditions that have, in the past, been tied to non-attainment, and thus, all else equal would likely be similar to periods in the future that would also have higher levels. If not a model(s), what other approach is preferable? The subgroup can not identify an alternative that is more attractive or scientifically as sound.

Option 1a. Do not use models for attainment or RFP demonstrations since they are not accurate enough.

Pros:	Addresses the issue of uncertainty and model limitations, and would minimize the gaming of results.

Cons:	Begs the issue as to how to determine the magnitude of control, or which sources should be controlled

Option 1b. Use models for attainment demonstration, but only for directional information.

Pros:	Addresses concerns about model limitations.

Cons:	Still begs the issue as to how to determine the magnitude of control, or which sources should be controlled.

Option 2a. Apply an appropriate model and use it for attainment and RFP demonstrations.

Pro:	Similar to current structure so is more readily accomplished. Provides a means for demonstrating attainment.

Cons:	Does not deal with the view that models are incapable of "demonstrating attainment" in that it does not deal with the issue of using models to provide answers that are viewed to be beyond their level of accuracy. It does not deal with what constitutes adequate performance. Can lead to gaming of results.

Option 2a: Apply an appropriate model, and attempt to get "performance" within some recommended guidelines. If the model performs adequately, it may be used for attainment.

Pro:	Similar to current structure so is more readily implemented. Provides a means for demonstrating attainment when model performance is adequate.

Cons:	Does not deal with the view, held by some, that models are incapable of "demonstrating attainment." Does not deal with poor model performance due to lack of appropriate data. Requires the development of performance guidelines that are themselves not universally accepted. Adequate performance can be region- and case-dependent. Use of rigid performance guidelines leads to model gaming that is ultimately harmful to the process. Hidden flaws. False confidence. Does not provide a mechanism to demonstrate attainment if model performance is inadequate or if appropriate data are unavailable.

Option 3a: Continue to use models similar to now.

Apply an appropriate model using the most appropriate inputs for the region (AOI), based on the nature of the air pollution problem. Use the model results along with other air quality analysis. If good model performance is achieved, the model is then used in attainment demonstration. If poor performance is found, reasons for poor performance should be identified and quantified. (Reasons for good performance should also be established.) Further, if poor performance is found, greater weight would be placed on other information used in the attainment demonstration process (e.g., statistical analysis of the response of air quality to emissions controls, observation based methods, etc.).

Pro:	An extension of the current structure so is more readily implemented. Provides a means for demonstrating attainment whether model performance is adequate or not. Provides a more robust approach to attainment demonstration. Deals with the view that models are incapable of "demonstrating attainment".

Cons:	Requires the development of performance guidelines that are themselves not universally accepted. Adequate performance can be region-dependent. Weighing different kinds of analyses is subjective. Does not prevent gaming if absolute predictions are still basis for attainment

Option 3b: Continue to use models similar to now after developing more rigorous scientific based performance criteria for acceptance of models, possibly including independent peer-review of results. This is similar to option 3, but the models would only be used for regulatory modeling after rigorous evaluation and testing of alternative control strategies.

Pro:	In addition to the pros of Option 3a, it would rely on much more extensive testing to increase the confidence of both the decision makers and the stakeholders in the use of the model results.

Cons:	Cons are similar to option 3a, though the chance of gaming is reduced significantly. It also increases resource demands for conducting a more extensive evaluation and outside peer-review.

Recommendation: Option 3b. Air quality models (and the air quality modeling process), when applied correctly and in conjunction with other information should provide the best means of assessing the necessary levels of controls. As discussed above, the process should be iterative and on-going, with periodic re-assessment. Developing new guidelines to include more extensive evaluation are key to improving confidence. As discussed below, it is also recommended that more advanced use of models (and more advanced models) are key to improved air quality management in the future.

Issue 2. Identification and Treatment of Uncertainty in the Emissions Inventorying and Meteorological, Emissions and Air Quality Modeling Process. Environmental decision making will be conducted in the presence of uncertainty, now and forevermore. Some of those uncertainties are inherent, while others exist due to the lack of effort to minimize them. In the past, the decision making process was not cluttered significantly by the knowledge of exactly how big those uncertainties (and biases) were (and continue to be). This was largely because of a lack of knowledge on the magnitude of the uncertainties, which is largely due to the perception that developing better, quantitative uncertainty estimates would not be used. On the other hand, it is now recognized that knowledge of such uncertainties is valuable for a number of reasons. Policy-makers can use such information to influence their decisions, for example to choose more robust strategies. The public can use it to understand the complexity of the issues involved. Scientists can use it to identify methods to reduce uncertainty in future studies, and provide upper and lower bounds to the assessment. Finally, quantitative knowledge of uncertainties and variabilities can be used to identify better air quality management practice and design control strategies, much like it is used in the design of automobiles to improve product quality. However, development of a better knowledge of uncertainties takes resources. A major problem with not addressing uncertainty is that, while it helps define what we don't know or should not say, abstract discussion of its impacts cloud what we do know and can say with greater confidence. Further, if the agencies (EPA and others) are not actively assessing uncertainties, other parties will, and use such information to argue for increased or decreased controls. EPA can take the lead in this endeavor.

Option 1a. Continue to focus primarily on best estimates for developing emissions inventories and air quality modeling, as opposed to developing best estimates and quantifying the effect from uncertainties and biases.

Pros:	Easiest to implement and least resource intensive.

Cons:	Does not allow good estimates of uncertainties in the overall process, or the determination of more robust approaches to air quality management. Inhibits identification of the most critical uncertainties in the process. More likely to allow major errors to exist. Does not, by itself, provide significant impetus to reduce uncertainties.

Option 1b. Continue to focus primarily on best estimates for developing emissions inventories and air quality modeling with some qualitative estimates of uncertainties, as opposed to developing best estimates and quantifying the effect from uncertainties and biases. (This most closely approximates the current State)

Pros:	Easy to implement and not significantly more resource intensive than option 1a.

Cons:	Does not allow good estimates of uncertainties in the overall process, or the determination of more robust approaches to air quality management. Inhibits identification of the most critical uncertainties in the process. More likely to allow major errors to exist. Does not, by itself, provide significant impetus to reduce uncertainties. Further, qualitative uncertainty analysis can be misleading.

Option 2. Require that quantitative uncertainty estimates be developed and documented for emissions, meteorological input and boundary condition estimates, and their impact on air quality modeling results.

Any modeling exercise that uses those estimates would also assess how those uncertainties impact the assessment of control effectiveness and the attainment/RFP demonstration. The results of that uncertainty exercise would then be communicated to the appropriate policy makers, as well as in the RIP/SIP/AOI determination. This process, should identify, to the degree possible, the upper and lower bounds for the proposed strategy, and can also be used to determine how RFP may be best measured.

Pros:	Begins to educate the community as to the magnitude and origin of uncertainties in the modeling process (for attainment demonstration or AOI determination). Allows decision-makers to better assess what options are most appropriate. Would ultimately reduce uncertainties in the process.

Cons:	Somewhat more resource intensive than options 1a/1b, and may discourage or impede firms or agencies from developing and reporting emissions estimates. As Stated, does not provide significant impetus to reduce uncertainties.

Option 3. Require that quantitative uncertainty estimates be developed and documented for emissions estimates, and other model inputs and parameters, and that these estimates be used in establishing attainment goals.

Any modeling exercise that uses those emissions would also include the emissions and monitoring uncertainties. If biases do exist, their impact would also be quantified. Any modeling exercise that uses those estimates would also assess how the individual and composite uncertainties and biases impact the assessment of control effectiveness and the attainment/RFP demonstration. The results of that uncertainty exercise would then be communicated to the appropriate policy makers, as well as in the RIP/SIP/AOI determination. This process, should identify, to the degree possible, the upper and lower bounds for the proposed strategy, and can also be used to determine how RFP may be best measured. Further, attainment/RFP demonstration would be based on showing attainment with the uncertainty taken into account, e.g., the attainment/RFP based on the best emissions estimates would be demonstrated to attain by a certain date would be likely with a certain degree of confidence based upon knowledge of the uncertainties and biases in the inventory.

Pros:	Begins to educate the community as to the magnitude and origin of uncertainties in the modeling process (for attainment demonstration or AOI determination). Allows decision-makers to better assess what options are most appropriate. Encourages minimizing uncertainties, and would reduce uncertainties in the process faster than other options. Ultimately provides more confidence in the air quality modeling process.

Cons:	More resource intensive than options 1a/1b or 2, and may discourage or impede some bodies from developing and reporting emissions estimates.

Recommendation: Option 2, moving to option 3. Uncertainty assessment is viewed as an important process, providing means to improve future efforts and to better assess risks and rewards that can accrue from the choice of alternative strategies. It is commonly done in most environmental assessments, and is now being done for air quality modeling by some parties. Further, with the current time lines, it is very feasible.

Issue 3. How Much Model Choice Flexibility Should be Allowed in the Attainment/RFP Modeling Process, and Should Proprietary or Closed Models be Used?

This is a two part issue, the first dealing with the general idea of whether a specific model should be given preference by EPA over others, and the second deals with the issue of whether proprietary and/or closed models should be used for regulatory decision making. In regards to the first issue, currently, the attainment modeling process is rather tightly prescribed. For example, the EPA modeling guidelines specify the default model to use (UAM-IV), and give modeling guidelines and performance measures. A number of reasons are given for such prescription. First, a preferred model (UAM-IV) has been set to deter the problem of 'dueling' models. Second, this promotes consistency. On the other hand, it leads to the use of a sub-optimal approach. For example, the preferred model is urban scale, and it is now recognized that regional transport is important. Further, it represents what many believe to be a less advanced model. In fact, the approach to use a regional model to provide boundary condition estimates to UAM-IV can (and has) resulted in the scientifically inappropriate endpoint of using what is generally considered a better model to provide information to a less good model that is then used in the attainment demonstration.

In addition, no single approach or model is best for all applications. Furthermore, this can lead to stagnation in advancing the modeling process, and the current regulatory photochemical model (UAM-IV) is not the most scientifically advanced approach, and there continue to be significant deficiencies. There are compelling arguments for a more flexible process, the first of which is that it can allow an area to use the best approach. Some areas will have resources (time, money, tools, and data) that are not available to another area. Prescription would lead to taking the lowest common denominator. Further, as is evidenced historically, air quality management plans evolve, and the development of emissions inventories and the application of air quality models, are part of that evolution. An attainment/RFP demonstration developed at one time will, if attainment is not reached in the interim, be revised at a later time using better information, and hopefully better tools as well.

Many believe it is preferable to bring as much information to bear in developing attainment plans without reliance on one approach alone. The dueling model problem is actually a benefit since it can improve air quality management, and can in essence occur even now. If one allows multiple models to be applied, and that the choice of how much to base controls on specific model results is done by the quality of the model application, then the system will evolve to use better approaches. Thus, the potential of using multiple models, including some that may be very different, should be encouraged.

Finally, models that are in a developmental stage, and therefore have not been used for regulatory application in the past, most appropriately address many of the issues being addressed in this committee. This is likely to continue to be the case. Even with the advent of Models 3, similar problems will remain. The best approach (including air quality models) may not be part of Models 3, and Models 3 may include approaches considered less sound. Given Models 3 structure, it should be noted that flexibility here includes choice of different advection algorithms, meteorological models, and chemical mechanisms, the choice being based on which ones provide the best approach to air quality management for that application and domain, and having had prior (preferably recent) scientific peer-review and acceptance. A part of the flexibility issue involves the use of proprietary and/or closed models, and a separate set of options is presented for this.

Option 1. Continue to have a more prescribed approach to regulatory modeling.

Pros:	This is easiest to implement, and easiest to monitor from a regulatory standpoint. It prevents dueling models, though State of the science air quality models, when applied correctly, should give similar results. If not, then this raises further concerns that should be explored.

Cons:	It may not allow the most appropriate approach, and there is no current approach for integrated strategy development.

Option 2. Increase the flexibility in regulatory modeling, emphasizing using the best set of tools to do the job.

Pros:	This should lead to the use of better air quality models and more rapid development of air quality models, along with a better understanding of the ozone and PM formation, transport and fate processes. It provides greater range of possibilities to deal with air quality modeling for integrated strategies across multiple pollutants.

Cons:	Places greater burden on EPA staff to review demonstrations. It could lead to dueling models, and may lead to States choosing a less appropriate model.

Option 3. Increase the flexibility in regulatory modeling, emphasizing use of the best set of tools and data to do the job. Commit to a frequent peer review and comparison (say, every five years) of approaches taken in the modeling process.

This periodic review would include how inventories are developed and manipulated, the applicability and strengths of available meteorological and air quality models, and how to interpret the results in regards to being able to demonstrate attainment. Possibly identify and justify a suite of recommended models, and procedures to justify the use of other models. (One could develop user-based groups, structured in a manner similar to the Inter-Agency Workgroup on Air Quality Models (IWAQM). These groups could help share information and analysis techniques and interpretation.)

Pros:	Would lead to the use of better air quality models and an improvement in the air quality modeling process in general, along with a better understanding of the ozone and PM formation and transport processes. It would also lead to more rapid development of air quality models. It provides a greater range of possibilities to deal with air quality modeling for integrated strategies across multiple pollutants.

Cons:	Places greater burden on EPA staff to review demonstrations. It could lead to dueling models.

Option 4. This option is similar to Option 3, except that each RAMP or AOI would have a Technical Advisory Panel (TAP) of modeling, emissions, and meteorology experts to review and advise on all aspects of the work from data gathering and analysis plans to protocol development through interpretation of results and uncertainty analysis. Further, a stakeholder group (representing industry, environmental, and other agency groups) would likewise be involved to advise on the process, possibly as part of the TAP.

Pros:	Would lead to the use of better air quality models and an improvement in the air quality modeling process in general. It would also lead to more rapid development of air quality models. It provides a greater range of possibilities to deal with air quality modeling for integrated strategies across multiple pollutants. In addition, it removes burden from EPA to review the details of the SIPs.

Cons:	May be difficult to identify or get persons with appropriate expertise to review, given limits on their time. There could be problems with potential conflicts of interest. The review may be time consuming and inconsistent with required regulatory time frames. Guidelines on panel selection would need to be developed, as well as how stakeholder participation would be structured. However, recent examples suggest that the stakeholder process can work very well when used from the outset. Authority for EPA to hand over such oversight would need to be investigated.

Recommendation for Issue 3: Option 3 or 4.

Options for the use of proprietary and/or closed models.

Here, proprietary model refers to its unavailability to any parties at a reasonable fee, and closed model refers to the unavailability of the source code for inspection and modification for testing purposes. (The definition of reasonable is, of course, subjective. A small fee to support the maintenance of the code and its transmission could be considered appropriate. However, the fee could not be so large as to significantly discourage its use.) Concern has been expressed over the use of a proprietary and closed model, particularly for regulatory policy development. It can be argued that, on one hand, one should be allowed to use the best available approach, and at times that may be a proprietary or closed model. However, the use of such a model can be dangerous. First, errors may exist in the approach that are not readily apparent and would be less readily identified if the tools are not fully open to inspection, testing, modification, and use. Second, it can minimize the ability of all the interested parties from developing their own knowledge base that could then be utilized in the process, and further inhibits the development of information within the community. It can also potentially put a monopoly on the ability of groups to be effectively involved by limiting access to such models either by the model's owners not allowing access or the group not having sufficient resources.

Option 1. Allow the use of proprietary and closed models.

Pros:	Can allow the use of the best tools, if in some case the best tool is proprietary.

Cons:	Will inhibit the open development of relevant information. Model errors and limitations are less readily identified. Inhibits widespread participation. Models will lack peer-review of science, uncertainties, etc.

Option 2. Not allow the use of proprietary and closed models in the development of SIPs and RIPs, though allow information derived in past efforts for regulatory policy development.

Pros:	Better ensures that environmental policy making will be an open process.

Cons:	May not allow the use of the best approach.

Option 3. Not allow the use of proprietary and closed models in the development of SIPS, TIPS, RIPs, etc., and not allow the use of information derived from such models for regulatory policy setting.

Pros:	Better ensures that environmental policy making will be an open process. Minimizes the temptation to use a proprietary or closed model with the hopes that such results would be used.

Cons:	Would not allow information derived historically. In some cases it may not allow the use of the best approach. However, given the fact that historic applications are for the one hour standard and that performance, and age of episode, may not make it defensible to use such modeling in development of regulations. Hence, this may not be a problem.

Recommendation on use of proprietary models: Option 2 or 3. At the recent Source Apportionment Workshop, the scientific panel commented that no closed or proprietary model should be used in regulatory decision making or attainment demonstration. Their view was that if the code is not generally open to inspection, use and modification, it seriously impugns the integrity of the process. Further, they suggested that any fee would inhibit the academic community from testing the models. Also, they Stated that peer-review means in the scientifically peer-reviewed literature, as opposed to panels and reports. This is somewhat stricter than the options chosen here.

Summary on Issue 3: Model Flexibility.

It is the general view that the concept of a guideline model should be dropped, and instead the use of the most appropriate set of models should be encouraged. Thus flexibility is strongly supported, and no preferred or default approach should be given. This flexibility is limited, however, to models that are generally available, and open to inspection and testing. Further, in line with other issues and Statements above, the models should be peer-reviewed in the scientific literature.

Issue 4. How can uncertainties in the emissions inventories be reduced?

One of the most critical limitations in the air quality management modeling process is the large uncertainty in many parts of the emissions inventories, even for ozone precursors. For PM-10 it is worse, and worse still for PM-fine. Little attention has been paid to developing spatially and temporally disaggregated emissions inventories for ammonia and primary fine particulate matter. To do a credible job, models will require detailed information of those emissions, as well as SO2, NH3, VOCs (speciated) and NOx. There is limited information on the size and composition of PM from many sources. Developing an inventory requires accurate information on the number of emitters in each source category, their activity levels and emission rates in terms of particle size and chemical composition. This is a significant undertaking, but it will provide the basis for greater confidence placed in the regulations emanating from such an effort. It is generally believed that one can devote the necessary resources now and do it right, or spend significantly more in the long run down the road making up for it. Four options have been devised from which current PM inventories might be constructed.

Option 1. Use existing State and national inventories along with default information.

Pros:	Analysis can start quickly.

Cons:	May misallocate control emphasis, since control information is weak or missing and defaults are not always appropriate.

Option 2. Launch a major research initiative to improve inventories.

Pros:	Strategies will be based upon most up to date information and should provide much better guidance on which controls should be most effective. Will allow for better identification of deficiencies in the models.

Cons:	May cause delays in the analysis, and costs may be substantial. However, given the new timelines, such an initiative should be viable and may not be an issue.

Option 3. Start with existing inventories for preliminary AOI definition and switch to refined inventories for final RIP/TIP/SIP.

Pros:	Analysis can start quickly, and final plans will be more credible. Early results can be used to improve the inventories.

Cons:	Cost of the refined inventory may be substantial.

Option 4. (Can be added to any of the above) Develop a mechanism to provide incentives to sources to better characterize their emissions. ('Ownership' of some emissions is a complex issue, e.g., natural emissions, road dust, etc. Such incentives would not be limited to industries.) This can be done as a fee based on uncertainty levels or the requirement to increase control levels to make up for uncertainties. Further, by involving the stakeholders in the process, historical evidence suggests that the incentive exists to ensure that major sources have well characterized emissions. Thus, they should be made active participants in the RAMPs and on advisory committees in the emissions and modeling process. Industry has led in past efforts when done in association with intensive data collection efforts.

Pros:	Can lead to better inventory characterization more expediently.

Cons:	Not popular with affected sources. Best incentive mechanism not apparent.

Recommendation 3+4 (?)

An issue that is not addressed above is that of roles and responsibilities. In part this is because the roles and responsibilities of some of the more important entities are not yet defined. For example, does a RAMP identify a preliminary AOI, then forms a body to do the planning for that AOI, or does the RAMP maintain responsibility for the planning for all AOIs in that area. Further, will a RAMP have the resources (money/expertise) to conduct the emissions inventory development and modeling? These issues impact who takes what role in developing the emissions inventory and modeling. One approach is to give EPA much of the responsibility for facilitating the modeling process, including emissions inventory development, choice of models for AOI determination, (and possibly AOI determination guidelines), and identifying the resources necessary. Another is to give the RAMPs (or AOIs) such responsibility.

Recent trends suggest that the role of EPA should change. Viewed from the perspective of lessons learned with modeling and emission inventories to date, and the increasing knowledge and expertise with models and inventories being developed within State and local air agencies, the regulated community, and academia, EPA need not, and should not, take the same lead role as in the past (e.g., to prescribe models and emission inventory techniques). It seems more important that they facilitate the best possible scientific and technical analyses, and let the analyses stand on their own merit. Thus, one recommendation is to invest more responsibility into RAMPs to get regionally credible solutions to regional problems.

Resources are a constant problem in air quality management, and the emissions inventory/modeling process is not an exception. However the stakes are big, and a number of stakeholders are involved. Involving the stakeholders early in the process can facilitate a successful outcome, and can be one vehicle for identifying the necessary resources. This has been seen in the execution of a variety of field campaigns and data analyses. Again, the more open the process, the better.

During OTAG, a new paradigm was used to provide the desired computational capabilities involving "Modeling Centers." This paradigm has been further suggested in a more recent EPA white paper. Each RAMP would have its center, though the "Center" could actually be virtual, i.e., it could be distributed amongst a number of organizations and locations. It is not apparent that this is the most effective approach (or given the fuzziness of what the centers might be, what this may or may not entail). In some instances, a consulting firm may be able to best provide the desired expertise and facilities, and other cases, an agency, industry or university (as is now the case). What is recommended is to open up the process (as was not the case in OTAG) to the community to provide both independent ideas and to act as centers if that is the chosen paradigm. The concern is that continuing with the current State of affairs will miss opportunities and will suffer from some of the baggage of OTAG. A particular concern with not changing significantly is that the current centers are set up to use a particular model, and the above recommendations suggest against both that model (without significant changes) and lack of flexibility. Again, there is a resource issue, not just for the RAMPs to do "their job," but also to develop the level of understanding necessary to evolve rapidly, and to lead cooperative efforts across stakeholders to conduct major studies. For example, in California, this role is played by CARB. In other areas, it is not apparent what organization(s) will take the lead.

It is not apparent that there needs to be a limited number of modeling centers (e.g., one per RAMP, or so). For example, different organizations can take the lead (and do the modeling) for different AOIs. Further, one organization may take the lead in the meteorological modeling, another for emissions, and another for air quality.

Another issue that has not been addressed is that of improved inventory methods. EPA has recently published a revised Protocol for Development of Emission Factors which invites Stakeholder participation in the factor development process. Another aspect of improved inventory methods is the improvement of area source emission estimation methodologies. EPA is currently beginning to undertake this project, and Stakeholder participation in this process is critical to its success. A third aspect that EPA is working on is the Emission Inventory Improvement Program, a joint effort with the States to develop improved inventory guidance. Further, EPA should continue to pursue more advanced methods for emissions inventory reconciliation with observations, e.g., top-down and bottom up intercomparisons and analysis.

Another issue not covered herein is that of emission inventory reporting. EPA is currently developing a rule to consolidate emission inventory reporting requirements under the CAA, in light of the revised Ozone and PM NAAQS. Appendix E (which is currently a separate file) contains the latest draft of this rule. The rule is currently undergoing external review and will be issued along with the implementation rules for the revised NAAQS.

Appendix A

DEFINITION OF TERMS

Area of Influence (AOI) - An AOI is a geographical area containing anthropogenic and natural sources that contribute to Areas of Violation (AOV). The identification of AOIs is divided into two phases:

Phase 1 - Preliminary AOI - The Preliminary AOI is the area which contains an adequate array of sources to prepare a Regionally Integrated Plan (RIP). Inclusion within a Preliminary AOI does not mean that a location will necessarily be subject to regulation.

Phase 2 - Final AOI - The Final AOI is the area in which regulatory control of emissions will be needed to bring an AOV back into attainment and ensure maintenance of the standards. The Final AOI is synonymous with a control region.

Area of Violation (AOV) - The AOV is the area which is determined to be violating the NAAQS. Class I areas will be treated in the same manner as AOV's for the purposes of defining AOI's and developing plans. Contrary to the current nonattainment designation process, an AOV is not necessarily assumed to be responsible for the observed violation. Thus, designation of an AOV does not carry with it automatic control requirements for the designated area, as is currently the case for a nonattainment area. However, the initial presumption is that the AOV will be a part of the preliminary AOI and final AOI. This presumption can be rebutted only upon a clear and convincing demonstration that anthropogenic sources in the AOV are not contributing to the violations being experienced. (Process for making the demonstration to be defined later).

Control Region - A control region (CR) is an area within an AOI for which emission reductions of PM, its precursors and/or precursors of ozone are deemed necessary to meet the NAAQS/RH Rule. Controls are not necessarily applied uniformly across the CR. (A CR may be associated with one or more proximate areas of violation and Class I areas included with the AOI.) The control region is synonymous with the final AOI's.

Control Target - A control target is the amount of reduction and/or the emissions budget estimated as necessary in a control region to be consistent with an identified cohesive strategy for meeting PM and ozone NAAQS in areas of violation. The target can be a reduction relative to an agreed upon base level of emissions or it can be in the form of an emissions cap.

Regional Air Management Partnerships (RAMP) - Contiguous multi-State areas that have common air quality characteristics and share common air quality concerns. The RAMP will have the responsibility to undertake appropriate air quality analysis needed to identify the Preliminary Area of Influence. There are numerous options for the number and size of the RAMPs.

Regionally Integrated Plan (RIP) - This has been referred to as a "spatially integrated plan (SPIP)" in earlier versions of the FACA Regional Partnerships/AOI issue paper. A RIP is prepared for a preliminary AOI. A RIP will define:

×	the Final AOI/control region,

×	in general terms, what level of control should be applied in the various portions of the AOI,

×	where transportation conformity should apply,

×	the structure and type of market-based incentives programs (where appropriate)

×	New Source control recommendations, and

×	other elements as appropriate.

The RIP is a set of recommendations to EPA and is to serve as the basis upon which EPA can develop SIP and TIP calls to the individual States and tribes. The RIP is not enforceable on its face. The RIP may also contain recommendations to EPA for action at the national or subnational level.

State Implementation Plan (SIP) - A State implementation plan is a plan submitted by individual States to reduce emissions consistently with the regionally integrated plan. It contains specific measures and identifies mechanisms by which the measures are enforceable by that State.

Tribal Implementation Plan (TIP) - A tribal implementation plan is a plan submitted by individual tribes to reduce emissions consistently with the regionally integrated plan. It contains specific measures and identifies mechanisms by which the measures are enforceable by that tribe.

Appendix B.

Current Modeling Practices and Comments on Improving the Process

This appendix is designed to provide an understanding of the current regulatory modeling practice and guidance given by EPA. In addition to spelling out the policies, the committee has also commented upon specific guidelines and procedures. Many of the comments are taken from the issues and recommendations presented in the paper, and are given here to provide specific guidance to EPA. Current policy regarding regulatory modeling for ozone is spelled out in three documents: "US EPA (1991) Guideline for Regulatory Application of the Urban Airshed Model, EPA-450/91-013", "US EPA (1994) Guidance on Urban Airshed Model (UAM) Reporting Requirements for Attainment Demonstrations, EPA-454/R-93-056", and "US EPA (1996) Guideline on Use of Modeled Results to Demonstrate Attainment of the Ozone Standard, EPA-454/B-95-007." In addition, the EPA identifies "guideline models" for various applications (see Guideline on Air Quality Models (Revised), 40 CFR Part 51, Appendix W). The Urban Airshed Model, version IV is identified as the EPA Guideline model for use in attainment demonstrations. EPA policy is to allow use of other photochemical grid models in place of UAM-IV if two criteria are met by those wishing to use an alternative model:

(1) There must be a discussion of why use of the alternative model is likely to be more sound scientifically than use of the UAM-IV and the EPA must agree with the reasoning presented, and

Comment: Most existing photochemical air quality models are viewed as scientifically equivalent or superior to UAM-IV. Guidance, however, is lacking as to what would constitute such a judgment to the EPA. As suggested throughout this paper, scientific peer-review should be the primary basis of such a determination.

2) The proposed alternative model must perform at least as well as the UAM-IV in replicating observed data. Comparisons performed in making this determination are decided on a case by case basis. Under current practice, such a comparison is made by also running UAM-IV for the same episodes and domains and then comparing model performance evaluation results.

The identification of a model as preferred along with Requirement 2 has been the subject of concern in the community, and is addressed in the issue paper. It has led to a relatively limited number of other models approved for use in attainment demonstrations. These include SAQM in the San Joaquin Valley applications and UAM-V in the LADCO and Atlanta applications. A request to use CALGRID in the New England application is pending. To date, use of urban scale analyses (e.g., horizontal grid resolution <~5km) has been recommended for all attainment demonstrations in the area which is the focus of the demonstration.

Comment: As an example of the problem of having such a prescribed approach, it is enlightening to examine the current situation in North Carolina. The State of North Carolina wants to use a scientifically more advanced model than UAM-IV in their attainment demonstration process, including one that can also better handle regional transport. However, they are required to apply UAM-IV, and then show equivalent or better performance when using a more advanced approach (e.g., MAQSIP). First, this increases the amount of work necessary. If it is generally accepted that MAQSIP is a "better" model, why use a less good one first? Second, suppose that UAM-IV performs "better." This presents some significant complications and conundrums. There is general agreement in the scientific and academic modeling community that

MAQSIP is a more advanced and scientifically comprehensive model;

uncertainties in the process (e.g. the emissions inventories and meteorology fields) are so large that changes in the inputs within the range of uncertainties could change the performance of both models greatly; and

the typical performance measures are inadequate, particularly if you use only ozone.

Thus, an agency could be saddled with doing more work, and then being forced to use a model that is considered to be less good, in part because it is better. The better performance achieved by the older model may be due to an inadequacy that will bias the assessment of control strategies. One direction that North Carolina could choose is to tune MAQSIP to improve performance, but that may introduce new biases. Thus, being strapped to a single model leads to the process having less integrity.

A second reason to not be overly prescriptive is that no single approach, or model, is best for all applications. Third, this can lead to stagnation in advancing the modeling process, and the current regulatory photochemical model (UAM-IV) is not the most scientifically advanced approach. There are compelling arguments for a more flexible process, the first of which is that it can allow an area to use the best approach. Some areas will have resources (time, money, tools, and data) that are not available to another area. Prescription would lead to taking the lowest common denominator. Further, as is evidenced historically, air quality management plans evolve, and the development of emissions inventories and the application of air quality models, are part of that evolution. An attainment demonstration developed at one time will, if attainment is not reached in the interim, be revised at a later time using better information, and hopefully better tools as well. Third, as mentioned above, it is preferable to bring as much information to bear in developing attainment plans without reliance on one approach alone. The dueling model syndrome can improve air quality management, and can in essence occur even now. If one allows multiple models to be applied, and that the choice of how much to base controls on specific model results is done by the quality of the model application (e.g. comprehensiveness of the modeling approach, model evaluation, etc.), then the system will evolve to use better approaches. (There is little to be gained by conducting an effort that is less defensible than other, "competing" efforts.) Thus, the potential of using multiple models, including some that may be very different, should be encouraged. In fact, this is recommended below (and is not too dissimilar to the ideals behind the weight of evidence directions being developed within the agency). Finally, models that are not being used for regulatory application at this time, and are in development most appropriately address many of the issues being addressed in this committee. This is likely to continue to be the case. Even with the advent of Models 3, similar problems will remain. The best approach (including air quality model(s)) may not be part of Models 3. Further, Models 3, by design, is supposed to represent a broad range of models, thus it is not a model but a modeling system, and models may be supported by Models 3 that may not follow the guidelines suggested in the issue paper (e.g., peer-reviewed).

The EPA guidance for performing modeled attainment demonstrations for ozone addresses 10 subjects. They are listed below, and a number of the issues in this paper address aspects of the guidance. Subgroup comments on the 10 subjects are added after the applicable point.

Development of a Modeling Protocol. This step requires the lead agency to describe the general procedures which will be followed in completing the 9 additional steps, as well as who will be participating and what arrangements will be followed in coordinating/communicating efforts. In cases where the domain encompasses more than one political entity, EPA urges participation of each in developing the protocol and in implementing subsequent steps.

Episode Selection. This is currently tailored toward the 1 ExEx NAAQS. Guidance suggests that exceedance days be divided into meteorological regimes. Within each regime, days are ranked according to severity of the exceedances. Episodes should be chosen from among the top three ranked days in each regime. Exceptions have been granted to allow simulation of days with intensive databases.

Comment: This guideline should be modified for the new standard. In particular, the period containing the annual fourth highest ozone concentration should be considered. However, periods from more prevalent meteorological regimes should be considered as long as the ozone concentrations are not substantially difference from the observed annual fourth highest value.

Preparation of Emissions Inputs. The starting point in our guidance is a 1990 emission inventory compiled by the participating States for a "typical summer day". These estimates are then adjusted to take account of VOC-speciation, day-specific meteorological conditions, day of the week and spatial/temporal variability patterns. Emissions preprocessor models are used for this purpose. The EPA has supported the EPS-2 processor for use for this purpose. However, other similar preprocessors (e.g., EMS-95) may also be used. BEIS1 is to be used to estimate biogenic emissions with CB4.2 chemistry. BEIS2 is recommended for use if the isoprene chemistry in CB4.2 has been revised to be consistent with the BEIS2 estimates.

Comment: This discussion suggests a practice that is considered contrary to the practice described above. In particular, it suggests using compensatory errors, i.e. an emissions inventory method (BEIS1) that is considered outdated and containing shortcomings, and a chemical mechanism (the old isoprene mechanism) that is also flawed. Modeling practice should be directed towards always using the best available information and tools. Further, the CB4 mechanism has not been peer-reviewed in the scientific literature for almost a decade, during which time other mechanisms have continued to evolve and be peer-reviewed. In line with the recommendations in the issue paper, a scientifically peer-reviewed chemical mechanism should be used.

Selection of Model Domain/Horizontal & Vertical Resolution. Size of the modeling domain is a function of the days selected to model. Guidance suggests the domain should be large enough to contain emissions originating in the center until at least 8 pm on the day of primary interest. Horizontal grid resolution of 2-5 km and use of at least 5 vertical layers within the planetary boundary layer are recommended.

Comment: The subgroup recommends that in coastal areas, the domain should encompass the extent of the return flows. To the extent possible, upwind emissions sources should be explicitly included in the domain rather than be represented by the boundary conditions, and the density of the vertical layers (i.e. number of layers), be dictated by model testing and data analysis. It is not apparent that one needs to have such fine (2 km) horizontal resolution.

Boundary/Initial Conditions. Use of a regional scale model is identified as the preferred approach for identifying boundary conditions. Interpolation of appropriate monitored data is suggested as an alternative. Default estimates are identified for clean areas. The simulation should begin at least one day prior to the day of primary interest in order to diminish importance of assumed initial conditions.

Comment: Default values must be more realistic than just the annual mean values because the clean area pollutant concentrations fluctuate with the weather conditions, time of day and season, and can vary spatially. Further, if a regional model is scientifically more advanced, and can have the appropriate resolution, there is no reason not to use the regional model for the analysis. This is discussed in the issue paper.

Preparation of Meteorological Inputs. The preferred approach is to use regionally derived windfields (for regional models) and observations as a starting point for the more detailed winds needed for urban scale analysis. This should be compared with winds generated from the UAM's diagnostic wind model. Use of the procedure, which seems to produce the most realistic winds is recommended. Rawinsonde data are suggested for use in estimating mixing heights. Dynamic meteorological models are mentioned, but not particularly encouraged. Since publication of the guidance, these have been used with increasing frequency.

Comment: It is quickly becoming apparent that prognostic models using FDDA are a powerful approach to developing meteorological fields. Generally, if the resources allow, their use should be strongly considered, and likely encouraged.

Performing Diagnostic Analyses. The guidance suggests these should be undertaken to increase confidence that results make physical sense and as a means for quality assuring inputs to the air quality model. Guidance is flexible regarding what these tests should be. Sensitivity tests are recommended as means for anticipating which components of the model's inputs are likely to be most influential in choice of control strategies.

Comment: This part of model evaluation, as well as model performance evaluation, should be strengthened considerably. Both mechanistic and diagnostic analysis should be done, both at the model and individual module level, to the extent practical. Future EPA guidance should reflect this.

Evaluating Model Performance. Use of computer graphics is recommended as qualitative means for assessing performance. Criteria for three numerical metrics are suggested to evaluate model performance in predicting ozone. These criteria are based on a series of urban analyses performed during the 1980's. EPA also encourages evaluation of ability to predict measured precursor levels. However, EPA has not placed much emphasis on this due to a lack of data and uncertainty about the spatial representativeness of the data that do exist.

Comment: As discussed above and in the issue paper, model performance should be made much more rigorous and extensive, but less prescriptive. Present guidance needs to be expanded so that "good" model performance is not due to underlying compensating errors. This can only be done with a much more complete data sets-air quality, meteorology, emissions-to test the model than is traditionally is available. Further, current guidance is based on historical model applications which may be flawed or have had compensatory errors, and new guidance is required to account for both the improvement in the modeling process and to inhibit the creation of compensatory errors and the motivation to game model results.

Attainment Test. Guidance recommends evaluating severity of selected episodes based on a climatological database. Subject to some limitations, predicted exceedances are allowed on severe episodes. Guidance suggests use of corroborative analysis in cases that are close to meeting the air quality goal. The corroborative analyses can be used in a weight of evidence determination to justify approval of a demonstration not quite meeting the goal. Guidance requires use of a mid-course review as a means for dealing with uncertainty. EPA recommends that modeling also be done for the intermediate period at which the mid-course review is conducted in order to facilitate the corrections.

Comment: Weight of evidence methods should be further studied such that they can be used with greater confidence. The current reliance on showing whether a model has a specific result, e.g. ozone concentration, should be evaluated, and reviewed as to whether the proper use of such tools warrant the implied level of reliance and to determine what alternatives exist. Much of the concern with the use of models in air quality management is due to the perceived over-interpretation of model results for developing regulations to meet attainment in the future.

Archiving/Reporting Information. Requires a description of procedures followed and results obtained. Also requires maintaining input and output files, documenting procedures for accessing them, identification of compatible hardware and ADP contacts for the application.

Comment: While EPA has added weight of evidence measures and encourages quality assurance/control, EPA guidance does not encourage as comprehensive analysis of the data which as is discussed in this document. However, results of the analysis from this data can provide an independent mechanism by which to assess the model performance prior to use in a regulatory setting. Future guidance should place more emphasis on data analysis beyond the level currently required for regulatory modeling.

PM Modeling

To this point, EPA guidance on NAAQS attainment has, of course, focused exclusively on PM10. It is described in the following document:

US EPA, (1987), PM10 SIP Development Guideline, EPA-450/2-86-001.

Three approaches are outlined in the following order of preference:

use dispersion models and receptor models in concert

use dispersion models alone

use receptor models alone, under limited circumstances (e.g., homogeneity of ubiquitous sources, etc.).

A number of Model Guideline (40 CFR 51 Appendix W) Gaussian models are suggested for various types of applications.

One immediate observation is that ozone modeling practice is more developed. This is an important observation as the community must deal with PMfine air quality management and in multi-pollutant integrated strategies. Further, the ozone guidelines were developed for the old, one hour standard, and new guidelines are expected in the summer of 1998 to address modeling for the new standard.

Appendix C

Appropriate Use of Models in Attainment Planning (or Regulatory Modeling in General): The Need to Strengthen Model Evaluation and Investigation of Alternative Control Strategies.

One issue addressed in the issue paper was the use of air quality models in developing strategies that would lead to attainment and to RFP analysis. Such activities are possible end points of the process, but should not be the sole endpoint. Additional goals should be to better understand the impacts of emissions and the resulting air quality, the dynamics of the pollutants in the area, etc. In regards of the discussions of how models are used for regulatory modeling (what has been historically referred to as attainment and RFP demonstration) it is useful to consider how one should go about the more broad activities of the air quality modeling process for management and planning. Good practice in the planning stage should be the basis for the actual modeling that is used to development of a strategy that is projected to lead to/towards attainment or to be used in RFP demonstration. Again, consider the effort as a process, beginning with collection and analysis of emissions, meteorological and air quality data. As stressed before, the whole process hinges upon the availability and use of such data at a level commensurate with the questions being asked. Given that the decisions to be made represent very large costs to industry (much larger than the costs of the data collection and modeling exercise) and similar health effects to the public, the data should be able to justify the resulting decisions. In some cases (more typical all the time) analysis of the data suggests that to do the job right, it is wise to conduct special field studies to augment the routine information. (Such analysis can include air quality modeling of past events.) The analysis of the routine and special data will then lead to selection of episodes for modeling, the appropriate model(s) and the appropriate modeling domain(s). At this point, meteorological modeling (e.g., prognostic modeling with four-dimensional data assimilation (FDDA), though diagnostic modeling may be applicable for some areas; prognostic modeling without data assimilation is not viable) is used to develop fields for the air quality model and emissions model. The meteorological fields are then evaluated and analyzed. (Our limited experience suggests that there can be significant uncertainty and errors in the fields, even with FDDA.) Emissions modeling, using the emissions inventory and meteorological fields is then conducted to develop spatially and temporally specific emissions in the modeling domain. Again, these should be evaluated (e.g. versus past results and observed data). The emissions fields and meteorological fields, along with the air quality data, are then used for air quality modeling. Air quality modeling results are then evaluated. This is a crucial point on which to elaborate.

Past model performance evaluation practice has been rather poor, both in the regulatory and scientific worlds. Here, we distinguish between model evaluation and model performance evaluation. In general, the current advanced air quality models are considered to adequately represent the atmospheric dynamics of pollutants such as ozone and PM. Their descriptions of individual processes (e.g. chemistry, transport, aerosol dynamics) have been evaluated both in the lab and in the ambient (the latter less so). This is not to say they are perfect, but adequate (or better). Thus, on the individual component level, the model has been evaluated. On the other hand, the ambient measurements and model inputs do not exist to seriously test the overall model, and to see how well it performs in a specific episode. This step is model performance evaluation, and will be episode and region specific. However, even though the testing of all of the individual components may show them to work quite well, that does not assure that the overall model will be able to simulate the observed concentrations.

One reason that model performance evaluation has not been as rigorous as desired is that the results are never as good as we hope for when looked at in detail. Another reason is that this step is somewhat burdensome, and only a limited evaluation has been necessary for regulatory acceptance. In particular, for ozone modeling, guidance, as discussed before, has concentrated on assessing if the simulated ozone concentrations and the observed levels match within some guideline rules. This is not as detailed as is suggested by the weight placed upon the results of the modeling. Some concepts about model performance evaluation follow.

Model performance evaluation is the process of testing the ability of a model to estimate observed measures of air quality over a range of meteorological, emissions, and air quality conditions. Model performance evaluation guides the effort of model development, data collection, model analysis, model testing, and model re-testing. For purposes of model evaluation, the model includes not only the air quality model, but also the other components of the system -- the emissions model, the meteorological model, the supporting aerometric and emissions data base, and any other related analytical and numerical procedures used to produce modeling results. In fact, studies to date suggest that model performance is more sensitive to the inputs, than the model formulation and structure. Further, other studies suggest that the inherent spatial variability leads to significant deviations between point measurements used in evaluation and the spatially averaged concentrations that are being simulated. Part of the model testing process is to identify flawed model components. Of note, it is possible for the meteorological and emissions modeling results to perform well, but the entire episode can be rejected from further consideration if the air quality modeling results fail to meet performance evaluation specifications.

Model performance evaluation has several components. An operational evaluation refers to an assessment of the ability of a model to estimate the correct answer whether or not the processes described in the model are accurate. It is an examination of how well a model reproduces an observed event consistent with the needs for policy analysis or regulatory decision making. A scientific evaluation examines the accuracy of the basic physical and chemical processes that are described within the context of the model. It involves testing the entire modeling system as well as components of the modeling system. The scientific evaluation is comprised of a diagnostic, mechanistic, and comparative evaluation.

A diagnostic evaluation is an assessment of the ability of a model to simulate processes which occur during an air quality episode (e.g., dispersion patterns, deposition rates). The diagnostic evaluation is conducted through a series of tests that are designed to stress the science in the model. A mechanistic evaluation is an assessment of an individual module's ability to reproduce observed phenomena that the module is intended to describe (e.g., advection, wet deposition). When applied to all process modules that constitute the full model, a mechanistic evaluation represents a test of the accuracy of the underlying science. A comparative evaluation refers to the intercomparison between different models or modules (e.g., wind field generators, mixing height algorithms, emissions factors models, chemical mechanisms) using observed data as the standard against which the intercomparisons are judged.

The desired outcome of a complete model performance evaluation is good performance and acceptance that the model is adequately representing the dynamics, fate and impact of the pollutants in the atmosphere from source to receptor. This is attained when a minimum acceptable level of agreement between model predictions and observations over a specified range of tests is achieved. Historically, acceptance is achieved through successful completion of a number of model performance evaluations, diagnostic analyses, and model modifications. Despite continued progress in the field of model performance evaluation, current model performance evaluation practice is hindered by:

>	Lack of adequate data bases to provide a thorough basis for model performance testing. In most instances, including OTAG, the available data sets consist almost exclusively of routinely collected aerometric data. Seldom are high resolution data bases available for intensive mechanistic evaluation;

>	Lack of consistency among performance evaluations that have been conducted by different groups. For example, various definitions of ozone cutoff levels or time periods over which model performance is quantified have been used. Further, results are often presented in a wide variety of graphical formats which leads to difficulty and confusion when attempting to intercompare model performance results;

>	Use of alternative, and sometimes conflicting, definitions of key performance measures such as bias, peak estimation accuracy, skill, and correlation;

>	Continuing inability of most statistical measures to reveal clearly the existence of model flaws, data base deficiencies, and compensatory error in models, the data, or both;

>	Reliance on statistical results of past UAM-IV modeling studies (many of which are now known to have been seriously flawed) to serve as the basis for judging whether current model results are acceptable; and

>	Lack of generally agreed upon criteria to judge acceptable model performance

As noted above, the guidelines were developed based on what is now getting to be rather historical practice, and there is evidence that there were compensating errors brought into the process to improve performance. New guidance is needed for a variety of reasons noted above, as well as performance of a model should likely be domain specific (if for no other reason than that the spatial representativeness of the air quality data may vary, and the abundance of monitors in one location versus another can vary as well). Further, ozone is not a very good diagnostic output for model testing. What is recommended is to greatly expand the model evaluation portion of the process. Close examination of the ability to simulate precursor levels is a much more stringent sign of whether the model is doing a good job, and is more likely to identify errors and biases. While, it is recognized that the same performance evaluation metrics that are used for ozone are not applicable to the precursors due to the greater spatial and temporal changes, and the fact that a point measurement (monitor) is being used to evaluate a volume averaged prediction, measurements of those precursors contain a wealth of information. This is a major reason for having the PAMS network, and the consideration of its expansion. At this point, this part of the evaluation can be used as a top-down check on the inventory as well. A number of questions may be asked: Are the ratios observed similar to that expected from the inventory? Does the mass balance look reasonable? Do any of the results seem to contradict what else is known (e.g., regional CO or NOx levels well above observations)? Also, at this point, other calculations can be done to better understand the model's behavior in this case, e.g., no emissions, clean boundary conditions, etc., just to make sure that results from these cases make sense. If possible (there is some doubt that this can be done very quantitatively), can it be shown that the model does or does not reproduce how air quality has evolved over time, e.g., do the ozone or PM levels replicate the historical trend? (The reason for saying, if possible, is that it takes a rather large perturbation in the emissions of a precursor compound to find a change in the atmospheric concentrations of that compound (about 10% or greater). Further, there is a less than proportional change in ozone, e.g., a 10% reduction in VOCs may result in only a 2.5% reduction in ozone. In this case, it may take upwards of a 40% reduction in VOCs to fully establish that a signal has been established above the noise due to variabilities in the meteorology. The signal can be further masked by geographic changes in the emissions.) Thus, what is suggested is that this is the point to step back and re-examine the process to this point and thoroughly examine what has been found, and the limitations. One result of this examination is to establish the confidence on which to base decisions, and what are the uncertainties. Another possible result is that it is determined that there are such significant gaps that a major undertaking (e.g., field experiment), is warranted. One problem in the past with doing a thorough examination of the process is that this is viewed as a delay in the process which is usually up against a fixed and looming deadline, and since it is not required, it is not done. Very pointedly, the goal of model evaluation should not be to improve model performance or to solely establish that the ozone (or PM) results are within guidelines. The former can introduce biases, and ultimately reduce confidence in the process and the results.

It is at this point the model can be used to assess the effectiveness of various control options. Not only should the impact on the peak levels be considered, but also considerations like population exposure and the like. Also, one can investigate some different factors as well, e.g., what happens if the meteorological fields are somewhat different than established for the "base case." This might be due to using different approaches to developing the fields. A variety of studies have shown this can be important. How does this affect the impact of various control strategies? Which set of inputs is better, and why? (Again, this question should not be answered by saying that the performance for ozone is better.) In essence, this is the time to ask the what-if questions, not only to explore various control options, but also other aspects of model inputs and outputs. The experienced gain from this exercise then provides the foundation upon which to go to the next step: Identification of a strategy (or strategies) predicted to lead to attainment, which is further discussed in Issue 1 in the paper.

Appendix D

Coordination Between Modeling for Control Strategy Identification for Attainment, Mid-Course Review, RFP, and Emissions Inventory and Modeling Process Review

Control strategy modeling, mid-course review, RFP, and emissions inventory and modeling process reviews are viewed as key parts of effective use of air quality models and emissions inventories in air quality management. While they can be done at disparate intervals, that would be inefficient and degrade the process considerably.

At present, attainment modeling is done at different intervals depending on the location and severity of the air quality problem, and generally the demonstration is prospective, i.e., it is using the modeling tools to show what types of controls will be necessary to bring an area into attainment. However, the current process does not do the retrospective analysis, i.e., to see if the area has realized the air quality improvements expected. As examined in the RFP paper, there should be a set of checks on the process to investigate whether an area is obtaining the air quality improvements expected and why or why not. While there are very serious questions as to how quantitatively this can be done using air quality models, this can be an important part of air quality management. Thus, even if it does not come out of the RFP paper as being suggested as part of the RFP process, it is recommended here, strongly, that a combined retrospective and prospective analysis be done as part of the air quality modeling process in air quality management and attainment demonstration. Further, that prospective analysis (Mid-course Review) should be synchronized with the retrospective RFP. Likewise, the modeling process review (e.g., as discussed in Issue 3, recommendation 3) should be conducted such that the results are available in time for the Mid-Course review. This will reduce the burden on the States, and should improve the process, including enhancing its evolution.

A possible scenario of events could be as follows. First, an area starts to develop an emissions inventory, as well as collects the necessary ambient data. The ambient data and emissions inventoried are analyzed. Ambient data analysis can include, but is not limited to statistical approaches (e.g., time-series, the Cox and Chu approach, etc.), review of past modeling, comparison of emissions inventories and air quality (again, it is valuable to conduct a top-down and bottom-up check using the emissions inventory and ambient air quality data, as well as temporal trends), and wind trajectory analysis. This information is used for air quality modeling (see the details in Appendices Band C). The results of the air quality modeling and the following adoption of rules to reduce emissions would be used to forecast expected air quality improvements, not only for the pollutant(s) of concern, but other species that can be used to better assess the effects of controls on emissions. One outcome of this step could be that it is found that the likely change in a secondary pollutant, like ozone, would be so small (say less than 10-15%), that the meteorological variability will make it difficult if not impossible to use that species to determine RFP. (As an example, recent studies have found it very difficult to quantitiatively assess the impact of oxygenated fuels on CO. The impact was found to be about 10%, though with a large uncertainty, and some of the studies could find no statistically significant impact. CO, coming primarily from one source and being a primary pollutant, would be relatively easy to find a direct impact of emissions reductions as compared to ozone. Thus, ambient measurements may provide little quantitative information unless very large changes in emissions are expected.) On the other hand, this step can identify what to look for, and guide future data analysis, and the timing of when controls should start becoming noticeable. Given that controls will take time to put in place, no immediate impacts may be seen. At this point, the protocol for conducting a retrospective RFP would begin development.

The next step in the process (again, the process should be continual and evolving, so there would actually be no discrete start and end of activities), would be to assess the tools used in the first emissions inventory and air quality modeling round, and this would include peer-review of the techniques (the peer-review could be conducted by a panel, based primarily on scientifically peer-reviewed work). This step should be completed such that the results are available for the mid-course review and RFP determination that would happen every five years, or so. The mid-course review and RFP demonstration would be a coordinated retrospective and prospective assessment of how well the controls are working and if they are on target for meeting attainment of the air quality goals as planned. This step is well laid out in the RFP paper. The key here is to coordinate the mid-couse review and RFP to maximize the information gained, and to make sure that the appropriate tools are available and reviewed to conduct this step. Outcomes of this step, or the initial data analysis stage, is to determine if the information available is sufficient to determine if emissions controls are working, if the modeling and emissions inventory process are able to be conducted adequately, or if there are unexpected changes to the domain as to need significantly more data than is available. If this is the case, a special study may be in order.

While following this recommendation may add a new level of effort in the short term, the improvements in our ability to identify effective control strategies should make the process more efficient, and will certainly add to confidence in the process. If coordinated with quantitiatve uncertainy analysis at both the initial modeling and data anlaysis stage and at the mid-course review, it is also possible to better assess where the major uncertainties lie, and their affect on determining how to judge the impact of controls.