STSWG Subgroup Revised Response to NRSWG Questions #1 and #2:

Reasonable Further Progress (RFP) Metrics

June 20, 1997

STSWG Subgroup Members:

Cyril Durrenberger	TNRCC

Steve Ziman	Chevron

U.S. EPA Support:

David Sanders

John Silvasi

Bob Stallings

The STSWG Subgroup on RFP Metrics recommends that the following response be offered to Sarah-Peirce Sandner's group working on the Reasonable Further Progress (RFP), with regard RFP Metrics. The paradigm presented here is based on the knowledge and experience gained from the analysis of ozone data; however, it is the belief of the STSWG membership that this same conceptual framework could also be successfully utilized for fine particulates.

Reasonable Further Progress (RFP) metrics should reflect an hierarchal approach, which would foster the development of an iterative approach. The focus is on a set of metrics, not a single metric, that utilizes the measurement of parameters that relate directly to the pollutant that is the subject of the State Implementation Plan (SIP). Tracking parameters that can be measured provide a superior way to establish if progress toward the air quality goals has been made. Some point source emissions can be tracked with continuous emissions monitors (CEMs) or parametric monitoring; but, similar direct measurement techniques are not available for most volatile organic compound (VOC) point sources, smaller NOx point sources (e.g. flares, internal combustion engines), on-road and off-road mobile sources and area sources. Thus, tracking estimates of emissions is in most cases not an exact measurement, is complicated by changes in the techniques used to estimate the emissions, and opens the door for manipulation of data and in the worst case game playing. Controls implemented in SIPs to control a pollutant can vary in different areas, so it would be difficult to develop a prescriptive approach to the RFP metrics. An integral part of the SIP is definition of the specific RFP metrics to be utilized to establish progress toward the standard. The "customized" SIP specific RFP process contemplates an iterative process where certain air quality-based metrics will be analyzed at pre-determined intervals to determine if the emission reduction programs are producing the desired improvements in ambient air quality. Implementing such an iterative process will allow states the flexibility of mid-course corrections/adjustments on critical techniques that are shown to be ineffective.

Certain metrics may be better-suited for prospective (planning) approaches, while other metrics may be best utilized under a retrospective (tracking) analysis. However, the use of these two approaches can not be treated in a stand alone fashion; both prospective and retrospective methods must be utilized to develop a definitive and quantifiable air quality management program. It is only through a prospective analysis that one can look at alternative strategies with the intent of optimizing for the metric. A retrospective application without any reference to how the emissions reductions were identified is incapable of telling us if we chose the best pathway to reduce the air quality metric. Additionally, it is possible to have different levels of success for the various metrics within a given air quality management region. Thus, an iterative approach will allow the individual states the option to develop a process which best meets their specific needs.

The planning aspect will define the future air quality targets based on modeling of the controls proposed in the SIP. Then the same metric, altered to utilize monitoring data, would be used to quantify the amount of progress has been made toward the air quality targets. If the air quality targets have not been met, then corrective action can be taken in the mid-course correction. This process can be continued until the target is attained, or it is determined that no further progress is economically or politically feasible. The process can also be utilized at well defined intervals in the future to insure that the air quality goals are maintained, and to make corrections if there is slippage from those goals.

The selection of one (or more) of these metrics relies on the strength of the underlying ambient monitoring network and the credibility of the numerical models chosen for the analyses. STSWG recognizes that the existing ambient monitoring networks currently operational (e.g., NAMS, SLAMS, PAMS, SPAMS, etc.) will not meet the data requirements of the proposed air quality metrics for some regions (perhaps most) of the United States. Additionally, STSWG understands the need for better model performance evaluation and most likely a finer model resolution than is currently accepted in EPA modeling guidance. The effort to refine model resolution and improve model performance evaluation should likewise lead to a revision in the foundational guidance for monitor network design and monitor siting practices. Thus, it is believed that through enhanced monitoring guidance and better modeling protocols, the proposed metrics will lead to improved air quality management practices for the United States. It must be noted that the requirements and caveats just expressed are equally applicable to the SIP process as it is now carried out, and will apply to future SIP processes. The RFP metrics do not of themselves, add a major additional requirement to the SIP process. It may require additional model simulations in order to look at alternative strategies, but a well thought our SIP modeling exercise should also include these simulations.

No one metric can be used to establish if progress to reduce the pollutant of concern has been made. The suite of RFP metrics can be divided into three classes: (1) air quality metrics; (2) administrative metrics; and, (3) emission metrics. The primary goal of the SIP is to attain the standard for the pollutant of concern, and air quality measurements are used to determine if the standard has been met. The metric used to make this determination is the peak value concentration air quality metric. If the emission reductions described in the SIP are attained, there is no guarantee that the standard will be attained. These reductions were implemented because they were either specifically mandated by the Clean Air Act, mandated by other rules (HON, NESHAPS, etc.), or they were developed as a result of modeling that showed that they would be effective in reducing the pollutant of concern. Therefore, the best approach to establishing if progress has been made is to utilize all three classes of metrics. The final strategy and its federally enforceable individual control measures (and hence, emissions reductions) will be a result of integrating all of the information gained by analysis of each of the proposed metrics. It is this integration which will define the most cost effective path to improvement in air quality, as one moves toward attainment.

Administrative Emissions Metrics

In many cases it is not possible to establish trends based on measurement; but, it may be possible to determine administratively if reductions can be expected. This metric is very "soft" and should be used with care. Its best use is to determine if the state has made "good faith" efforts to implement the control programs described in the SIP. Specific details of this metric will be SIP specific, but some guiding principles for these weight of evidence tests are:

(a).	Were the rules to control the sources adopted?

(b).	Were the rules adopted in a timely fashion?

(c).	Were the subject sources inspected for compliance with the rules?

I.	How many sources were subject to the rules?

ii.	How many sources were inspected?

iii.	What was the frequency of the inspections?

iv.	How many of the inspected sources were in compliance with the rules?

(d).	What impact did population growth and/or shifts have?

Planning - The planning aspect involves careful development of rules to implement control strategies so that appropriate record and tracking mechanisms are established.

Tracking - The tacking aspect involves careful development of appropriate parameters to track source performance.

Emission Metrics

In many cases it is difficult to obtain direct measurements that can be used to indicate the performance of control programs. The first, and most important source reduction metric used should determine if precursor emissions have been reduced based on air quality monitoring. The ultimate goal of this approach is to optimize the strategy that best meets the air quality metrics that have been chosen, as a locale moves toward attainment. Again, it is necessary to state that, even though these are not the recommended primary metrics, the SIP will contain federally enforceable control measures and the timing of implementation of those control measures. The selection of the measures and the order of implementation of those measures needed to attain the standard will be determined by use of the air quality metric. The following principles apply:

(1) Direct Emission Metrics - This metric applies to specific point sources whose emissions can be directly determined. The best method is to utilize continuous emissions monitors (CEMs), but this data has to be processed. Some of the reporting protocols require that maximum concentration data be inserted when the CEM is not functioning, or other modifications of the data, including times for calibration. Important aspects to consider in processing the CEM data to determine source emission trends and compliance with control measures are:

(a).	Identification of times when the source was not operating

(b).	Identification of times when there was an upset, start up, or shut-down in progress.

(c).	Identification of times when the CEM was not functioning.

(d).	Seasonal or annual period.

(e).	After controls have been applied and proved to be functioning as intended, a minimum of one whole analysis period (season, year) will be necessary to establish a trend. Similarly, a minimum of one analysis period will be needed to establish a base from which to evaluate reductions.

(f).	Parametric monitoring may be used. This involves the determination of emissions based on monitoring a set of parametric parameters. The parameters selected will be based on scientific analysis of the process and the exact relationships of the parameters to the emissions will be established for each specific source empirically using extensive test data and operations research methods.

Planning - The amount of reduction expected from a control program can be estimated during the planning stages.

Tracking - When applying a source control to a specific source the actual reductions may vary depending on source configurations and parameters. These variations will have to be accounted for during the use of this metric.

(2) Indirect Emissions Metrics - There are many sources for which it is not practical or possible to install CEMs or establish parametric monitoring programs. Examples are area sources (e.g., gasoline marketing, consumer products, surface coating (paint), etc.) and mobile sources. This type of metric applies to sources whose emissions cannot be directly determined, but that can be estimated based on other parameters that can be directly measured. These metrics will be specific to the source category being controlled and in some cases specific to the nature of the rule to control the emissions. These metrics are SIP specific and must be described within the SIP. In most cases, record keeping or data gathering requirements of this metric will also have to be specifically defined in SIP. The very nature of this metric will require that only generalized guidelines can be provided. This metric is "soft" and should be used with care. Below are listed some examples of indirect emission metrics.

(a).	Surface Coating (e.g., painting)

(1)	The type and amount of solvents used in the coating operations such as auto refinishing can be determined. The amount of solvents in the coating that will evaporate into the atmosphere can be determined. Then the amount and type of compounds emitted can be determined. These values tend to be averages over the time period for which the records were kept.

(2)	For consumer surface coating products (e.g., house paint), economic information on sales of paint products can be used to evaluate the necessary reductions.

(b).	Inspection and Maintenance (I&M)

(1)	Records from the I&M inspections can be used to determine the amount of pollutant reduced by the detection and correction of excess emissions from automobiles.

(2)	Carefully planned sampling of the automobile population with remote sensing instruments can be used to estimate mobile source reductions.

(3)	Tunnel studies and other source reconciliation evaluations with Photochemical Assessment Monitoring Sites (PAMS) and special monitoring can be used to estimate mobile source reductions.

(3) Air Quality Emission Metrics - One approach to emission metrics is to determine from air quality data if emission reductions have been made. The advantage of this metric is that it is based on actual measurements and measurements in the atmosphere where, if applicable, the reactions occur. The disadvantage is that atmospheric chemical reactions can occur rapidly and not provide accurate information on emissions trends. Also air quality measurements may be subject to permutations caused by meteorology and random error. The exact pollutants to track and the value to track are SIP specific. This metric must be used with care and used in conjunction with the direct emissions metric and the administrative emissions metric. The following are aspects that should be considered:

(1)	The expected trend (usually a target percentage reduction in the atmosphere that results from the application of controls) is established by modeling to determine the response to the metric.

(2)	Trends are based on a minimum of 5-6 consecutive years of data.

(3)	The trends are established by first removing perturbations caused by variable meteorological conditions. Seasonal data may be required. For example, some fuel programs are implemented only during summer months.

(4)	The actual trend is established from air quality monitoring to determine if the expected trend has been met.

The air quality metrics depend upon ambient air quality measurements. These metrics are recommended based on the existing ozone monitoring networks that are in place around the country; however, implementation of these metrics for other pollutants (e.g., fine particulates) would require a significant modification or expansion of the existing monitoring networks and associated monitoring guidance. It is recognized that in some locations the lack of monitoring data will make it impossible to apply some these metrics. However, if there is sufficient monitoring data to declare the area nonattainment (or to establish an area of violation), there should be enough data to apply at least one of the air quality metrics.

The air quality measurements can be subject to variation in meteorology and must be statistically sound (remove random noise).

(1)	Establishment of a Trend - The STSWG RFP Subgroup unanimously agrees that one year of air quality data is insufficient for establishing a trend, and for making judgments about the efficacy of any emissions reduction program. The Subgroup recommends that a minimum of 5-6 years of ambient air quality data be used to establish such a trend. However, trend analysis can be performed on an annual basis once enough data is collected.

(2)	Statistical Descriptions of Air Quality - The STSWG RFP Subgroup unanimously recommends that the use statistical techniques (e.g., the t-test methodology of David Chock) in the processing of air quality data to be used in the air quality metrics.

Air Quality Metrics

Each of the following metrics measures different aspects of progress toward attainment of the standard, and no one is sufficient to establish if real progress in reducing the air pollutant of concern has been made. The best approach is to use all three of the following metrics:

(1) Peak Value Concentrations - This metric is useful, but only addresses changes to the maximum concentration. It can not measure the reductions obtained over a large area. It is possible to have significant decrease in the frequency for which the standard is exceeded and the area over which the standard is exceeded, but not have a change in the peak value. This metric does relate to the design value as used in the existing Clean Air Act. This metric should not be used alone, but in conjunction with the areal exposure and/or population-weighted exposure metrics. It does, however, allow an understanding of whether there are hot spots, so that these hot spots are not ignored when looking at the population weighted metric, as it is not the intent of that metric to minimize that concern.

Planning - The maximum concentration modeled over the domain will be used to determine this metric. It is possible that this value is significantly higher that the highest concentration measured, because a monitor may not be located in the vicinity of this peak value. In many cases the location of the maximum concentration shifts when controls are modeled. Since it is not now possible to model a large number of episodes, the maximum concentration over the modeled days is the only practical value for this metric. However, these results can be placed in perspective with weight of evidence analyses.

Tracking - The maximum concentration, second high, or some other value from the set of monitors that cover the domain can be used for this metric. As with tracking this metric may not indicate that significant progress has been made since the location of this value can change.

This is a poor metric to use by itself for planning. Many effective control programs show extensive improvement in pollutant exposure over a large area, but may not reduce the maximum substantially. Conversely some control programs reduce the maximum in the domain, but increase the exposure of the population to the pollutant over a large area. Such an approach may not improve public health.

PROS If the extreme site meets the standard, other sites in the area will also meet the standard; consistent with the form of the standard. CONS Poor utilization of data; inherent large fluctuations in the metric; not representative of the outdoor exposure of the public at large; a reduction in concentration at the extreme site may necessitate an increase in exposure in the overall population. (This possibility must be weighed against the current notion that ozone and PM health effects have no threshold.

(2) Areal Exposure - This approach should be used if the population weighted exposure is not used, or as an addendum to the population weighting approach. It includes both spatial and temporal variability in ambient air quality data. It is best used when a target level is established and the variation above the target level is determined.

Planning. The metric values would be determined for each grid cell for each day modeled. For each grid cell and for each hour when the concentration exceeds the target concentration, the target concentration is subtracted from the pollutant concentration, then these differences summed for each hour of the day.

Tracking. The metric values would be determined for each monitor for each day being analyzed. For each monitor and for each hour when the monitored concentration exceeds the target concentration, the target concentration is subtracted from the monitored concentration, then these differences summed for each hour of the day. The limiting factor for this method is that the monitoring network be sufficiently dense enough to establish a reasonable measure of the areal exposure. For the details of this methodology, please refer to the following discussion on the population-weighted exposure metric.

Areal Exposure has been utilized in planning analyses in Texas and is currently being evaluated as a prospective tool in California. A modification of this metric has been utilized extensively in the Ozone Transport Assessment Group (OTAG) process.

PROS Better utilization of data; statistically more stable; better representation of overall ambient air quality in an area; better characterization of potential dosage above a given threshold concentration/level. CONS Different from the form of the standard; A reduction in the areal exposure metric does not guarantee a uniform reduction in exposure or dosage in the area.

(3) Population-Weighted Exposure - This approach would be the preferred approach, given that a region had sufficient ambient monitoring information to support such a methodology. This tool is preferable in that it allows the design value to be tracked, while also tracking exposure above a certain level. This methodology allows for the resolution of both the temporal and spatial characteristics of the surface pollutant concentrations coupled with the extent and magnitude of population exposure. It should be noted that this approach mimics the approach that EPA used in their analysis of alternative ozone standards when preparing their health risk assessment.

Planning. The metric values would be determined for each grid cell for each day modeled. For each grid cell, the population in the cell would be multiplied by the hourly pollutant concentration above the standard(s), then summed for each hour of the day.

Tracking. The metric values would be determined for each monitor for each day being analyzed. For each monitor the population within the monitors area of influence would be multiplied by the hourly pollutant concentration above the standard, then summed for each hour of the day. The area of influence for each monitor would have to be carefully determined and the areas for two monitors could not overlap. The limiting factor for this method is that the monitoring network be sufficiently dense enough that most of the population lives within a reasonable definition of monitor area of influence. For example, a good criteria would be that 90 percent of the population live within 15 kilometers of some monitor. Also, one monitor should not cover too large a fraction of the population. For example, a good criteria would be that no more than 10 percent of the population live in the area of influence of one monitor. Specific monitoring guidance would be needed to develop the network design; methodologies to develop this guidance exist.

This metric has been used for planning in Texas and has been used for tracking in California. The metric is recommended as part of modeling analysis by the CARB in their guidance on photochemical model use. This metric is very valuable in developing a control program that will minimize exposure of the population to the pollutant being addressed in the SIP. It should be noted that there is clear recognition that this metric, as used in a prospective sense, can not be applied to all future exceedances. However, EPA's SIP modeling guidance requires states to focus on those episodes which are typical of the episodes experienced in an area, and recommends modeling for more than one episode. At the present time, major regulatory decisions on the measures to be included in the SIP on made on this basis of limited modeling. This metric would only change the mechanism by which the individual measures are selected and the timing for implementation, as it would emphasize measures which would try to optimize ozone exposure on the way to attainment. The emissions reductions measures chosen this way would be included in the SIP and be federally enforceable. Thus, there is still the direct link to emissions reductions and attainment.

PROS Better utilization of data; statistically more stable; better representation of outdoor exposure of public at large; an improvement in the metric means a reduction in overall population exposure; and, can be made consistent with the form of the standard with the daily max being weighted by the population distribution. CONS A reduction in the population-weighted exposure doe s not guarantee a reduction in concentrations at the extreme site.

EXAMPLES

Example 1:	The ozone control strategy is to only reduce NOx. The SIP would identify the sources of NOx to be reduced and the amount of NOx that would be reduced. Modeling would identify the level of ozone to be reached and indicate the level of NOx that would be expected at each monitoring station. For the time period to when attainment was required, primary tracking would be performed for ozone based on monitoring data using the Population-Weighted Metrics, Areal Exposure, and Peak Value Air Quality. If the expected reductions were not attained, then NOx emissions reductions be would be tracked with the Direct Emissions, Administrative Emissions, and Air Quality Emissions Metrics where applicable. If it appeared that the expected NOx reductions had been made and these reductions could not be identified in the atmosphere, then further study would be warranted to determine a mid-course correction.

Example 2:	The ozone control strategy includes reduction of VOC. The SIP would identify the sources of VOC to be reduced and this could be used to identify reductions of specific compounds that can be measured in the ambient air. As in Example 1 above, modeling would be performed to identify the amount of reduction of ozone required and the amount of specific compounds that would be expected to be reduced. Progress would be tracked with the applicable Air Quality and Emission Metrics. For example, acetylene could be used to track mobile source exhaust emissions. The tracking would be performed as above, except that the data would come from the Photochemical Assessment Monitoring Station (PAMS) network. If the Air Quality metrics did not show the expected improvements, then the Emission Metrics could be used. If the Administrative Emission Metric indicated that the program should show benefits, but the Air Quality Emissions Metric did not show improvement, then further study would be necessary. It could be that the emissions calculation methods were inaccurate or that there were problems with the design of the program.

Example 3:	The PM fine control strategy is to reduce sulfates by reducing only SO2. Modeling would be performed to indicate the expected levels of PM-fine, sulfates, and SO2. The levels of PM-fine, sulfates, and SO2 would be tracked with Air Quality and Emissions Metrics as indicated in Example 1. If it was found that the levels of sulfates and SO2 matched the expected levels, but the PM-fine level did not meet the expected level, then a mid-course correction would be needed to determine the additional controls needed to attain the standard.

Comment Submitted by Bruce Hill (STSWG Member):

In the process of setting up 5 PM/ozone and visibility monitoring sites in NH, I have not had adequate time to address the metrics proposed in the May 28 draft in a comprehensive way. However, I have read the draft and wish to squeeze in a few comments prior to its submission to NRSWG. I hope that there will be opportunity at the June FACA STSWG meeting to more fully understand the proposed options and to comment on them more substantively. You may distribute to STSWG.

(1)	All of these RFP methods should be secondary to emissions reductions: progress in achieving real, permanent, quantifiable and enforceable emissions reductions. Secondary metrics might be useful in determining if partially implemented control strategies are effective. I have still not had an adequate response to the question I have raised at FACA and in the work groups about why current attainment demonstrations do not already provide this function, especially given the 5-6 year intervals are called for in establishing a trend (p.2).

(2)	It seems to me that the population based exposure metric tells you little or nothing about progress in attaining the NAAQS design value. Moreover, the method depends on models that, as I am led to believe, do not have adequate precision to accomplish the task, especially in areas with inhomogeneous concentration contours. And I wonder: is this method much closer to going after the real "prize"--measurable change in actual air quality-- if it utilizes concentrations cooked up by modeling? Another problem is that the population-based method is also self- described as invalid in rural areas like much of New England. The method would systematically give more weight to progress in urban areas and less in rural areas like New Hampshire, Maine and Class-I areas (where I monitor) that have less (or no) population but nasty air. As such, it would not seem to be useable as a prescribed national RFP metric. Finally, the method would "incentivize" gaming in monitor network design.

(3)	What happened to the Zurbenko/Rao method of tracking increments of change in baseline concentrations? This, would, at least have the advantage of using monitoring data.

(4)	Do any of these metrics address the simple question of tracking statistical change in the design value at each monitoring site?