MODTOMON Full Paper

I. Introduction

This document describes the results of a model-to-monitor comparison we conducted for a subset of the 33 urban HAPs.  We view this comparison as a evaluation of not only ASPEN, but also its inputs: emissions data from various sources including the National Toxics Inventory (NTI), the Emissions Modeling System for Hazardous Air Pollutants (EMS-HAP), and meteorological data.

For most of the pollutants examined, we found that the model's estimates tend to be lower than the monitor averages at the exact locations of the monitors.  However, for some HAPs, there are usually high modeled concentrations in the vicinity of the monitor.  So it may be presumptuous to say that the modeling system underestimates the monitors.  It may just be finding a peak concentration in a different place from the monitor.

However, in general, it appears that the modeling system is underestimating the monitors.  We discuss some possible reasons for this.  The National Toxics Inventory (NTI) is missing some emissions sources, and for many of the sources in the NTI, some of the emissions parameters are defaulted or missing.  It is also possible that emissions rates are being underestimated in many places.  We believe the ASPEN model itself is contributing in only a minor way to the underestimation.  We think this because output from the antecedents of the ASPEN model compared favorably to monitoring data in cases where the emissions and meteorology were accurately characterized and the monitors took more frequent readings; and in simulations we ran, the ASPEN model's estimates compared favorably to the estimates derived from a more meticulous model.  Monitor siting may have contributed to the underestimation, in that they are normally sited to find peak pollutant concentrations, which implies that errors in the characterization of sources would tend to make the model underestimate the monitors.  Finally, we are not sure of the veracity of the monitor averages, which have their own sources of uncertainty.

Our results suggest that the model estimates are uncertain on a local scale; i.e., at the census tract level.  We believe that the model estimates are more reliably interpreted as being a value likely to be found within 30 km of the census tract location.

These results are consistent with the results of an evaluation of 1990 ASPEN output,¹ which was conducted as part of the Cumulative Exposure Project (CEP).  There are many differences between the evaluation for the 1990 assessment and the 1996 assessment–different evaluation methods, different emissions inventory, different list of pollutants assessed–but most of the model estimates there, as here, were lower than the monitor averages.

Because of all the uncertainties involved, we focus a good portion of our energies in this document on assessing the uncertainties.

II. Raw Materials

A. List Of Pollutants.

1. they are a subset of the 34 air toxics considered in the national-scale assessment;
2. they represent a range of pollutant types (i.e., organic, volatile, particulate matter);
3. they are influenced by different combinations of mobile, area and point sources, and include reactive and non-reactive compounds; and
4. EPA considers the available 1996 monitoring data for these pollutants to be adequate for a national-scale evaluation.

B. Monitoring Data.

i) Completeness check and computation of annual averages.

1.  A measurement was considered below the method detection limit (MDL) if either it is indicated as being below the MDL (e.g., value of –888 in the data archive), or if it was specified with a numeric value that is lower than the reported MDL value for that pollutant/monitor/time combination.  If there was no reported MDL, the lowest reported value for the pollutant/monitor combination was assumed to be the “plausible MDL”.  Measurements below the actual or plausible MDL were assigned one of three values depending on the averaging statistic being calculated:

1. One-half the value of the actual or plausible MDL
2. The value of the actual or plausible MDL
3. Zero

2.  For each pollutant/monitor combination, an annual average was calculated stepwise from temporal averages of shorter durations, as indicated below.  At each step the data set was assessed for completeness and retained for further processing only if the completeness criteria were met for the given averaging period.

    a.  Daily average.  A day was considered complete if the total number of hours monitored for that day is 18 or more (i.e., 75 % of 24 hours).  For example, 18 hourly averages, 3 six-hour averages, or 3 eight-hour averages would satisfy the daily completeness criteria.

    b.  Quarterly average.  Calendar quarters are Winter (Jan-March), Spring (April-June), Summer (July-Sept), and Fall (Oct-Dec).  A calendar quarter was considered complete if it has 75 % or more complete days out of the expected number of daily samples for that quarter, and if there were at least 5 complete days in the quarter.  To determine the expected number of daily samples, the most frequently occurring sampling interval (days from one sample to the next sample) was used; in cases of ties, the minimum sampling interval was applied.

    c. Seasonal average.  The seasons are composed of 2 quarters: Winter/Fall and Spring/Summer.  A season was considered complete if it had at least 1 complete quarter.

    d. Annual average.  An annual data set was considered complete if it had 2 complete seasons.

ii) MDL.

For these low concentration sites, we do not have much confidence in the monitored annual averages, because of the uncertainty introduced when replacing values below MDL with one-half the MDL.  Because of this uncertainty, only pollutant/monitor combinations with at least 50% of data above the MDL were included in the comparison.  In the CEP model-to-monitor comparison,¹ only pollutant/monitor combinations with at least 90% of data above the MDL were included.  The cutoff percentage we chose was a compromise between ensuring high quality monitoring data and ensuring that our sample size was large enough to compute meaningful statistics.  In general, we believe that monitored averages for pollutant/monitor combinations which have more than half their observations below the MDL are too uncertain to be used in a model-to-monitor comparison.

iii) Flagging.

• We have reason to doubt the accuracy of the monitoring average.
• The site is very close to an international border.
• We know that a source in the vicinity of the monitoring site is missing from the emissions inventory.

b) Border Sites.  Some of the monitoring sites are very close to the US-Mexico border or the US-Canada border.  Because we are not using emissions data for Mexico or Canada, we do not have confidence in the model estimates for these sites.  This is especially true of sites in Calexico, CA; El Paso, TX; Brownsville, TX; and Bellingham, WA.  These sites are close to large cities on the other side of the US border.

Even though we discarded these sites from the comparison, it should be noted that we are using model estimates at census tracts along the border in NATA.  In preliminary runs, the model estimates were much lower than the monitor averages at these sites for all pollutants.  The model estimates will almost certainly be low for census tracts near international borders, especially when large sources are located on the other side of the border.  The absence of emissions data from Mexico and Canada is a weakness of NATA.

c) Known Missing Sources.  In a few cases, by the end of the study, we discovered that a source near a monitor had been missing from the initial emissions inventory.  We discarded these pollutant/monitor combinations because they skewed our results (the model estimate is near zero, but the monitor average is significant) and because they do not provide a meaningful test of model performance.  We will add these missing sources to the NTI for the future.  Only a few pollutant/monitor combinations were discarded for this reason.

Future evaluations of the ASPEN modeling system should not exclude sites along the border or sites associated with missing emissions sources, in order to better evaluate the performance of the modeling system in its entirety.  We recommend including these sites in the analyses and explaining any disagreement between model estimates and monitor averages, instead of discarding them.

C. Emissions Inventory.

i) NTI.

ii) EMS-HAP.

Temporally, EMS-HAP allocates the annual total emissions into eight three-hour periods within an annually averaged day.  Each day of the year is allocated the same emissions.

EMS-HAP spatially allocates non-point, county-level emissions to the census tracts within each county, as required by the ASPEN model.  EMS-HAP allocates these emissions based on surrogates such as population, land use, roadway miles, etc., depending on the source category.  For point source emissions, spatial allocation is not performed because most point sources in the NTI already have exact latitude/longitude coordinates.  When point source geographic coordinates are missing, EMS-HAP defaults them where possible.  For example, if the geographic coordinates for a point source are missing but the zip code is not, EMS-HAP assigns the lat/lon coordinates of the zip code's centroid to the source.  If the zip code is missing but the county is not, EMS-HAP assigns to the source the lat/lon coordinates of a census tract centroid chosen randomly from the county.

We will discuss some of EMS-HAP's allocation and defaulting techniques in the uncertainties section of this document.

D. Model Results.

i) Estimates at monitoring sites.

ii) Estimates at census tract centroids.

III. Model-to-Monitor Comparison Analysis Methods

A. Graphical.

i) Scatter plots.

ii) Ratio box plots.

Ratio box plots show the same data in the scatter plots, in a different way.  Each box shows the distribution of model-to-monitor ratios.  So if we have 50 monitors for perchloroethylene, we will have 50 model/monitor ratios to compute.  We then compute percentiles and the mean of these 50 numbers, and create a box plot.

The plots will show the median, 25th, and 75th percentiles of the ratios.  If the model is performing well, the box plots will be short, and centered at around 1.

We decided not to show more extreme percentiles (e.g., 10^th and 90^th) of the ratios because often the extreme percentiles were far from the center of the distribution.  This is an unfortunate property of ratios.  When the denominator is near zero, the ratio can be enormous.  In our study, sometimes model estimates of near zero are paired with significant monitor averages, due to missing sources or sources with defaulted or incorrect locations.  These result in ratios near zero, which skew the distribution of the ratios.  Less often, we'll see high model estimates paired with low monitor averages.  But because ratios take extreme values so often, we decided to only show the interquartile range of the distribution.

There is an additional reason for leaving out the extreme percentiles.  Given values with large uncertainty, high percentiles tend to be biased high and low values tend to be biased low.⁴  This is because the values with large positive errors will collect in the high end of the distribution, and values with large negative errors will collect in the low end of the distribution.  The ratios are fraught with uncertainties of all kinds.

We decided to use a logarithmic scale for the vertical axis, for the following reason.  If we use a regular arithmetic scale on the vertical axis, then a ratio of 2 is twice as far from 1 as a ratio of 1/2.  But as said before, modelers typically speak of estimates as "within a factor of x."  An underestimate by a factor of x should look just as erroneous as an overestimate by a factor of x.  The logarithmic scale makes the overestimation and underestimation the same distance from the horizontal line where the ratio is 1.

The ratio box plots in this document will be shown side-by-side, one for each pollutant.  This will allow us to see easily which HAPs are being overestimated and underestimated, and which are being estimated consistently and inconsistently.

B. Statistical.

i) Number of sites.

ii) Median of ratios.

iii) Percent of sites estimated "within a factor of x".

iv) MAXTOMON.

This technique compares the MAXimum model estimate within r kilometers of the monitor TO the MONitor average.  All model estimates are considered in finding the maximum–both estimates at monitor sites and estimates at census tract centroids (see section II.D above).  This is an example of a point-to-range tool.  We use this tool to test whether the frequent underestimation by the model at monitoring sites was due to location uncertainties, or due to systematic underestimation.  To explain further, let's say the model estimate for a certain pollutant/monitor combination was much lower than the monitor average.  We might wonder if the model predicted a concentration similar to the monitor average anywhere near the monitor site.  If it did, it is possible that the underestimation at the exact monitor site was due to uncertainties in the inputs to the model, especially in source locations, instead of due to systematic underestimation by the model.

In general, we hope to see very few monitor averages underestimated by the model as r gets large.  One weakness of the MAXTOMON test is that there is a sparser network of model receptors in rural areas than in urban areas, because the census tracts are larger.  Thus, the MAXTOMON test might have more difficulty finding a peak concentration in a rural area than an urban area.

IV. Uncertainties

A. History Of Model-to-Monitor Comparisons With ASPEN.

Pooler⁷ described one of the first attempts to employ numerical methods for automating the computations (an IBM 650 computer) to provide estimates of monthly average concentration values for comparison with observations of sulphur dioxide (SO₂) collected daily from November 1958 through March 1959 at 123 sampling sites in Nashville, TN.  We have to temper the evaluation results, as regressions were performed with the observed concentration values to provide best estimates of the variation of the monthly emission rates from the known sources.  When we digitized Pooler's data for reanalysis we only found 122 data values, not 123.  That said, the model overestimated the observed values by a factor of 1.37 with a correlation coefficient (r²) of 0.95.  (Note: In all descriptions of past model-to-monitor comparisons in this section, the reported factor of over or under estimation and correlation coefficient (r²) were deduced through a linear regression with the intercept forced to be at the origin.)  We found 110 of the 122 values within a factor of 2 of the observed values, with 74 of the 122 values within 30%.

Martin⁸ summarized the dispersion model used to numerically compute (an IBM 1130) winter season estimates of the average SO₂ concentration values for comparisons with observations collected daily from December 1964 through February 1965 at 40 sites in the St. Louis area.  Removing comparison results for the five most suspect locations reduced the overestimation to 1.47 and increased the correlation coefficient to 0.95.  Thirty-four of the 35 values were within a factor of 2, with 14 within 30%.  A reanalysis of these same data was performed by Calder⁹ using the Climatological Dispersion (CDM) model, using a revised characterization of the area source emissions by Turner and Edmisten.¹⁰  A major enhancement within the CDM over the model employed by Martin was to incorporate an improved treatment for characterizing dispersion from area sources, employing an algorithm based on the narrow plume hypothesis.  In spite of the attempts to improve the characterization of area source emissions and the dispersion from these low-level sources, the comparison results were similar to those achieved by Martin.  The CDM tended to overpredict concentration values by a factor 1.54 with a correlation coefficient (r²) equal to 0.92.  Thirty-five of 39 estimates where within a factor of 2, with 17 within 30%.  Possible factors contributing to the tendency to overestimate the observed concentration values were: an inherently crude emissions inventory, no day versus night variation in emission rates, and the crude estimates of mixing height employed.

Turner et al.¹¹ summarized the results obtained in applying the CDM model to estimate annual average particulate and SO₂ concentration values for the New York area for 1969.  SO₂ observations were available for comparison at 75 locations and total suspended particulate matter observations were available for comparison at 113 locations.  This version of the CDM employed the Briggs¹² plume rise algorithms (in contrast to use of Holland¹³ algorithms used by Martin and Calder in the St. Louis comparisons).  For SO₂ it appears the CDM tended to slightly overpredict concentration values by a factor of 1.11 with a correlation coefficient (r²) equal to 0.90.  Seventy-one of the 75 values were within a factor of 2, with 47 values within 30%.  For particulates it appears the CDM tended to slightly underpredict concentration values by a factor of 0.93 with a correlation coefficient (r²) equal to 0.94.  111 of the 113 values were within a factor of 2, with 94 within 30%.

Irwin and Brown¹⁴ summarized the results obtained in applying the CDM model to estimate 1976 annual average SO₂ concentration values for the St. Louis area.  There were 13 sites, but omission of a lead smelter near one site precluded use of data at two sites for model performance comparisons.  The emission inventory and monitoring results were obtained as part of the St. Louis Regional Air Pollution Study.  These simulations differ with those computed by Turner et al. in that urban dispersion parameters (based on tracer studies conducted in St. Louis, McElroy and Pooler¹⁵ and Gifford.¹⁶  It was determined that although the area source emissions constituted only 3.5% of the total area and point source emissions, estimated concentrations from area sources ranged from 14 to 67% of the total concentration estimated at the monitoring sites.  For the 11 sites it was found that CDM slightly overpredicted concentration values by a factor of 1.10 with a correlation coefficient (r²) equal to 0.96.  Nine of the 11 sites have estimates within a factor of 2, with 3 values within 30% of those observed.  This same inventory was simulated using the RAM model,¹⁷ which employed hourly specification of the meteorology and the emissions.  For the 11 sites it was found that RAM slightly overpredicted concentration values by a factor of 1.10 with a correlation coefficient (r²) 0.96.  For the RAM estimates, all 11 sites had estimates within a factor of 2, with 10 values within 30% of those observed.

The version of CDM applied by Irwin and Brown is similar to the Industrial Source Complex (ISCLT) Long-Term model.¹⁸  The major difference in ISCLT versus CDM is that the area source algorithm is better than that employed in CDM.  ISCLT's area source dispersion characterization nearly approximates what is obtained when one computes area source impacts using an hour-by-hour simulation (which employs a double integral over the area and hence is our best expression of dispersion from an area).  The emphasis on improving the treatment of area source impacts reflects the recognition that area source emissions (if present) often account for a major portion of the simulated impacts, as discussed in the previous paragraph.

In the studies summarized in the table below, it is important to remember that the long-term models have evolved, with the adoption of improved characterizations for the dispersion and for treatment of area sources.  Except for the simulations for Nashville by Pooler and for St. Louis by Martin and Calder, the average bias has been slight.  The CDM versus RAM comparisons offer an interesting clue that perhaps the time variation of the emission rates (which Calder was the first to offer as a major concern) is of importance, although there are other differences between CDM and RAM that also might offer explanation of the differences seen.  Regardless of the model employed, the estimates are generally within a factor of 2 of those observed.

B. Uncertainties Affecting Our Comparison.

i) Monitoring uncertainties.

a) Temporal gaps.  Most of the 1996 data in the Archive were not collected every day.  Instead, they were produced every 12th day or every 6th day throughout the calendar year.  Our completeness criteria discussed in section II.B.i above filter out pollutant/monitor combinations which have many missing sample days, but they do not filter out pollutant/monitor combinations based on the number of missing calendar days.  Our only requirement for the calendar days is that we have at least five days in either the winter or fall quarter and in either the spring or summer quarter.  This is not a very rigorous standard.  The temporal gaps are especially problematic for pollutant/monitor combinations which have widely varying concentrations throughout the year.  This might be true for pollutant/monitor combinations where the weekday and weekend concentrations are very different, or those near sources whose emissions vary widely from day to day.

b) Spatial gaps.  For some of the pollutants in this study (section II.A above), we have very few sites across the country.  The table below shows the number of monitors and the number of states covered for each pollutant.

We only have 20 complete monitors for cadmium, 13 of which fall in Illinois and New York.  Acetaldehyde and formaldehyde have exactly the same set of 32 monitors, 28 of which fall in the northeastern and Great Lakes states.  Of the 36 chromium monitors, 32 are in California, New York, and Illinois.  All 10 chromium monitors in New York are in Staten Island.  The situation for benzene and lead is much better, as the table shows.  In general, the southeastern, northwestern, Great Plains, and Rocky Mountain states are very sparsely monitored.  The northeastern and mid-Atlantic states, Great Lakes states, California, Texas, and Louisiana are fairly well monitored.

c) MDL.  As discussed in section II.B.ii above, values below MDL add uncertainty to the annual averages.  We dealt with this by eliminating from the comparison all pollutant/monitor combinations for which less than 50% of the daily observations were above the MDL.  Still, many of the pollutant/monitor combinations have barely more than 50% of daily observations above the MDL.  Of the 493 pollutant/monitor combinations, 36 (7.3%) have between 50% and 60% of daily observations above the MDL.  Chromium is the most uncertain pollutant in this respect: 13 of the 36 monitors (36%) have less than 60% above the MDL.

d) Monitor siting.  The monitors used in the study were located considering a wide variety of objectives.  Some were placed near sources, to monitor sites with high pollutant concentrations.  Others were located in residential areas, intended to measure more typical exposures.  Still others were located in rural areas, in order to find a background concentration.  A few were run for special studies.  The Archive has information on siting objectives for some of the monitors, but not all: 179 (36%) of the 493 pollutant/monitor combinations used in this comparison had missing site objective codes.  Also, there are no specific guidelines for when an air toxics monitor should be classified as "source-oriented", "population-oriented", "background-oriented", etc., so even for those monitors with siting objective information in the Archive, we cannot be sure of the quality or consistency of such data.
    Generally, most monitors are sited to find peak concentrations.  We use this assumption in the interpretation of the MAXTOMON results.

e) Location.  How trustworthy are the exact (latitude,longitude) coordinates of the monitors used in this comparison?  This is difficult to assess.  One way to assess this is to see how often the monitor's lat/lon coordinates fall in a different county from the one reported in the Archive.  Another is to take a Global Positioning System (GPS) out to some monitors and check its readings with the values in the Archive.
    We performed the first test using a Geographic Information System (GIS) software package and the US counties coverage which is included with the software.  Of the 493 pollutant/monitor combinations, only one has lat/lon coordinates which fall more than 5 km from the reported county.  This monitor is in Louisiana.  We didn't look at monitors which are outside of the reported county by less than 5 km, because these discrepancies are likely due to lack of geographic detail in the counties coverage rather than erroneous lat/lon coordinates.  The one in Louisiana only missed by 7.5 km, so even that one might be due to lack of detail in the coverage, rather than erroneous lat/lon coordinates.
    We performed the second test using three lead monitors near a source in Herculaneum, Missouri.  Mick Daye, an EPA modeler from Region 7, took GPS readings at each monitor.  We compared Mick's coordinates to those in the Archive, and here is what we found:

These are significant differences for a model-to-monitor comparison, which we will see in the emissions uncertainties section.  Since most lead emissions come from widely spaced industrial plants, with little contribution from well dispersed sources, there are often steep concentration gradients near a source.  A difference of 50 meters might not have much of an effect, but a difference of 170 meters would.  We will discuss this further in the emissions uncertainties section: it is very important to get the source-receptor geometry correct when using a model to estimate concentrations near an isolated point source.

ii) Emissions uncertainties.

a) Point sources.  Of the pollutants we investigated, the point source contribution to the modeled concentrations is highest for the three metals.  Thus, our investigation of point source uncertainties will focus on the metals.  Some of the emissions data variables that are important for accurate modeling are location, release height, and emissions rate.  We discuss uncertainties surrounding the first two variables below.

1) Location.  We will now give a brief explanation of the way EMS-HAP processes the point source location data from the NTI.   It does this in two major ways.  First, if the geographic coordinates of the source are missing or out of range (i.e., outside the U.S., Puerto Rico, and Virgin Islands), it places the source at the centroid of the zip code if that is nonmissing.  If the zip code is also missing, it places the source at a census tract centroid chosen randomly from the county.  Second, for the sources with nonmissing geographic coordinates, it checks if the lat/lon coordinates fall within the reported county.  If not, but the coordinates are within a distance of 5.4 times the county radius (the county is approximated as a circle of area equal to the county area) from the county centroid, EMS-HAP leaves the lat/lon coordinates as is.  If the coordinates are outside this circle, EMS-HAP will move the source to the zip code centroid if available.  If not available, it will move the point to a randomly selected census tract centroid in the reported county.  For more detail, please consult the EMS-HAP user's guide.

There are two ways we have assessed the location uncertainties of the point sources used in ASPEN.  First, we found out which source lat/lon coordinates fall more than 5 km outside their reported counties, just as we did for the monitor locations.  Second, we found out which sources have locations that were defaulted based on either the zip code or county defaulting methods.  We did this for each of the three metals.

In the table below, we looked at the percent of emissions (by mass) from each state/metal combination coming from uncertain locations.  Each cell in the table is a link to a pie chart, which shows in more detail the breakdown of emissions into categories of location uncertainty.  We also noted the states which we feel are likely to have missing sources.  As said above, some states were more forthcoming than others with respect to submitting a point source inventory.  The less forthcoming states are considered more likely to have sources not included in the NTI.  Missing sources often cause drastic underestimates by the modeling system.

The table below shows the percent of emissions falling into each location category, when summing across states.  The NTI has exact geographic locations for most of the point source chromium emissions, most of which fall in the reported county; but about 13 percent of lead emissions and 25 percent of cadmium emissions were defaulted.

In section V.D below, we'll discuss some of the results of the location defaulting for lead in more detail.  We did a detailed study of some of the monitoring sites which were matched up with extremely low model estimates.

2) Stack parameters, fugitive vs. stack.  The other stack parameters are as important to model results as the locations.  Slight changes in the stack parameters can cause widely varying model results.  We'll focus mainly on the release heights.  Most of our monitors measure the concentration of a pollutant on the ground; and the ASPEN model does the same.

            EPA modelers agree that the release height is important, because emissions released from high stacks have more air to pass through on their way to the ground than emissions released from ground-level.  As a result, we might expect positive errors in release height to lead to model underestimates, and vice versa.  Studies have shown that ground-level concentrations are 5 to 8 times more affected by low level emissions than by elevated emissions.^20,21

In part to check the accuracy of the stack release heights in the NTI, members of EPA visited a lead smelter in Herculaneum, Missouri.  According to the TRI, 89.91 tons of lead were emitted from the 550-foot stack in the center of the facility, and 7.66 tons were "fugitive" emissions: that is, escaping from the facility through open doors, windows, etc.  However, the emissions for an ongoing 2-month study at the facility suggest that the “fugitive” emissions are of order 50 tons rather than 7.66 tons.  We can not generalize to all other sources from one site visit, but this does reveal the possibility of the types of emission characterization uncertainties that can occur.  All other factors being equal, an increase from 7.66 to 50 tons in low-level emissions would likely increase the predicted annual average for this site by a factor of 3.

b) Spatial and temporal allocation in EMS-HAP.  One of the tasks of EMS-HAP is to allocate NTI emissions summarized at the county level both temporally and spatially.  EMS-HAP makes numerous assumptions in these allocation processes, which add to the uncertainty of the model estimates.  The EMS-HAP user's guide will be available to the public soon.  This guide will describe these allocation techniques in detail.  For our purposes, it is important to note that pollutants for which area sources contribute significantly to model estimates have more uncertain model estimates.

iii) Model uncertainties.

Dispersion Calculations
ASPEN was constructed using Version 2 of the Industrial Source Complex Long-Term model (ISCLT2).  The major changes between Versions 2 and 3 (the current version) of ISCLT were: a new area source algorithm, a revised dry deposition algorithm, a wet deposition algorithm, COMPLEX1 algorithms, a pit retention algorithm.  None of these changes would affect ASPEN, as ASPEN does not use the ISCLT area source algorithm, dry deposition algorithm, or wet deposition algorithm.  Hence, for the simple point source case, one would expect ASPEN and ISCLT to provide similar results.  To test this, a series of runs with both ASPEN and ISCLT were made, to provide a means for comparing the respective model's estimates.

ASPEN Calculations
The ASPEN calculations are actually distributed over a series of separate processors.  The first (ASPENA) reads in the emissions data, and computes for each source the concentration for a set of receptors distributed around the source location.  Receptors are located along 16 radials outward from the source at 12 locations (ranging from 100 m to 50 kilometers).  The 16 radials are defined in a clockwise sense about the source, with the first radial pointed due north.  The interpolation procedures are all computed within ASPENB, which reads in the results from ASPENA, and combines this with a listing of where concentration values are desired.  In the simplest of situations, when no spatial averaging is employed (valid for all sources once the receptor is no longer located within the same census tract as the source), concentrations are interpolated using the values computed along the 16 radials.  Linear interpolation is used for a point between radials at a fixed distance downwind.  Log-log interpolation is used for determining concentration variations as a function of distance downwind.  We will not attempt here to describe the averaging procedures used for computing point source and area source impacts for receptor locations within the census tract of the source.  In principal, the impacts to receptors that are within the same tract as the emission are essentially computed as an area-weighted average concentration for all ASPENA receptors found within the census tract.

Test Case Meteorology
Both ASPEN and ISCLT use a Stability Array to describe the frequency of occurrence of wind speeds and stability conditions as a function of sixteen wind sectors going clockwise from the north.  For the test cases to be described, only winds from the north were specified.  Only neutral stability was allowed.  The mixing height was set at 1000 m and the annual average temperature was set at 279.65 K.  The frequency of occurrence of the six wind speed categories was specified as:

a) Interpolation.  We specifically investigated whether the interpolation scheme used within ASPEN might be underestimating the actual modeled impacts.  This concern arose because a "net" of receptors is employed by ASPEN, and then concentrations at specific points are estimated by interpolating within the "net".  We wondered whether ASPEN might underestimate peak ambient concentrations because it "averages out" the peak values by combining them with lower concentrations nearby.
    To do this, we simulated three types of emissions sources, and compared the ASPEN estimates downwind from each source to the estimates derived from a more recent, detailed version of ASPEN, the Industrial Source Complex Long-Term Model Version 3 (ISCLT3).  The simulations were run under a variety of wind speed conditions.

The first simulated source was a point source with a 10-m stack height.  The stack gas temperature was set to be nearly the same as the ambient temperature, and the exit velocity was set at 1 m/s.  In this case, there would be no plume rise.  Figure 1 depicts the results obtained from the two models.  It is evident that at 250 m downwind, ISCLT provides a concentration that is greater than what ASPEN would provide using its log-log interpolation procedures.  However, at most distances the two models provide similar results.

In these comparisons, we see no systematic bias in the ASPEN calculations that would cause it to significantly underestimate concentration values in comparison to what ISCLT can provide.  If anything, it may be that ASPEN may provide higher concentration values for receptors near or within tracts with area source emissions.

b) Deposition.  The ASPEN model simulates the effect of dry deposition of particulate by adding an additional decay term to the emission rate in calculation of ambient concentrations. The decay term is a function of the deposition velocity, downwind distance from the source, and plume dimensions with respect to the mixing height.  Deposition velocity is also a function of the particle size, wind speed, and the land-use type.  The ASPEN model allows different deposition options for fine and coarse particulate and urban/rural environments.  In order to analyze the effect of these options on the modeled ambient concentrations, we performed test case simulations using lead emissions from mobile non-road sources in Colorado.  We used different compositions of fine/coarse fractions and held the total emission rate constant.  Five different scenarios were used for this test case: 10% fine and 90% coarse, 25% fine and 75% coarse, 50% fine and 50% coarse, 75% fine and 25% coarse, and 90% fine and 10% coarse.  Emissions from 17 pseudo-point sources of 10 m height, 1 m/s exit velocity, and T = 295 K were considered.  For fine particles, the deposition velocities are generally similar and scattered around the 1:1 ratio line.  The deposition velocities for coarse particles are much higher for ASPEN than for ISCST3.  The effects of these differences was extrapolated to the national scale.  For the entire U.S., the total lead emissions were 66.5 g/s and the percent contribution from different source categories was as following: 49% all lead emission were accounted for by major sources, 28% by area sources, less than 0.01% by mobile on-road, and 23% by mobile non-road sources.  For the ASPEN simulations this means that about 50% of all lead emission sources were treated as point sources and about 50% as pseudo point sources.  We estimated that ASPEN has a bias to predict average lead concentrations in the air 20–30% lower than one would typically predict, because it is using coarse particle deposition velocities that are higher than one would usually use.
    The same logic applies to the other particulate HAPs in the comparison (cadmium and chromium).  We expect that the percent of emissions in the coarse category for these other two is less than that for lead, however, so we would expect the underestimation bias also to be less.

V. General Results

A. Overall.

For comparison to the results from the historical studies listed in section IV.A above, the table below gives the percent of sites estimated within 30%, and within a factor of 2.  It also reports the percent of sites which are underestimated by the modeling system.

These results are surprising given the results of the historical studies.  The agreement in our results on a point-to-point basis is comparable to the agreement from historical studies for benzene only.  The rest of the pollutants show poor agreement on a point-to-point basis, with the model estimates systematically lower than the monitor averages.  From the ratio box plot graph, we can see that this is especially true for the three metals, which all have ratio medians of less than 1/5: on "average", they are underestimated by more than a factor of 5.  This is most interesting for lead, because this is a well-studied criteria pollutant, for which we have extensive monitoring experience and an emissions inventory which has been assembled over many years.

Because the past model-to-monitor studies in section IV.A show much better agreement than ours, something must have been different in our study from historical studies–and we do not think the underestimation is due to the model itself, because the model we used here is very similar to the model used in historical studies.  We can think of two possible explanations for the underestimation:

1. The emission rates are systematically underestimated and/or many sources are missing from the emissions inventory.
2. Many of the monitors likely were sited to find peak concentrations.  Often, the ambient concentration falls off quickly around the peak area.  Even given a perfect model and perfect monitors, if the monitor is right at the peak and the emissions or meteorological inputs are off even a little, the model will underestimate.  This is especially likely for pollutants dominated by point sources with elevated releases, because any errors in release height, exit velocity, or emissions location might cause the model to find a peak concentration area different from the true peak.

We are currently designing a national Air Toxics monitoring network.  To get an idea of how to proceed with the national network, we are conducting pilot monitoring studies in a few cities across the US.  The monitors in these studies are sited in high, medium, and low concentration areas, with a future model-to-monitor comparison as one of the goals.  We believe in terms of a model-to-monitor comparison, the new network will be more useful than the current collection of monitors.

B. Benzene.

As expected from Figure 4 and Table 8, most of the points in the scatter plot fall between the 2:1 and 1:2 lines.  The high concentration monitors seem to be estimated a little less reliably: most of the points falling outside the "factor of 2 wedge" are those with high monitor concentrations.  Some of the misses are low and some are high, but nothing misses by much.  The largest model-to-monitor ratio is 2.45 and the smallest is 0.34, so all monitors are estimated within a factor of three.

There are several reasons why we would expect good agreement for benzene:

• It is a widely distributed pollutant which is emitted from point, area, and mobile sources.  Thus, if the model is biased in the way it handles any one of these source categories, the bias will be dampened.  For example, if a point source near a monitor is mislocated away from the monitor, there will nevertheless be significant mobile source emissions near almost every monitor location which will dampen the underestimate.
• We have an estimated background concentration for it.
• There is a large number of monitoring sites for it (87), giving an adequate sample size for the statistics in the comparison.
• Monitoring technology for it has a long history, suggesting that the monitoring data is reflective of actual ambient concentrations.
• Its emissions have been tracked for a long time, so there is some confidence in emission estimates.

C. Other Gases.

Perchloroethylene is dominated by area sources.  For the average census tract across the US, area sources are responsible for 49% of the perchloroethylene model estimate.  Modeled concentrations of both aldehydes are dominated by mobile sources, both onroad and nonroad: the percent contribution of mobile sources is 69% for formaldehyde and 90% for acetaldehyde.  For area sources and mobile sources, we are heavily dependent on the spatial allocation methods of EMS-HAP, which add uncertainty to the model estimates.

Because of the uncertainty involved in the spatial allocation methods, it is possible that the model is estimating a concentration at or higher than the monitor average nearby, but not at the actual monitor location.  Thus, let's look at the MAXTOMON results for all three gases.  We include benzene also for comparison purposes.

The percent of monitors underestimated drops off quickly for perchloroethylene.  The model's estimate is low on a point-to-point basis 86% of the time; but there are modeled concentrations nearby which are greater than or equal to the monitor average for many of the monitors.  Less than half the sites are underestimated if we go out 10 km; and less than a quarter are underestimated if we go out 20 km.  This suggests that uncertainties in the location of the nearby area sources may be responsible for the underestimation on a point-to-point basis.

The effect is less dramatic for the two aldehydes, but still evident.  If we go out 20 km, 31% of the formaldehyde and 38% of the acetaldehyde monitors are underestimated, compared with 88% and 91% at the actual monitor locations.

The aldehydes differ from the five other HAPs examined in this comparison because a significant portion of their ambient concentrations were secondarily formed.  The other HAPs are relatively inert, which makes them easier to model.  However, the ASPEN model simulates atmospheric chemical reactions for the aldehydes in a simple manner.  This is an additional source of uncertainty.

It is possible that the ASPEN model underestimates the amount of secondarily formed aldehydes.  Analysis of ASPEN modeled nationwide mean values for formaldehyde and acetaldehyde suggest that 23% and 58% (respectively) of the total modeled concentrations are attributable to secondary formation.  A more recent study using OZIPR (a photochemical grid model) suggests that secondary formation generally accounted for approximately 90% of the ambient formaldehyde and acetaldehyde.²²

These results loosely suggest, especially for perchloroethylene, that the model is not necessarily systematically underestimating ambient concentrations: it may just be finding the peak concentration in the wrong place.

D. Metals.

i) Lead.

Since the underestimation at the source-oriented monitoring sites is especially severe, we investigated more closely the model estimates at some of these monitor sites.  We did not look at all 106 monitors which were labeled as source-oriented in the Archive, however.  The decision to define a monitor as source-oriented is somewhat subjective.  A monitor placed well downwind of a source in a residential neighborhood might be considered source-oriented by some but population-oriented by others.  Because of this, we chose to look closely at a subset of the monitors which we are more certain fit the mold of a source-oriented monitor, subsetting by monitored concentration.  Experience suggests that a monitor with an annual average of 0.3 mg/m³ or higher is certain to be source-oriented.

Of the 242 monitors used in the comparison, 42 are both labeled as source-oriented and have monitored annual averages greater than 0.3 mg/m³.  The median underestimation factor for these 42 monitors is 16.7.  What caused this underestimation?  We have two theories:

1. The Re-entrainment Hypothesis: Near sources which have been emitting lead for many years, there may be lead particles in the soil, which can reenter the air when the soil is stirred up by wind or human activities.  There may also be lead in the soil from the days when leaded gasoline was the norm.  These "re-entrained" particles are read by the monitors, but they are not accounted for in the NTI, and thus would not contribute to the model estimates.  We also do not have a background concentration for lead, which could include re-entrained particles.
2. The Location Uncertainty Hypothesis: This theory was introduced in section V.A. above, but may be especially true for lead.  Many of the lead sources are isolated facilities in rural areas, so the area of high concentration might be very small.  Any errors in emissions locations, release heights, etc., may cause the model to miss the peak, causing an underestimate at the exact monitor location.

We could think of no convenient way to test the first hypothesis.  We can look at the second hypothesis by using MAXTOMON methods and at the percent of defaulted emissions near the source.  We did both.  In order to focus more closely on what caused the underestimation, we looked only at the 30 of the 42 monitors which were underestimated by a factor of 10 or greater.  For the sake of brevity, we will call these the "discordant monitors".

a) MAXTOMON results.  The model receptors used in the MAXTOMON tests included exact monitor locations and tract estimates.  But since many of the lead sources are in rural areas, away from urban areas and their small census tracts, the network of receptors used in the MAXTOMON test is not that dense.  Thus, it is possible that the model could be simulating a peak concentration near the monitor, but none of the tract centroids or monitor locations are near this peak.  For this reason, we feel that the MAXTOMON test is not as effective (without adjustment for receptor network size) in areas of low population density.  Here are the MAXTOMON test results:

So even a search radius of 30 km brings only one discordant monitor out of "underestimated" status.

b) Location uncertainty results.  The following table shows the percent of emissions falling into each of the four location categories for each of the discordant monitors.  Included are all sources with reported locations within 50 km of the monitor.

For some of the other monitors, we were able to do a "before and after" test.  Six of the source-oriented monitors are near three large lead sources which were defaulted in the penultimate model run, but located more accurately in the final model run.  We can compare the agreement between model and monitors before the sources were located accurately to after.  Some other changes to the modeling system were made between the penultimate and final model runs, but we believe the correction of defaulted locations is the most important change affecting the model estimates for these source-oriented lead monitors.

1) The Missouri Source.  The source in Missouri (NTI Site ID ES0912) emits 2.90 tons/year of lead and is close to the two discordant monitors in Holt County, Missouri.  Defaulted, the source was about 10.6 km from the monitors.  The true location of the source is within .7 km of both monitors.  There are no other sources nearby.

The model estimates are still much lower than the monitors, but the agreement is improved.

2) The Tennessee Source.  The source in Tennessee (NTI Site ID ES099) is in Memphis.  It emits 2.70 tons/year of lead and is close to two source-oriented monitors in county 47157 (Shelby County, Tennessee).  One monitor is discordant, i.e., underestimation factor of greater than 10, and the other is not.  Defaulted, the source was about 13 km from the monitors.  Again, the true location is right next to the monitors - within .1 km.  The only other large source in the area (3.2 tons/year) is about 10 km away.

In the first two before-and-after cases, when we located the source correctly, the model estimates were much improved, but still lower than the monitor averages.  The best agreement we found was an underestimation by a factor of six, for monitor ID 471570044.  This result suggests that location defaulting can contribute significantly to underestimation for source-oriented lead monitors, but it is not the only cause.

One of the most interesting discordant monitors is 170310068.  It is in Chicago, very close to Lake Michigan and the Indiana state line.  This site and the surrounding area are shown in the map below.

There are six large (>1 ton/year) sources in Indiana which fall within 50 km of this monitor.  These are circled in the map.

One possible explanation for the underestimate at this site is that these six large Indiana sources are mislocated.  In general, many of the Indiana sources have reported locations which place them in Kentucky and Ohio, probably south and east of their true locations.  So some of the Indiana sources are offset to the southeast.  If the six large Indiana sources in the circle are moved northwest, they will be much closer to the monitor than they are now, which would presumably increase the model estimate at the monitor site.  Overall, of the 1333 lead emissions sources identified as belonging to Indiana, 155 (11.6%) have latitude/longitude coordinates that place them outside of Indiana, accounting for 5.7% of the emissions.  This would also explain the underestimate at the monitor in Indiana (180350008).

Further support for the theory that these six sources are mislocated comes from the surrounding area.  There are no big cities near the circled sources.  If they are moved northwest, they move into the heavily industrialized cities of Hammond and Gary.

In light of these results, it is possible that the current state of the NTI’s source location data precludes point-to-point model-to-monitor comparisons for point source oriented monitors, particularly for metals concentrations which tend to decay rapidly with distance from the source due to their high particle size and weight.  We wonder if point-to-point model-to-monitor comparisons for source-oriented monitors should be done only on a small scale, where emissions and source locations can be more accurately characterized.

Lead seems to stand out as the most underestimated pollutant in the comparison.  This may be due to the large number of source oriented monitors, but even the other monitors are underestimated, and the MAXTOMON methods did not significantly change this.  We wonder if there really is reentrainment going on, or something else causing lead emissions to be underestimated nationwide.

ii) Cadmium.

The 6 New York monitors are all underestimated by a factor of five or greater.  Two of the 7 Chicago monitors are estimated within a factor of 2, with the other 5 being underestimated.  However, there are higher modeled concentrations within 30 km of all but one of these 13 monitors.  Nationwide, 15 of the 20 monitors have higher modeled concentations within 30 km.

iii) Chromium.

The ten monitors in Staten Island are all underestimated by factors of between 6.64 and 8.30.  The monitor averages are very similar among the ten sites.  There are no census tracts on Staten Island with modeled concentrations similar to the monitors, but there are tracts within 20 km of the monitors with modeled concentrations higher than all 10 monitor averages.

However, the eight monitors near Chicago are generally estimated accurately at the monitor locations.  The model-to-monitor ratios are all between 0.65 and 1.13, so all are easily estimated within a factor of 2, and all but one are estimated within 30%.

The California monitors are estimated inconsistently.  Only 2 of the 14 are estimated within a factor of 2 at the exact monitor locations.  Of the other 12, 2 are overestimated and 10 are underestimated.

These state-to-state differences suggest differences in the state inventories.  The results for cadmium suggest the possibility that New York has low emissions estimates for cadmium as well.  Possible differences in monitoring must also be considered.

VI. Conclusions

By "modeling system", we mean to emphasize that the estimates are the result of:

We tried to use the MAXTOMON test to see whether the model was mislocating the peak concentrations, or systematically underestimating.  On a pollutant-by-pollutant basis, here are the percent of sites underestimated at different radii, out to 50 km:

Each pollutant has a certain distance which we starred: in our opinion, this is where the effect of the uncertainty distance starts to wane.  In general, we might say that the modeling system has uncertainty in locating source impacts to within 20 km.  But even as we go out to 50 km, many of the pollutant/monitor combinations are still underestimated for all HAPs except benzene and perchloroethylene.  This suggests systematic underestimation of the aldehydes and metals.

We look forward to receiving the monitoring data collected from the pilot cities.  This monitoring data will have fewer uncertainties surrounding it than most of the data currently in the Archive.  We can compare the monitoring data from this network to future model runs.

In general, we think that the most effective way to improve agreement between the model estimates and monitor averages is to improve the emissions inventory.  One possibility would be to conduct a study on a small sample of sources, to see if the emissions rates are accurately estimated.  But if these model estimates are used on a local scale, we think it is crucial to:

1. For pollutants dominated by point sources, get better data on the source locations and releases.
2. For pollutants dominated by area sources, work on either improving our spatial allocation methods or ceasing the pattern of aggregating to the county level in the inventory, then disaggregating for modeling purposes.
3. Accurately estimate background concentrations, possibly on a regional basis instead of nationwide.

VII. References

²Rosenbaum, A.S., Stiefer, P.S., and Iwamiya, R.K., (1999): Air Toxics Data Archive and AIRS Combined Data Set: Data Base Descriptions.  Prepared for US EPA by Systems Applications International, SYSAPP 98/05r.

³Code of Federal Regulations, Title 40, Part 136, Appendix B, Revision 1.11.

⁴Curran, T.C., and Steigerwald, B.G., (1982): Data Analysis Consequences of Air Quality Measurement Uncertainty.  Paper presented at the 75^th Annual Meeting of the Air Pollution Control Association, New Orleans, Louisiana, June 1982.

⁵Meade, P.J., and Pasquill, F., (1958): A study of the average distribution of pollution around Stayhorpe.  Int. J. Air Pollution.  Pergamon Press.  Vol. 1, pp. 60-70.

⁶Lucas, D.H., (1958): The Atmospheric Pollution of Cities. Int. J. Air Pollution.  Pergamon Press.  Vol. 1, pp. 71-86.

⁷Pooler, F., (1961): A prediction model of mean urban pollution for use with standard wind roses.  Int. J. Air and Water Pollution.  Vol. 4(3/4):199-211.

⁸Martin, D.O., (1971): An urban diffusion model for estimating long term average values of air quality.  J. of the Air Pollution Control Association.  Vol 21(1):16-19.

⁹Calder, K.L., (1971): A climatological model for multiple source urban air pollution.  Paper presented at First Meeting of the NATO/CCMS Panel on Modeling.  Paper published in Appendix D of User’s Guide for the Climatological Dispersion Model.  EPA-R4-73-024.  Office of Research and Development, U.S. Environmental Protection Agency, Research Triangle Park, NC 27711, pages 73-105.

¹⁰Turner, D.B. and Edmisten, N.G., (1968): St. Louis SO2 dispersion model study - description of basic data.  (Unpublished report, Division of Meteorology, NAPCA).  Atmospheric Sciences Modeling Division, U.S. Environmental Protection Agency, Research Triangle Park, NC 27711.

¹¹Turner, D.B., Zimmerman, J.R., and Busse, A.D., (1971): An evaluation of some climatological models.  Paper presented at Third Meeting of the NATO/CCMS Panel on Modeling.  Paper published in Appendix E of User’s Guide for the Climatological Dispersion Model.  EPA-R4-73-024.  Office of Research and Development, U.S. Environmental Protection Agency, Research Triangle Park, NC 27711, pages 107-131.

¹²Briggs, G.A., (1969): Plume Rise.  TID-25075. U.S. Atomic Energy Commission, Available from the National Technical Information Service, U.S. Department of commerce, Springfield, VA 22151, 85 pages.

¹³Holland, J.A., (1953): A meteorological survey of the Oak Ridge area: final report covering the period 1948-1952.  U.S. AEC Report ORO-99.  Technical Information Service, Weather Bureau, Oak Ridge, TN. pages 554-559.

¹⁴Irwin, J.S. and Brown, T.M., (1985): A sensitivity analysis of the treatment of area sources by the Climatological Dispersion Model. J. of the Air Pollution Control Association. Vol 35(4):39-364.

¹⁵McElroy, J.L. and Pooler, F., (1968): St. Louis Dispersion Study Volume II-Analysis.  National Air Pollution Control Administration.  Publication Number AP-53.  U.S. Department of Health Education and Welfare, Arlington, VA, 51 pages.

¹⁶Gifford, F.S., (1976): Turbulent diffusion-typing schemes: a review.  Nuclear Safety. Vol 17(1):68-86.

¹⁷Turner, D.B. and Novak, J.H., (1978): User’s Guide for RAM, Volume 1, Algorithm Description and Use.  EPA-600/8-78-016a.  Office of Research and Development, U.S. Environmental Protection Agency, Research Triangle Park, NC 27711, 70 pages.

¹⁸U.S. EPA, (1995): User’s Guide for the Industrial Source Complex (ISC3) Dispersion Models, Volume I - User Instructions.  EPA-454/B-95-003a.  Office of Air Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC 27711, 320 pages.

¹⁹Turner, D.B. and Irwin, J.S., (1983): Comparison of sulfur dioxide estimates from the model RAM with St. Louis measurements. Air Pollution Modeling and Its Application II, (Edited by C. De Wispelaere), Plenum Press, pages 695-707.

²⁰Gifford, F.A. and Hanna, S.R., (1973): Modeling urban air pollution.  Atmospheric Environment.  (7):131-136.

²¹Hanna, S.A., Briggs, G.A., and Hosker, R.P., (1982): Handbook on Atmospheric Diffusion.  Available as DE82002045 (DOE/TIC-11223) from the National Technical Information Service, U.S. Department of Commerce, Springfield, VA, 22161, 108 pages.

²²U.S. EPA, (1999): A simplified approach for estimating secondary production of hazardous air pollutants (HAPS) using the OZIPR model.  EPA-454R-99-054.  U.S. Environment Protection Agency, Research Triangle Park, NC 27711, 86 pages.