![[logo] US EPA](../gif/logo_epaseal.gif){kind=logo}
Limitations in the 1996 National-Scale Air Toxics Assessment: Uncertainty
Note: EPA no longer updates this information, but it may be useful as a reference or resource.How was the Uncertainty Analysis Conducted?
(Technical Level)
The uncertainty analysis EPA conducted for the national-scale assessment consisted of both quantitative and qualitative stages. First, for each of the sources of uncertainty identified in the section on Components , the uncertainty in that individual parameter or model was assessed quantitatively to the extent possible. Where quantitative assessment was not possible (either because data were not available or because the source of uncertainty is not quantitative in nature), EPA performed a qualitative assessment of the uncertainty.
EPA then combined each of these individual judgments and/or characterizations of uncertainty to assign measures of confidence to the calculation of ambient air concentration, to the relationship between ambient air concentration and exposure, and to the relationship between exposure and risk. EPA combined the judgements for emissions, ambient concentration modeling, and exposure modeling to determine an overall confidence. Finally, EPA combined the judgments for all the steps of the national-scale assessment into a composite judgment of confidence: (i) a judgment for ambient air concentration, (ii) a judgment for exposure, and (iii) a judgment for risk. The discussion that follows first discusses how uncertainties for individual parameter values and/or models were assigned, then how these were combined to form the various judgments of ambient air concentration, exposure and/or risk. Results are presented in the main section on Uncertainty as well as the section providing more details about the assignment of overall confidence.
Ambient Air Concentration
Considering first the predictions of ambient air concentration, the specific contributors to uncertainty considered in the uncertainty analysis were:
Uncertainties due to the emissions parameters. It was not possible to assign quantitative estimates of uncertainty to these parameters. As mentioned in the section on Components, slightly less than 10% of the point source sites were assigned a default location. Still, at least one spot check of emissions at a lead smelting facility indicated the possibility of uncertainties on the order of a factor of 3 for an individual source. A qualitative judgment of the confidence in emissions parameters was made for each air toxics compound based on previous experience.
Uncertainties due to stack parameters. It was not possible to assign quantitative estimates of uncertainty to these parameters. As mentioned in the section on Components, approximately 63,000 of the 97,000 vertical stacks were assigned one or more default values for these parameters. EPA did not consider uncertainties in this aspect of the assessment further.
Uncertainties due to particle size and reactivity parameters. It was not possible to assign quantitative estimates of uncertainty to these parameters. Uncertainties in this aspect of the assessment were not considered further.
Uncertainties due to terrain parameters. It was not possible to assign quantitative estimates of uncertainty to these parameters. Uncertainties in this aspect of the assessment were not considered further.
Uncertainties due to background concentration parameters. It was not possible to assign quantitative estimates of uncertainty to these parameters. A qualitative judgment of the confidence in applying background values was made for each air toxics compound based on previous experience. Specifically, if background contributed strongly to overall exposure, but it was not possible to verify the true background value for an air toxics compound, the confidence in ambient air concentration estimates was assigned a Medium or lower.
Uncertainties due to meteorological parameters. To partially quantify the uncertainty due to this parameter, the ASPEN model was run with two different meteorological databases. These produced differences in predicted ambient air concentration ranging from minus 17% to plus 84%. This suggests that uncertainty in meteorological parameters introduces uncertainty into estimates of ambient air concentration on the order of 50%, or a factor of 1.5.
Uncertainty due to the ASPEN dispersion model equations. The ASPEN model uses a Gaussian dispersion equation to calculate ambient air concentration, taken from Version 2 of the Industrial Source Complex Long-term (ISCLT2) model. The uncertainty in the ISCLT2 model has been studied extensively, and generally indicates the model predictions are accurate to within a factor 2 when emissions characteristics are known.
Uncertainty due to the ASPEN chemical transformation equations. The results of the ASPEN model were compared to the results of a more detailed model (OZIPR) for formaldehyde and for acrolein. The OZPIR model is significantly more accurate in predicting secondary formation (i.e. production of an air toxics compound due to chemical reactions in the atmosphere). The study indicated that the ASPEN-based predictions generally produce lower estimates of secondary formation than does the more detailed OZIPR model. For example, ASPEN predicted that secondary formation of formaldehyde would account for 23% of the total ambient air concentration; OZIPR estimated this fraction to be 90%. ASPEN predicted that secondary formation of acrolein would account for 44% of the total ambient air concentration; OZIPR estimated this fraction to be 85%. This suggests that ASPEN probably is underestimating chemical formation in the atmosphere, which would underestimate ambient air concentration by as much as a factor of 2.
In addition to assessing the uncertainty in each parameter and model, the overall uncertainty in predictions of ambient air concentration was assessed by comparing ASPEN model predictions against available measurements at the same locations. Monitoring data were available for seven compounds: benzene, perchloroethylene, formaldehyde, acetaldehyde, cadmium, chromium and lead. Separate comparisons were performed for stable gases, reactive gases and particulates, since the model should perform better for stable gases than for reactive ones (due to uncertainties in reactivity discussed previously), and better for stable gases than for particulates (since there is uncertainty in the settling velocity for particulates).
The results of this assessment showed that the ASPEN model tended to under-predict the concentration found at monitors. A model-to-monitor ratio was calculated by dividing the monitor result by the model result at the same location, and then repeating this over many locations. The mean value of this ratio was 1.19 for benzene; 2.26 for perchloroethylene; 2.28 for formaldehyde; 2.69 for acetaldehyde; 15.37 for lead; 12.53 for cadmium; and 6.05 for chromium. Note that this ratio is consistently greater than 1, indicating that the measured concentration was higher than that predicted by the model.
There are several possible explanations for this. First, sources could be missing in the model, or perhaps the background concentration used in the model was too low; this would produce a ratio greater than 1. Second, the monitors tend to be located geographically at points of highest concentration. If the model incorrectly predicts the location of this point, the ratio will tend to be greater than 1. This explanation has some credence because if the monitor results are compared against the highest concentration predicted by the model within the 50 km radius of the source (on the assumption that the model correctly predicts the peak concentration but not its location accurately), the model-to-monitor ratios are significantly closer to 1.
To further quantify uncertainty in predictions of ambient air concentration, EPA made multiple comparisons between monitor results and model results for stable gases, for reactive gases and for particulates. This produced a series of model-to-monitor ratios for each category. Percentiles of the cumulative distribution functions (CDFs) for these sets of ratios then were produced. Results were:
- For stable gases, the 5th, 50th and 95th percentiles were 0.78, 1.4 and 2.6, respectively. If the distribution of confidence is assumed to be lognormal, this indicates a median of 1.4 (i.e. a systematic 40% underprediction of ambient air concentration by the model) and a geometric standard deviation (GSD) of slightly more than 1.3. This is consistent with the statement given previously that the Gaussian model tends to be accurate to within a factor of 2.
- For reactive gases, the 5th, 50th and 95th percentiles were 0.88, 2.0 and 4.3, respectively. If the distribution is assumed to be lognormal, this indicates a median of 2 (i.e. a systematic 2-fold underprediction of ambient air concentration by the model) and a geometric standard deviation (GSD) of slightly more than 1.5. This is consistent with the statement given previously that the ASPEN model and its corrections for reactivity tend to under-predict secondary formation for more reactive gases.
- For particulates, the 5th, 50th and 95th percentiles were 1.4, 4.9 and 16, respectively. If the distribution is assumed to be lognormal, this indicates a median of 4.9 (i.e. a systematic 5-fold underprediction of ambient air concentration by the model) and a geometric standard deviation (GSD) of slightly more than 1.8. This is consistent with the statement given previously that the ASPEN model tends to under-predict ambient concentrations of particles due to use of a large settling velocity.
The above model-to-monitor comparison provides some insight into the uncertainty in predictions of ambient air quality. It does not, however, fully characterize the uncertainty. On the one hand, it may overestimate uncertainty since it generally compares measured concentration against predicted concentration at the peak. As mentioned previously, the ASPEN model does a better job of estimating peak concentration than it does of locating precisely the geographic point of peak concentration. This suggests it will be better at predicting an average or typical exposure (which is the use to which it is put in the national-scale assessment) than it will be at predicting exposure at a specific geographic point. On the other hand, the national-scale assessment uses a single predicted ambient air concentration to calculate exposures to everyone in a census tract. There can be significant variation in air concentrations across a census tract, which will tend to increase uncertainty in the estimate of exposures to the population since not everyone lives at the centroid.
All of the above considerations were incorporated into a qualitative judgment of the confidence with which predictions of ambient air concentration may be made for each air toxics compound. This process involved forming separate judgments on the confidence assigned to (i) the emissions and (ii) the ASPEN dispersion and transformation modeling, as well as the implications of the model-to-monitor results (for compounds where this was available), followed by combining these three judgments to yield an overall judgment of the confidence with which ambient air concentration can be estimated. These judgments were given as Higher (meaning there is higher confidence in reliability of the estimate of ambient air concentration), Medium (meaning there is medium confidence in the reliability of the estimate of ambient air concentration) and Lower (meaning there is lower confidence in the reliability of the estimate of ambient air concentration). These judgments are not the same for all compounds since the database available differs between compounds. Results of these judgments may be seen in the section providing details about the assignment of overall confidence.
Exposure
Considering next the predictions of exposure, the specific contributors to uncertainty considered in the uncertainty analysis were:
- Uncertainty due to microenvironment factor parameters. It was not possible to assign quantitative estimates of uncertainty to these parameters. A previous peer review of the database on which the parameters were estimated indicated that the uncertainty was significant, especially for individuals. The national-scale assessment, however, does not focus on individuals but rather on geographic regions over which uncertainty in the parameters for individuals might be “averaged out” to some extent. On the other hand, there is considerable uncertainty about even the average relationship between indoor and outdoor air concentrations for many of the microenvironments considered in the national-scale assessment. EPA made a qualitative judgment of the confidence in microenvironmental factors for each air toxics compound based on the above information and previous experience.
- Uncertainty due to population cohort parameters. It was not possible to assign quantitative estimates of uncertainty to these parameters. There should be uncertainty due to both limitations in the database and the assignment of national averages to individuals in all census tracts, rather than census-tract-specific values (which were not available). A qualitative judgment of the confidence in population cohort parameters was made for each air toxics compound based on previous experience.
EPA incorporated all of the above considerations into a qualitative judgment of the confidence with which predictions of exposure may be made for each air toxics compound when ambient air concentration is known. These judgments were given as "Higher" (meaning there is higher confidence in the reliability of the estimate of exposure when ambient air concentration is known), "Medium" (meaning there is medium confidence in the reliability of the estimate of exposure when ambient air concentration is known) and "Lower" (meaning there is lower confidence in the reliability of the estimate of exposure when ambient air concentration is known).
Risk
Considering finally the predictions of risk, the specific sources of uncertainty considered in the uncertainty analysis due to dose-response relationships (in addition to those considered for ambient air concentration and exposure) were:
- Uncertainty in hazard identification. As mentioned in the section on Components, there is uncertainty as to whether a specific compound has been classified properly as a carcinogen, or determined to produce non-cancer effects. There is no method available to quantify this uncertainty.
- Uncertainty in dose-response models for carcinogens. The linear dose-response model is the default model in regulatory risk assessment, and was the default model used in the national-scale assessment. The uncertainty introduced by use of this model has not been studied extensively for most of the air toxics compounds considered here. In general, uncertainties introduced by the choice of model to perform extrapolation to low doses are on the order of a factor of 3 or more, especially when data are scarce. The possibility of thresholds in the dose-response data, particularly for compounds that act through promotional mechanisms, means that the confidence intervals for risk might include 0. EPA made a qualitative judgment of the confidence in dose-response for each air toxics compound based on the information above and previous experience.
- Uncertainty in Unit Risk Estimate parameters. As mentioned in the section on Components, UREs for some compounds are based on best (maximum likelihood) estimates of the slope of the dose-response relationship based on reliable data, and in other cases are based on "upper-bound" estimates (i.e. the slope is not the best estimate, but is a conservative value which is likely to lead to overestimates of risk). For some compounds, the data are from human exposures, while for others they are from animal exposures. As a result, the uncertainty in URE depends on the specific compound considered, as does the judgment as to whether the uncertainty is symmetrical around the estimate used in the national-scale assessment (which might be true when the maximum likelihood estimate was used) or non-symmetrical (as will be true when an "upper-bound" estimate was used).
As an illustration, the EPA produced a cumulative distribution function (CDF) for the URE value applied to benzene, based on the confidence interval reported in IRIS. If this CDF is considered to be lognormal, the results indicate a geometric standard deviation (GSD) of approximately 2.7. This suggests the URE for benzene might be "accurate to within a factor of 3". It must be recognized that this statement applies strictly to benzene, but does give a sense of the uncertainty in URE values for moderately well characterized air toxics compounds. The uncertainty might be smaller for better characterized compounds, and greater for poorly characterized compounds. EPA made a qualitative judgment of the confidence in URE for each air toxics compound based on the information above and previous experience. - Uncertainty in Reference Concentration parameters. The uncertainty in RfC depends on the quality of the data used in the development of the RfC, which varies between air toxics compounds. In general, however, the RfC is described by the EPA as having "uncertainty spanning perhaps one order of magnitude". If this is taken to mean a 95% confidence interval, the difference between the 5th and 95th percentiles would be a factor of 10. The ratio of the 97.5th percentile over the median would then be a factor of greater 3, which indicates a GSD in the neighborhood of 2. EPA made a qualitative judgment of the confidence in RfC for each air toxics compound based on the information above and previous experience.
- Uncertainty due to additivity. For cumulative risk estimates (the sum of cancer risks from carcinogens, or the Hazard Index for non-cancer effects), there is an additional assumption of additivity of risks. In general, two compounds might interact either additively (the individual cancer risks or hazard quotients may be summed); synergistically (the risk from two carcinogens is higher than that predicted from summing the individual cancer risks; the hazard from two non-carcinogens is greater than would be expected by adding their hazard quotients); or antagonistically (the risk from two carcinogens is lower than that predicted from summing the individual cancer risks; the hazard from two non-carcinogens is lower than would be expected by adding their hazard quotients). The uncertainty in this assumption cannot be estimated at present.
All of the above considerations were incorporated into a qualitative judgment of the confidence with which predictions of risk may be made for each air toxics compound when exposure is known. These judgments were given both as Higher (meaning there is higher confidence in the reliability of the estimate of risk when exposure is known), Medium (meaning there is medium confidence in the reliability of the estimate of risk when exposure is known) and Lower (meaning there is lower confidence in the reliability of the estimate of risk when exposure is known).
More Details About the "Overall Confidence" RankingsWhat are the components of uncertainty?
Which components of uncertainty did the national-scale assessment include?
Return to the main Uncertainty Page