Upon thoughtful consideration of the sample survey approach, several questions may come to mind. This section answers several commonly asked questions concerning data analysis. These questions are addressed in fairly general terms. As noted in the introduction, additional technical details are in a series of methods manuals. (Back to Frequently Asked Questions )

Data Analysis

• Doesn't enhancing the sampling intensity for an area of special interest bias the overall estimates?
• For 305(b) reports, I need to estimate the total number of stream miles in my EPA Region that are degraded. Can I do this from sample survey data?
• I am accustomed to seeing estimates of average condition instead of estimates of proportion. Can sample survey data be used to estimate average condition?
• EMAP's objectives state that estimates are made with known confidence. What is "known confidence"?
• What are the most important factors affecting the size of the confidence interval?
• Doesn't the size of the target population affect confidence in the estimates?
• States conduct targeted monitoring for many purposes; the locations were selected for various reasons. How can the data from these monitoring programs be integrated with probability sample survey data?

Doesn't enhancing the sampling intensity for an area of special interest bias the overall estimates?

No. Sampling units inside an area of special interest usually have a higher chance of being selected than sampling units outside the special interest area. Within each stratum, however, the chance of selecting any location is equal; therefore, a separate (unbiased) estimate can be computed for each stratum, as well as for the entire resource.

With stratified random sampling, estimates are generated first for individual strata, then the stratum-specific estimates are combined into an overall estimate for the whole target population. Stratum-specific estimates are combined by weighting each one by the fraction of all sampling units that are within the stratum. For the simple two-stratum example given above, the weights would be 200/1000 for stratum 1 and 800/1000 for stratum 2. So, if the stratum-specific estimates are 0.5 for stratum 1 and 0.25 for stratum 2, the overall estimate is 0.30 [(O.5 x 2/10) + (0.25 x 8/10)]. This approach ensures that the overall estimate is corrected for the intentional selection emphasis within a particular subpopulation. Go to Top

For 305(b) reports, I need to estimate the total number of stream miles in my EPA Region that are degraded.  Can I do this from sample survey data?

The number of degraded stream miles can be calculated in two steps. First, the proportion of stream miles that are degraded is calculated as illustrated earlier. Then, that fraction is multiplied by the total number of stream miles in the population. The total number of stream miles is available from the sampling frame (for example from EPA's River Reach File, Version 3), which delineates all members of the target population.

Defining "degraded" is an important part of the calculation, regardless of whether it is for percent or absolute number of stream miles. "Degraded" can be defined if a threshold value or goal for each measurement variable can be established. Most of the variables measured in stream surveys, such as an Index of Biotic Integrity (IBI), have continuous ranges of response. Calculating the proportion of stream miles that are degraded requires converting this continuous data into binary, or yes/no (e.g., degraded or not degraded) form. The question of how many stream miles are degraded, therefore, must be rephrased to include a threshold value for the relevant measurement variable. For an IBI, the question might be rephrased as "What are the total number of stream miles in my Region with IBI below a score of 45?" Go to Top

I am accustomed to seeing estimates of average condition instead of estimates of proportion.  Can sample survey data be used to estimate average condition?

Yes, estimates of average condition, such as the average IBI in a watershed, provide valuable information and can be calculated with sample survey data as a simple mean. The principles of survey sampling, particularly the emphasis on selecting a representative sample, also apply to estimating population mean. Just as an estimate of the percent of stream miles in a Region in which IBI is below 40 is biased if data are collected only from sites downstream of sewage outfalls, so is the estimate of mean IBI. Furthermore, estimates of various other properties can be made from the sample survey results, such as median scores, various percentiles, or frequency distributions and their shapes.

EMAP emphasizes estimating spatial extent (e.g., percent of river miles) because it has several advantages over estimating the mean alone. For instance, a Region with an average stream IBI of 45 might be composed entirely of streams with an IBI of 45; however, the same average would occur if half the streams have an IBI of 55 and the other half an IBI of 35. Estimating the spatial extent of the resource that fails to meet some standard (e.g., IBI of at least 45) provides more information about the condition of the resource and is consistent with EPA initiatives to establish environmental goals and measure progress toward meeting them. Go to Top

EMAP's objectives state that estimates are made with known confidence.  What is "known confidence"?

An estimate of a population parameter is of limited value without some indication of how confident one should be in it. Scientists typically describe the appropriate level of confidence in an estimate derived from a sample survey by defining confidence limits or margins of error. This description of statistical confidence is used frequently in reporting the results of opinion polls using statements such as "this poll has a margin of error of ± 4%". Provided random sampling is used, similar statements can be made about estimates from biological sample surveys.

Sample surveys provide estimates that are used to make inferences about parameters for the population as a whole. Two types of estimates are commonly provided: the point estimate and the interval estimate. For example, the estimated proportion of voters that support a party is a point estimate. It is important to know how likely it is that such a point estimate deviates from the true population parameter by no more then a given amount. An interval estimate for a parameter is defined by upper and lower limits estimated from the sample values. A confidence interval is constructed so that the probability of the interval containing the parameter of interest can be specified. We do not know with certainty whether an individual interval, specified as a sample estimate plus/minus a margin of error, includes the true population parameter. For repeated sampling, however, the estimated 95% confidence intervals would include the true parameter 95% of the times. The length of the confidence intervals is a measure of how precise the parameter is being estimated: a narrow interval signifies high precision. The margin of error is often used for defining the upper and lower limits of the confidence interval; it is half the width of the confidence interval. Thus, if a poll estimates that 55% of the population will vote Democratic and the margin of error is ± 4%, then the estimated 95% confidence interval ranges from 51% to 59%.

A great advantage of using a random sampling design is that statisticians have developed procedures for calculating confidence intervals for the estimates. For most sample surveys, in which the goal is to estimate the proportion of the resource that is degraded, a standard probability distribution known as the binomial distribution can be used as an estimate of the upper and lower bounds of confidence intervals. Go to Top

What are the most important factors affecting the size of the confidence interval?

The sample size (# of sampling units collected) and the proportion of yes answers are the primary factors affecting the size of the confidence interval with binary (yes/no) data. The effect of sample size can be illustrated with a pre-election poll of voters. If only 30 people are sampled, and 14 indicate that they will vote Democratic, it would be unwise to predict the winner. With such a small sample size, the margin of error would be about ± 18% for a 95% confidence interval. The degree of confidence would be higher if 140 people out of a sample of 300 say they will vote Democratic (47% ± 6%), and higher still if 1400 people out of a sample of 3000 say they will vote Democratic (47% ± 2%). In this example, the estimated proportion of sampled voters who will vote Democratic stays the same (p = 47%), but the width of the confidence interval decreases with increasing sample size.

Confidence intervals for estimated percentages (p) are affected to a lesser degree by the proportion of yes answers (P) in the population. The widest confidence interval occurs for P equal to 50%. For values of P ranging from 20% to 80%, the margin of error will not vary much with P; it will be determined mainly by the sample size. The fact that there is a maximum margin of error for binomial estimates of proportions is very useful for planning a survey. If we plan for the worst case (i.e., when half of the population is in the yes category) we can select a sample size that ensures that the confidence interval for P will be smaller than a specified limit. Go to Top

Doesn't the size of the target population affect confidence in the estimates?

The size of the target population theoretically affects the precision of the estimates. For most sample surveys, however, the effect is negligible because the sampled fraction of the target population is so small. When the sampled fraction is small, the size of the sample rather than the size of the target population determines the precision of the estimate. Polling 1000 people in the state of Rhode Island, for example, would yield as precise an estimate as polling 1000 people in the state of Texas, or the nation as a whole. In these cases, a very small proportion of the total population is polled.

If the sample includes a large proportion of the population, in contrast, the accuracy of the estimate is improved. For instance, if a local town has a population of 1400 people, then a sample of 1200 people would produce a substantially more accurate estimate than a sample of 1200 people from a population of 100 million. As the size of the sample approaches the size of the population, statisticians adjust the confidence interval using the finite population correction factor. In practice, however, most sampling efforts don't sample a large enough fraction of the population for this correction factor to become important. That is why pollsters interview approximately the same number of people for a local election as for a presidential election.

For sample survey projects, the fraction of the population that is sampled is generally very small. Fish assemblages, for example, are generally sampled from short segments (100 - 400 meters). If 50 such samples are collected from a Region with 1000 kilometers of streams, the sampled fraction is 0.0001 - 0.0004.3 Go to Top

To characterize the status of aquatic resources in a watershed, ecoregion, or state, an unbiased site selection process is important so that valid inferences about the resource can be made. Sites selected for targeted monitoring are usually selected with a good purpose in mind, often to answer very site specific questions. Consequently, they might be very biased relative to the resource as a whole. In general, we do not know what the bias might be in using targeted monitoring sites to make population estimates.

One path toward resolving this issue is to divide the population of interest into two parts. One part is the set of sites (or proportion of the resource) that has been censused through the targeted monitoring. The other part is the portion of the population that is not censused and can be surveyed though a sample survey. Combining the results of the two parts produces an estimate of the condition of the entire resource. The extent to which targeted monitoring will influence the outcome of the picture of the resource as a whole will depend on the proportion of the resource that is censused through targeted monitoring. In general, this is a relatively small part of the entire resource, consequently the results of targeted monitoring won't influence the overall description of the resource.  Go to Top