Jump to main content.

Aquatic Resource Monitoring - Frequently Asked Questions - Survey Design

Upon thoughtful consideration of the sample survey approach, several questions may come to mind. This section answers several commonly asked questions concerning survey design. These questions are addressed in fairly general terms. As noted in the introduction, additional technical details are in a series of methods manuals. (Back to Frequently Asked Questions )

Survey Design

What is a survey?

Governmental organizations have monitored the aquatic environment for many years. This includes monitoring of estuaries, coastal waters, streams, rivers, lakes, reservoirs, and wetlands. Most, if not all, of these monitoring efforts have been designed to fulfill a specific purpose; e.g., is a municipal treatment plant in compliance. More recently, monitoring programs have been asked to address more regional questions. Examples of the types of question that these monitoring programs are asked to address are:

This type of question requires answers that apply to all of the aquatic resource of interest that occur in a geographic area. Monitoring for results, such as under the Governmental Performance for Results Act (GPRA), is one motivation for asking this type of question. Another is the 305(b) reporting requirements contained in the Clean Water Act. Both of these motivating agents require a "fair or representative" picture of the resource.

In general there are two approaches to obtaining information on the aquatic environment.  Historically, the most common approach was to collection information at locations that were chosen based on a variety of judgmental factors, e.g., thought to be representative, in areas of special interest, logistical considerations such as access and ease of sampling, etc.  The second approach, historically employed more frequently in other social and natural resource arenas, relies upon a statistical methodology to provide quantitative information about the aquatic resource.  The interpretation of the sampling results in the judgmental situation, relies on best professional judgment, or in some cases modeling approaches, to address the questions of interest.  The statistically based approach utilizes the scientific methodology developed for surveys to provide quantitative answers and uncertainty measures for the sampled resource.  The section on sampling survey versus census provides a fuller discussion of this aspect.

The questions are phrased as the general public might initially pose the issue of concern and are a basis for initiating the design of a monitoring program. As the monitoring program is developed, the questions will be found to be too general to complete the design. For example, what is meant by condition? Answering this question impacts the selection of what indicators should be measured? What is a lake? A stream? An estuary? Although everyone assumes that they know what a stream is, the design requires an explicit definition for a stream. It may be that only streams that are perennial and wadeable are of interest in the study. In some cases, the geographic region requires further clarification. For example, what is meant by "the Nation"? Does this include American Samoa, the Virgin Islands, Puerto Rico, Hawaii, Alaska? The information that follows focuses on the survey design portion of the general questions. We will only discuss what will be measured, i.e., the indicators, in terms of a general discussion on how the field plot design for measuring an indicator is related to the survey design. Go to Top

What is a target population?

Before monitoring program design process can begin, a clear, concise description of the aquatic resource is needed. In statistical terminology this description is called the target population. The target population refers to a different concept than that associated with the community, population, individual, genetic concept of biological systems. Both the "target population" for which information is wanted and the "elements" that make up the target population must be rigorously defined. The target population is the collection of elements about which information is wanted (Cochran, 1987; Särndal et al, 1992). A number of examples will give a better understanding of the concept of a target population and its elements. Go to Top

Lakes as a discrete target population

Assume that a study of lakes in Oregon is to be conducted and that questions focus on determining the number of lakes that have a particular characteristic, such as its trophic status. The target population might be all lakes in Oregon that are greater than 1 hectare and are wholly within the boundaries of the state. The elements of the target population are the individual lakes. Note that any lake which is partly in an adjoining state would not be included in the target population. To be rigorous it further requires that a definition be included for what constitutes a lake. For example, does the definition of a lake include man-made reservoirs? What if the lake is no deeper than 1 meter and is over 50% covered by vegetation? Are such lakes of interest for the study? The definition must be sufficiently rigorous and explicit that it can be clearly determined if a body of water is part of the target population. Note that in our example the target population is discrete, i.e., there is a finite number of lakes (elements) that comprise the target population. It would be possible to sample every lake, i.e. a census, however, that is generally logistically and economically unfeasible, an alternative being a sample survey.  Each indicator of trophic status determined for a lake results in a single value for the lake. Consequently, the summary information about the target population focuses on the number or proportion of lakes that have a particular trophic status. Go to Top

Lakes as a continuous target population.

Assume that a study of one of the Great Lakes, e.g., Lake Ontario, requires an estimate of the percent of the lake area with an unacceptable concentration of dissolved oxygen. In this case the target population is the entire surface area of Lake Ontario and the elements of the target population are all points within the lake. Conceptually, dissolved oxygen can be measured everywhere on the lake (an infinite number of points). Practically it is impossible to do so; consequently it is natural to sample only a limited number of locations. That is, it is impossible to complete a measurement on every element of the target population. A rigorous definition for the target population would likely involve questions of exactly what the boundaries are for Lake Ontario and whether the boundary definition involves a minimum water depth. These considerations are important for a field crew who must visit a proposed sample site and determine whether the sample site is included in the target population. Go to Top

Streams as a continuous target population.

A study is proposed to answer the question: What proportion of streams and rivers in Wyoming have a fish index of biotic integrity (IBI) greater than 50? The target population consists of all streams and rivers within the state of Wyoming. First, some initial questions must be addressed. Does this definition include the portion of Wyoming in Yellowstone National Park? Is the target population restricted to only perennial streams? Streams may be thought of as a linear network, such as is generally used to represent streams on maps and in geographic information systems (GIS). The elements of the target population are all the points within the linear stream network. In this case the target population consists of an infinite number of elements. This is similar to the Lake Ontario example, except that the elements occur on a linear network rather than over a two-dimensional area. Field samples would be collected at a sample of locations (elements) from the stream network and a fish IBI determined at each location. Go to Top

Estuaries as a continuous target population.

Suppose that a study of the estuaries in California is planned to determine the concentration of contaminants in sediments. For example, the state wants an answer to the question: What proportion of the estuarine area in California has unacceptable concentrations of mercury in sediments? The target population is all estuaries in California. For the purposes of the study an estuary is defined as any water body that is tidally influenced, saline, and has less than 50% of its perimeter adjacent to the Ocean. As a result, an estuary is defined at its lower boundary by its articulation with the Ocean or another estuary and its upper boundary by the head of tide. This definition results in approximately 75 different estuarine areas along the California coast. They range in surface area from 1092 km2 for the main body of San Francisco Bay to 0.09 km2 for Sweetwater River. The elements of the target population are all locations of sediments within the bounds of the estuaries. An infinite number of locations exist within each estuary and the entire target population consists of the collections of all the locations across all the sometimes disconnected estuaries along the California coast. Go to Top

What are Subpopulations and Why are They Important?

The target population defines the main aquatic resource of interest. Usually subpopulations of the target population are also of interest. For example, in a study of all lakes in Oregon two potential subpopulations might be natural lakes and man-made lakes (i.e., reservoirs). The study may also be designed to compare all lakes between 1 to 50 hectares versus all lakes greater than 50 hectares. Often an estimate of the number of lakes in each trophic status category is desired for each subpopulation. Note that the subpopulation of natural lakes overlaps the subpopulation of lakes between 1 to 50 hectares. Subpopulations do not need to be non-overlapping. They only need to be of interest in the study and explicitly defined.

Why is the definition of subpopulations important in the planning of a survey? Subpopulations arise from the questions that a study must answer. For example, the need for answers on natural and man-made lakes arises from questions posed at the initiation of the study indicate that the trophic status may differ between man-made and natural lakes. If such a difference exists, then different management strategies may be taken for the two types of lakes. During the initial planning of a study, it is typical for many subpopulations of interest to be identified. In each case a strong rationale can be given as to why the information on the subpopulation is important. A study can only meet the expectations of those requiring the information if clarity is reached on what subpopulations estimates will be provided.

Subpopulations are also sometimes referred to as domains of study or reporting units. These phrases imply that the subpopulations are known to be of interest prior to the conduct of the study. They are sufficiently important that the study would be viewed as incomplete if estimates for them did not appear in a report of the study's findings. During the statistical analysis of the study other subpopulations may be identified and reported on, but they would be viewed as providing additional information rather than essential to the study. Examples of subpopulations determined prior to sampling: geographic areas (States, counties), close to urban areas, resource definitions (wadeable, non-wadeable), ecoregions, etc.

Others can be determined following sampling, usually the information is only available at the time of sampling: salinity, substrate types, landuse category, small scale habitat features.  In general these are not available during the design phase, information that would allow allocation of samples in each subpopulation to assure sufficient sample sizes.

How does identifying subpopulations impact the design of the survey? The expectation is that the survey will provide estimates with acceptable precision for each subpopulation. Achieving acceptable precision requires having a sufficient number of sample sites occur within a subpopulation. Assume that 50 samples are needed to meet the precision requirements. If only the target population is of interest, then a total sample size of 50 is all that is required. However, if three non-overlapping subpopulations are identified, then 150 total samples would be needed: 50 in each of the three subpopulations. Note that in this case the precision for the target population will be better than required since it has at least 150 samples. Many times subpopulations overlap so that multiplying the number of subpopulations by the required sample size results in many more samples than is actually needed. For example, splitting lakes into man-made and natural and into 1 to 50 hectare and greater than 50 hectare may only require a total of 100 samples as long as the survey design results in 50 samples in each of the four subpopulations. The major impact on the design of the survey is the increased sample size requirements and the need for the survey design to make sure that the each subpopulation receives the minimum required number of samples to meet precision requirements. How survey designs can accomplish the allocation of samples to subpopulations is discussed under Common Survey Designs. Go to Top

When would I use stratification versus subpopulations?

Some of the usual reasons to stratify include: 1) administrative or operational convenience, 2) particular portions of the target population require different survey designs, and 3) increase precision by constructing strata that are homogeneous.  Designs for such strata tend to create more independent among strata and often contain subpopulations within them.  Impacts on the sample size needed to achieve acceptable precision are the same under both approaches, i.e. increases in the number of desired estimates also increases the number of samples needed.  Creation of subpopulations is usually undertaken to support unequal weighting, which provides a method for allocating sample to the subpopulations.  It also can improve the precision of the resulting estimates.  Creating subpopulations requires auxiliary information for each member of the target population during the design process.  For additional information on subpopulations, see the previous answer addressing subpopulations.  Go to Top

What is a sample frame?

Särndal et al (1992) define the frame or sampling frame as any material or device used to obtain observational access to the target population. Continuing, they state that the frame must make it possible " to (1) identify and select a sample in a way that respects a given probability sampling design and (2) establish contact with selected elements." The definition is abstract, suggesting that many options are possible to construct a sample frame. That is the case. Generally, the more information available for use in electronic form, the easier it is to develop a survey design that meets requirements of a study.

What does the definition imply when studying aquatic resources? The answer depends upon what information in a usable form currently exists about the location of the target population and all of its elements. For example, in the study of all lakes in Oregon, a list of all the lakes and their location may exist in a computer database. This list could then be used as the sample frame. Sample sites would be selected from the list and since their location is known they could be visited to obtain the desired measurements. This lake sample frame is very simple and easy to use. However, the list only has information on the lake name and its location. If the survey design required additional information, such lake size, to complete the sample selection, then the frame would be inadequate and an alternative sample frame for lakes would be needed.  Currently, for streams and rivers in the US, the RF3 GIS computer files of stream reaches is available as the frame for designs for these resources.  The primary requirements for the frame is that it cover the entire target population and be reasonably accurate. Go to Top

Sample survey versus census

Most people are aware of the term "census" from its use in relation to the decennial counting of the population of the United States as required by the constitution. Operationally the objective is to contact every individual in the United States to elicit basic demographic facts about them: sex, age, race, etc. The U.S. Bureau of Census devotes appreciable resources every ten years to carry out the census constitutional mandate. Can a census be completed for aquatic resources? The practical answer is rarely. A census of lakes in Oregon would involve visiting every lake in Oregon and obtaining on each lake the measurements specified for the study. The measurements from all lakes would be used to determine summary characteristics about all lakes. Although completing a census of lakes may be possible, available funds, logistics and personnel will likely make it impractical. A census of streams and rivers in Wyoming is not only impractical, but essentially impossible. When streams are viewed as a continuous resource in a linear network, a infinite number of elements exist in the target population making it impossible to visit each element. The same is true for estuaries viewed as a continuous population.

A sample survey is a way of collecting information on a subset of the elements of the target population with the intention of using the information to determine summary characteristics about the population. The summaries differ from the same summaries determined from a census in that they contain uncertainty. The uncertainty arises from the simple fact that not all elements in the target population were visited. How the sample is selected determines whether it is possible to know the uncertainty of the estimate.

A sample is any subset of the target population, i.e., any collection of its elements. Sampling methods may be classified into either probability-based sample methods or non-probability-based sampling methods. Probability-based methods are discussed in subsequent sections. Non-probability methods include chunk samples, expert choice samples, and quota samples. Go to Top

Chunk Samples. Scientists often draw conclusions using an arbitrary or fortuitous collection of sites. The sites are gathered haphazardly or "happen to be handy." Often the scientist implicitly assumes that the sites are typical for a larger universe of sites about which conclusions are desired. Such an assumption has only the individual's judgment as a basis and can not be easily defended. The sites are an unknown "chunks" of the target population and consequently no basis exists to make a scientific inference to the target population without invoking assumptions that can not be verified. Go to Top

Expert Choice Samples. Expert choice sampling is a form of judgment sampling that is a more developed form of non-random selection. An expert, or experts, may define a set of criteria to be met for a site to be included in the sample. Not all sites that meet the criteria are included. Criteria usually result in the designation of "typical" sites for the study. A fairly good sample may result given that the expert was skillful in defining the criteria and locating typical sites that met the criteria. However there is no way to be sure. A different expert would probably use different criteria or pick different sites that et the criteria. Without invoking additional assumptions, no basis exists to make inferences to the target population and know the uncertainty associated with the inference. Go to Top

Quota Samples. Quota sampling is commonly used in market research. The target population is divided according to one or more characteristics, e.g., age, sex, and geographic area. For two age groups, two sexes, and three geographic areas, a total of 12 population cells are defined. The cells are similar to strata in stratified random sampling. A quota sample then contains a pre-determined number of individuals in each of the 12 cells. The interviewer then simply "fills the quota" for each cell. The individuals may be the first individuals encountered or the interviewer may have the option of using judgment in selecting the individuals. The sample of individuals in a cell is either a chunk sample or a judgment sample. Individuals may refuse or be unavailable, but new individuals are contacted until the quota of individuals is achieved. However, the problem of selection bias due to non-response still remains. Hence as before there is no basis for an inference to the target population. Go to Top

What is a probability sample?

A probability sample is a sample where every element of the target population has a known, non-zero probability of being selected. That is, it is possible for every element of the target population to be in the sample. Two important features of a probability sample are that the probability selection mechanism (1) guards against site selection bias and (2) is the basis for scientific inference to characteristics of the entire target population.

Many alternative approaches are available to select a probability sample. Which approach is used depends on the objectives of the survey, the available auxiliary information, logistical or operational constraints in conducting the sampling, the characteristics of the sample frame, and the complexity of the statistical analysis. A few alternative designs are described below. Go to Top

Simple random sample. A simple example of a probability sample is one that gives every sample of a fixed size the same probability of being selected. This is simple random sampling without replacement. This is the simplest type of probability sample. Its major advantage is its simplicity not only in design but in statistical analysis of the survey results. Statistical analyses do not require any special procedures; consequently, users can analyze the data essentially ignoring the probability design. In reality the probability design results in the same assumptions that are used in standard statistical analyses. Its major disadvantage is that the design does not incorporate any information about the target population which would improve the efficiency (precision) of the survey and does not necessarily provide a sample that will address all the survey objectives. Go to Top

Stratified random sample. A stratified random sample may be the most common probability survey design used. When auxiliary information is available on the target population or the survey has multiple objectives, that information can be used to define strata. For example, a survey conducted over multiple states may have an objective to provide estimates for each state as well as for all states combined. In addition, it may be operationally convenient to have the sample for each state be selected independently from other states. This can be achieved by defining each state as a stratum and then selecting a simple random sample within each state. Strata may also be defined using a known characteristic of each element in the target population, i.e., auxiliary data. For example, streams may be categorized according to Strahler order in a GIS coverage that serves as the sample frame. Survey objectives may require that approximately an equal number of samples come from 1st, 2nd, 3rd, and 4th and higher Strahler order categories. This may be achieved by defining four strata based on the Strahler order of each stream. A stratified sample can be considered as a set of independent simple random samples, i.e., each stratum has a simple random sample. Consequently, the statistical analyses are similar to those of a simple random sample. Each stratum is analyzed as a simple random sample; then the estimates are combined across the stratum. The latter step must be completed correctly to avoid biased estimates. Go to Top

Unequal probability sample. An alternative to a stratified random sample is an unequal probability sample. An unequal probability sample is achieved by assigning a probability of selection to each element of the target population, usually depending on auxiliary information. For example, Strahler order could be used to assign a probability of selection to each stream segment where 2nd order streams would be twice as likely to be selected as 1st order streams, 3rd order streams four times as likely as 1st order, and 4th and higher order eight times as likely as 1st order. This type of design provides enormous flexibility in designing to meet objectives as well as a mechanism to increase precision. Their statistical analysis is more complex and requires that all analyses use weights derived from the unequal probability of selection. Go to Top

How many sample sites to use?

The most commonly asked question is: How many sample sites do I need? This is an important question as it directly determines the precision of any statement derived from the sample data. An answer requires detailed information on all the estimates that will be produced from the survey, the precision desired for each estimate, and knowledge of the variability expected. A reality faced in most studies is that the number of objectives creates a need for many more sample sites than budget and operational constraints allow. Consequently, the total number of sites in many situations is known from these constraints and the question is which objectives are the most important. It is usual to have some sub-objectives dropped due to sample size limitations.

Sample size calculations are available in most survey sampling textbook and will not be discussed here. One specific situation of interest is when the objectives call for the estimation of a proportion, e.g., proportion of stream length that meets a designated use. In this case, sample size calculations depend only on the proportion, precision required, and confidence required. Approximate precision estimates for proportions can be obtained by assuming the survey designs are simple random samples. Under this condition the estimated precision can be estimated using procedures given by Cochran (1987) for proportions. Go to Top

Precision, as a percent, is determined from precision = Z1-" * 100 * Sqrt[ p(1-p)/n]

To calculate precision requires knowledge of p, the proportion to be estimated. However, a conservative estimate of precision can be obtained by assuming p to be 0.5, which gives the maximum variance. Z1-" is related to the level of confidence required for the estimate. If desire 90% confidence, then use 1.645. If desire 95% confidence, then use 2.

Table 1. Precision to achieve 90% confidence in estimates of selected proportions.

Assumed

Proportion

(percent)

Precision with 90% Confidence

for alternative sample sizes

Precision with 95% Confidence

for alternative sample sizes
 

25

50

100

400

1000

25

50

100

400

1000

20%

±13

±9

±7

±3

±2

±16

±11

±8

±4

±3

50%

±17

±12

±8

±4

±3

±20

±13

±10

±5

±3

If the survey designs are actually based on the spatially-restricted survey designs, the actual precision estimates are expected to be lower (better) than those stated. Go to Top

How do Sites get selected?

Several processes lead to the selection of sites.  The first process identifies the resource characteristics and target population and results in a sample frame that contains all sites within the target population.  The second process establishes a spatial grid and hierarchical structure that result in cells containing single, or a small number/area of sites.  These two results are then combined resulting in each site, or small number/area of sites assigned a hierarchical cell address.  Randomization and statistical weighting processes produce a sequence of all sites from which a systematic random sample is selected.  For additional details see information on Survey design, Discrete Grid, and step-by-step site selection example. Go to Top

Why a sample size of 50?

A general objective of stream or lake surveys is to estimate the proportion of the population (of sample units) that meet a specific index of condition. For example, what proportion of the wadeable stream miles in a state don't meet their aquatic life use designations; or how many lakes in the state are eutrophic. Given that a statistical probability survey design is used to select a sample of lake or stream units from the entire population of several thousand such units, the estimate will have uncertainty associated with it. One measure of the uncertainty is its precision. The estimate is a proportion, p (i.e., the proportion of stream length that is impaired; or the proportion lakes that are eutrophic). Under a general assumption that the survey design is a simple random sample, we know the variance for the proportion depends only on the true proportion of units meeting the criterion and the sample size, n. Since the true proportion is unknown, we can make a conservation estimate of the variance by assuming the true proportion is 0.5, where the variance is a maximum. Under this scenario, with a sample size of 50, the precision when 50% of units meet the criteria (i.e., 50% of stream length is impaired) will be +/-12% with 90% confidence. If only 20% of the units meet the criterion then the precision will be +/-9%. If only 25 units are sampled, then the precision changes to +/-17% and +/-13% respectively. If 250 units are sampled over a 5-year period, then the precision changes to +/-5% and +/-4%. It is necessary to assume that conditions during the 5-year period remain constant. Note that for 250 units, the precision is +/- 6% and +/-5% with 95% confidence.

A critical element in any discussion of precision is the number "subpopulations" for which estimates are needed. If estimates are required for each of say, 5 ecoregions, then with 25 units sampled in each of the ecoregions the precision would be +/-17% and +/-13% (at p=.5 and p=.3, respectively) for each ecoregion and +/-5% and +/-4% for the entire study area. The sub-regions do not need to be ecoregions, but could be watersheds or some other subdivision of the landscape of interest. Another possible grouping of the units might be by percent of public or private ownership Hence a critical element on determining total sample size will be determining how many and what type of groups (subpopulations) will be of management interest. If the 25 samples are obtained over a 5-year period (that is, sample 5 units each year in each group), then the precision after five years is as stated. For any one year, the precision is +/- 37% when p=0.5.

Precision will be important when a determination is to be made on whether the proportion meeting the criteria differs between two different years, say 5 years apart. If the true proportion changes by 10%, e.g., from 20% to 30%, then what is the chance that the monitoring will detect this change? The better the precision the more likely the change will be detected. The probability of detecting the change depends not only on the sample size but also the specific survey design to be implemented. Decreasing the sample size decreases the ability to detect the difference. Is 50 a sufficient sample size? That depends on how confident we must be in detecting a change or estimating the true proportion. As an example, if the baseline proportion is 50% and after 5 years the proportion changes to 30%, then with a sample size of 50 the estimated difference would be 20% +/- 16% with 90% confidence. Since the confidence interval does not include 0, the conclusion is that a significant (at 90% confidence) change has occurred. This assumed that the sample units between the two periods were not paired, i.e., the same in both time periods. If they are the same, then other procedures can be used and are expected to be able to detect smaller differences between the time periods. Go to Top

Response designs - plot level design considerations

A monitoring design may be divided into two design parts: survey design and response design. The survey design selects which sites to visit during any particular sampling period, while the response design determines what and how to collect information at the selected sites. All the prior discussion concerned the survey design. Multiple response designs are likely to be used at each site. Typically, each field measurement requires its own specific field plot design, measurement protocol, laboratory protocol (if required), and calculation procedure to obtain a metric or indicator value of interest. Response designs may designate that only a single site visit each year will be made during an index time period. This is typical for monitoring designs conducted over large regional areas such as States. Nothing prevents the response design from requiring monthly site visits during a year or even collecting continuous data throughout the year. Choice of a response design is driven by the objectives of the study.  An example of a detailed protocol for a response design for wadeable streams is available on the web site:  www.epa.gov/wed/pages/EMAPManual.pdf  A diagram of that response design is included in this web site.Go to Top

Local Navigation

Jump to main content.