Review of NMFS' Draft Cumulative Risk Analysis Addendum, document ISAB 99-6

Review of the National Marine Fisheries Service' Draft Cumulative Risk Analysis Addendum:

"An Assessment of Lower Snake River Hydrosystem Alternatives on Survival and Recovery of Snake River Salmonids"

October 19, 1999 | document ISAB 99-6

EXECUTIVE SUMMARY

Introduction
In a letter dated May 28, 1999, from Mike Schiewe, the National Marine Fisheries Service (NMFS) requested that the Independent Scientific Advisory Board (ISAB) review the scientific adequacy of the April 14, 1999 draft document, "An Assessment of Lower Snake River Hydrosystem Alternatives on Survival and Recovery of Snake River Salmonids," prepared as an appendix to the US Army Corps of Engineers (COE) document, the "Lower Snake River Juvenile Salmonid Migration Feasibility Study." The NMFS document, generally referred to as the Anadromous Fish Appendix (AFA), attempts to evaluate a set of hydrosystem management options for their likely effects on listed stocks of Snake River Salmon.

The charge to the ISAB, in the letter from NMFS, stated a list of eight questions to help guide the review. These were:

Are the analytical approaches and methods appropriate?
Are data complete and of high quality?
Do the results support the conclusions?
Does the report accurately and fairly capture the degree of uncertainty in the projected outcomes?
Are the uncertainties adequately described? Is the degree to which research can resolve these uncertainties clearly identified?
To what degree did the failure of the PATH models to consider extinction affect the projected outcomes? Does this issue warrant further analytical attention?
Although the analytical focus of the A-Fish Appendix is the evaluation of whether specific breach or non-breach alternatives would meet survival and recovery criteria, did the analysis adequately consider other factors affecting salmon recovery?
Is the report balanced in its treatment of risks and uncertainties?

ISAB Responses
The ISAB found much to praise in the clarity and conciseness of the AFA, but our answers to many of the NMFS questions are not affirmative. Briefly,

The analytical approaches and methods suffer from:

Use of peculiar measures of performance (poorly related to survival, and recovery, as we understand these terms) in evaluating the consequences of each action,
Insufficient NMFS ownership of the analysis,
Over-reliance on optimistic scenarios, and
Isolation of hydrosystem decisions from other management decisions.

Some important inputs to the modeling were derived quantities, but were used as if they were data; and the data themselves had serious gaps that in fact account for much of the uncertainty in the analysis.

The three main qualitative conclusions -- (1) that dam breaching would probably make a greater contribution to salmon recovery than would leaving the dams in place and maximizing transportation; (2) that there is great uncertainty; and (3) that the uncertainty could be reduced in the future with further data collection -- are easy to agree with, but the attempts in the AFA to quantify the first two of these conclusions are unconvincing. We suspect that the uncertainties have been understated, and we are very concerned that insufficient attention has been paid to trying to quantify analysis of the third conclusion, advancing a delay option.

The implicit argument in the AFA in favor of delaying the hydrosystem decision does not adequately evaluate the biological costs or benefits of delay, and does not specify the substantial commitments and investments that would be necessary in order for the benefits of delay to be realized.

In subsequent sections of this report we elaborate on these brief comments, explain our answers to each of the NMFS questions, and present our own perspective on how the scientific support for the upcoming decision on hydropower operation might be better marshaled.

INTRODUCTION

In a letter dated May 28, 1999, from Mike Schiewe, the National Marine Fisheries Service (NMFS) requested that the Independent Scientific Advisory Board (ISAB) review the scientific adequacy of the April 14, 1999 draft document, "An Assessment of Lower Snake River Hydrosystem Alternatives on Survival and Recovery of Snake River Salmonids," prepared as an appendix to the US Army Corps of Engineers (COE) document, the "Lower Snake River Juvenile Salmonid Migration Feasibility Study." The NMFS document, generally referred to as the Anadromous Fish Appendix (AFA), attempts to evaluate a set of hydrosystem management options for their likely effects on listed stocks of Snake River Salmon.

The charge to the ISAB, in the letter from NMFS, stated a list of eight questions to help guide the review. These were:

Are the analytical approaches and methods appropriate?
Are data complete and of high quality?
Do the results support the conclusions?
Does the report accurately and fairly capture the degree of uncertainty in the projected outcomes?
Are the uncertainties adequately described? Is the degree to which research can resolve these uncertainties clearly identified?
To what degree did the failure of the PATH models to consider extinction affect the projected outcomes? Does this issue warrant further analytical attention?
Although the analytical focus of the A-Fish Appendix is the evaluation of whether specific breach or non-breach alternatives would meet survival and recovery criteria, did the analysis adequately consider other factors affecting salmon recovery?
Is the report balanced in its treatment of risks and uncertainties?

The ISAB report on its review of the AFA is organized into sections treating each of these NMFS questions.

ISAB RESPONSES

QUESTION 1. Are the analytical approaches and methods appropriate?

In the most general sense, the analytical approach underlying the PATH analysis used in the AFA was to attempt to fit a population model to a time series of data on population numbers and environmental variables (including the presence and operation of the dams), and then make predictions based on that model, in the context of alternative actions, to evaluate the predicted consequence of each alternative for the population. Because the fit to the population data was ambiguous, there was high uncertainty in the predictions. An attempt was made to represent this uncertainty quantitatively in the predictions by using a distribution of values for the model parameters, and then cumulating distributions of predictions. This general approach is logical, though it is well known that extreme circumspection must be exercised with such an approach to ensure that the modeling and statistical analyses stay in touch with common sense and elementary facts.

In detail, it is our judgment that the implementation of this approach was not satisfactory in the case at hand. We have four main misgivings about the actual implementation of the analysis in the AFA.

(a) We are not persuaded that the two measures of response of the populations, as scored in the model runs, were informative or appropriate.

The two measures of population response employed in the AFA were called a "survival criterion" and a "recovery criterion." The AFA uses these as indicators of the predicted success or failure of an action. However, we do not think that these criteria were adequate indices to the actual probability of survival or of achieving recovery.

These two criteria, used by PATH, in the PATH analysis on which NMFS relies in the AFA, were inherited from a 1995 effort of the "Biological Requirements Work Group" that was part of the resolution of the litigation over the 1994 Biological Opinion, and which served as the basis for the jeopardy criteria in the resulting 1995 Biological Opinion. NMFS was a party to the effort of the Biological Requirements Work Group, and of course was the signatory authority on the 1995 Biological Opinion, but it is not clear to us whether NMFS has officially committed to continue to use these particular criteria in its own decision processes under ESA for determining jeopardy, designating critical habitat, or determining recovery. In our reading of the criteria proposed by the Biological Requirements Work Group, we find these criteria insufficiently connected to the ESA considerations of probability of extinction, as we understand them.

Further, because other management actions besides manipulation of the federal hydropower system are under consideration for attempts at recovery of the listed salmon stocks, it is only reasonable that the evaluation of manipulation of the hydropower system should be in terms of that manipulation's contribution to recovery of the stocks, in conjunction with other management actions, which is rather different from asking whether this one action by itself is likely to bring about recovery. NMFS itself has issued a Biological Opinion on effects of artificial production, NMFS is involved in ongoing decisions about harvest, and many agencies are considering habitat and land use (e.g., the Interior Columbia River Basin Ecosystem Management Plan of the federal land management agencies.)

For these reasons, we believe better criteria could be drafted for purposes of Snake River salmon ESA decision making by NMFS. We recognize that the AFA was prepared to support an EIS that the COE is submitting under NEPA, so in this narrower context ESA standards may not be crucial. However, we also understand that this EIS is part of a suite of federal evaluations that will culminate, probably within a year, in a new "Decision for Hydropower Operation" as committed to in the 1995 Biological Opinion. This hydropower decision will be a jeopardy decision, by NMFS, under ESA, so it would be well for NMFS to develop survival criteria that will be appropriate in that context, as soon as possible. Similarly, there is an ongoing endeavor to draft a multispecies recovery plan that will include the Snake River stocks, and this too should include definitions of recovery consistent with the spirit of ESA. We do not believe that the "survival criterion" and "recovery criterion" used in the AFA are in tune with the current best scientific understanding for crafting survival criteria and recovery criteria for ESA applications.

(b) We are uncomfortable with the element of "black box" that results from NMFS deriving its AFA conclusions on the basis of a modeling exercise that was conducted by PATH, not by NMFS.

PATH has gone to some length to document its modeling efforts, but the modeling was extraordinarily convoluted, and we do not believe that the present documentation of either the models or the data sets is actually adequate for a third party to duplicate, or fully diagnose the analyses. In fact, it is our understanding that, at this time, NMFS is not in a position to duplicate the analysis independently.

Full ownership of the analysis seems especially important in light of the strong influence of several alternative assumptions, or assumption sets, used by PATH, in affecting the results. It is clear that making a choice between the two different "passage models", or making a choice between the several different assumptions about the "extra mortality" that is thought to affect Snake Basin stocks, can appreciably drive the analyses. This puts a premium on carefully diagnosing the influence of the assumptions on the model results, dissecting the evidence supporting the respective alternative assumptions, weighting each alternative appropriately, and dealing thoughtfully with the impasse that results if the choices cannot be resolved on the basis of statistical evaluation of available data. It is difficult to carry out such investigation at arm's length, which, however, was the relation between NMFS and PATH.

NMFS was a participant in PATH. But NMFS as an agency did not take a lead role in the process, and NMFS personnel did not take lead roles as individuals in the PATH modeling exercises that are at the heart of the PATH results used by the AFA.

Given the probable role of the AFA analyses in an eventual new "Hydropower decision" that is solely NMFS' responsibility under ESA, it would be appropriate for NMFS to be in a better position to explain, take responsibility for, and make accessible, the modeling assumptions, the modeling results, and underlying data analyses that it will use at that time.

We are rather surprised at the optimism reflected by the high "relative probability" accorded to population recovery, as that is very much at odds with the simple fact of unmistakable downward trends in many of these stocks over the recent past. We suspect that this discrepancy between the predictions of a complicated model and common sense observations about simple facts is ascribable to choices made during the course of the modeling, or in the decisions about assumption sets. These choices and decisions had the effect of "stipulating" the probability of certain features of the predicted future in the model, rather than letting all those probabilities be determined automatically from the statistical fit of the model to past data.

We appreciate that PATH carried out its predictions in the context of a kind of Bayesian decision analysis, where uncertain parameters were represented as distributions rather than fixed "estimates." This is a sophisticated treatment of uncertainty, which in general we think useful to advancing the discussions. We would point out, nevertheless, that certain details of the implementation of this approach still involve choices that can strongly color the conclusions. These details need to be understood thoroughly and treated thoughtfully.

In the present case, certain important parameters used in the predictive analysis were given distributions that were not the result of a formal Bayesian posterior analysis: i.e., these distributions were not formally shaped by a relationship to data. Rather, these uncertain parameters were treated as unresolved "hypotheses" that were thought capable of taking on a small number of discrete values; then, in a spirit of "fairness" these alternative values were assigned equal probability. For example, use of FLUSH or CRiSP estimates for passage mortality, use of FLUSH or CRiSP estimates for migration travel time, and use of the "alpha" or "delta" forms for the relationship between driving variables and the stock-recruitment function, were not weighted in accordance with their fit to data.

We wish to caution that this process, though it looks "fair," does not carry the force of scientific inevitability. The outcome could be manipulated by introducing or removing hypotheses, forcing the distribution one way or the other. In other words, the mere constraint of equal probability for the "hypotheses" does not guarantee objectivity or consistency. The choices of the hypotheses themselves drives the prior distribution used in the predictive analysis, and the prior distribution in turn can greatly influence the predictions. In other words, the list of "hypotheses" in this kind of decision analysis plays the same role as "assumptions" in less sophisticated modeling, and merits the same skeptical scrutiny.

(d) Perhaps the most disturbing of the shortcomings of the AFA analysis, in our opinion, is the attempt to isolate decisions about the hydrosystem from decisions about environmental management affecting the rest of the Columbia ecosystem as it impinges on the life cycle of salmon.

Basically, the AFA is an attempted analysis of the probable consequences for the Snake River salmon populations of a suite of actions restricted to: breaching or not breaching some of the Snake River dams, and eliminating or maximizing barge transportation. Implicit in this analysis is the proviso, "all other things being equal," where the list of all other things includes hatchery operations, harvest regulations, tributary habitat conditions subject to potential restoration or additional degradation, estuarine habitat conditions subject to the effects of flow management, mainstem predator populations, and ocean conditions. Of this list of other factors, the PATH analysis considered a very restricted subset of scenarios that does not exhaust the reasonable and prudent range of possibilities.

Construed most narrowly, the EIS "by definition" is concerned with decisions about transportation or possible breaching, which are activities that would be implemented by the COE. But this restricted scope of possible COE actions does not legitimize taking all other possible changes in management off the menu considered by NMFS in analysis of recovery options. We note that the common currency of results of the analysis reported in the AFA are predictions of the "relative probability of recovery" and "relative probability of survival" of index stocks. Poor management of the rest of the Columbia ecosystem could result in essentially zero probability of survival and recovery, regardless of the decisions to breach or not, or transport or not. Similarly, it is at least imaginable that extreme investment in ameliorating losses and enhancing the rest of the ecosystem might essentially guarantee survival and recovery, regardless of the decisions to breach or not, or transport or not. Thus, the implicit assumption of status quo with respect to management of the rest of the ecosystem is, at a minimum, disingenuous.

A proper analysis of probable responses to breaching and transportation management at least should be embedded in a spectrum of scenarios that considers different management regimes for the rest of the ecosystem.

We are especially concerned that the PATH analysis operated from a script in which habitat had only a very small part. It is worth remembering that the PATH exercise analyzes time series of data covering approximately the last 50 years. By the time this 50-year period began, the largest part of the decline of the Snake River salmon populations had already taken place, and much of the habitat degradation in major tributaries and headwater streams had already occurred. Even the relatively pristine, wilderness-area spawning grounds of some of the spring chinook stocks probably have had their productivity lowered because of reduced fertilization by carcasses, and the outmigrants from these wilderness areas still have had to negotiate passage through the rest of a river system that was considerably altered, physically as well as biologically, before the mainstem dams built in the last 50 years altered it even more.

In other words, the data series cannot be construed as including both "before" and "after" (or "treatment" and "control") with respect to habitat, since they begin after substantial habitat loss had occurred. It is not surprising, therefore, that the analysis does not reveal marked changes in population dynamics ascribable to habitat influences during the period of the data. This expected result of limitations of the data coverage does not justify an analytical conclusion that habitat is not important to the status of the Snake Basin stocks.

The PATH analysis, and therefore the AFA analysis as well, focus on the last set of insults to the Snake Basin ecosystem, but the last impacts are not necessarily the largest, the most important, or the most responsive to attempts at amelioration.

The AFA acknowledges the possible importance of the "other" factors, and states that NMFS is working on a more comprehensive analysis, called a cumulative risk model, that will take management of hatcheries, habitat, and harvest into account. We hope that this more comprehensive analysis will be available for use in the eventual 1999 Hydropower Decision. We would feel better about the situation if such an analysis had been available to serve as a basis for the present appendix to the COE EIS.

Even viewing the AFA most narrowly, as an appendix to an EIS, we might still question whether that EIS considered a broad enough range of alternatives. As those experienced in preparing NEPA Environmental Impact Statements recognize, selection of the alternatives for analysis can pre-ordain the result.

The operationally limited set of alternative actions considered in the AFA may have originated in an attempt to achieve focus, which of course has its place in complicated decision processes. However, the guidelines for environmental impact statements from the Council of Environmental Quality require a full exploration of alternatives. The Anadromous Fish Appendix, as a component of the Corps' impact statement for operation of the FCRPS, would thus seem bound to explore more numerous alternatives for their effects on target fish stocks than the set that was actually analyzed.

In thinking about the "missing alternatives" we find ourselves drawn to several that raise important ecological, genetic, and evolutionary issues that deserve attention, and that merit inclusion in the EIS to see how their implications might propagate through the biological and economic analyses. We might roughly consider these as elaborations on the current "short list" of alternatives, specifically extending the scope of analysis of enhanced status quo, transportation, drawdown to natural river grade, and the restoration of habitat.

Alternative A1 (Current hydrosystem operations under the 1995 Biological Opinion Interim Action) should be expanded to consider additional specific enhancements to the hydropower system without removing dams. This alternative assumes the present hydropower system remains intact and receives marginal improvements in the future. These marginal improvements are those indicated in the 1995 Biological Opinion, but the degree and timing of their implementation is not specified. They could be used individually or together, with compound applications most likely. It seems relevant and important for the appendix, and the EIS, to evaluate the performances of these specific improvements and their possible additive effects (at least in a general way).

Models for salmon management need to be flexible enough to evaluate multiple intervention alternatives. Specific system improvements that might have been analyzed are:

Fish-Friendly Turbines. Engineering development by the Corps, Department of Energy, and industrial partners portends major improvements in fish survival during passage through turbines. Exactly how much improvement is uncertain until further prototype testing is completed. Substantial reductions in mortality are anticipated. If the main source of mortality at many projects is in the turbine rather than in the tailwaters or forebays, and in-reservoir mortality is low in the Snake River (as indicated by some of the NMFS PIT tag studies), then this reduction could contribute to increasing the overall outmigration survival of smolts.

Surface Bypass Systems. Surface bypasses for smolts that are more effective than current bypass systems can likely be developed, based on continuing and planned research. Surface bypasses are considered in Options A6 and A6' (In-river passage options) but only with addition of flow augmentation. In Option A6 augmentation is as in the 1995 Biological Opinion with an added 1 million acre-feet of water from the upper Snake River; in Option A6' it is with that amount reduced by 427,000 acre-feet. Surface bypasses deserve to be analyzed on their own, without flow augmentation (which is of questionable benefit). Surface-bypass research involves both prototype development at dams and biological studies of behavioral guidance using several methods of providing guidance cues (turbulence, strobe lights, etc.). There is room to hope that such bypasses can be engineered to attain performance levels near that of Wells Dam, where 90 percent of the smolts pass through the bypass with less than 10 percent of the bulk river flow. It is important to learn whether the results at Wells Dam can be extrapolated to other dams. It would make a considerable difference to the analysis if large amounts of conventional spill were not necessary to ensure high fish survival, allowing more electricity production and higher fish survival (assuming bypass outfalls in dam tailraces are located away from predator concentrations). Effective smolt passage at dams might reduce hydropower-related mortalities to levels that also obviate the need for other alternatives such as transportation and breaching. The prospect of effective surface bypasses without flow augmentation deserves more detailed analysis, and flow augmentation itself needs to be tested over a larger range of flows.

Combinations of Actions (fish-friendly turbines, surface bypasses, intake screens). A combination of fish-protection devices will likely be used at most projects under the "status quo + improvements" option. This multiple use will result both from history (many projects will have had turbine-intake screens installed) and from new initiatives for installing surface bypasses and upgraded turbines. The combined applications will likely increase the biological diversity of species and stocks that are protected, and will add protection benefits for the major target stocks and peak run times. Further analysis seems desirable.

Flow Management and Withdrawal-depth Management. Flow stabilization has had positive influence on the productivity of the Hanford Reach fall chinook. Flow stabilization, tighter than achieved with the operating constraints already in place, could be evaluated for the Snake River, especially downstream from the Hells Canyon complex, where smolt mortality appears to be high even though riverine conditions persist. Flow management and re-engineering of withdrawal systems should be examined for the promise they might hold for achieving more favorable temperature regimes.

Other Improvements. While maintaining the basic system configuration, other improvements could be evaluated. Some have been successful, while others have not. Flow augmentation from headwater reservoirs has not had definable effects on either significant amounts of added water in the lower river or fish population increases. Most measures adopted so far, both alone and in combination, have not been successful in stemming stock declines. Evaluation and discussion of unsuccessful measures would be an informative part of the appendix to help us understand the prospects that further refinements of the present system might be more successful.

Alternatives A2 and A2' (which are A1 + maximization of transportation without or with surface collectors) propose increased transport, by barge and truck, of juvenile anadromous salmonids to below Bonneville Dam, thus circumventing the cumulative mortalities from sequential passage through multiple dams and reservoirs. A program operationally similar to the one currently in use is anticipated except that the A2' option considers increased use of surface collectors.

The ISAB recently evaluated the existing transportation program. It was our conclusion that the system selectively favors certain species and life histories. The resulting reduction in

species and life history diversity is not desirable. The ISAB provided technical recommendations for improving the effectiveness to protect diversity of species and stocks. The NMFS appendix does not include specific analyses of program modifications recommended by the ISAB, such as decreased reliance on trucking, minimization of holding times, etc. A more detailed analysis in the appendix or impact statement of the biological effectiveness and related costs of such potential modifications to the transportation program would seem to be needed. This extended analysis would elevate the level of discourse by bringing attention to biologically crucial issues, such as the role of stock diversity, and metapopulation organization, in ensuring long term survival of the salmon, and in providing mechanisms to buffer the total salmon population against natural variation, such as changing ocean conditions.

Alternative A3 considers drawing down all of the four lower Snake River reservoirs to natural river elevation through breaching of the dams. A related option, B1 includes full drawdown of John Day dam as well as the four Snake River dams. Both PATH and the NMFS appendix consider only full drawdowns and not any partial or seasonal drawdown approaches. This limited range of natural river options is technically unsatisfactory considering the widely different benefits to fish from other related options.

Partial Drawdowns. Although much discussed in the recent past, the option of partially drawing down reservoirs (one, several, or all) to a new, year-around, maximum pool elevation was dismissed as an option for biological analysis. Partial drawdown of Snake River reservoirs would convert the upper reaches of the present reservoirs to a somewhat more riverine condition. This change would increase spawning areas for fall chinook salmon, increase the rate of passage for spring and summer migrants, reduce lacustrine predators in dam tailwaters, re-establish natural interchanges between subsurface and surface waters that keep temperatures low, and likely achieve other benefits that deserve evaluation. The list of potential benefits seems to justify a search for creative solutions to the problems that partial drawdown will create for the functioning of engineered structures, such as the adult fish ladders, the juvenile bypass system, and the turbines themselves, that were originally designed on the assumption that reservoir levels would stay nearer to full pool.

Seasonal Drawdowns. Seasonal drawdowns of one or more of the Snake River reservoirs during the spring outmigration were also excluded from the biological analyses in the AFA. Yet seasonal drawdowns have been much discussed in the past decade, and a drawdown test for engineering purposes only was conducted on one reservoir in the early 1990s. This omission is inappropriate, in our view, because engineering and economic feasibility should be judged in the light of a clear presentation of biological benefits (or detriments). The decade or more of biological debate over seasonal drawdowns could at least be summarized, and it would be even better if the analysis were quantitative. Seasonal drawdown would undoubtedly speed the outmigration of yearling spring migrants. It could also delay return of spring-migrating adults with potentially negative results. Fall migrants would not be provided the needed spawning habitat that would come from partial, full-year drawdown, but more timely (earlier and cooler) outmigration of underyearlings could be stimulated. The interactions of these effects, both positive and negative, could benefit from further analysis using quantitative methods and modeling.

Selective Drawdowns (not all reservoirs). The relative benefits of breaching only one or a combination of Snake River dams, short of all four, have not been analyzed in the AFA. The recruits-per-spawner histories reconstructed by PATH show intriguing short episodes of productivity that are substantially higher and lower than the trend line during the 50-year period that they span. It would be worthwhile to consider the possible incremental effects of specific events such as the closure of John Day Dam. Conceivably, breaching a different selection of dams (rather than taking out the four lower Snake River dams as a group) could provide significant biological benefits.

There could be great value in an experimental selective drawdown of one reservoir as a test of benefits before other drawdowns are considered further. For example, only Lower Granite Dam could be breached with observations of the responses of salmon and steelhead populations (both short term such as adult spawning and juvenile migration speeds, and longer term population numbers). If this experiment were carried out in the context of a well-designed monitoring program, the results would contribute very substantially to our understanding of the effects of the dams on salmon.

Drawdown of John Day Reservoir Alone. Option B1 is for adding full drawdown of John Day Reservoir to the breaching of the four Snake River dams. However, the benefits of altering John Day Dam and reservoir alone were not analyzed. PATH analyses indicated a divergence of recruits per spawner between upriver (both Snake and Columbia) and lower river stocks shortly after John Day pool was filled, and before the filling of pools of Lower Monumental, Little Goose, and Lower Granite dams.

Both Columbia and Snake River stocks must pass through John Day pool, whereas the lower river index stocks used by PATH do not. The John Day River stock enters immediately upstream of John Day Dam, and thus experiences effects of dam passage but not reservoir passage in this project. For consideration of reservoir effects, therefore, this stock has more affinity to lower river stocks than upper river ones.

Considering that some of the NMFS PIT tag studies show high survival of smolts in the lower Snake River reservoirs, another location for the most damaging effects (or at least their expression as mortalities) might be hypothesized. The locale could be the John Day pool per se or the combination of that pool and the McNary tailwater. Effects of passage through the upper reservoir are currently indistinguishable from effects occurring in the immediate tailwater. Drawdown of John Day pool, totally or partially, would have major positive effects on fall chinook salmon spawning habitat, reopening one of the historically most productive spawning reaches in the basin. Travel time reduction for outmigrating smolts could be reduced, probably far more than the comparable benefit from breaching Snake River dams. As the longest reservoir on the mainstem, John Day pool takes the most time for spring and fall migrants to pass through.

Tributary Habitat. Biological effects of modifications to the FCRPS analyzed in the AFA do not include any measure of benefits relative to improvements in tributary habitats that are underway or planned. There is a strong opinion among many (but not all) biologists that the quantity and quality of salmonid habitat in tributary rivers, lakes and streams is more important than the mainstem for anadromous fish that migrate as yearlings. If this opinion is correct, then removal of the Snake River dams or enhanced transportation will be either relatively futile (if tributary habitat is not sufficiently improved) or unnecessary (if tributary habitat is improved sufficiently to raise the recruit/spawner ratio high enough to sustain populations). Either way, it seems critical for satisfactory evaluation of the selected alternatives that status and change in tributary habitat quality and quantity be included as either a discrete option or a major analytical component of analyses of other alternatives. We believe that analysis of effects of tributary habitat deserves another look, with an open mind as to the effects of habitat both on the carrying capacity term and the productivity parameter in a model, with attention to the possibility that habitat may influence spawning success, fry survival, rearing, overwintering, growth rate, and smolt condition, and accounting for the tendency of harvest policy to absorb increases in recruitment.

Time Scales for Habitat Recovery. The recovery standard used in the AFA is 48 years. Many types of habitat improvements may take longer than that to have their fullest effect. The analysis does not consider temporal variability in habitat, which, like variability in ocean conditions, is a major factor influencing recovery.

Estuarine Habitat. The AFA does not include evaluations of estuarine habitat restoration. Specific measures could have been evaluated. One example is the relocation of the Caspian tern colony on Rice Island. With the Columbia River estuary receiving attention from the EPA National Estuary Program, it seems logical that there are environmental restoration measures under discussion that would be relevant to salmon survival.

QUESTION 2. Are data complete and of high quality?

The data used in the analysis reported in the AFA are very limited, and some of the uses to which the data are put in the analysis are something of a stretch. Moreover, the PATH analysis, upon which the AFA discussion is based, used derived quantities, which were actually outputs of models, as inputs to some of the analyses. On close inspection it emerges, for example, that the "run reconstructions" and "passage survival values" were used in the PATH modeling as if they were "data," but these quantities actually were several long steps removed from direct observation or measurement.

The central set of salmon population numbers used in the PATH analysis is restricted to run reconstructions based ultimately on redd counts in particular reaches of stream for a small number of "index stocks." Various expansions and conversion factors were applied, in a very complex process, to convert the observed redd count numbers into estimates of returning adults at the Columbia River mouth. The population dynamics modeling treated the estimates of adult numbers like "data."

The "index stocks" are not necessarily representative of the listed stocks, or ESUs. The number of "index stocks" was rather modest, and when these were separated into groups according to "Snake River", "Mid-Columbia," or "Lower Columbia," the number of stocks in each group was definitely small. Nevertheless, comparisons between the dynamics of these groups of stocks formed the basis for estimation of a particular effect, called "extra mortality," that proved to be very influential on the outcome and interpretation of the analysis. We believe there is room for further exploration of the statistical validity of comparing such a small number of stocks for this purpose. It would be worth comparing the within-group variation to the between-group contrast that is responsible for the impression of "extra mortality." It would also be valuable to examine, on a larger sample of populations (and not just in the Columbia system), how patterns of variation in recruits per spawner cluster according to geographic proximity in general.

The run reconstructions use all the redd count data that are available for the index stocks, but this may not be all the census-like data that bears on the population dynamics of the listed stocks. And there are definitely data of other sorts, besides run reconstructions based on redd counts, that are pertinent to an evaluation of the status of the listed stocks.

The data used in the run reconstructions are of as good a quality as are now available for these stocks. But these data are still quite inadequate for answering the questions that the AFA (and PATH) pose. The run reconstructions cover a period from about 1950 to the present. The big changes in the Columbia and Snake system during this period were construction of most of the mainstem dams, massive hatchery production, smolt transportation, and possibly changes in ocean conditions. At most, a statistical analysis covering this period can attempt to determine the correlation between these factors and the population dynamics. In the actual event, the statistical analysis was not able to resolve the relative roles of these respective factors in accounting for the changes in population dynamics because all the factors, and the population dynamics, exhibited trends that constituted similar large-scale signals in the data. Small scale patterns in the data are ambiguous, for the possible signal does not stand out from a background of considerable noise, and the relation between possible causal factors and demographic results is clouded by uncertainty about possible time lags in the biological effects on salmon.

"Passage survival" in the PATH analysis was taken directly from the output of two other independent models. These two models gave different results, were calibrated to different data, and their fit to data did not influence the way they were weighted in the PATH predictive modeling. One of the passage survival models was calibrated to data on reach survival, but the number of reaches (and years) with such data was much smaller than the number of reaches for which survival rate predictions were made (and used). The other passage survival model was calibrated essentially to data from a predation study that was very limited in the time and space coverage of observations. A third approach, by NMFS, but outside PATH, to estimating more recent downstream passage survival was based on PIT tag detection estimates of reach survival. Estimates from actual tag detection data spanning about half the migration distance was extrapolated to the rest of the distance, though the reaches for which there are data are the upstream (and Snake River) half, while the reaches that are extrapolated are the downstream (and Columbia River) half. It is worth noting that the uncertainty over the so-called "D" value, which also has a crucial role in driving the AFA analysis, derives largely from uncertainties and disagreement about estimates of downstream passage survival.

The NMFS extrapolations of survival estimates have the good property of a very clear paper trail establishing their relationship to data. But the interpretation of these estimates, and their attribution to various routes of passage, is not as straightforward as one might think, and merits a few words of caution. The fundamental unit of smolt survival estimation, using the new PIT tag technology, is a survival rate estimate over a reach. The survival rate estimate for a reach requires observations of PIT tag detections in three locations: at the head of the reach for which the survival estimate is calculated, at the tail of the reach for which the survival estimate is calculated, and a location (or multiple locations) below the tail of the reach for which the survival estimate is calculated. The detections below the reach for which the survival estimate is calculated are used to estimate a detection rate for the station at the tail of the reach of interest, according to the model variously attributed to Jolly, Seber, or Cormack. This model, in its classical form, assumes that mortality is independent of detection, which of course does not apply to the salmon PIT tag system with detectors restricted to the bypass structures. This is because the premise of the bypass system was that it would provide higher passage survival than turbine passage, perhaps as high as spill passage.

In the reality of the present Columbia River PIT tag detection system, with passive detections, the survival estimate for the reach in question is based entirely on fish that were all bypassed at the head of the reach, but the detection rate estimate is based on a population of fish that took all available routes of passage at the tail of the reach. According to the Columbia-specific model of "tailrace to tailrace" reach-survival estimation, it is assumed that from the tailrace at the head of the reach in question to the dam face near the lower end of the reach in question, survival rates are unaffected by the route of passage at the head of this reach. In other words, it is assumed that survival rates of the bypassed fish that are being tracked are the same, over this distance, as the survival rates of fish that were undetected at the head of this reach. Next, it is assumed that mortality rates may differ by route of passage between the dam face and the tailrace at the tail of the reach in question, but it is assumed that all mortality in the bypass system occurs prior to the detection location, that fish that die in the bypass structure do not get detected, and that all route-of-passage-specific mortality associated with any of the passage routes is completed by the point of mixing in the tailrace where all passage routes converge. Under this set of assumptions, the detection rate estimated from the detections downstream of the reach in question is not confounded by passage history at the dam at the tail of the reach in question. Therefore, under this set of assumptions, the standard Jolly-Seber-Cormack formula for the surival estimates is an estimate of the average survival rate for the entire population of fish moving from "tailrace-to-tailrace" over the reach of question, with route-specific mortalities weighted in this average according to the fraction of the migrating population taking each respective route.

This is a mathematically pleasing result, and the label "tailrace-to-tailrace average" sounds good. But it must be noted that the assumptions are quite at variance with the mortality mechanisms that are believed to be operating in the various passage routes at the dams. Most specifically, it is now believed that a substantial portion of the dam passage mortality is mediated by predation in the tailrace and outfalls, and further there is now suspicion that route of passage affects subsequent mortality for some time and distance post-passage.

These complications rule out simple and straightforward calculations mixing "tailrace-to-tailrace average" reach survival estimates with route-specific estimates obtained from paired release experiments: i.e., one cannot simply factor out a measured route-specific estimate from the "average" in the hopes of back-calculating the route specific rates for the other routes. Likewise, the "tailrace-to-tailrace average" reach survival cannot legitimately be used in place of an unknown route-specific survival rate for undetected (and therefore spilled or turbined) fish, in an attempt to back-calculate, for purposes of SAR estimation, the initial size of a group of smolts that passed the dams undetected.

Because of these complications, our actual information base for estimating route-specific passage mortality, or even genuine population average passage mortality, over the entirety of the migration corridor, is remarkably thin, not withstanding the large numbers of smolts tagged and the large numbers of detections at a few of the dams. Such data as we have are mostly confined to slightly ill-defined averages over reaches that account for approximately half the span of the hydrosystem, and that account for approximately one third of the total migration corridor from Lower Granite Dam to the estuary. Errors owing to mismatches of assumptions, real variation specific to particular reaches and dams, or small sample sizes, compound badly when estimates for reaches with data are raised to a power of the number of reaches that must be "extrapolated."

Rectification of these data gaps, for survival estimates and for run reconstructions, will require very serious attention to monitoring design, deployment of detectors at new locations, and coordinating and merging of different kinds of monitoring results (e.g., hydroacoustic, radio-tag, and PIT tag).

QUESTION 3. Do the results support the conclusions?

This question does not specify which "results" and "conclusions" are to be evaluated. At the most general level, it seems to us that the conclusions of the AFA are three: (a) that breaching is more likely to achieve recovery of fish than leaving the dams intact and maximizing transportation,

(b) that there is enormous uncertainty about the probability of successful recovery, and time to achieve it, under any of the proposed actions, and (c) that there is a reasonable prospect for the uncertainty to diminish significantly over a time horizon of 5 to 20 years if intelligent investments in monitoring and experimentation are made during that period. At this level of generality the results (and the data behind them) do support the conclusions.

At the next level of detail and specificity, we are not sure that the message that the AFA seems to be sending is adequately supported by the information presented in the document. The implicit message of the AFA is that, not withstanding the acknowledged uncertainty and limitations of the technical analysis, the analysis presented really does provide useful scientific support for the pending "hydrosystem decision." The AFA seems to imply that it provides a basis for the salmon extinction risk considerations that bear on the decision that eventually needs to be made about dam breaching and transportation. Further, the AFA seems to imply that it provides a basis for evaluating the merits of possibly deferring any commitment on the dam breaching decision on grounds that important uncertainties may be resolved in the interim. We are not entirely comfortable with that message.

The AFA acknowledges that there are large uncertainties that are not technically captured as probabilities in the results of the PATH analysis. For that reason, the AFA proposes to interpret the PATH results as "relative probabilities" of the outcomes of the respective alternative actions, rather than treat these results as literal probabilities of outcomes. Mathematically, a set of relative probabilities are numbers that could be converted to literal probabilities just by rescaling or renormalizing. In this process, rank order, for example, would be preserved.

We wish to point out that some of the practical shortcomings of the PATH analysis, such as the exclusion of scenarios in which hatchery, habitat, and harvest were managed differently from present, do not result in "relative probabilities" if the assumption of "all other things being equal" is relaxed. It is theoretically possible that the rank ordering of the breach/not and maximize transportation/not alternatives would change under various alternatives for hatchery, habitat, or harvest management.

More importantly, it needs to be understood that relative probabilities are not an adequate basis for decision making when the alternatives carry significant costs. For example, if there were a 60 percent actual probability that an expensive alternative A would lead to recovery, while a cheaper alternative B conferred a 20 percent actual probability of recovery, the decision makers, in a larger social and economic context, might conclude that the increased cost of alternative A was justified by the large increase in expectation of recovery, from 20 percent to 60 percent. The ratio of relative probability in this case is 60/20=3.

Now consider a situation where expensive alternative A confers a 0.006 percent actual probability of recovery, and the cheaper alternative B confers a 0.002 percent actual probability of recovery. The ratio of relative probability is still 3 because 0.006/0.002=3. But it is easy to imagine that decision makers might well conclude that the added cost of alternative A was not justified, as the increment in probability of recovery is small, and even with the more favorable alternative the probability of recovery would remain extremely small. So, the bottom line is that "relative probability" is not an adequate basis for decision making under uncertainty. Actual probability, by contrast, is a satisfactory way to represent uncertainty in a fashion that can be used for optimizing a decision according to the rules of classical decision theory.

The inadequacy of the AFA analysis for evaluating the "delay" option, which the AFA puts on the table, is even more stark. A decision maker who wishes to responsibly weigh the merits of choosing "delay," needs to know the following quantities that should be supplied by technical analysis:

What is the probability that any of the stocks will go extinct because of the delay?

What specific commitment to additional monitoring and experimentation (including commitments to specific designs and sample sizes) are proposed to ensure that the delay period will be used effectively to reduce uncertainty?

What quantified reduction in uncertainty is predicted to result from implementation of that design?

What is the probability that the better informed decision at the end of the delay period will be a different decision than would have been made if delay had not been adopted?

With the answers to these questions in hand, the decision maker could take this technical scientific information, considered in its social and economic context, and decide whether the balance of biological risks and economic costs favor delay, or argue against delay. The AFA does not provide this technical scientific information to support analysis of the "delay" option. Further, it must be understood that the probabilities required for technical analysis of the delay option are actual probabilities, not relative probabilities, so the AFA is not dealing in the right currency.

QUESTION 4. Does the report accurately and fairly capture the degree of uncertainty in the projected outcomes?

We suspect that the actual uncertainties of the outcomes are rather larger than what was quantified in the PATH analysis. Our answer to NMFS Question 1 explains some of our misgivings about the implementation of the technical approach; our answer to NMFS Question 2 explains some of our misgivings about the data; and our answer to NMFS Question 3 explains our misgivings about the use of "relative probability."

QUESTION 5. Are the uncertainties adequately described? Is the degree to which research can resolve these uncertainties clearly identified?

The technical analysis is adequately described in the AFA. Indeed, we applaud the document as a concise and thorough representation of a chain of complicated analysis that heretofore has been somewhat shrouded in mystery, at least from the perspective of outsiders, perhaps because of too voluminous a documentation. This reporting of the technical analysis in the AFA does lay out much of the causes of uncertainty, though we suspect that the uncertainties are larger than the quantification of uncertainty reported in the AFA. Where we part company with the AFA is in the implicit argument that the uncertainty supports a decision to delay the "real" hydropower decision in the hopes that future research will reduce the uncertainties.

In principle we agree that the delay option is worth considering, specifically because the present uncertainties are very large, and there is a potential for reducing some of the uncertainty with additional research. However, we are not satisfied that the evaluation of this option in the AFA has adequate depth or specificity. In particular, we do not find in the AFA a sufficient explanation of exactly what future investments in research would be needed to resolve the important questions that are now in play, nor do we find an explanation of the administrative structure that might make a commitment to this investment. One of the lessons of the historical interlude since the 1995 Biological Opinion should be that the mere passage of time does not guarantee the accrual of necessary data. And our own appraisal of the seriousness of the present data gaps suggests to us that the integration of experimental design, monitoring design, and monitoring implementation necessary to solve the data problem would require coordination on a scale never before achieved in the Columbia. We think it fair to ask how this will be done, and who will do it.

QUESTION 6. To what degree did the failure of the PATH models to consider extinction affect the projected outcomes? Does this issue warrant further analytical attention?

We are not comfortable with the failure to quantify extinction probabilities. This issue needs further attention in order to address the concerns we raise in our answers to NMFS Questions 1 and 3.

QUESTION 7. Although the analytical focus of the A-Fish Appendix is the evaluation of whether specific breach or non-breach alternatives would meet survival and recovery criteria, did the analysis adequately consider other factors affecting salmon recovery?

Though the AFA acknowledges other factors, we do not think these factors were sufficiently taken into account. The theoretical implications of insufficient attention to "other factors" are explained in a generic way in our answers to NMFS Questions 1 and 3. More specifically, we believe that eventually NMFS and the region will have to embrace a far more integrated approach, rather than attempting to make decisions one "factor" at a time.

Considering just the breach and transportation factors relating to the hydrosystem leads to rather superficial judgments, because with the current uncertainty neither can be convincingly construed as a "make or break" decision. With the available information we are led to think that dam breaching would be a noble experiment in the abstract if costs were not an issue. After all, the dams could well be a contributing factor to the decline of the stocks; it is unlikely that breaching the dams will harm the stocks in the long run; and dam breaching on mainstem projects is one intervention that has not yet been tried in the Columbia system. But the real decision cannot be made in such a cavalier way because there are very significant social and economic costs.

The real questions that NMFS and the region need to address are: (1) What combination of interventions shows the best promise for achieving recovery of the stocks at an acceptable social and economic cost? and (2) What combination of interventions has the lowest social and economic cost among those combinations that have an acceptable probability for achieving recovery? We recognize that the policy component of defining what is "acceptable" in either of these questions will be extremely difficult, but it must be determined. The scientific component of both questions is also a challenge. Addressing the scientific component of the two questions will require serious analysis of the effects of combinations of interventions in many aspects of the ecosystem, and of course we must expect interaction among the interventions.

We wish that the draft of the AFA that is under review had shown more interest in this kind of analysis of multiple interventions throughout the ecosystem. The right analysis will take into account a broad menu of possible interventions besides dam breaching and transportation. The menu should include: fish-friendly turbines, surface bypass improvements, intake screen improvements, flow stabilization, partial drawdowns, seasonal drawdowns, selective drawdowns, drawdown of John Day alone, tributary habitat improvement, estuarine habitat improvement (as affected by flow, in particular), harvest management, and hatchery management. The present draft of the AFA does not do justice to this list.

QUESTION 8. Is the report balanced in its treatment of risks and uncertainties?

The AFA is honest in its admission of risks and uncertainties. But the AFA does not treat risk or uncertainty in a sufficiently quantitative way for meaningful judgment about "balance." More seriously, the treatment of risk and uncertainty in the AFA is probably too vague for the report to be much real help to the decision makers who may want to rely on it for the decisions that the EIS addresses, as we explain in our answer to NMFS Question 3.

In particular, the discussion of the "delay" option, which will doubtless prove attractive to hard-pressed decision makers, is much too casual. In that discussion, for example, the mention of new data bearing on revised estimates of D verges on the anecdotal. Matters that are this influential on the main conclusion of the analysis deserve the same sort of formality, documentation, rigor in analysis, and quantification of uncertainty as did the PATH analysis itself. We note that PATH did not formally analyze the "delay" option; so it is all the more worrisome that the AFA throws this option on the table without sufficient analysis. We are not enthusiastic about a decision spiral in which the "delay" option is chosen simply to provide time for properly analyzing the "delay" option.

To ISAB 99-7

^ top

Review of the National Marine Fisheries Service' Draft Cumulative Risk Analysis Addendum:

"An Assessment of Lower Snake River Hydrosystem Alternatives on Survival and Recovery of Snake River Salmonids"

Contents

EXECUTIVE SUMMARY

INTRODUCTION

ISAB RESPONSES