Executive Summary
“However beautiful the strategy, you should occasionally look at the results.”
Winston Churchill
“Failure is the tuition you pay for success.”
Walter Brunell
In 2007 the Independent Scientific Review Panel (ISRP) completed over twenty reports containing reviews of Fish and Wildlife Program projects. This Retrospective Report focuses on how projects are changing their objectives, strategies, and methods based on learning from the results of their actions. We accomplish this by looking at themes that emerged in previous ISRP retrospectives, examining a subset of projects that were reviewed in Fiscal Year 2007, and investigating how sponsors applied the results of their past projects to proposed future actions and monitoring. Some of our findings include:
Effectiveness Monitoring
Habitat restoration projects should include some form of effectiveness monitoring in their plans. This is essential to demonstrating that the project is achieving desired habitat results. Effectiveness monitoring does not have to be costly; in fact, showing some evidence of a beneficial habitat trend is usually sufficient for the ISRP’s needs. We do not in most cases expect individual habitat project sponsors to demonstrate target population benefits. However, demonstrations of population benefits at the watershed scale using approaches such as intensively monitored watersheds are always helpful. We do not wish to place an unfair monitoring burden on individual project sponsors, and we encourage collaboration among habitat restoration participants in documenting physical habitat improvements and population benefits. The ISRP continues to emphasize that each project without an effectiveness monitoring plan represents a lost learning opportunity.
Artificial Production
The ISRP understands that early in 2008 the Ad Hoc Supplementation Work Group will submit a final report outlining recommendations for evaluating the demographic and long-term fitness effects of supplementation. We recognize the significant progress made by the regional scientists and look forward to reviewing the report. We specifically anticipate that a plan for a comprehensive evaluation of supplementation is close to implementation. Such monitoring and evaluation should provide information to guide decisions on the efficacy of supplementation and its contribution to the restoration of salmon and steelhead.
Mainstem
In 2007 the ISRP and ISAB completed a review of the Comparative Survival Study’s (CSS) Ten-year Retrospective Summary Report , one of the largest M&E projects in the Fish and Wildlife Program. The CSS is a field study of the survival of PIT-tagged spring/summer Chinook and PIT-tagged summer steelhead through the Snake and Columbia River hydrosystem from smolts through returning adults, with a focus on relative survival of smolts with different migration routes (e.g., in-river, transported, different routes of dam passage, and different numbers of dams passed). For the most part, the CSS Ten-Year Retrospective was effective in answering the concerns posed by the ISAB’s review of the CSS 2005 Annual Report (ISAB 2006-3). It also provided improved clarity in the presentation and explanation of the sophisticated methodologies used in analyses of CSS data.
However, the ISRP, with concurrence of the ISAB, found that one of the major project objectives (Upriver/Downriver Comparisons) did not meet scientific review criteria, because of inevitable confounding from other factors in establishing cause(s) of upriver/downriver differences that may be detected, regardless of sample size and detection power that could be achieved.
Project and Program Reviews
The ISRP’s major effort for the year was conducting follow-up reviews to address unresolved issues from the review of 540 FY 2007-09 proposals. In addition, we reviewed proposals seeking to test innovative technologies and methods; completed a retrospective review of results reporting in FY 2007-09 proposals; provided a potential framework for future project reviews; and with the ISAB jointly reviewed a handbook of salmonid field protocols. In this Retrospective Report, we examine a subset of projects that were reviewed in FY07, and assess how the project sponsors learned from their past actions and, based on lessons learned, proposed future actions and monitoring.
Adaptive Management in the Fish and Wildlife Program
Almost without exception, project sponsors state that they use adaptive management to modify the tasks and work elements in proposals. At the same time, however, project proposals almost never provide (1) an experimental design to identify whether biological objectives have been met by employing specific strategies or (2) a decision tree that would be used to modify management based on updated scientific information.
Often project sponsors propose to continue to employ tasks and work elements even when monitoring data indicates that biological objectives are unattainable because of environmental and biotic conditions. Sponsors of these projects are understandably reluctant to abandon these efforts in which they have invested much time and energy. A structured or integrated decision management process, linking the biological requirements needed to sustain fish populations together with the legal and cultural setting, is one approach to create the human dimension required to confront unrealistic biological objectives. Exploration of decisions and alternatives requires clear problem definition, establishing agreed objectives, development of alternatives over several iterations, and evaluation of consequences. The latter requires stakeholder and expert input, aided by simulation modeling.
