MEMORANDUM
TO: Doug Marker, Fish and Wildlife Division Director, Northwest Power Planning Council
FROM: Rick Williams, ISRP Chair
At your request, the ISRP summarizes its interactions with the Action Agencies and NOAA Fisheries' Research, Monitoring and Evaluation (RME) effort. Three primary observations emerge from our interactions thus far. First, the Council and BPA have not agreed at a policy level how the RME effort will be integrated with the Fish and Wildlife Program. Second, the ISRP's review role in the RME effort is not established, and consequently ISRP and RME group interactions are characterized by informal, but hurried, requests for reviews with inadequate time provided for ISRP review of RME group products and for the RME group to address ISRP concerns. Third, while the RME effort potentially offers the top-down RME framework and leadership that is needed to ensure the region is collecting critical data and developing appropriate experiments to address key uncertainties, the RME plan and process need to be developed further and independently reviewed to ensure the greatest likelihood of success. Success of the RME effort is dependent on assurance of sound scientific methods, as well as regional and local buy-in beyond the Action Agencies.
The RME group is currently interested in short turnaround ISRP reviews of three items: a) Request for Proposals (RFPs) and Requests for Qualifications (RFQs); b) a subsequent review of proposals to meet the RFPs; and c) a broader review of the draft RME plan titled, "Research, Monitoring & Evaluation for the NMFS 2000 FCRPS Biological Opinion." The ISRP awaits the Council's direction on whether to undertake those reviews and hopes this memo spurs the Council and BPA to formalize a review process and interactions between the ISRP and the RME group. Certain topics and products generated by the RME effort may also be appropriate for ISAB review.
To date, ISRP interactions with the RME group have occurred primarily through the Mainstem and Systemwide project solicitation and review process. The ISRP was officially introduced to the RME effort in late March 2002, when the Council and BPA requested the ISRP to review draft guidelines for action effectiveness research proposals developed by the RME group for the Mainstem and Systemwide solicitation. On April 19, 2002, the ISRP provided a memo to the Council concluding that the draft RME guidance document was a useful first step at developing necessary guidelines for planners, investigators, and reviewers, but it was too narrow and insufficiently targeted toward actual information needs to facilitate universal evaluation of action effectiveness research proposals (see ISRP 2002-5). The ISRP recommended two types of revisions. First, revision as a scoping document for planners and administrators was needed to provide clear top-down guidance that actually stipulates overall design specifications to address the need for collecting data to answer the BiOp check-in questions about effectiveness of mitigation actions on salmon survival. Second, revision as a more methodology-oriented document intended for use in a bottom-up fashion by researchers and technicians was needed where guidance on alternative methods, statistical approaches, and statistical design requirements are given in detail.
On April 24, 2002, five days after the ISRP memo was released, the Mainstem and Systemwide solicitation was open to the public. It is the ISRP's understanding that the RME group did not have time to revise the RME guidelines to incorporate the ISRP review comments and instead, the original RME guidelines with the ISRP comments attached were released as part of the Mainstem and Systemwide solicitation. Consequently, potential project sponsors were given mixed messages from the funding agency (BPA) and the review body (the ISRP). This initial scenario is symptomatic of ISRP and RME group interactions in general. It appears that ISRP review is not factored into the initial RME group schedule to develop products, but is added at the last minute. This prohibits the ISRP from reviewing the RME product in the context of the larger RME effort and providing the most useful service. It also does not allow the RME group to adequately address the ISRP concerns.
On May 20th, 2002, the ISRP met with the RME group and discussed the April 19 memo. Discussions at that meeting confirmed that the RME group had the potential to lead the development of a basinwide RME plan for ESA listed anadromous species. However, the ISRP and RME group had real differences of opinions on methods, which have yet to be resolved. In addition, the RME group emphasized that it was under extreme pressure to meet 2003 BiOp check-in points and suggested that the RME group may need to develop and solicit projects on such a rapid timeframe that ISRP review would not be tractable. The ISRP was not comfortable with this last assertion and notes that significant progress has occurred over the past five years in developing an accountable and scientifically rigorous project review process. The ISRP's review role is a key part of the region's accountability process and therefore, it is important to maintain ISRP review of projects funded through the Fish and Wildlife Program. Assumedly, all projects funded through the Fish and Wildlife program have gone through this rigorous ISRP, CBFWA, Council, BPA, and public review. The ISRP noted that its reviews, if scheduled in advance, are efficient and rapid.
The Mainstem and Systemwide Review — Summary of ISRP and RME review process:
The Systemwide/Mainstem solicitation placed emphasis on projects that would meet the Action Agencies' responsibilities under the National Marine Fisheries Services' FCRPS 2000 Hydro Biological Opinion (BiOp), especially those responsibilities associated with Reasonable and Prudent Alternative (RPA) actions 179-199. The review process included a front-end review of projects by the RME group (comprising scientists from the Action Agencies and NMFS) to provide some preliminary information to the ISRP and project sponsors on the ability of proposals to meet the RME needs identified in the Biological Opinion, or as further defined by the RME group. The process was intended to aid in the development, selection, and funding of a suite of integrated projects that will meet the intent of BiOp RPA actions in the most effective and economic way.
To their credit, BPA and the RME group participated in the Mainstem and Systemwide project selection process from start to finish at a greater level than during any of the preceding rolling reviews for the other provinces. This participation was an important step towards grounding the RME effort in the Fish and Wildlife Program and specifically attempting to meet Reasonable and Prudent Actions (RPAs) associated with RME efforts through existing projects and an ongoing Fish and Wildlife Program process. Unfortunately, the exact mechanism and schedule for integrating the RME group and ISRP review was not adequately discussed before the review began and opportunities to consolidate the review effort with the ISRP were not explored. Instead, both groups made the best of an awkward and difficult situation and collaborated to the extent time allowed to provide context and critiques of each group's reviews to project sponsors. The ISRP's final Mainstem and Systemwide report summarized programmatic issues related to its review role and the RME effort that arose during the review process.
There was some initial concern that the RME group and the ISRP would provide inconsistent comments requesting divergent approaches from the project sponsors in the response loop. Consequently, the ISRP agreed to review the RME group comments for consistency with the ISRP review team comments. The RME group noted that approximately 40% of the 104 submitted proposals potentially addressed implementation of the RME BiOp RPA action items. For these proposals, the RME group provided written comments on the extent to which the proposed project would meet the RME requirements of the BiOp. Some of the comments identified shortcomings in the proposal relative to BiOp requirements and proposed modifications to more directly meet the intentions of the RPA actions. The RME group comments were first released to the ISRP and the project sponsors during the week of July 22 and were included in the ISRP's preliminary report for the relevant proposals following the ISRP comments.
In our preliminary report, the ISRP remarked on the RME group comments on the 43 projects reviewed by the RME group. For the most part, the ISRP and RME group comments were consistent or addressed different criteria and were not in disagreement. Most project sponsors adequately addressed the initial RME concerns; subsequent RME comments acknowledged this. RME comments were generally helpful and consistent with ISRP comments.
One issue that may need policy attention deals with the fix-it loops and the RME review process. The ISRP found some proposals designed to meet RPA/RME needs technically inadequate. However, according to RME Group comments, some of these proposals, once reworked, were critical components of their plans to meet RPA needs. This raises a process issue of whether to revise proposals until they meet ISRP criteria and RME needs or to solicit another RFP. Does the Council make an exception to established process for proposals because of the perceived BiOp priority and timeline? There is a fairness issue associated with such iterative "fix-it" reviews for these particular projects, given the single pass through of the fix-it loop for other FWP proposals under ISRP review.
Another difficult issue deals with potential conflict of interest for RME members in the review process. Because NMFS authors the BiOp and RPAs, there is significant potential for conflict of interest when the RME group, whose membership includes NMFS personnel, makes RPA and RME recommendations on NMFS' and competing proposals. This internal review is analogous to the CBFWA review of its own members' projects, which at times are specified as measures in the Fish and Wildlife Program. One of the reasons Congress created the ISRP through the 1996 amendment to the Power Act was to provide an independent check against this inherent conflict of interest. The potential for conflict of interest with NMFS is particularly acute given the threat of ESA non-compliance. Formalizing the ISRP / RME program review process may provide a useful safeguard here, especially given the funding and implementation priority that BiOp and RPA actions have recently been given.
Finally, the ISRP provided extensive review comments on the need for a basinwide coordinated RME effort, focusing in particular on the Collaborative Systemwide Monitoring and Evaluation Program (CSMEP) project (#35033), proposed by CBFWA, which the ISRP reviewed favorably. The CBFWA proposal provided an environment for developing and coordinating common data collection protocols and standards. The ISRP continues to believe that there is a useful role for CBFWA in developing and implementing a coordinated basinwide RME program. As compared to the NMFS/Action Agencies RME proposals in the Mainstem and Systemwide review, the CBFWA proposal was broader, both in scope and participation and, therefore, had a higher probability of success. Consequently, the ISRP recommended that it receive priority for immediate funding.
Recently we met with the RME group (29 January 2003) for a quick briefing on development of the RME work plans. The briefing was interesting but adequate time was not available to address the many ISRP questions raised by it. Clearly, the RME group has put in substantial effort on developing the draft RME framework document and identifying potential gaps that remain after the Mainstem and Systemwide project review process. On the latter point it is important to note that the RME effort identified that BiOp RPAs associated with research, monitoring, and evaluation are largely met by ongoing and new work reviewed and recommended by the ISRP, CBFWA, Council, BPA, and NMFS in the rolling review process. Pursuing a targeted solicitation approach to meet remaining gaps is a logical course that the ISRP has recommended before. However, at first glance, the ISRP was not convinced that the RME group's selection of gaps (and suggested RFPs) was appropriate given that projects reviewed during the rolling review process addressed the same uncertainties. Perhaps the RFPs and RFQs are justified, but for the ISRP to provide meaningful review comments and suggestions, the ISRP would need time to review any RFPs or RFQs in the context of the draft RME framework document and the ISRP's previous project reviews.
As a role between the ISRP/ISAB and RME group is formalized, the ISRP notes it works as an independent consensus body. It is ISRP/ISAB policy that individual ISRP/ISAB members are not enlisted to work on workgroups whose product may be later reviewed by the ISRP/ISAB. Similarly, the ISRP does not sit on project review panels with participation beyond the membership of the ISRP and its Scientific Peer Review Group. That said, as evident in the rolling reviews, the ISRP review process operates at its best when the process is transparent and there is an avenue for dialogue between the ISRP and those being reviewed. The programmatic nature and long-term implications of the RME effort call for a creative review process that provides adequate time for dialogue, independent ISRP comments, and subsequent time for the RME group to address those comments.
Finally, the ISRP offers our best interest in making this process work and our willingness to assist Council in any further questions or reviews related to it.
