Fish & wildlife Program FY 1999 Annual Implementation Work Plan

"Cost-effective measures" determination October 1998   |   < back to work plan contents

Under the 1996 amendment to the Northwest Power Act, the Council must, in making its fish and wildlife project funding recommendations to Bonneville, "determine whether the projects employ cost effective measures to achieve program objectives." The scientific review by the Independent Scientific Review Panel (Scientific Panel, also known as the ISRP) and the Council's cost-effectiveness review together should greatly increase the likelihood that projects funded will be biologically promising and economically sound.

The legislation did not specify any particular approach to cost-effectiveness analysis. It does not require, for example, the use of a single measure of biological effectiveness as a basis for comparison among projects, nor the use of strictly quantitative analysis. Because of this, the Council took several steps last year to understand the state of the art in natural resource economics and cost-effectiveness analysis to make the determination required by the Power Act.

First, the Council established an Independent Economic Analysis Board (Economic Board, also known as the IEAB) to provide advice on and improve economic analysis of fish and wildlife recovery measures. The Economic Board members have substantial experience in areas of natural resource economics; irrigation and agricultural economics; water use and policy; river transportation economics; fishery economics; local-area economic impact assessment; non-market valuation of natural resources; electricity system configuration and economics; and U. S. Army Corps of Engineers project evaluation. The Economic Board has helped the Council determine how to analyze the proposed projects and make funding recommendations that are consistent with the statute's call for cost-effectiveness review.

Second, the Council produced, with the Economic Board's help, a discussion of "Methods of Economic Analysis for Salmon Recovery Programs" (July 30, 1997, Council Document No. 97-12), for the purposes of initiating the cost-effectiveness review in Fiscal Year 1988. The Economic Board reviewed the paper and supported the analysis and conclusions. The methods analysis concluded that several problems make it difficult for the Council to undertake a quantitative cost-effectiveness comparison among fish and wildlife projects using a single, quantified, measure of benefits to determine which projects produce the greatest benefit per dollar. The problems include the lack of agreement on measures of biological effectiveness; the fact that the complex life-cycle of anadromous and resident fish makes it difficult to isolate the biological effects of particular activities or to compare different biological effects of different kinds of projects; and the fact that in the prioritization process, different project sponsors provide different kinds of cost and economic information, which makes cost comparisons difficult.

Based on the methods analysis and the Economic Board's advice, the Council concluded last year that it could not undertake a quantitative cost-effectiveness comparison of the projects, primarily due to the inability to quantify the expected benefits of particular projects. The Council concludes the same this year. Whether this will be possible in future years is still not known. A quantitative cost-effectiveness comparison would require a far greater understanding of the biological effectiveness of actions than we have now. It would also require a better defined set of biological goals and objectives for the Council's Program to be able to make a quantified analysis of whether projects are cost effective in meeting "program objectives." The Council, the Scientific Panel and the Economic Board have all urged the development of a more coherent analytical framework for the Fish and Wildlife Program. A clearer, more comprehensive Program framework could provide a sounder basis for establishing measures of effectiveness, perhaps allowing in the future for a multi-variable quantitative cost-effectiveness comparison of projects as described in the Council's methods paper. Beginning at the end of Fiscal Year 1998, and continuing into Fiscal Year 1999, the Council, in cooperation with the other governments in the basin, has initiated such a framework development process.

As highlighted last year, however, the Council's methods analysis noted that there is more to cost effectiveness than a quantitative comparison of the costs of alternative ways to achieve a single biological objective. Cost-effectiveness review may suggest procedures for project review, selection and management that emphasize efficiency and accountability, making it more likely that projects funded will be effective and efficient, even if these changes cannot be reliably quantified. The methods analysis recommended four strategies to help improve the cost-effectiveness of projects proposed for funding: emphasizing the role of independent scientific review in increasing the cost effectiveness of the Council's Program; improving the level and nature of cost information provided; evaluating the results of specific projects; and improving contract selection and management procedures. Most of last year's "cost-effectiveness determination" from the Council consisted of a description of what the Council has done and is doing to implement these strategies. That is also the bulk of this year's statement.

Role of independent science review in cost-effectiveness review. As noted last year, the central change brought about by the 1996 Power Act amendment -- independent scientific review -- is also the most important element in improving the cost effectiveness potential of this project selection process. The purpose of the Scientific Panel is to provide an independent scientific assessment of the biological effectiveness of the proposed projects. The independent science review process has proven useful in raising questions about the effectiveness of certain types of projects, project management and funding priorities. As the Council did last year, it used the Scientific panel's report and a staff review of the proposed funding package to bring public scrutiny on a series of funding issues over the summer of 1998, detailed in the Council's decision document for its Fiscal Year 1999 funding recommendations.

One example concerns a specific criticism of watershed habitat projects that the Scientific Panel couched directly in term of cost-effectiveness. The Panel objected that watershed project proposals did not give sufficient evidence of being preceded by watershed assessments, by which the Panel meant some indication that the habitat restoration proposals had been considered "in the context of the drainage system as a whole and had used existing watershed information to develop a technical justification for their efforts." Without this information, the Panel was "unable to judge whether the projects were worthwhile, or whether they would result in expensive, high-maintenance projects whose effectiveness was limited." The Council accepted the Scientific panel's prescriptions for change -- to begin addressing in Fiscal Year 1999 a set of "questions and concerns" identified by the Panel that should produce the type of information that would result from the evaluation of how a project relates to a watershed assessment, and to provide notice that within two to three years no watershed project will be recommended for funding without a demonstrated link to watershed assessment information.

Last year, which was the first year of the new procedure, the Scientific Panel had time only to get organized and provide some programmatic considerations. This year, the Panel had the opportunity to review and comment on all projects proposed for funding. Perhaps the most important reaction from the Scientific Panel this year was that many of the project proposals are not described appropriately for independent scientific review, leading to a Panel characterization of a significant percentage of the projects as "inadequate proposals." The Scientific Panel also criticized many of the projects for failing to report results, and the managers overall for inadequate efforts at reporting accomplishments or progress towards meeting the objectives of the Program and at explaining how vast amounts of information acquired each year through the Program are used to improve the Program. The Council responded by calling on project sponsors and Bonneville to address the technical criticisms of projects in the statements of work at contracting for this fiscal year, and by calling on the fish and wildlife managers to work with the Scientific Panel over the next year to improve the quality of project proposals and to report results and progress towards objectives in a more meaningful and extensive way.

The Scientific panel's criticism with regard to reporting and evaluation of results is consistent with the Power Act amendment's charge to the Panel to focus attention on improving monitoring and evaluation (that is, the panel's funding recommendations are to include a determination that projects have "clearly defined objectives and outcomes with provisions for monitoring and evaluation of results"). This is a further step toward improving our long-term understanding of the effectiveness of projects that implement the Council's Program. As indicated, over the last few years the participants in the Program have greatly increased the amount of information that is collected as part of fish and wildlife project implementation. What still needs to be developed is a more systematic framework for evaluating this growing collection of information. Part of the process of developing a revised Program framework should include producing a functional monitoring and evaluation framework for the Program. The coded-wire tag projects funded by Bonneville out of the direct program budget provided an example this year of what needs to be done in this area. Coded-wire tags provide a wealth of information, but the relevance of that information to the Council's Fish and Wildlife Program is unclear, and one of the obstacles to knowing how to reform the coded-wire tag effort is the lack of a sophisticated monitoring and evaluation plan for the entire Program. After a review of the coded-wire tag program, the Council recommended the development of an appropriate protocol for the use of coded-wire tagging as part of analyzing the broader monitoring and evaluation needs under the Council's Program. This is but a first step in addressing this problem.

Improved cost information/increased fiscal review of capital investments and operation and maintenance obligations. A second strategy recommended by the methods analysis has been to bring better cost information into the decisionmaking process. Council staff has worked with Bonneville, the managers and the project sponsors to develop and provide better cost information about projects and about the Program allocations in general. This is exemplified by, for example, the improved cost information in the project submissions, in the cost and budget allocation information displayed in CBFWA's draft implementation workplan and subsequent comments, and in the Scientific panel's report.

More important than the simple display of cost information, however, continues to be increased scrutiny of the components of those costs and their long-term financial implications for the Council's Program. Consistent with the Council's recommendations in Fiscal Year 1998, the past year has seen increased fiscal scrutiny by the Council, Bonneville and others during the final design phase of production projects, to ensure more attention to capital investment needs and operation and maintenance cost expectations; better descriptions of project-specific operation and maintenance costs overall; compliance with the Council's direction that the operation and maintenance costs set forth in project descriptions be treated by Bonneville, the managers and the project sponsors as estimates, with the actual budget allocation for operation and maintenance expenses to be determined during contracting; better efforts to develop and stick to a more realistic schedule for the capital investments; and increased consideration of the procedures of "value engineering" to find more cost-effective ways of completing major capital improvements.

The Council continues to be concerned about the Program's ever-increasing obligation for funding operation and maintenance, which may not be adequately accounted for, described and limited. In last year's funding recommendations, the Council recommended that Bonneville work with Council staff and project sponsors to develop better long-term operations and maintenance projections for projects, including standards for determining what are appropriate expenditures for operations and maintenance. The Scientific Panel noted this year that project proposals for similar tasks carry very different budget estimates, and the Panel recommended that the Council "systematically evaluate budgets among projects for consistency and reasonableness."

Some progress has been made. The Wildlife Working Group in particular developed rigorous standards for determining appropriate operation and maintenance costs for wildlife projects. In other parts of the Program, CBFWA asked project sponsors to estimate out-year expenses, as part of developing a proposed ten-year budget estimate. But the work to standardize operation and maintenance expenditures and develop better operation and maintenance funding projections based on those standards still needs to be done. For this reason, the Council recommends this year that the rest of the Program follow the lead of the Wildlife Working Group -- Bonneville is to work with the managers and project sponsors (and the Council staff) to develop better long-term operations and maintenance projections for projects, including "standards for determining what are appropriate expenditures for operations and maintenance," with something similar occurring for the appropriate application of overhead costs. The Council expects progress on this recommendation before next year's Council decisions on project funding.

Project review. A third strategy recommended by the Council's methods analysis is to evaluate the record of existing projects. To reiterate from last year, projects that have been ongoing for some time should have yielded measurable effects or have contributed concrete knowledge about fish and wildlife problems. A sampling of projects could be evaluated to determine what benefits they have yielded for the money expended. This exercise should introduce accountability into the process as well as provide a better understanding of how to specify measurable objectives in future project information sheets. On-going project reviews are essential for an adaptive management approach to Program design and implementation.

Thus in the months leading up to the Council's Fiscal Year 1998 funding recommendations, the Council scrutinized a set of projects for concerns about effectiveness, questions about cost, or both. These included the predator control program, the law enforcement program, the Fish Passage Center, the white sturgeon program, the Lake Pend Oreille fisheries study, the Integrated Hatchery Operations Team, the hatchery PIT-tag study, Squaw Creek and Fifteenmile Creek habitat projects, and the Creston Hatchery kokanee project. The Council then made use of the results of these reviews in making its funding recommendations this year, as shown throughout the discussion of general policy issues.

A similar effort occurred during Fiscal Year 1998 -- the Council initiated reviews of several projects or program areas, reviews which resulted in (or may yet result in) better defined and more efficient projects and contributed to the Fiscal Year 1999 funding recommendations. One example is the review of the coded-wire tagging activities discussed above. For another example, concerning the predator control project, the project sponsors, Bonneville and the Council were able to define and reduce funding for certain aspects of that effort, reducing the project budget significantly with little apparent reduction in effectiveness. In a third example, the Council led an interagency review of the captive propagation effort, to discuss whether the region needed a more reflective approach to what had become a series of ad hoc but very significant investments in an uncertain technology. One outcome of that review was an agency commitment to develop protocols and objectives to guide decisions as to whether and how to use this technology, which are expected soon. Other specific examples can be found in the funding recommendation decision.

Probably more important than these specific project reviews has been the Council's overall approach to artificial production investments under the Council's Program and in the basin as a whole. The Council has been leading a comprehensive review of artificial production in the basin, called for by the Council's Program, Congress and independent scientific panels. The end of the review is not expected until June of 1999; the results of the review should, the Council hopes and intends, provide the basis for a coordinated, comprehensive production policy for the basin. In the interim, the Council has pushed for and facilitated increased scientific and fiscal scrutiny of new production initiatives.

Improvements in contract selection and management/contract management review and audits. A fourth strategy developed out of recommendations in the economic analysis paper has been to improve the procedures for selecting and managing contracts and reviewing contract management. One example has been to increase competition in the solicitation and procurement process. To experiment with this idea in Fiscal Year 1998, the Council recommended the initiation of three new research initiatives by describing the scope of work needed and recommending the use of a competitive selection process to choose the contractor. For two of the projects the competitive selection process has been completed and contracts awarded; the third is still under development. In the Fiscal Year 1999 decision, the Council recommended the extension of this practice as the preferred way to initiate new work under the Program -- to define gaps and needs in Program implementation and then use a competitive selection process to accomplish the tasks. At the same time, the Council accepted a Scientific Panel recommendation to reserve a small amount of the budget to award "scoping grants" to promising new ideas for better implementation of identified Program needs.

Reviews of contract management procedures and audits of specific contracts are another tool for increasing the overall efficiency of the Program. During Fiscal Year 1998, the Council retained an independent accounting firm, Moss-Adams LLP, to review a representative set of Bonneville fish and wildlife contracts. The purpose of this review was not to evaluate these projects as much as to evaluate Bonneville's contract management process, to see if the appropriate controls and procedures were in place. The Council asked the auditor to investigate whether the contracting process contained the procedures necessary to manage a project's cost and effectiveness, such as whether projects are held to schedule and cost estimates, whether projects are allowed to change in scope, activities, or budget without policy review, and whether there are clear review responsibilities for contract managers, especially to see if projects are reviewed to determine if project sponsors report results consistent with the stated objectives for the project. Moss-Adams completed the report in December, 1997, "Columbia River Basin Fish and Wildlife Program, Management Review of Contracting Processes." Moss-Adams recommended that Bonneville make several changes in its contract selection and management procedures, such as make more use of a competitive solicitation process to request proposals for implementing priority projects; modify its agency purchasing instructions to strengthen procurement and contract administration; develop a transition mechanism for funding maintenance and operation costs; establish minimum information requirements for contractor statements and progress reporting; and evaluate certain opportunities to reorganize contract administration personnel to improve the efficiency and cost-effectiveness of contracting activities.

Building on the Moss-Adams report, and in order to oversee the efforts to address technical concerns, the Council has now recommended that Bonneville fund an audit of a subset of direct program contracts for technical, fiscal and contract management purposes. More precisely, the Council will work with Bonneville and CBFWA to develop and recommend a procedure for an independent audit of a significant number of projects approved for funding in Fiscal Year 1999. Developing the audit procedure will include describing the purposes of the audit, developing criteria and guidelines for the audit, describing a process to select the auditor, and selecting a set of contracts for review or developing a set of guidelines to assist the auditor in selecting contracts for review. The Council expects the auditor's report not only to help make changes in on-going projects and in contract management to improve the efficiency of the Program but also to help guide the Fiscal Year 2000 project selection process and be used in developing guidelines and reviewing programs for multi-year funding. Such an audit might investigate whether the technical concerns raised by the Scientific Panel are addressed in contracting between Bonneville and the project sponsor; whether funding for staff positions is being linked to projects and project-specific tasks; whether project staffing levels, time spent, and equipment purchases are appropriate and consistent given the work to be accomplished; whether the levels of administration and coordination are reasonable and consistent for specific projects and between similar projects; whether operation and maintenance needs are tied to capital or restoration investments, and reasonable and consistent across similar projects; whether monitoring and evaluation components exist, and whether they appear appropriate in terms of adequacy to evaluate results given project scope; whether reporting requirements are adequate and meet the needs of the project sponsors, Bonneville and the Fish and Wildlife Program; and whether projects remain consistent with their original proposed scope, and that contract and budget modifications are appropriate.

What all of these activities add up to is that the Council's Fish and Wildlife Program, as implemented and funded through the set of projects, is being more closely scrutinized than before in terms of effectiveness, accountability, cost, and efficiency, although much still needs to be done. One result should be a Program that is more cost-effective, satisfying the direction of Congress in the 1996 Power Act amendment. The Council makes this conclusion while recognizing that improvements in cost-effectiveness have not and cannot be quantified. What is especially lacking is a satisfactory way of understanding and measuring the biological effectiveness of particular projects or of the Program as a whole, as well as a comprehensive and consistent framework of goals and objectives for the Program that could be based on a better understanding of biological effectiveness. That is why nothing is more critical to the long-term efforts to improve the cost-effectiveness of the Program than the process now underway to develop a multi-species scientific framework for the Council's Program and fish and wildlife restoration in the basin as a whole.

^ top