Measuring Progress:
Process Evaluation Resources

Topics Covered:

The following list of reports and on-line resources highlights materials authors have produced to evaluate the characteristics (or strengths and weaknesses) of project processes. This bibliography of resources is not an exhaustive list of materials on the subject, but rather is designed to present a representative sample of the kinds of reports available on process evaluation. The resources are organized alphabetically according to subject headings representing the different types of objectives or purposes evaluators have brought to evaluating a project's process characteristics. These categories include:

 

1. Integrating findings to understand relationships between process characteristics and outcomes
2. Assessing whether or not to participate in a collaborative initiative
3. Evaluating the effectiveness of meetings, mediators, and programs
4. Assessing how outcomes and characteristics of collaborative process compare to other decision-making mechanisms
5. Identifying what process characteristics/factors facilitate/inhibit project success
6. Final category of resources highlighting materials that do not fit in any of the above groupings

An asterisk (*) next to these sources connotes resources featured in EMI's literature review entitled Objectives & Approaches to Evaluate Process in Collaborative Initiatives: A Literature Review on Existing Frameworks, Methodologies & Recommendations to Measure Process Success.

 

Integrating Findings to Understand Relationships Between Process Characteristics and Outcomes

Recent Trends in Ecosystem Management - Brush, M., A. Hance, K. Judd, and E. Rettenmaier. School of Natural Resources and Environment, University of Michigan, Ann Arbor, MI., 2000.
Note: Researchers designed a 10-page survey instrument with nearly 200 open- and closed-ended questions to be administered to ecosystem management project managers. The authors present in tabular form the major questions/categories covered in their survey instrument, many of which highlight process themes

An Evaluation of Selected Watershed Councils in the Pacific Northwest and Northern California - Pacific Rivers Council.
Chuck Huntington and Sara Sommarstrom prepared a three-part study to "evaluate the effectiveness of watershed councils in addressing environmental and fisheries problems, particularly salmonids habitat needs and water quality concerns. All three parts of the study are integrated and designed to provide insights into the connections between process and outcome achievements

* Innes, J. 1999. "Evaluating Consensus Building." Pages 631-675 in The Consensus Building Handbook. L. Susskind, S. McKearnan, and J. Thomas-Larmer (Eds.). Sage Publications, London.
Innes considers the role and challenges of evaluation as well as different methods and types of evaluation. She also identifies criteria and expected outcomes against which to measure the success of a collaborative endeavor. Innes developed two lists of evaluation criteria to aid evaluators in conducting more meaningful and systematic assessments of consensus-building efforts.

The New Watershed Source Book: A Directory and Review of Watershed Initiatives in the Western United States- Kenney, D., S. McAllister, W. Caile, and J. Peckham. Natural Resources Law Center, University of Colorado School of Law, Boulder, CO, 2000..
Authors developed and conducted a survey of participants in watershed initiatives and presented their results in 117 detailed case studies of select efforts. The statistical survey covered themes such as identifying natural resource problems, understanding the composition of participants, assessing specific goals and activities, determining funding and related resources, and evaluating accomplishments. Many process issues relevant to these themes were featured in the team's survey instruments.

Watershed Partnerships in California & Washington: Final Report for the Watershed Partnerships Project- Watershed Partnerships Project, Dept. of Environmental Science and Policy, University of California, Davis, CA, 2002.
Authors conducted a case study analysis of fifty watershed partnerships active between 1995 and 1999 in California and Washington. They sought to "understand the factors affecting the ability of watershed partnerships to resolve resource management controversies within the watershed and then to implement those decisions through restoration projects." They developed several process-based questions and insights during the course of the project.

 

Assessing Whether or Not to Participate in a Collaborative Process

Collaboration: Guide for Environmental Advocates - Dukes, E., K. Firehock, University of Virginia Institute for Environmental Negotiation, The Wilderness society, and the National Audubon Society. Charlottesville, Virginia, 2001.
Authors highlight numerous criteria to help environmental advocates consider the characteristics of effective collaborative processes and the benefits and costs of engaging in such efforts. The report also presents several questions/criteria to help parties design and administer effective and enduring collaborative processes that result in implementable and adaptable agreements over time.

Assessing Whether to Participate in a Collaborative Evaluation: Detailed Evaluation - Yaffee, S. and J. Wondolleck., Material prepared for Collaboration Skills and Strategies: A Workshop for Environmental Leaders, Denver, Colorado, 2001.
Authors develop a framework list of questions to help parties (e.g., agencies, businesses, citizen groups, and environmentalists) identify and evaluate the challenges and opportunities of engaging in a collaborative initiative. They organize their framework according to three major themes: 1) assessing your group's objectives and options, 2) assessing your group's organizational considerations, and 3) assessing the situation.

 

Evaluating the Effectiveness of Mediators, Meetings, and Programs

Elliot, M. 1999. "The Role of Facilitators, Mediators, and Other Consensus Building Practitioners." Pages 199-240 in The Consensus Building Handbook. L. Susskind, S. McKearnan, and J. Thomas-Larmer (Eds.). Sage Publications, London.
Elliot argues that facilitators must be able to build working relationships among diverse parties, promoting greater communication between individuals that may have traditionally been unable to communicate or hostile to one another. Among the tasks a facilitator must be able to do include: to be able to develop and lead an agenda for each meeting, enforce ground rules of conduct, promote healthy interaction and communication during meetings, and work with parties to bring an issue to closure. Elliot argues that a group is most effective when its members are committed to a clear goal, organized to achieve specific results, are technically competent, are motivated to contribute to the effort, and share leadership responsibilities throughout the process. It is the facilitator's challenge to create a climate in which the group can work effectively and achieve results.

* Straus, D. 1999. "Managing Meetings to Build Consensus." Pages 287 - 323 in The Consensus Building Handbook. L. Susskind, S. McKearnan, and J. Thomas-Larmer (Eds.). Sage Publications, London.
The goal of this work is to help parties plan for and run a successful meeting; however, Straus also dedicates a portion of the text to present a framework in which to evaluate meeting success, which involves several process themes.

The Montana Consensus Council. 2001. Effective Negotiations: Measuring Participant Satisfaction. Helena, Montana.
MCC has developed a participant satisfaction scorecard that is given to every participant they work with to evaluate the consensus building process participants experienced at MCC. The scorecard is designed to help MCC's facilitators improve their services by evaluating past performance; several process questions are featured.

Toupal, R. and M. Johnson. 1998. Conservation Partnerships: Indicators of Success. NRCS Social Sciences Institute, University of Arizona, AZ.

U.S. Institute for Environmental Conflict Resolution- U.S. Institute's Program Evaluation System. Tuscan, AZ. 2002.
The Institute developed a detailed program evaluation system in order to better define, measure, and improve program performance. A major component of the Institute's evaluation program involved the development of multiple survey instruments to be distributed to different parties who have participated in conflict assessments, ECR processes, or public meetings managed by the Institute. The survey questions for each evaluation theme (conflict assessments, ECR processes, public meeting facilitation) are based on the Institute's "best practice factors and desired outcomes" for each programmatic area.

Assessing how Outcomes and Characterists of Collaborative Processes Compare to other Decision-Making Mechanisms

Evaluating Dispute Resolution as an Approach to Public Participation- Beierle, T. and J. Cayford. 2001. Resources for the Future Discussion Paper 01-40. Washington, DC: Resources for the Future.
Authors review 239 published cases of public involvement in environmental decision-making in order to evaluate dispute resolution as a form of public participation. Their analysis serves two different evaluation goals. First, they aim to measure how effective dispute resolution processes are in achieving various social goals or outcomes. Second, they sought to compare public dispute resolution processes' ability to achieve these social goals relative to other forms of public participation in order to discover the strengths/weaknesses of both approaches.

Rethinking Consensus: Is Agreement a Sound Basis for Regulatory Decisions?- Coglianese, C. Paper presented at a conference on environmental contracts and regulation: Comparative Approaches in Europe and the United States, University of Pennsylvania Law School, September 24-25, 1999.
Coglianese raises numerous questions/concerns about the government's reliance on consensus in the making of public policy. Implicit in his analysis is that consensus processes should be measured not only by the outcomes they achieve, but also by how these achievements compare to other decision-making structures not based on consensus. With this in mind, he argues that consensus-building processes may not measure up to other participatory processes that do not aim for consensus.

 

Identifying what Process Characteristics/Factors Facilitate/Inhibit Project Success

Toward Understanding New Watershed Initiatives - Born, S., K. Genskow. A Report from the Madison Watershed Workshop, July 20-21, 2000, Madison, Wisconsin. University of Wisconsin-Madison, 2000.
Authors convened a workshop for twelve researchers and practitioners who had collectively conducted assessments of or worked with watershed initiatives across the United States. The participants came together to discuss their experiences and viewpoints on evaluating success among watershed partnerships. In this report about the workshop, they highlight framework elements for evaluating watershed partnerships as well as key factors that influence success of watershed initiatives, many of which are process needs.

* Colby, B. In print. "Economic Characteristics of Successful Dispute Resolution Outcomes." Chapter x in Evaluating Environmental and Public Policy Dispute Resolution Programs and Policies. R. O'Leary, ed.
Colby proposes new ways to apply economic expertise to evaluating environmental and public policy disputes. The body of her report highlights ten economic and financial criteria that may lead to successful conflict resolution outcomes and recounts the results of a pilot study applying the criteria to Western U.S. water conflicts.

* Jackson, L. 2002. "Consensus Processes in Land Use Planning In British Columbia: The Nature of Success." Pages 1-90 in Progress in Planning. D. Diamond & B. Massam (Eds.). Elsevier Science Ltd., Oxford, UK.
Jackson set out to determine what makes consensus work in resource management decision-making. Jackson interviewed representative stakeholders who had participated in consensus-based process involving the Canadian resource ministries in order to identify success factors for consensus planning. Based on her research, Jackson developed a diagnostic evaluation framework for consensus building efforts to enhance planners understanding of these processes.

Making It Work: Keys to Successful Collaboration in Natural Resource Management. Environmental Management- Schuett, M., S. Selin, and D. Carr, 27 (4): 587-593, 2001.
Authors evaluate the keys to success in thirty different collaborative initiatives in which the Forest Service played an active role. The results of their study provide a preliminary framework of issues or "keys to success" to consider when initiating or working within a collaborative initiative.

Stahl, A. 2000. "Ownership, Accountability, and Collaboration." Pages 194-199 in Across the Great Divide: Explorations in Collaborative Conservation and the American West. Brick, P, D. Snow, and S. Van De Wetering (Eds.). Island Press, Washington, DC.
Stahl presents a critical analysis of collaborative groups such as citizen councils that, at least in some forms, may exercise too much power over decision-making. Stahl's work highlights the challenges of ownership, accountability and authority that surface among parties in collaborative initiatives. While Stahl's report does not focus on evaluation, his commentary may be useful for evaluators. Evaluators may explore the extent that the common challenges Stahl considers surface in a particular collaborative being evaluated.

* Wondolleck, J. and S. Yaffee. 2000. Core Principles of Collaborative Resource Management. Ecosystem Management Initiative. School of Natural Resources and Environment, University of Michigan, Ann Arbor.
The authors take a somewhat different approach to evaluating collaborative processes, suggesting examination of the degree to which overarching principles are met by the structure and functioning of the process. Based on their research and experiences in collaboration, they have identified six core principles that a process should embody. These core principles may be useful for any party (e.g., agencies, businesses, citizen groups, and environmentalists) who may engage in a collaborative initiative.

* Wondolleck, J. and S. Yaffee. 2000. Making Collaboration Work. Island Press, Washington, DC.
This work provides insights on a variety of topics related to collaboration; it also functions as a prescriptive evaluative guide to would-be collaborators highlighting recommendations for people beginning or engaged in collaborative initiatives. The heart of the book is centered on a set of eight key factors that, according to the author's analysis of roughly two hundred collaborative initiatives, are critical to the effectiveness of a collaborative effort.

 

Final Category of Resources Highlighting Materials that do no Fit in any of the above Groupings

Evaluating the Collaboration Process- Clark, R.. Ohio State University Fact Sheet.

Rating Member Satisfaction- Community Toolbox.

Kenney, D. 2001. "Are Community-Based Watershed Groups Really Effective? Confronting the thorny issue of measuring success." Pages 188-193 in Across the Great Divide: Explorations in Collaborative Conservation and the American West. Brick, P, D. Snow, and S. Van De Wetering (Eds.). Island Press, Washington, DC.

Konisky, D. and T. Beierle. 2001. Innovations in Public Participation and Environmental Decision Making: Examples from the Great Lakes Region. Society and Natural Resources. 14: 815-826.

Leach, W., N. Pelkey, P. Sabatier. 2000. Conceptualizing and Measuring Success in Collaborative Watershed Partnerships. Report prepared for delivery at the 2000 Annual Meeting of the American Political Science Association. Department of Environmental Science and Policy, University of California, Davis.

McCool, S. and K. Guthrie. 2001. Mapping the Dimensions of Successful Public Participation in Mesy Natural Resources Management Situations. Society and Natural Resources. 14: 309-323

Smith, P. and M. McDonough. 2001. Beyond Public Participation: Fairness in Natural Resource Decision Making. Society and Natural Resources. 14: 239-249.

Go back to the Measuring Progress resources page
Return to the Measuring Progress home page

Page last updated 4/15/03. If you encounter technical difficulties using this site (broken links, missing pictures, etc.), please contact the EMI Webmaster. For more information on EMI, contact us. © EMI, SNRE 2004.