Báo cáo Nghiên cứu khoa học Monitoring and evaluation procedures

January 2010 MONITORING AND EVALUATION PROCEDURES VERSION 3 Collaboration for Agriculture & Rural Development Program Vietnam January 2010 CARD – M&E Procedures January 2010 TABLE OF CONTENTS Executive Summary List of Abbreviations 1 CARD M&E FRAMEWORK ............................................................................................ 2 1.1 Introduction ................................................................................................................. 2 1.2 Overview ..................................................................................................................... 2 1.3 Project Level M&E...................................................................................................... 3 1.4 Program Level M&E ................................................................................................... 3 1.5 Institutional Level M&E.............................................................................................. 4 1.6 Where M&E Fits in the CARD Project Cycle............................................................. 4 1.7 Purpose of this Document............................................................................................ 5 2 BASIC PRINCIPLES OF MONITORING AND EVALUATION.................................... 5 2.1 Overview ..................................................................................................................... 5 2.2 The Five Key Questions .............................................................................................. 5 2.3 Different Projects, Different Approach ....................................................................... 7 3 M&E IN THE CARD PROGRAM..................................................................................... 7 3.1 Overview ..................................................................................................................... 7 3.2 Logical Framework (Logframe) Methodology............................................................ 7 3.3 Intermediate and Final Outcomes/Impacts .................................................................. 9 3.4 Designing for Impact ................................................................................................... 9 3.5 When to Monitor and Evaluate?................................................................................ 10 4 TECHNIQUES OF M&E AND IMPACT ASSESSMENT............................................. 11 4.1 Performance Indicators.............................................................................................. 11 4.2 Information Sources and Timing of Impacts............................................................. 12 4.3 Using Negative Findings ........................................................................................... 13 4.4 Baseline Information ................................................................................................. 13 4.5 Options for Comparison ............................................................................................ 15 4.6 Contribution Analysis ................................................................................................... 15 4.7 Specific M&E Tools.................................................................................................. 16 4.8 Impacts to be Assessed .............................................................................................. 18 4.9 Environmental Monitoring ........................................................................................ 19 4.10 Judging a Project’s Success ................................................................................... 20 5 BENEFIT COST ANALYSIS IN M&E........................................................................... 20 5.1 Overview ................................................................................................................... 20 5.2 Identifying Benefits and Costs .................................................................................. 21 5.3 The Representative Farm Concept ............................................................................ 22 5.4 Enterprise Budgets..................................................................................................... 23 5.5 Comparing Benefits and Costs .................................................................................. 23 6 WORKPLAN FOR EVALAUTION OF THE CARD PROGRAM ................................ 24 6.1 Overview ................................................................................................................... 24 6.2 Mid-Term Reviews.................................................................................................... 25 6.3 Project and Program Completion Evaluations........................................................... 25 6.4 Ex Post Evaluations ................................................................................................... 26 6.5 Further Training needs............................................................................................... 26 6.6 Conclusions ............................................................................................................... 27 Attachment 1: Terminology and Definitions Attachment 2: Procedures and Terms of Reference for Mid-Term Reviews CARD – M&E Procedures January 2010 Attachment 3: Procedures and Terms of Reference for Project Completion Evaluations Attachment 4: Enterprise Financial Analysis Templates LIST OF ABBREVIATIONS AMC Australian Managing Contractor APR Annual Progress Report AusAID Australian Agency for International Development BCA Benefit Cost Analysis BCR Benefit Cost Ratio CARD Cooperation for Agriculture and Rural Development EIA Environnemental Impact Assessment EMP Environnemental Management Plan EOI Expression of Interest IFAD International Fund for Agricultural Development M&E Monitoring and Evaluation MARD Ministry of Agriculture and Rural Development MTR Mid-Term Review PCC Project Coordinating Committee PCE Project Completion Evaluation PCR Project Completion Report PMU Program Management Unit STED Science and Technology and Environment Department (of MARD) TAP Technical Advisory Panel TOR Terms of Reference ACKNOWLEDGEMENTS The CARD M&E strategy and procedures described in this document are based on AusAID recommendations for project monitoring and evaluation as described in AusGuide which is downloadable from www.ausaid.gov.au/ausguide. Material was also derived from “A Guide for Project M&E” produced by International Fund for Agricultural Development (IFAD) and downloadable from www.ifad.org/evaluation/guide. CARD – M&E Procedures January 2010 CARD – M&E Procedures January 2010 Sequence of Activities EOI Expression of Interest Feedback and Lessons Learned TAP Technical Adivisory Panel PCC Project Coordinating Committee M&E Stages PCR Project Completion Report Ex ante evaluation begins here And continues through the project design process Contract defines outputs, outcomes and milestones Monitoring continues through implementation period Initial evaluation undertaken at project completion Ex post evaluation follows some time later Project evaluations aggregated to evaluate overal portfolio Monitoring Data Used In Evaluation Process MONITORING AND EVALUATION IN THE CARD PROJECT CYCLE Proponent Organisation(s) Identify Project Ideas Prepare EOI and Submit to CARD TAP Evaluates EOIs Proponents Prepare Project Proposals and Submit for Peer Review Proposal Modified as Necessary and Submitted to TAP TAP Evaluates Proposals TAP Recommendations to PCC Final Review and Approval by PCC CARD & Proponent Agree Contract and Payment Milestones Project Implementation Undertaken by Proponents and Self- Monitored Independent Project Completion Evaluation Independent Ex-Post Evaluation Evaluation Reports - Relevance - Effectiveness - Efficiency - Impact - Sustainability - Lessons Learned Monitoring Reports - Baseline Study - Six Monthly Reports - Outcome Milestones - Technical Reports - PCR Lessons Learned Inform Future Rounds of Project Design and Implementation Independent Evaluation of Entire CARD-MARD R&D Portfolio PCC Reviews EOIs and Prepares Shortlist TAP Recommendations to PCC PCC Feedback Revise & Resubmit EOIs PCC Feedback Peer Feedback CARD – M&E Procedures January 2010 EXECUTIVE SUMMARY One of the main objectives of CARD is to strengthen the capacity of MARD to manage agricultural technology and knowledge development programs. Sound management of such programs depends on being able to monitor and evaluate Programs in terms of their relevance, effectiveness, efficiency, impact and sustainability. This recognises that R&D is an investment which needs to be evaluated alongside other investment opportunities, in order to ensure that the best investments are chosen from widely differing alternatives. CARD is in the process of building the capacity of MARD to undertake M&E of R&D projects, beginning with the projects currently supported by the CARD Program. As part of this process CARD provided the services of a M&E Specialist to undertake training and facilitation services for a group of personnel from MARD and its affiliated institutions. The first round of training was completed in April 2007, a second round was undertaken in September-October 2008, and a M&E review workshop was conducted in January 2010. The CARD M&E framework caters for M&E requirements at project level as well as the institutionalisation of CARD processes within MARD. Its key elements are:  Project M&E which aims to assess the progress and impact of collaborative research projects on raising smallholder productivity and competitiveness;  CARD Program M&E which aims to assess the progress and impact of the Program as a whole, both in benefits to smallholders and raising the capacity of research institutions, to undertake effective research projects;  M&E at the MARD institutional level in assessing the improvement in capacity in MARD (STED) in organisation and management of the MARD research Program. The CARD Program has been focused on monitoring at the individual project level through milestone reports and site visits. During 2008 and 2009 CARD initiated evaluation of eight on-going and 14 completed projects using procedures for Mid-Term Reviews (MTRs) and Project Completion Evaluations (PCEs) which were provided in an earlier version of this document. This represents the first step towards evaluation of the overall CARD Program. This document sets out the proposed approach and procedures for individual project evaluations based on the monitoring data already accumulated, and for subsequently aggregating these evaluations up to Program and finally to institutional level. The document was developed in parallel with a training program for a group of staff from MARD and its affiliated institutions which involved three workshop sessions and 21 case studies. The training represents significant progress in developing a group of competent evaluators which is capable of evaluating all CARD projects in the first instance, and subsequently strengthening the evaluation of the entire MARD R&D portfolio. CARD is approaching the end of its seven-year duration and it is therefore appropriate to carry out a number of systematic PCEs. This document sets out a schedule for undertaking these evaluations during the remaining life of the Program. CARD will assist by providing coaching and guidance to contracted evaluation teams as well as further on-the-job training in evaluation methodology. The benefits will be improved project design, the identification of areas of high (and low) return R&D investment, improved relevance and impact, and improved accountability and transparency in the allocation of resources to agriculture and rural development. CARD – M&E Procedures January 2010 2 1 CARD M&E FRAMEWORK 1.1 Introduction One of the main objectives of CARD is to strengthen the capacity of MARD to manage agricultural technology and knowledge development programs. Sound management of such programs depends on being able to monitor and evaluate Programs in terms of their relevance, effectiveness, efficiency, impact and sustainability. This recognises that Research and Development (R&D) is an investment which needs to be evaluated alongside other investment opportunities, in order to ensure that the best investments are chosen from widely differing alternatives. CARD is in the process of building the capacity of MARD to undertake M&E of planned and ongoing R&D projects, beginning with the projects currently supported by the CARD Program. As part of this process CARD provided the services of an International Monitoring and Evaluation Specialist to work with CARD’s in-house national M&E Specialist to undertake training and facilitation services for a group of personnel from MARD and its affiliated institutions. 1.2 Overview The purpose of M&E is to learn so that future development interventions can be more effective. M&E is not an examination or test. It is not an audit. Negative outcomes have value provided we learn from them. The learning process is essentially internal within CARD and its partner institutions. However internal learning needs to be balanced with external accountability. Projects have responsibilities to stakeholders and Vietnam society at large to account for expenditures, activities, outputs and impacts. There is an extensive literature on project M&E for agricultural development which is mainly oriented towards long term impacts of major investment projects such as those funded by the World Bank, ADB and IFAD. CARD comprises a suite of relatively small projects which are intended to generate benefits for stakeholders in both the short and long term. In this regard, CARD needs an approach to M&E which somewhat different to the standard textbook models, and which is certainly cheaper and less complex. The CARD Program M&E framework (see chart at the front of this document) requires an approach which caters for the M&E requirements of the program and project level as well as the institutionalisation of the CARD processes within MARD. The key components of the framework are:  research Project M&E which aims to assess the progress and impact of collaborative research projects on raising smallholder productivity and competitiveness;  CARD Program M&E which aims to assess the progress and impact of the Program as a whole, both in benefits to smallholders and raising the capacity of research institutions, to undertake effective research projects – this is in effect a sum of all project implementation impact; and  M&E at the MARD institutional level in assessing the improvement in capacity in MARD (STED) in organisation and management of the MARD research Program. The purposes of the M&E framework for the Program are to: CARD – M&E Procedures January 2010 3  make available timely and relevant information to support effective management decisions by the PMU, Program Coordinating Committee (PCC), AMC and AusAID.  measure progress of the projects and identify issues for which management can take necessary actions; and  provide information for internal management and external reporting. 1.3 Project Level M&E The M&E strategy at the project level revolves around the development of a simplified logframe for each project, and from these the development of output and outcome milestones and deliverables for each project contract. Output milestones are six-monthly progress reports which record achievement against the logframe activities and identify highlights and issues in project implementation. Outcome milestones focus on impacts at the smallholder and institutional levels. A key milestone for each project is the establishment of baselines on current practices, production levels and profitability. The final milestone for each project is validation of the impact of the project in relation to baseline levels and production of the Project Completion Report. Project reporting involves collaborating institutions self-assessing achievements against their own specific performance measures, detailed in the logframe and project milestones. The PMU monitors projects through appraisal of project output and outcome milestones. Once projects are completed, independent case studies of selected projects are commissioned to evaluate potential economic, social and environmental impacts. At the project level output milestone reporting includes, 6-monthly and Annual Progress Reports (APR) and Project Completion Reports (PCR). Each year the APR and at the end of the project, the PCR, reports against achievement of their defined performance measures. Delivery and payment of these milestones is tracked on the PMU database and any implementation issues are addressed through discussions between the PMU and the collaborating institutions. Outcome milestones include impact assessment at the smallholder and institutional capacity levels. Using baseline data on knowledge, skills and practices; and at the smallholder level physical and financial performance, all projects are required to validate their project outcomes at the completion of the project. 1.4 Program Level M&E At the program level the M&E strategy is to assess the sum of project impacts and to assess changes in research institutional capacity to prepare and implement high quality R&D projects. A key outcome milestone for all projects involves assessment of improvements on competency levels of research and extension workers. PMU project site visits to evaluate implementation and impact using standards assessment formats are integrated with TAP site visits to assess the quality of project selection. At the program level the PMU prepares an Annual Report for the financial year (July – June) to be presented to the PCC in March for finalisation prior to June 30 each year. The Annual CARD – M&E Procedures January 2010 4 Report provides the context for development of the Forward Annual Plan and enables the PMU/Technical Coordinator to take into account any significant interventions that will improve the ownership of CARD in MARD and through that CARD’s sustainability. In this way M&E is part of the planning process and is expected to result in continued improvement in implementation of the CARD Program. The Annual Report includes:  Implementation highlights, issues and options.  Number of projects started and the status of implementation of each project in each year.  Significant outputs from completed projects.  Research project summary sheets including objectives and milestones and acceptance and payment of project milestones.  Significant impacts of research outputs and capacity building arising from the Program as measured by research institution self-assessment, case studies and PMU Progress reports.  Summary of resource inputs and activities achieved against logframe estimates as well as qualitative ratios established from activity analysis (achievements against the CARD Program logframe activities.  Summary of institutional capacity building arising from analysis from internal and external activities involving MARD.  Issues, problems and recommendations. 1.5 Institutional Level M&E At the institutional level a series of performance indicators has been established to monitor and evaluate the institutionalisation of governance and management structures and processes within MARD. A key aspect of this is to measure the change in attitude, beliefs, behaviour and practices within MARD (STED) in relation to R&D policy, organisation and management. A survey was conducted early in the life of the Program to establish the baseline status of institutional capacity. This survey will be repeated during the second half of 2010 to assess the degree of institutionalisation of CARD systems, procedures and management practices into MARD and its affiliated institutions. 1.6 Where M&E Fits in the CARD Project Cycle CARD projects are designed and implemented according to a sequence of steps which can be described as the “CARD Project Cycle”. The second of the charts at the front of this document details the steps in the cycle. M&E begins early in the design process where the project logframe identifies the indicators of achievement and the means of verification. The project is also subject to independent ex ante evaluation firstly at expression of interest (EOI) stage, and again at proposal stage, to assess its likely outputs, outcomes and impacts. During implementation, projects are expected to gather baseline information and self-monitor their activities to provide the data needed for subsequent evaluation. Mid-Term Reviews (MTRs) CARD – M&E Procedures January 2010 5 may be undertaken when about half the milestones have been reached. Independent project completion evaluation (PCE) takes place at the end of the project implementation period, and ex post evaluation is conducted some time later when the full impacts of the project are likely to be apparent. The various M&E reports produced in this process are then available to undertake overall Program evaluation. 1.7 Purpose of this Document Until now the CARD Program has been focused on monitoring at the individual project level. Monitoring is on-going with submission and appraisal of milestone reports and site visits. MTRs and PCEs have been carried out in 2008 and 2009. No further MTRs will be undertaken, but a number of PCEs will be undertaken in 2010 which is the final year of the Program. This will allow aggregation of individual project impacts to the program level. This document sets out the proposed approach and recommended procedures for undertaking individual project evaluations based on the monitoring data that have been accumulated during implementation, and for subsequently aggregating these evaluations up to Program and finally to institutional level. 2 BASIC PRINCIPLES OF MONITORING AND EVALUATION 2.1 Overview M&E is an essential tool in the management of programs and projects in agricultural research and development and is an important part of the CARD implementation framework. This section describes the basic principles of M&E as applied in the CARD Program and is followed by a section which describes a number of techniques which can be used. Monitoring is defined1 as the regular collection and analysis of information to assist in timely decision-making and provide the basis for evaluation and learning. It is a continuous function that generates data to provide project management and stakeholders with early indicators of progress and achievement of objectives. Monitoring provides data to generate insights about impact as part of the evaluation process. Formal monitoring involves gathering data about selected indicators and performance measures. However informal monitoring involving valuing and sharing impressions is also an important ingredient of the process. There can be no evaluation without some form of monitoring Evaluation is defined as a systematic (and objective as possible) examination of a planned, ongoing or completed project. It aims to answer specific management questions and judge the overall value of a project and generate lessons learned to improve future planning and decision-making. 2.2 The Five Key Questions Evaluations commonly seek to determine the relevance, effectiveness, efficiency, impact and sustainability of the project (see Box 1 below). Evaluation should provide information 1 See Attachment 1 for a complete list of M&E terminology and definitions. CARD – M&E Procedures January 2010 6 that is credible and useful and offer concrete lessons learned to help partners and funding agencies make better decisions. Box 1: The Five Key Evaluation Questions 1. Relevance: the extent to which the objectives of a project are consistent with the target group’s priorities and the Governments policies.  Were the objectives clear, realistic and measurable?  Is the project design adequate to achieve the objectives? 2. Effectiveness: a measure of the extent to which a project attains, or is expected to attain, its objectives in a sustainable manner.  Progress in achieving objectives  Quality of outputs  Extent of benefit to the target population 3. Efficiency: a measure of how economically inputs are converted to outputs.  Timeliness and appropriateness of project design and implementation processes  Efficiency of implementation by the contractor(s).  Strength of partner support and value of dialogue.  Quality of CARD management and PMU support 4. Impact: The change in the lives of rural people, as perceived by them and their partners at the time of evaluation, plus sustainability-enhancing changes in their environment to which the project has contributed. The assessment of impact is a particularly important part of the M&E process since impact is the ultimate objective of any agricultural R&D initiative.  Impacts may be social, financial, institutional, technological or environmental in nature.  Where possible undertake benefit-cost analysis to estimate the magnitude of financial benefits.  Possible impacts on policies should be highlighted. 5. Sustainability: the likelihood that the positive effects of a project (such as assets, skills, facilities or improved services) will persist for an extended period after the project is completed.  Sustainability of benefits.  Need for ongoing recurrent costs or further investments.  Sustainability of institutional capacity. Evaluation must address all five of these key questions in order to identify lessons learned. Lessons Learned: knowledge generated by reflecting on experience, that has the potential to improve future actions. Lessons learned include broader implications of the evaluation results in relation to sectoral policies and future project design and implementation modalities with a focus on strengths and weaknesses in project design and implementation that affect the achievement of objectives. CARD – M&E Procedures January 2010 7 The five key evaluation questions are normally scored on a scale of 1-5 with 1 representing the worst assessment and 5 being the best. The same questions are used for both MTRs and PCEs. Guidelines for application of the scoring system are given in Attachment 2 (MTR) and Attachment 3 (PCE). M&E is essentially an internal learning process which relies on a constructive and questioning attitude – but it also helps ensure external accountability to funding agencies and other stakeholders. It is often a rather subjective exercise because of the difficulties of attribution; i.e. identifying of likely causal relationships between project inputs and outputs. It calls for the application of perceptive observation and common sense in telling a believable story about why particular activities generate particular outcomes. M&E is based on design logic which defines the causal relationship between project inputs and outputs. If the design logic of a project is weak or uncertain, it usually proves very difficult to monitor and evaluate. 2.3 Different Projects, Different Approach Different projects have to be evaluated in different ways because the nature of the project activities and the benefits and costs they engender vary. Therefore it is not possible to provide a standardised set of procedures for evaluation. At the beginning of each evaluation exercise the evaluators have to think carefully about the information they will need and how they can best obtain it. Indirect or proxy measures of performance are sometimes used where direct measurement proves impossible. The first step in any evaluation exercise is the planning process including questionnaires, checklists and analytical formats. These will vary for example between capacity building and technology dissemination projects, between annual and perennial cropping activities, between cash and subsistence crops, or between projects that involve intensive training to selected farmer groups and projects which disseminate information through the mass media. 3 M&E IN THE CARD PROGRAM 3.1 Overview Within the CARD Program, impact assessment will be used to assess individual CARD- supported projects as well as the overall CARD Program, to identify reasons for success or failure and the lessons learned. This will help decide whether to expand or replicate the CARD approach to R&D across the entire MARD R&D portfolio. CARD projects activities are generally self-monitored by the project proponents through the system of progress reporting and milestone reports specified in the contract between CARD and the proponents. The self-monitoring is supported by site visits and informal progress reporting by collaborating partners. As CARD projects are completed it is appropriate to begin the evaluation process. This will be undertaken with the help of external facilitation using trained project evaluators. Although evaluation should be a participatory process, external facilitation is important to ensure objectivity and gain insights which may not be apparent to those who have been closely associated with the project activities. 3.2 Logical Framework (Logframe) Methodology CARD – M&E Procedures January 2010 8 The logframe is the means of describing the design of projects and forms the basis for subsequent monitoring and evaluation. CARD projects utilises a simplified version of the logframe methodology structured as follows: Narrative Information Required Performance Indicators Performance Measures Assumptions and Risks Objectives Outputs Activities Inputs The key elements of the CARD simplified logframe are shown in Box 2: Box 2: Key Elements of the CARD Simplified Project Logframe Objectives: a statement detailing the desired outcomes of a project at different levels (short to long term). Objectives should be impact oriented, measurable, time bound, specific and practical. Outputs: tangible, measurable and intended results produced through provision of project inputs in order to undertake project activities. Activities: actions taken or work performed in a project to produce specific outputs by using inputs such as funds, technical assistance, machinery and other types of resources. Inputs: the financial, human and material resources necessary to produce the intended outputs. Outcomes and Impacts are detailed under the Performance Indicators and the means of measuring these are described under Performance Measures. Outcomes are estimates or measures of what changes are expected to take place as a result of project implementation. Impacts describe the change in the lives of rural people, as perceived by them and their partners at the time of evaluation, plus sustainability-enhancing changes in their environment to which the project is expected to contribute. Defining the logical connections between the different elements of the logframe is the key to successful application of the logframe methodology, and by implication, to successful monitoring and evaluation. The following is an example of design logic defined in a logframe context: 1. Objectives: improve incomes, living standards and nutrition amongst rice farmers 2. Outputs: release of new high yielding rice variety 3. Activities: plant breeding Program to develop a new variety 4. Inputs: staffing, seed, equipment, field plots, fertilisers etc 5. Outcomes: Improved crop yields in farmers fields 6. Impacts: farmers have more to eat and sell resulting in higher incomes and living standards CARD – M&E Procedures January 2010 9 Objectives, outputs, activities and inputs are specified in the logframe and are relatively easy to estimate or measure. However outcomes and impacts are more difficult to define, measure and evaluate. This usually requires a degree of judgment about future adoption rates and the level and nature of benefits to stakeholders. Sustainability is also a key issue in assessing outcomes and impacts. At the time of evaluation, often we can only infer or anticipate what impacts might be forthcoming after the project benefits have had time to be fully realised. This may be a number of years after project completion and can only be finally and objectively assessed through ex post evaluation (see below). 3.3 Intermediate and Final Outcomes/Impacts CARD supports projects intended to benefit rural small-holders through promotion of enhanced productivity, efficiency and sustainability that result in improved farmer income, food security and welfare. These benefits can be considered as the final outcomes and impacts. Hence M&E must directly measure or seek for signs that productivity, efficiency, sustainability, incomes, food security and welfare have in fact improved, and that such improvements can be attributed (in full or in part) to the project initiatives. CARD also supports other activities such as capacity building development of extension materials, training of trainers, etc. The results arising from such activities are considered intermediate rather than final outputs which are intended to improve the capacity to deliver final outcomes and impacts in terms of productive activities undertaken by smallholders. The only thing that matters to farmers is the end result, and M&E should therefore focus on evaluation of outcomes and impacts at the farm level. 3.4 Designing for Impact M&E can only be a useful tool if projects are designed to achieve specific identified impacts. Designing for impact is critical to the quality of project design and for subsequent monitoring and evaluation. At Expression of Interest (EOI) stage designing for impact requires proponents to:  describe expected outputs, benefits and impacts;  indicate time-frame for application of the technology; and  describe how outputs/benefits will be sustained. At Project Proposal stage proponents are expected to present a stakeholder/beneficiary analysis which specifies:  benefits expected and timeframe;  need for baseline information;  procedures to collect baseline information;  procedures to measure benefits; and  performance indicators and performance measures. Project Proposals should also:  describe expected impacts – social, financial, environmental, institutional etc;  describe how progress and impact will be assessed; and  describe how the project will gather and analyse information for measuring progress and impact and explain reasons for success and failure. CARD – M&E Procedures January 2010 10 At Contract Stage the contract between CARD and the lead proponent specifies the output milestones and outcome milestones which are the basis for disbursement of CARD funds to the proponents. 3.5 When to Monitor and Evaluate? As shown in Chart 2 in the front of this document, the process begins during the project design phase. This is known as ex ante evaluation where the project is assessed according to its expected outcomes and impacts as specified in the logframe. At the EOI stage ex ante evaluation is mainly concerned with relevance and potential benefits. At the proposal stage the evaluation focuses on impacts, the likelihood of success and sustainability and value for money. The ex ante evaluation process also specifies the performance indicators and performance measures which will be used to monitor, and eventually evaluate, the outcomes and impacts. Ex ante evaluation involves the project development team from the proponent organisation as well as a Technical Advisory Panel (TAP), peer reviewers and the CARD Project Coordination Committee (PCC) which makes the final decision on which projects to support. Monitoring takes place during the implementation of the project (normally two to three years) and often includes a baseline survey to define the situation before project activities begin. Monitoring reports include some or all of the following: baseline studies, six-monthly progress reports, outcome milestone reports, various technical reports, and the Project Completion Report (PCR). Together these provide the CARD Program Management Unit (PMU) and the manager(s) of the project itself with regular information on how the project is proceeding towards its objectives. If routine monitoring suggests that modifications to project designs are necessary, these can be undertaken at any stage with the approval of the PMU. A Mid-Term Review (MTR) is a useful tool in project monitoring which may also involve initial efforts to undertake evaluation and preliminary impact assessment. The MTR should be undertaken as a collaborative exercise involving external reviewers working in partnership with the project team. A suggested format, checklist and standard terms of reference for undertaking MTRs is given in Attachment 2. The MTR is an implementation support procedure which involves an interim assessment of the project to assess progress in undertaking activities and generating outputs, identify problem areas and propose solutions. In some cases this may suggest changes to the project design and budget2 in order to respond to changing circumstances and lessons learned so far, or to remedy deficiencies in the original design. The MTR should assess operational aspects such as project management and implementation of activities, and the extent to which objectives are likely to be achieved. It should focus on corrective actions needed for the project to achieve impact, but will generally be conducted before impacts are apparent. The MTR should also evaluate plans in place for end-of-project impact assessment and the resources available to undertake it. The MTR will also help to identify “problem projects” at an early stage where things are not going according to plan, and where remedial action is warranted. In extreme cases where it 2 CARD’s operational procedures preclude increasing budgets, but it is possible to transfer funds between expenditure categories if this will improve the likelihood of achieving project objectives. CARD – M&E Procedures January 2010 11 becomes clear that the objectives will not be achieved, early termination of the project may be recommended. Project evaluation takes place at the end of the project implementation period, known as project completion evaluation (PCE); and again some time after project completion when the outcomes and impacts of the project have had time to fully evolve. This is known as ex post evaluation and usually takes place several years after project completion. 4 TECHNIQUES OF M&E AND IMPACT ASSESSMENT This section of the document provides a description of the main tools and techniques and how they may be applied to CARD projects, and eventually to evaluation of the overall CARD Program. There is a range of tools and techniques available for M&E and impact assessment. These are very extensively described in easily accessible literature. The most comprehensive description of tools and techniques applicable to agricultural and rural development projects is the IFAD “Guide for Project Monitoring and Evaluation3”. 4.1 Performance Indicators The CARD standard logframe must specify performance indications and the means by which they are to be measured. Performance indicators are qualitative or quantitative factors/parameters that provide a simple and reliable basis for assessing achievement, change or performance. Each objective, output, activity and input included in the logframe must have performance indicators in order to be successfully monitored and evaluated. Performance indicators enable managers to track progress, demonstrate results and take corrective action; and enable evaluators to assess impact. If possible, key stakeholders should be consulted in selecting indicators to ensure that the M&E system measures things that are important to them. In order to limit the time and effort required to obtain information about indicators it is preferable to select indicators which can be measured from existing data sources or from routine project monitoring data. The temptation to have too many indications should be resisted by applying the “need to know” test – see Box 3 below. Box 3: The “Need to Know” Principle Many project M&E systems are excessively complicated and expensive and collect a lot of non-essential information which is never used and may even conceal things that are really important. The “need to know” principle distinguishes between what is really essential, and that which is merely interesting or informative. The following are some useful guidelines  Keep the number of performance indicators as small as possible.  Focus on things that are essential to know to monitor and evaluate the project.  Adjust M&E effort to the scale of the project: small projects = small M&E effort.  Keep it as simple as possible.  Focus on the farmers and how they are affected by the project. The different types of indicators used in M&E include the following: 3 This can be downloaded in pdf format from www.ifad.org/evaluation/guide. CARD – M&E Procedures January 2010 12  Simple quantitative: e.g. No of people trained, average crop yield.  Complex quantitative: e.g. food consumption per household, crop gross margin.  Indices: e.g. cropping intensity.  Proxy indicators; e.g. % of households owning motorcycles.  Open-ended qualitative: e.g. what stakeholders think about performance.  Focused qualitative: e.g. perceptions about a specific technology. A common error in selecting performance indicators is to confuse indicators of performance with explanatory measures. Indicators measure actual performance, whereas explanatory measures provide explanations or insights into why certain levels of performance were achieved. Performance indicators must measure final outcomes and impacts, or at least proxy measures thereof. Explanatory indicators are usually about intermediate outcomes and impacts. An example of the difference between performance indicators and explanatory measures is:  Performance indicator: Total milk produced per farm  Explanatory measures: Number of cows per farm, milk yield per cow, supply of forage, vaccination coverage etc. There is nothing wrong with collecting information on explanatory measures of impact, provided the true performance indicators are not overlooked, and provided the effort used in collecting the explanatory information does not diminish the project’s ability to measure the performance indicators. 4.2 Information Sources and Timing of Impacts Sources of Information: As far as possible monitoring activities should be confined to gathering, analysing, and reporting on information that is necessary for managing the project and the CARD Program in an efficient and effective manner. Evaluation should be based on the same set of information, but sometimes it is necessary to obtain additional information, over and above that required for project/program management. In the best case, routine reporting and management information systems will generate sufficient information for evaluation. More commonly however, some additional factfinding is needed to verify activities undertaken and the impacts they have generated. For projects where farmers are directly involved, primary data collection including farmer surveys are usually needed as part of the evaluation process. Timing of Impacts: In a few cases project impacts on the target beneficiaries are observable and measurable before project completion. However, more commonly there will only be preliminary indications of impact apparent during the implementation period, and in many cases impacts on beneficiaries will not be evident or measurable until later. Considering both information sources and timing of impacts CARD projects can be categorized according to the ease and simplicity of M&E, and consequently the amount of resources required for the task. As shown in the following chart, projects falling in the A1 cell of the box are the easiest to monitor and evaluate and those in C3 are the most difficult. As with all R&D projects which are intended to benefit smallholder farmers the majority of CARD projects fall towards the difficult end of the range in terms of both information requirements and timing of impacts. Consequently evaluations conducted at project CARD – M&E Procedures January 2010 13 completion usually involve estimation of benefits which are likely to eventuate rather than actual measurement of impacts. Information Sources Timing of Impacts 1. Impacts on target beneficiaries observable and measurable before project completion. 2. Preliminary indications of impact apparent during implementation period 3. Impacts on beneficiaries will not be evident or measurable until later. A. Routine reporting and management systems provide sufficient information for evaluation.  No additional data collection needed.  Evaluation at completion is adequate  No additional data collection needed.  Follow-up assessment needed after project completion  No additional data collection needed.  Full ex post evaluation is essential B. Some additional factfinding needed to verify activities undertaken and impacts.  Some additional factfinding needed.  Evaluation at completion is adequate  Some additional factfinding needed.  Follow-up assessment needed after project completion  Some additional factfinding needed.  Full ex post evaluation is essential C. Primary data collection including farmer surveys required to assess impact.  Primary data collection needed.  Evaluation at completion is adequate  Primary data collection needed.  Follow-up assessment needed after project completion  Primary data collection needed.  Full ex post evaluation is essential An example of a CARD project which would fall into the C3 cell is project 055/04 “Enhancing Small Holders Access to Agribusiness Services in the Central Region of Viet Nam”. This project involves capacity building among agribusiness service providers, for which the impact on farmers could not be expected for some time after project completion. Primary data collection including farmer surveys will be required to assess impact. A project that would fall somewhere near the top left part of the box is project 01/04 “Diagnosis and control of diarrhoea in suckling pigs”. This project is preparing diagnostic tests and vaccines for testing and demonstration in farmer’s pig herds. Results will be visible and measurable within a few weeks allowing reasonably confident evaluation of the likely impact on the profitability of smallholder pig production. 4.3 Using Negative Findings Whenever project evaluation is done rigorously and objectively it will identify failures as well as successes. But failures are rarely total, and usually some benefits are generated in terms of lessons learned and knowing what doesn’t work. This is especially true in R&D projects such as CARD where new technologies are being tested and evaluated. Evaluation procedures should therefore extract as much benefit as possible from the so-called failures as well as draw attention to the positive outcomes of the successes. The evaluation should be forward- looking and constructive. Where mistakes have occurred or performance has been disappointing it is vital to identify the reasons why and the lessons learned. 4.4 Baseline Information The purpose of baseline studies is to provide an information base against which to monitor and assess progress during implementation and after the project is completed. Baseline studies are the first step in M&E and focus on the indicators and performance measures detailed in the logframe. The MTR, PCE and other evaluations judge progress largely by comparison with the baseline data. CARD – M&E Procedures January 2010 14 Baseline information comprises facts and figures collected during the initial phases of a project that provide a benchmark for measuring progress in achieving project objectives. Most CARD contracts require the proponent to conduct a baseline survey or compile existing baseline data to provide a factual basis for later evaluation of the project. The most important aspect of baseline data collection is to be highly discriminating in deciding what information to collect. Good baseline information is relatively rare, because it is either not collected at all, lost by the time it is needed, or because the wrong questions are asked. The logframe should be used to determine the baseline information to be collected. The two columns detailing Performance Indicators and Performance Measures show what is essential to record in the baseline data. Other information may be gathered at the same time, especially if this provides insights into the reasons underlying success and failure, but this should only be done if it does not interfere with, or deflect attention from, the key performance indicators and measures. Useful baseline data sometimes can be found from existing sources but usually it will be necessary to undertake primary data collection which is tailored to the precise data needs of the project. Baseline studies should be undertaken as early as possible during the life of the project, but not before project objectives and activities have been well defined, along with the target population. Baseline studies conducted during the project design process run the risk of asking the wrong questions. Baseline studies can be used to measure changes attributable to project interventions in two ways: (i) “before and after” comparisons; and (ii) “with and without” project comparisons. These two approaches have advantages and disadvantages as follows: Before and After Comparisons With and Without Comparisons Advantages  Need to collect data from only the project area, so demands fewer resources.  Allows a combination of monitoring and evaluation functions  Provides a stronger motivation for participatory monitoring and evaluation. Advantages  Increases the likelihood of identifying causal factors in change.  Allows a clearer measure of the amount of change Disadvantages  More difficult to identify causal factors in change, especially where other activities are being undertaken in the same location  Assumes that change will be a linear progression.  Only provides two snapshots in time, one at the beginning and the other at the end, and ignores what happens in between. Disadvantages  Difficult to find truly comparable areas in terms of agro-ecology and socio- economic conditions.  Can be compromised by the activities of other donors, local government and community organisations in the “without” location.  Requires more advanced statistical skills and software.  Is more expensive.  Does not provide information that is useful in monitoring.  People in the “without” location may CARD – M&E Procedures January 2010 15 object to missing out on benefits. What happens if, at the time of project completion or ex post evaluation, the baseline data are found to be absent or inadequate in some way? First of all, this seriously reduces the rigour and value of the evaluation process; but there are usually ways to salvage some value from the exercise. For example, where the lack of baseline data makes “before and after” comparisons impossible, “with” and “without project” comparisons can be useful provided pairs of comparable individuals, groups or regions can be compared. Whilst there will always be problems in attributing differences to project interventions, this applies equally to “before and after” comparisons. Likewise official statistics can sometime make up for lack of baseline data for both “before and after” and “with and without” comparisons. Proxy indicators can sometimes be obtained retrospectively to make up for lack of baseline data. If all else fails, there is always anecdotal evidence based on what stakeholders and observers remember of the pre-project situation. 4.5 Options for Comparison The process of evaluation always involves comparisons, since the assessment of outcomes and impacts tries to identify changes that can be attributed to project interventions. There are three main types of comparison which can be used:  “Before and after” – this requires the collection and storage of accurate baseline data on the performance indicators specified in the logframe followed by collection of information on the same indicators at or after project completion.  “With and without” – this involves comparison of project and non-project areas which are otherwise similar in agro-ecological and socio-economic characteristics. The comparison sheds light on the question of “what would have happened in the project areas in the absence of the project” and adds weight to the attribution of benefits to the project interventions.  “Participants and non-participants” – this involves comparison of participating and non participating households within the project area in the search for evidence that project interventions did in fact make significant changes to peoples’ lives. However, in some cases this method may under-estimate project benefits if there has been “leakage” of benefits to non-target households, as tends to occur with very popular and easily disseminated technologies like improved crop varieties. The evaluation will have added credibility if more than one type of comparison is used producing similar findings. 4.6 Contribution Analysis The standard methodologies for M&E have been developed for investment projects where there is generally a strong causal association between the investment and the expected results which can be demonstrated ex ante as part of the feasibility/design process, and ex post as part of the evaluation. Whilst the basic concepts of M&E apply equally to R&D and other types of investment, some variations in the approach are necessary. Firstly, with R&D investments there is much greater uncertainty about results because of both the un-known outcome of the research, and because of the many other factors at play in translating research results into CARD – M&E Procedures January 2010 16 agricultural production outcomes, and ultimately to impact on people’s lives. In addition the results at outcome and impact level always take some time, often many years, to emerge and become measurable. Because of these uncertainties and time lags attribution is always problematic in monitoring and evaluating R&D. In the usual event that there is no rigorous means of linking cause and higher level effects it is necessary to resort to contribution analysis (see Box 4), recognising the reality that impacts usually have multiple causes which cannot be unscrambled. Box 4: Contribution Analysis Contribution analysis is used where cause-effect relationships are diffuse or indirect and where a particular course of action contributes to the achievement of certain results, but is not in itself sufficient to deliver the results. It recognises that in most development contexts there are multiple influences on the achievement of results and direct causal linkages can rarely be proven. Contribution analysis aims to reduce the level of uncertainty about contribution, by providing a credible and logical explanation of causes and effects. The essential elements of the approach include:  Acknowledging and accepting the problem of attribution.  Presenting the logic (usually in chart form) to explain why certain actions influence outcomes.  Identifying and documenting changes that provide evidence of contribution.  Using performance indicators which are appropriate for the nature of the expected results.  Tracking performance over time or comparing performance between locations.  Acknowledging and testing alternative explanations.  Gathering additional evidence such as expert opinions and case studies.  Aiming to tell a credible story which provides evidence, rather than absolute proof. Source: Mayne (1999) “Addressing Attribution Through Contribution: Using Performance Measures Sensibly. Office of the Auditor General, Canada 4.7 Specific M&E Tools The literature on M&E lists more than 30 tools which can be used to monitor and evaluate agricultural and rural development projects. The best tool, or combination of tools, varies from project to project, and according to the time and resources available. Whatever method(s) are chosen they should be suitable for both monitoring project implementation to provide information valuable to management, and for subsequent project completion or ex post evaluation. Some of the tools which are likely to be useful in evaluating R&D projects in Vietnam are discussed below.  Documentation Review: The first step in evaluating any CARD project for the purposes of mid-term, project completion or ex post review should be study of existing documents held at the CARD PMU and by the implementing organisations. If the project has been well designed and well monitored much of the necessary information will be obtained from the existing documents. This process also helps the review team to understand the project including what to look for and where to look for evidence of outcomes and impacts.  Sample Survey Methods: These involve first selecting the sample and then designing the questionnaires or checklists. The sample may be a random sample, a stratified random sample, or a non-random/targeted sample. The survey questions need to be CARD – M&E Procedures January 2010 17 carefully phrased and tested to ensure that people understand them correctly and that the questions themselves do not bias the results. Sample surveys are a specialised skill and an expert in this field should be consulted before proceeding. The “need to know” principle needs to be firmly applied here.  Direct Observation: This is a basic but effective means of assessing outcomes and impacts which should almost always be used to cross-check or verify other sources of information. Photographs add significantly to the value and interest of M&E reports. However, evaluation teams should be careful to avoid observation bias such as only observing the more readily accessible and more successful farms.  Key Informant Interviews: In any project context there are always key individuals who have especially valuable knowledge or opinions. These people may be members of the implementing agencies/proponents, beneficiaries, other stakeholders or simply well informed observers. Structured interviews with such persons should always form part of the evaluation process. This also adds to the participatory nature of the evaluation.  Bio-Physical Measurements: In some cases the key performance indicators may be expressed in bio-physical terms such as crop yields, amount of land terraced, number of animals vaccinated etc. The key here is to use simple but accurate measures which can be compared with the baseline date in order to provide solid evidence of cause and effect.  Benefit Cost Analysis (BCA): This method draws on information obtained through other means to compare the total benefits and cost of a project. Further detail on BCA is given in Section 5.  Semi-Structured Interviews: These are face-to-face interviews with individual stakeholders or small groups using a series of open-ended questions and topics to guide the conversation. Such interviews are critical in gaining an in-depth understanding of why things happened (or did not happen), and what people feel about the relevance and impacts of a project. Sometimes the interviews will identify project outcomes and impacts which nobody had previously thought about or expected.  Case Studies: These are detailed assessments of selected individuals or groups which are believed to be broadly typical or representative of a larger group. Detailed case studies can reveal deeper insights about project outcomes and impacts but should always be used in conjunction with methods which interact with a larger cross section of stakeholders.  Focus Groups: These are small groups of people (say 5-10) who are selected on the basis of their special knowledge or understanding and are brought together for facilitated discussion on project outcomes and impacts. Focus groups are more about obtaining opinions or views than concrete factual information. One problem with focus groups is that vocal participants with strong opinions can dominate proceedings and provide misleading impressions. Skilled facilitation is needed to ensure that the full range of views is expressed. CARD – M&E Procedures January 2010 18  SWOT Analysis Undertaken in Groups: This is an easily applied technique to identify strengths of a project (things that have worked well), weaknesses (things that didn’t work so well), opportunities (to build on strengths and remedy weaknesses), and threats (from external forces) that may damage future outcomes. SWOT analysis is very useful in identifying lessons learned.  Rapid Rural Appraisal (RRA): This is really and approach to M&E rather than a specific tool, since various combinations of the above methods may be used. RRA represents a quick low-cost way to gather information from stakeholders and involves key informant interviews, focus groups, semi-structured interviews, direct observations, mini-surveys etc. Because RRA is a quick process it can provide useful information for management decision making and engaging more closely with beneficiaries. However the approach is less valid and accurate than formal surveys and requires well developed skills in group facilitation, observation and non-directive interviewing. On the positive side, RRA is strongly participatory and actively involves stakeholders in the evaluation process. 4.8 Impacts to be Assessed The ultimate purpose of project evaluation is to assess outcomes (what changes have taken place), and impacts (how such changes have affected peoples lives). CARD projects deliver three types of impacts to their target beneficiaries and stakeholders:  Financial Impacts4: things which improve farmer’s incomes or assets through increasing revenues and/or reducing costs. The value of produce consumed by the farm household is generally counted in estimating revenues and the value of un-paid family labour is considered a cost. Ways of estimating financial impacts include: - Benefit cost analysis based on information provided by farmers - Household income and expenditure surveys - Proxy measures of financial wellbeing: e.g. size of house, ownership of motorcycles - Case studies and anecdotes Positive financial impacts are critical to the sustainability and wider dissemination of agricultural technologies or innovations. Without clear financial advantages smallholders will not embrace change or even sustain changes that have already taken place. This makes financial impact of overriding importance in impact assessment.  Social Impacts: things that are not measurable in financial terms but which affect the quality of people’s lives. Assessment of social impacts requires careful consideration of issues such as the following: 4 Conventional M&E methodologies also refer to economic impact. This is related to, but different from financial impact. Economic impact refers to the overall impact on the economy of Vietnam, whereas financial impact refers to the impact on the incomes of farmers or other target beneficiaries. In reality the two a closely related and highly correlated and CARD deliberately focuses only on financial impacts. Disparities between financial and economic impacts arise where there are major distortions in the markets for inputs and outputs. Whilst such distortions do exist in Vietnam, they are generally fairly small and tend to cancel each other out. CARD – M&E Procedures January 2010 19 - How did the project affect people’s lives? eg improved nutrition/food security, reduced labour input, better health (food safety) etc. - Who are the beneficiaries? men/women, rich/poor households? - Who were the target groups and did they benefit? - Is the technology accessible/affordable? - Is there evidence of “elite capture” or exclusion?  Environmental Impacts: these are positive or negative environmental consequences of CARD projects which need to be identified during the project design process, monitored during the life of the project, and evaluated at the end. Examples of positive environmental impact include reduced pesticide use and improved cultivation methods which reduce soil erosion. Negative environmental impacts of agricultural projects can include things such as water pollution from animal waste and biodiversity reduction from plantation forestry. Further details on environmental monitoring and evaluation are given in the following section. Identifying impacts in the field is much easier if we know what to look for. This is why it is important to have at least one technical specialist in each project evaluation team who can brief the other members on the key indicators of success for a particular agricultural enterprise. Direct observation of crops or animals can tell a lot about their productivity to those who have the required technical expertise. Other people are skilled in assessing farmer attitudes and enthusiasm, which are useful indicators of sustainability. 4.9 Environmental Monitoring Project proposals require proponents to assess positive and negative environmental impacts at EOI stage. Initial environmental assessment at this stage should categorise the project according to the level of environmental risk using the internationally accepted A/B/C rating system:  Category A are Projects with possible serious environmental consequences. Such Projects should be subject to environmental impact assessment (EIA) before approval and necessary safeguards and monitoring procedures specified in an Environmental Management Plan (EMP).  Category B are Projects with possible environmental consequences but which are readily manageable using some simple safeguards.  Category C includes Projects where no environmental consequences are foreseen. Few negative environmental impacts are expected on the CARD Projects and this expectation has been verified with project proposals submitted for funding. Most would be classified as category B or C. However all projects should be monitored and evaluated from an environmental perspective to identify and describe actual negative and positive environmental consequences. In most cases observation will be the most effective tool for environmental impact assessment but in some cases measurements (e.g. water quality, pesticide residues) may be needed. Field monitoring through site visits conducted by the PMU should review EIAs and EMPs where these exist. In addition, Project progress reports should report against environmental CARD – M&E Procedures January 2010 20 performance indicators in progress reports and PCRs. The PMU reports at the Program level and will raise any adverse environmental issues with MARD and the PCC. 4.10 Judging a Project’s Success The primary issue in judging success is the extent to which the project achieved its objectives and the degree to which outcomes are likely to be sustained. Issues such as the level of financial return, the impact on poverty reduction, the sustainability of benefits, and the implications for the government's budget also need to be described and assessed, along with social and environmental impacts. For projects designed to boost agricultural production or prices, the long-term effect on the national economy is a major basis for judging performance and success. This effect may be quantified and expressed as the financial Benefit Cost Ratio (BCR). In cases where detailed financial analysis has not been undertaken during design or at project completion, it is not normally feasible for the evaluation team to undertake detailed financial analysis. However, it may be possible to undertake financial assessment of one or more key components and to make a general assessment of overall financial impact. Many CARD projects focus on institutional strengthening. In these cases, quantitative objective assessment may be difficult unless baseline surveys were conducted, the basis for comparison established and clear performance indicators put in place. In this case, judgments will need to be made of the form and content of information to be used in assessing performance. 5 BENEFIT COST ANALYSIS IN M&E 5.1 Overview Benefit Cost Analysis (BCA) is a decision-support tool to be used in conjunction with other measures of project impact. It is a key tool in explaining the underlying logic of a project. BCA can be applied at various levels including policies, sectoral strategies, programs, projects and project components. BCA is widely used by international development assistance agencies as a key criterion for allocating resources. One of CARD’s objectives is to help MARD improve the allocation of resources amongst its portfolio of R&D programs. BCA is seen as valuable tool in supporting better allocation decisions. BCA is the comparison of project financial benefits or impacts (direct and indirect) attributable to a project, with the investment and recurrent costs of implementing it. BCA is the basic analytical tool for assessing the financial (and by implication, economic) impacts of CARD projects, and ultimately the entire CARD Program portfolio. BCA can be conducted before, during or after a project is implemented with a progressively increasing level of precision. Pre-project (ex ante) BCA is based on expectations of the magnitude of benefits and costs. Post-project (ex post) BCA is based on the best available estimates or measures of actual benefits and costs. It can be applied at the micro-level relating to an individual beneficiary, or at the macro level relating to an entire project. In all cases the objective is to estimate the Benefit Cost Ratio (BCR) which provides a quantitative measure of project financial impact. CARD – M&E Procedures January 2010 21  Project benefits are positive financial, environmental and social consequences that can reasonably be attributed to a project investment. In BCA we are only concerned with financial benefits.  Project costs are the total value of resources used in generating benefits, including the value of un-paid family labour.  BCR = (the sum of benefits) ÷ (the sum of costs); expressed as a ratio (eg: 1.4:1) BCR provides a single figure estimate for comparing projects in terms of their financial impact. It shows how much value is created for each VND invested. A BCR >1.0 represents a worthwhile investment, and amongst alternatives, the project with the highest BCR represents the best investment. 5.2 Identifying Benefits and Costs This is the first step in assessing financial impact. Some benefits and costs can be easily measured and quantified – others can only be described. Impact assessment often fails to recognise all of the benefits and costs and this can produce misleading results. There also needs to be a consistent approach between different projects to identifying and estimating benefits and costs to allow form meaningful between-project comparisons. As well as identifying benefits and costs it is important to determine who receives the benefits and who pays the costs. CARD is primarily concerned about generating financial benefits for smallholder farmers. However, the analysis should not overlook benefits received by other parties: e.g. traders, input suppliers, consumers, labourers etc. These can be a significant share of total benefits Financial benefits take several forms. Financial benefits to farmers may come in the form of cash income arising from increased revenues and/or reduced costs, or the value of farm produced consumed by the household. Primary or direct benefits to CARD stakeholders may include some or all of the following:  incremental (increased) value of production – resulting from higher yields and/or quality;  decremental (decreased) cost of production due to improved technologies; and/or  value added from improved transport processing and marketing. External or secondary benefits may accrue to persons other than primary stakeholders, such as consumers who benefit from cheaper or better quality food and traders who market incremental production. In estimating financial costs it is necessary to distinguish between a number of different cost categories as follows:  Public costs incurred by government institutions – these may be funded by regular budget, CARD or donors.  Private costs incurred by farmers and other private sector actors. CARD – M&E Procedures January 2010 22  Investment costs – occur once only, usually at the beginning of a project: e.g. machinery, equipment and training.  Recurrent (operating) costs – these continue into the future: e.g. salaries, building maintenance.  Cash costs – e.g. fertiliser or pesticides purchased by farmers.  Non-cash costs – e.g. value of unpaid family labour used (see opportunity cost below).  Financing costs – interest paid on money borrowed for investment or working (seasonal) capital. The concept of opportunity cost is also relevant to BCA in some cases. Opportunity cost is the loss of income by using resources which would otherwise have been productively employed elsewhere. For example, if a farmer uses land to grow a new crop, the cost of the land is the income he would have earned if he had grown the usual crop. Opportunity cost is also used in estimating the value of un-paid family labour. The value of the labour is the amount it would have earned working on the next-best alternative. This value can vary quite markedly between different seasons and locations according to the level of employment opportunities. BCA is only concerned with costs that change as a consequence of the project. These include increased or incremental costs and decreased or decremental costs. If costs are the same before and after the project, or with and without the project, they are of no relevance to BCA and do not have to be estimated at all. It is also necessary to distinguish between current and constant prices in estimating costs and benefits. Current prices (also known as nominal prices) are the actual amount of money paid or received. Constant prices (also known as real prices) are prices which have been adjusted to remove the effect of inflation. BCA normally uses constant prices so that when costs and revenues are projected forward it is assumed that today’s prices will continue into the future. Sometimes it is not immediately clear whether something is a cost or a benefit. The test to apply here is that: (i) anything which increases Vietnam’s GDP is a benefit – regardless of who receives it; and (ii) anything which reduces Vietnam’s GDP is a cost – regardless of who pays it. The confusion usually arises in relation to labour, whether paid or un-paid. Employment is regarded as a social benefit to the employee, but in BCA it is always treated as a cost to the nation. Using the opportunity cost concept labour is a cost, since if not employed on project activities it would be contributing to Vietnam’s GDP in some other way, however small. In situations of high unemployment or underemployment the opportunity cost of labour may be very low, but it is never negative. 5.3 The Representative Farm Concept Most CARD projects engage a number of smallholder farmers who are involved in trials, demonstrations and field testing of new technologies. It is not feasible to undertake a separate CARD – M&E Procedures January 2010 23 BCA for each participating farmer, and in any case the results would have to be re-aggregated to make an estimate of overall project impact. The best approach is to formulate a model of a “typical” or “representative” farm. This is done because it is impractical to complete a financial analysis for each individual’s enterprise. Instead, the evaluation team must develop a picture of what is “typical” or “representative” of the group as a whole. In some cases this will be a simple average, for example, the number of trees or fish ponds per group member. In other cases the judgement may be more complex. For example if farmers are growing different types of vegetables, the team will have to decide on a small number of crops as typical or representative examples of the enterprise. The financial BCA is then carried out for one or more representative farm models. 5.4 Enterprise Budgets Generally a key starting point in BCA is the development of enterprise budgets for the crop, livestock or aquaculture enterprise being analysed. Usually there will be one budget representing traditional or conventional farming practices, and another representing the improved practices which the project has developed and/or disseminated to farmers. The information required to construct enterprise budget generally has to be gathered from several sources including farmers themselves, research and extension personnel, and technical specialists who are members of the evaluation team. Information on costs and prices often needs to be obtained from commercial sources such as traders and input suppliers. Enterprise budgets for annual crops or short-cycle livestock and aquaculture enterprises are the simplest to develop. Longer-term activities such as perennial crops and forestry are more demanding to analyse and must cover the complete life cycle of the enterprise. Standard formats for annual and perennial enterprises are given in Attachment 4. 5.5 Comparing Benefits and Costs5 The standard format for comparing benefits and costs is given in the following table. All benefit cost analyses have to produce a table in this general format in order to estimate the BCR. 5 The approach presented here compares benefits and costs without consideration of the period of time between incurring costs and receiving benefits. Further training is required to enable evaluators to apply discounting procedures so that future benefits can be discounted to today’s values (Present Value). For the time being CARD will use un-discounted measures of costs and benefits for project evaluations, and apply discounted measures of impact at the aggregate Program level. CARD – M&E Procedures January 2010 24 There are two ways of comparing project benefits and costs in the above table: (i) “with” and “without” project comparisons; and (ii) “before” and “after” project comparisons. These do not always produce the same results. There are two reasons for this. First, if the “without project” scenario represents a non-static situation (such as improving crop yields in the absence of project initiatives), which is often the case, this needs to be taken into consideration in estimating the incremental benefits to the project. The second reason is that “before” and “after” comparisons only represent two snapshots in time and ignore what happens in between, including investments needed to achieve the actual outcome. In the above example the “before” and “after” project comparison shows net benefits increasing from 20 in Year 0 to 90 in Year 5, an apparently very large improvement. However, this is misleading because it ignores the “without project” scenario and the significant investment costs incurred in Year 1. A more valid measure of project financial impact is to divide incremental benefits over the life of the project (107) with incremental costs (40) to produce a BCR of 2.7. The conclusion is that “with” and “without” project comparisons are the only valid way of comparing benefits and costs which gives a reliable estimate of project financial impact. “Before and after” comparisons have some value but need to be interpreted with some caution. 6 WORKPLAN FOR EVALAUTION OF THE CARD PROGRAM 6.1 Overview The current phase of CARD commenced in March 2004 and will end in December 2010. The final batch of projects was approved in 2007, taking the total to 38 projects. The number of projects at different stages in January 2010 was as follows: Year Number of Projects at January 2010 Approved MTR Completed PCE 2004 11 1 8 8 2005 11 4 6 6 2006 10 2 0 0 2007 6 1 0 0 Without Project Scenario a/ Year 0 Year 1 Year 2 Year 3 Year 4 Year 5 Total Costs A 100 100 100 100 100 100 600 Benefits B 120 122 125 130 133 133 763 Net Benefits C=(B-A) 20 22 25 30 33 33 163 With Project Costs D 100 200 90 85 85 80 640 Benefits E 120 130 140 150 160 170 870 Net Benefits F=(E-D) 20 -70 50 65 75 90 230 Incremental Costs G=(D-A) 0 100 -10 -15 -15 -20 40 Incremental Benefits H=(E-B) 0 8 15 20 27 37 107 Incremental Net Benefits I=(H-G) 0 -92 25 35 42 57 67 Benefit Cost Ratio 2.7 a/ Year 0 represents the "before project" situation. Standard Layout for Benefit Cost Analysis CARD – M&E Procedures January 2010 25 Total 38 8 14 14 About half of the 2004-2006 projects are two-year project and half are three-year projects. All of the 2007 projects will run for two years. Most of the 2004 and 2005 projects have been completed or are nearly complete, and PCEs have been undertaken for 14 of them. Eight projects have been subject to MTRs. All CARD projects should be evaluated at completion. However, given that CARD will close in December 2010, it will not be possible to undertake PCEs for all 24 of the remaining projects. This is because the PCU does not have the capacity to oversee such a large number of evaluations during the remaining 11 months, and also because some of the projects will not be completed until the latter part of 2010 when the PCU will be fully engaged in winding up the Program. An achievable target would be to undertake about 16 PCEs at the rate of two per month between March and October 2010. 6.2 Mid-Term Reviews The purpose of the MTR is to check that projects are running according to plan, identify “problem projects” and provide the necessary support where needed. Resource constraints have limited the number of MTRs to eight projects to date. No further MTRs will be undertaken during 2010. 6.3 Project and Program Completion Evaluations Project completion evaluations will take place 3-6 months after project completion to allow for time for all documents including the project completion report to be submitted to the PMU. It is advisable not to attempt evaluation until all of the documentation is completed. The completion evaluations will be undertaken in batches by trained project evaluators and relevant technical specialists working in teams of 3-4 persons. Mid-Term Review of the CARD Program took place during the second half of 2007 by the independent Technical Advisory Group (TAG). Completion evaluation will be undertaken in the second half of 2010 and will involve review and aggregation of all completed project evaluations at that time. This will provide valuable information for the Project Completion Report to be prepared during the final three-months of the CARD Program. Project evaluations will be undertaken by selected teams of evaluators drawn from those who have participated in the March 2007 and October 2008 training workshops and the January 2010 M&E Review Workshop. In some cases it will also be necessary to engage technical specialists who have not undergone evaluation training. The time needed to complete the evaluations will vary widely depending on the nature and geographic scope of the projects. At the lower end of the scale there are projects mainly concerned with capacity building and training which can be evaluated thoroughly in a day or two. At the upper end there are projects working directly with farmers and other stakeholders in a number of different provinces. These will require extensive travel and field work to assess the impact at farm level. An indication of the range of resource requirements is given in the following table. Indicative Work Program for Completion or Ex Post Evaluation Tasks Days Responsibility CARD – M&E Procedures January 2010 26 1. Prepare Terms of Reference for evaluation. 0.5 PMU2. Identify and recruit Evaluation Team members. 0.5 3. Briefing for Evaluation Team. 0.5 4. Desk review of information held in the CARD PMU: project EOI and Proposal, progress reports, publications, completion report etc. 2 Evaluation Team and Partner Institutions 5. Prepare checklist and work plan for evaluation. 1 6. Factfinding:  Interviews with key informants including project directors (Australian and Vietnamese) and project staff.  Interviews with other participating agencies, NGOs etc.  Meetings (group and/or individual) with project beneficiaries. 1-2 1-2 1-5 a/ 7. Analysis and report preparation. 3-5 8. Review of draft evaluation report by project partners and stakeholders 1-2 Partner Institutions 9. Finalisation of evaluation report taking partner and stakeholder comments into consideration. 1 Evaluation Team a/ Depends on the number and geographic location of beneficiaries. Ideally evaluation should be undertaken in close consultation with both the Australian and Vietnamese partner organisations. However, since the exercise will generally take place several months after project completion, it will not usually be possible for the Australian partner to participate, other than by commenting on the draft evaluation report as indicated in Task 8 in the above table. 6.4 Ex Post Evaluations Ex post evaluations should be conducted after outcomes and impacts of a project have had time to fully mature, so that sound judgements can be made on the success of the project and lessons learned, particularly with regard to sustainability. Often initiatives which appear successful and sustainable at project completion have disappeared without trace several years later. In other cases ex post evaluation identifies spontaneous and widespread adoption of new technologies far beyond the original target group of beneficiaries. With CARD due to end in December 2010 it will be fully engaged in conducting PCEs until then and it will not be possible to conduct ex post evaluations. Projects will have to undergo ex post evaluation under MARD supervision after CARD completion. 6.5 Further Training needs The ultimate aim of M&E training is to develop a group of skilled and experienced evaluators within MARD and its affiliated institutions which is capable of evaluating all CARD projects in the first instance, and subsequently strengthening the evaluation of the entire MARD project portfolio as a means of improving resource allocation decisions. The training workshops conducted in March 2007 and October 2008 introduced around 40 MARD and STED staff to basic evaluation methods and provided some practical experience through the conduct of case studies. This has provided a nucleus of trained evaluators within MARD, with at least half of those attending the training showing considerable aptitude and enthusiasm for the task. Further practical experience has been gained on the eight MTRs and CARD – M&E Procedures January 2010 27 14 PCEs completed. There is now sufficient number of trained personnel to carry out the scheduled evaluations. There are now 29 people who have participated in PCEs, nine of them as team leaders. The immediate training need is to gain further practical experience. With a number of evaluations scheduled for the final quarter of 2008 and in 2009, this provides an opportunity to further develop the practical evaluation skills of the nucleus group. Another training priority is in the area of Benefit Cost Analysis. The 14 PCEs undertaken so far showed that whilst the concepts are generally understood, the teams struggled with the practicalities of BCA. This suggests a need to further develop BCA skills. The best way to do this will be through case studies and examples, based on the evaluations to be undertaken in the coming 11 months. The teams selected for these evaluations should collect the data needed for BCA and complete the analysis with the support of the PMU M&E Specialist. 6.6 Conclusions CARD is approaching the end of its seven-year duration and the first 14 projects have been completed and evaluated. During the remaining life of the Program, systematic PCEs will be undertaken to learn lessons that can be applied to the ongoing management of the CARD Program as well as the wider agricultural R&D agenda across MARD and its affiliated institutions. To do this MARD needs to expand its team of skilled evaluators through practical experience. CARD will assist by providing further coaching and guidance in evaluation methodology. The benefits will be improved project design, the identification of areas of high (and low) return R&D investment, improved relevance and impact, and improved accountability and transparency in the allocation of resources to agriculture and rural development. CARD – M&E Procedures January 2010 1 ATTACHMENT 1: TERMINOLGY AND DEFINITIONS To avoid ambiguity in monitoring and evaluating projects, the following standard terminology has been adopted by CARD. CARD-SPECIFIC TERMINOLOGY Term Definition Expression of Interest (EOI) First stage in the CARD project cycle in which the project rationale, objectives and components are presented for the purpose of initial evaluation and short listing. Peer Review Review of Project Proposals by specially selected professionals to assist in preparing high quality proposals. Program Used to refer to the overall CARD Program – as distinct from individual projects supported by CARD. Project An intervention that consists of a set of planned, interrelated activities designed to achieve defined objectives within a given budget and a specified period of time. Project Contract Contract between CARD and the proponent organisation(s) which defines the project to be implemented and the milestones at which funds will be disbursed. Project Coordinating Committee (PCC) The governance body which is responsible for R&D policy and priorities, and for making the final decisions on which projects to be financed by the CARD Program. Project Management Unit (PMU) The operational centre of CARD which is also the contracting party and the R&D program management agency responsible for monitoring, evaluation and impact assessment. Project Proposal The final submission by project proponents for consideration and approval by the PCC. Proponent The organisation with primary responsibility for designing and implementing a CARD Project. Short List Projects which are invited to submit full Project Proposals. Technical Appraisal Pannell (TAP) Independent group of experts engaged under contract to provide specialist technical advice on the quality of EOIs and Project Proposals. Feedback Comments and suggestions made to project proponents by the TAP, peer reviewers and PCC to assist proponents improve the quality of EOIs and project proposals. GENERAL MONITORING AND EVALUATION TERMINOLOGY Term Definition Activity Action taken or work performed in a project to produce specific outputs by using inputs such as funds, technical assistance, machinery and other types of resources Assumptions/Risks External factors, explicitly recognised in the logframe, that could affect the success of the project. Attribution Description of a likely causal relationship between project inputs and outputs. Baseline Information Facts and figures collected during the initial phases of a project that provide a base for measuring progress in achieving project objectives. CARD – M&E Procedures January 2010 2 Benefits Positive financial, environmental and social consequences that can reasonably be attributed to project investment. Benefits to farmers may come in the form of cash income or value of household consumption. Benefits may also include institutional capacity enhancements which may contribute to future financial, social and environmental benefits. Beneficiaries Individuals, groups or organisations who, whether targeted or not, benefit directly or indirectly from the project – also referred to as “primary stakeholders” Benefit-Cost Analysis The comparison of project benefits or impacts (direct and indirect) attributable to a project, with the investment and recurrent costs of implementing it. Capacity The ability of individuals and organisations to perform functions effectively, efficiently and in a sustainable manner. Causal Relationship A logical connection or cause-and-effect linkage in the achievement of results. Generally the term refers to plausible linkages, not statistically accurate relationships. Completion The final phase in the project cycle when a project completion report (PCR) is produced. Lessons learned are identified and the various project completion activities take place including a project completion evaluation. Costs The total value of resources used in generating benefits, including the value of un-paid family labour. See note below on opportunity cost. Design Logic A logically connected hierarchy comprising objectives, outputs, activities inputs, and outcomes. Effectiveness A measure of the extent to which a project attains, or is expected to attain, its objectives in a sustainable manner. Efficiency A measure of how economically inputs are converted to outputs. Evaluation A systematic (and objective as possible) examination of a planned, ongoing or completed project. It aims to answer specific management questions and judge the overall value of a project and generate lessons learned to improve future planning and decision-making. Evaluations commonly seek to determine the relevance, effectiveness, efficiency, impact and sustainability of the project. Evaluation should provide information that is credible and useful and offer concrete lessons learned to help partners and funding agencies make better decisions. Ex Ante Evaluation Evaluation conducted before or at the beginning of the project implementation period. Ex Post Evaluation Evaluation conducted after completion or at the end of the project implementation period. Impact The change in the lives of rural people, as perceived by them and their partners at the time of evaluation, plus sustainability-enhancing changes in their environment to which the project has contributed. Impacts can be positive or negative, intended or un-intended Indicators Qualitative or quantitative factors/parameters that provide a simple and reliable basis for assessing achievement, change or performance. A single objective may have more than one indicator. Input The financial, human and material resources necessary to produce the intended outputs of a project. Lessons Learned Knowledge generated by reflecting on experience, that has the potential to improve future actions. A lesson learned summarises knowledge at a point in time, while learning is an on-going process. Logframe A 4x4 logical framework matrix that summarises objectives (what the project intends to do), and how (necessary inputs and outputs), what the key assumptions are, and how the outcomes will be monitored and evaluated. Means of Verification The source(s) of information underlying the indicators. Mid-Term Review (MTR) An evaluation performed around the middle of the project implementation period with the objective of making necessary changes to the project strategy – also known as a Mid-Term Evaluation (MTE). CARD – M&E Procedures January 2010 3 Milestone A verifiable indicator that shows a specific output has been generated (output milestones) or outcome has been achieved (outcome milestones). In CARD projects milestones are specified in the Project Contract as the trigger points for disbursements of funds to proponents and partner organisations. Monitoring The regular collection and analysis of information to assist in timely decision-making and provide the basis for evaluation and learning. It is a continuous function that generates data to provide project management and stakeholders with early indicators of progress and achievement of objectives. Objective A statement detailing the desired outcomes of a project at different levels (short to long term). Objectives should be impact oriented, measurable, time bound, specific and practical. Opportunity Cost Loss of income by using resources which would otherwise have been productively employed elsewhere. Output Tangible, measurable and intended results produced through provision of project inputs in order to undertake project activities. Outcome Estimates or measures of what changes have taken place as a result of project implementation. Participatory Evaluation A term for the involvement of primary and other stakeholders in evaluation. Primary Stakeholders The main intended beneficiaries of a project – in CARD projects these are normally smallholder farmers. Also known as the target group. Project Cycle The sequence of activities involved in the design and implementation of a project including monitoring and evaluation. Project Design An iterative process beginning with a project idea and proceeding through a progressive formulation and review procedure until the project is fully designed and costed. Proxy Indicator A conveniently measurable indicator that is used to represent a less easily measurable one. Relevance The extent to which the objectives of a project are consistent with the target group’s priorities and the Governments policies. Self Evaluation Evaluation by those who are involved in implementing a project. Stakeholder An agency, organisation, group or individual with a direct interest in the project, or who affects or is affected by the project outcomes Sustainability The likelihood that the positive effects of a project (such as assets, skills, facilities or improved services) will persist for an extended period after the project is completed Target Group See primary stakeholders. Triangulation Use of a variety of sources and methods to cross check and validate data, where a single method does not provide an acceptable degree of accuracy. January 2010 ATTACHMENT 2: PROCEDURES AND TERMS OF REFERENCE FOR MID-TERM REVIEWS 1. OBJECTIVES AND SCOPE OF THE MID-TERM REVIEW As part of its routine monitoring and quality assurance procedures CARD carries out a review of projects during the implementation period, generally around the mid-point. This is known as the mid-term review (MTR). The MTR is an implementation support procedure which involves an interim assessment of the project to assess progress in undertaking activities and generating outputs, identify problem areas and propose solutions. In some cases this may suggest changes to the project design and budge