- Open Access
- Open Peer Review
Applying the results based management framework to the CERCA multi-component project in adolescent sexual and reproductive health: a retrospective analysis
Reproductive Health volume 15, Article number: 24 (2018)
Adolescent sexual and reproductive health (SRH), problems such as unplanned pregnancies are complex and multifactorial, thus requiring multifaceted prevention interventions. Evaluating the impact of such interventions is important to ensure efficiency, effectiveness and accountability for project funders and community members. In this study, we propose Results Based Management (RBM) as a framework for project management, using the Community Embedded Reproductive Health Care for Adolescents (CERCA) as a case study for RBM. The CERCA Project (2010-2014) tested interventions to reduce adolescent pregnancy in three Latin American countries, Bolivia, Ecuador and Nicaragua. Activities were designed to increase adolescent SRH behaviors in four domains: communication with parents, partners and peers; access to SRH information; access to SRH services; and use of contraception. When the project ended, the outcome evaluation showed limited impact with concerns about accuracy of monitoring and attrition of participants.
We reviewed and analyzed a series of CERCA documents and related data sources. Key findings from these documents were organized within an RBM framework (planning, monitoring, and impact evaluation) to understand how CERCA methodology and performance might have reaped improved results.
Strengths and weaknesses were identified in all three elements of the RBM framework. In Planning, the proposed Theory of Change (ToC) differed from that which was carried out in the intervention package. Each country implemented a different intervention package without articulated assumptions on how the activities of intervention would bring about change. In Monitoring, the project oversight was mainly based on administrative and financial requirements rather than monitoring fidelity and quality of intervention activities. In Impact Evaluation, the original CERCA evaluation assessed intervention effects among adolescents, without identifying success and failure factors related to the outcomes, the nature of the outcomes, or cost-effectiveness of interventions.
This analysis showed that multi-country projects are complex, entail risks in execution and require robust project management. RBM can be a useful tool to ensure a systematic approach at different phases within a multi-country setting.
La salud sexual y reproductiva de los adolescentes (SSR), como problemas de embarazos no planificados son complejos y multifactoriales, por lo que requieren múltiples intervenciones para su prevención. Evaluar el impacto de estas intervenciones es importante para asegurar la eficiencia, la eficacia y la rendición de cuentas para los donantes de proyectos y los miembros de la comunidad. En este estudio, proponemos la Gestión Basada en Resultados (o su sigla en inglés RBM) como un marco para la gestión de proyectos, utilizando el Community Embedded Reproductive Health Care for Adolescents (CERCA) como un caso de estudio para RBM. El Proyecto CERCA (2010-2014) puso a prueba las intervenciones para reducir el embarazo adolescente en tres países de América Latina, Bolivia, Ecuador y Nicaragua. Las actividades fueron diseñadas para mejorar los comportamientos adolescentes en SRH en cuatro aspectos: comunicación con padres, compañeros y pareja; acceso a la información sobre SSR; acceso a los servicios de SSR; y uso de anticonceptivos. Cuando el proyecto terminó, la evaluación de resultados mostró un impacto limitado sobre todo por la precisión en el monitoreo y pérdidas en el seguimiento de cohortes.
Se revisó y analizó una serie de documentos CERCA y fuentes de datos relacionados. Las principales conclusiones de estos documentos se organizaron en el marco lógico de la Gestión Basada en Resultados (planificación, monitoreo y evaluación del impacto) para comprender cómo la metodología y el rendimiento de CERCA podrían haber obtenido mejores resultados.
Se identificaron fortalezas y debilidades en los tres elementos del marco de RBM. En la Planificación, la propuesta Teoría del Cambio (ToC) difiere de la que se llevó a cabo en el paquete de intervención. Cada país implementó un paquete de intervención diferente sin supuestos articulados sobre cómo estas actividades de intervención producirían cambios. En el Monitoreo, la gestión del proyecto se basó principalmente en requisitos administrativos y financieros en lugar de supervisar la fidelidad y la calidad de las actividades de intervención. En la Evaluación de Impacto, la evaluación original del CERCA evaluó los efectos de la intervención entre los adolescentes, sin identificar los factores de éxito y fracaso relacionados con los resultados, la naturaleza de los resultados o el análisis de costo-efectividad de las intervenciones.
Este análisis mostró que los proyectos multinacionales son complejos, implican riesgos en la ejecución y requieren una gestión del proyecto sólida. La Gestión Basada en Resultados puede ser una herramienta útil para garantizar un enfoque sistemático en las distintas fases de un proyecto bajo un contexto multi-país.
Plain English summary
Adolescent pregnancy can negatively affect the health and well-being of adolescents. Complex problems like adolescent pregnancy require multi-faceted prevention interventions that are carefully planned, implemented and evaluated in order to satisfy funders and community members. The Community-Embedded Reproductive Health Care for Adolescents (CERCA) Project (2010-2014) set out to lower adolescent pregnancy in Bolivia, Ecuador, and Nicaragua through multiple ways: increasing communication; increasing the number of adolescents who can get sexual and reproductive information and services, and increasing the number of adolescents using birth control. When the project ended, CERCA had not met all of its goals. In this paper, we studied the CERCA project using a specific project management framework called Results Based Management (RBM) to learn more about the project’s strengths and weaknesses. RBM is a system that can enhance effectiveness and verify that the project is going in the right direction using feedback loops at each level. Using the RBM framework, we found important lessons in the 3 main elements of RBM, which are (1) planning, (2) monitoring and (3) evaluation.
(1) We learned that what was proposed in the project differed from what was carried out. (2) We also learned that the way the project’s progress was monitored did not meaningfully allow for an understanding of what was being done. (3) We also learned that measuring the impact left some unanswered questions. This analysis using RBM sheds new light on lessons learned from the CERCA project and may be useful in future adolescent SRH initiatives.
Multi-component, community-integrated health intervention packages are necessary to address complex social problems such as adolescent pregnancy . However, these intervention packages are often themselves complex and require intensive planning, sound implementation, and rigorous evaluation to ensure effectiveness, community acceptance, and return on investment by non-governmental organizations (NGO) and government agencies [16, 27]. Multi-country programmes have the opportunity to create broad, population-based change, but without the use of evidence-based approaches to create intervention packages and appropriate management of interventions, their impact may fall short ([11, 18, 22, 25].
In Latin America, the need for health intervention packages is particularly pressing in the area of adolescent sexual and reproductive health (SRH) as adolescent birth rates are among highest in the world . The adolescent birth rate has remained persistently elevated, declining only from 77 to 72 per 1000 women of 15-19 years of age between 1990 and 2005, while access to contraceptive methods and comprehensive sexuality education remains restricted among adolescents . Bolivia, Ecuador and Nicaragua face particularly high adolescent birth rates and associated poor health and psychosocial outcomes. In Bolivia, the adolescent birth rate is 89.1 per 1000 women (2005), Ecuador 99.6 (2002) and Nicaragua 92 (2010), thus are settings warranting special consideration for complex interventions .
A recent initiative titled Community-Embedded Reproductive health Care for Adolescents (CERCA) Project (2010-2014) aimed to reduce adolescent pregnancy rates and improve SRH of adolescents through a comprehensive strategy to modify health behaviors related to communication, SRH information-seeking, SRH care-seeking, and promotion of safe sexual relationships. To achieve these changes, CERCA implemented a strategy involving multiple stakeholders such as adolescents, parents, teachers, health personnel, health and education authorities, community and local authorities in three settings: Cochabamba (Bolivia), Cuenca (Ecuador) and Managua (Nicaragua).
CERCA was implemented from March 2010 and concluded in March 2014. The key institutions leading the implementation were The International Center of Reproductive Health-Ghent University in Belgium as the general project coordinator) and South Group in Bolivia, University of Cuenca in Ecuador, CIES and ICAS in Nicaragua, Kaunas University in Lithuania, and the University of Amsterdam in The Netherlands as the country implementers [4, 8]. The quantitative evaluation of CERCA, using a pre- and post-test controlled design, did not demonstrate effects after twenty months of intervention in the following key areas: reinforced behaviour, knowledge, attitudes, skills, and self-reflection. This lack of measured success was primarily attributed to challenges in data collection described in the CERCA report  amid others. A qualitative post hoc impact evaluation, supported by the World Health Organization, was carried out from June 2014 until March 2015 and aimed to expand upon the quantitative evaluation of the CERCA project (Crescendo ). This qualitative analysis revealed that a concerted application of a multi-faceted strategy reduced taboo in a conservative setting, sensitized the community and authorities, enabled the environment for adolescent interventions, and that subsequent, multi-country projects warrant a more thorough preparatory phase .
In the past, many health projects have been implemented without a clear idea of whether they did, in fact, achieve the expected results due to incomplete monitoring or inadequate evaluation phase . However, more recently, a number of bilateral agencies and United Nations’ programs have adopted the Results Based Management (RBM) framework to demonstrate aid effectiveness and accountability from project planning to completion [1, 26]. We aimed to understand whether an RBM framework might have helped CERCA produce evidence of tangible and positive results by allowing country implementers to more effectively address challenges in design, implementation, monitoring, and evaluation.
The objectives of this paper are to apply the RBM framework retrospectively to the CERCA project in order to reflect on lessons learned and provide recommendations for future similar multi-country projects with complex, community-integrated interventions. This will be conducted through the application and analysis of three main components of RBM framework: 1) Planning, 2) Monitoring and 3) Impact evaluation. Findings may offer project managers, implementers, evaluators, and funding agencies valuable information regarding adolescent SRH intervention packages and improve effectiveness in management, accountability, and creation of evidence in subsequent SRH projects.
RBM is a system designed to support efficiency and effectiveness at different stages of the project cycle starting from planning, designing the intervention package, monitoring, and supporting impact evaluation. RBM uses feedback loops at each of these levels. In complex interventions, this framework helps ensure that the project implements a package of interventions that is necessary and sufficient to achieve the expected result [29, 30]. Its goal is to enhance effectiveness and provide avenues to verify if the project is progressing appropriately through three phases: planning, monitoring and evaluation. Planning relies on a Theory of Change (ToC) framework which is the basis for the intervention package (IP). The ToC explains how a project will achieve a series of outcomes, by outlining causal relations based on assumptions to reach behavior change . It involves choosing strategic interventions that are necessary and sufficient to reach the expected outcome [29, 30]. Monitoring tracks the process of change and how to ascertain success, tracking the implementation progress against the expected results of ToC . Quantitative and qualitative evidence is collected for defined indicators to help determine the expected results from the intervention. Impact evaluation compares the control and intervention groups to assess the degree to which the impact can be attributable to the project after controlling for all external variables. Impact evaluation allows for decisions about whether project activities should be continued, replicated, scaled-up and improves the overall understanding about why certain changes have happened ([12, 21]. RBM is versatile enough to accommodate other research strategies within its qualitative arm such as community-based participatory research (CBPR). CBPR strategies foster empowerment in the community with the creation of knowledge as well as engagement, capacity building and quality assurance .
RBM can be used both retrospectively and prospectively to sharpen the strategic focus of the project. Organizations that use it for their project management are the Swiss Agency for Development and Cooperation and the United Nations Population Fund [24, 28]. In order to perform a retrospective analysis, we reviewed and analyzed a series of CERCA documents and other data sources (detailed in Appendix 1) structured by applying the RBM framework in order to see in what way the CERCA project performance might have reaped improved results. This included CERCA methodology design, final CERCA report 2014 (Crescendo ), internet report , post-hoc evaluation process report 2015 , international published papers [3, 4, 6,7,8, 13, 14, 17] and the A-form for results. Records and discussion notes from two meetings (recorded without transcript) were also reviewed: 3-day meeting in Cuenca-Ecuador (February, 2014), and 3-day meeting in Ghent-Belgium with country project leaders, researchers from Ghent and the study advisor from WHO-HRP (December, 2014).
Findings related to planning and a theory of change (ToC) framework were divided into three key elements: 1) outputs, intermediate- and long-term outcomes, 2) identification of the preconditions to create change and of articulated assumptions, and 3) the intervention package, which contains a sequential pathway of change with indicators to assess the performance.
Outputs, intermediate- and long-term outcomes: The CERCA project proposal included interventions targeting multiple outcomes, including improved access to quality sexual and reproductive services, a more supportive and enabling environment, and strengthened adolescent competence to make reproductive health choices (Fig. 1). The post-hoc evaluation revealed outcomes during the process of implementation, and a new ToC was developed based on the CERCA documents (Fig. 2) , more accurately reflecting what actually occurred in project implementation. This retrospective ToC was: a) flexible and could adapt to the needs of every country but included different outcomes than the ones originally defined in the proposal; b) did not display a causal analysis on how the activities of intervention would bring about change in a logical and chronological manner.
Identification of the preconditions to create change and of the articulated assumptions: Our review revealed that identification of preconditions and articulated assumptions did not occur due to construction of the ToC with different actors of the community. These community actors were involved from the beginning of the project until the end in order to promote credibility and buy-in with every activity. The ToC changed as activities were added and modified over time based on Community Based participatory research (CBPR) insights. This CBPR approach involved countries having exclusive meetings with every group of stakeholders without a common stakeholder meeting. This lead to the modification of country-specific interventions without adherence to the original ToC.
Although CERCA carried out two extensive data collections at the beginning of the project, the results were not used to identify articulated assumptions or preconditions to create change. One was a cross-sectional study of around 9000 adolescents aged 14-18 years old. The second was a qualitative study among health care providers. Due to the project size, there was insufficient time reserved for its analysis in depth; thus, this was deferred to the end of the project, by 2013  and at the end of 2014  consecutively.
Intervention Package (IP). The IP was constructed by stakeholders using a CBPR approach but without a realistic timeframe to achieve the outputs and outcomes with each activity. Further, there is evidence that effective interventions were chosen based on literature review. There was no comprehensive guideline on how to design, implement or monitor the intervention activities (being the only ones found: SMS multi-country guideline and Guideline for SRH workshops in Bolivia). Each country had the flexibility to develop its own interventions and to determine the frequency of each activity based on the needs of target groups, CBPR insights, experience and disciplinary strengths of the local teams with different backgrounds .The IP was different in all three countries and not all activities were linked to the indicators of the intervention; hence, conditions to produce behavioral change were not created .
Table 1 and Appendix 2 demonstrates the variability of intervention activities in more detail. Table 1 demonstrates that: 1) each country had a different frequency of activities in every site; 2) interventions did not have a common pattern of activities in terms of coverage; 3) there are some IP features that could not be elicited with the reporting documents such as the duration of each activity or the how it was delivered. Appendix 2 demonstrates the finding that most activities focused in improving access to SRH information with less attention to the other three objectives of intervention: improved comfort discussing SRH; increased use of health services; and improved contraceptive use. Further, it shows that in Bolivia and Ecuador, the greatest emphasis was placed on workshops (in Bolivia 67% and Ecuador 81% of the total activity resources in the country were dedicated to workshops). Workshops contained the 4 objectives of intervention while in Nicaragua, 2 activities covered all the 4 objectives with 64% of the total activity resources. The percentage of coverage was under 100% in most of the activities, but in one case, coverage extended beyond to the target population by 639%.
Findings related to monitoring were organized into three key elements  : 1) planning, 2) evaluation on how and to what extent the objectives are reached, 3) identification of failures to produce outputs.
Planning. We found no plan within the original CERCA documents to validate a monitoring approach. Furthermore, there was no pilot period to test whether the project was feasible or to check if the collected data was adequate to track change. Skill evaluations of the implementers following the intervention design were not found.
We learned that, due to a conservative society with taboos and machismo , country partners decided to first focus on activities to break down taboos and cultural gender issues, followed by activities to transmit awareness and sensitization about SRH. Although these steps were deemed necessary, they did not correspond with the ToC plan; thus, these activities were not monitored but still affected the timeframe of other planned activities (Ghent meeting).
Evaluation of objectives. The monitoring system envisioned by CERCA was mainly based on administrative and financial requirements (control per activity and trimester), rather than monitoring to track the level of fidelity to the intervention model, quality of service delivery or degree of behavioral change and community perception on SRH.
In addition, the coordination office prepared a multi-country monitoring template that gave priority to certain objectives of the ToC over others. For example, when reporting activities to authorities, the template tracked number of workshops, TV/radio presentations, meetings and people reached but not degree of sensitization or improvement of communication skills on SRH. Data collected for monitoring was not used for technical feedback (e.g. objective-outcome-correct path to reach change) that may have led to corrective actions. For instance, data was collected over 3 years at health care facilities to track the number of adolescents that visited the health care facility before, during and after the project but this data was never used to analyze if access to SRH services was improving. A project member in Ecuador suggested that both limited skills of some team members and inadequate monitoring templates contributed to evaluation challenges.
“…Not all members of the project knew how to monitor activities. There was a format to check options and that is what they did but did not register what was really done, did not tell the whole process of the activity…”
Identification of failures. In the absence of a monitoring plan, failures were not systematically identified; therefore, they were not reported. The 3-day meeting in Ghent revealed that when unexpected events occurred during interventions and corrective measures were taken, there was no report to account for changes that took place (Table 1 and Appendix 2). For instance, our analysis revealed that peer-to-peer activities (FOY) in Nicaragua reported the number of visits to adolescents per trimester, but did not report how many times each FOY visited each adolescent nor was there data on quality of interaction with each adolescent to evaluate its content and effectiveness for behavior change. We also found that open-air activities reported the number of participants but it did not detail if the persons stayed throughout the whole activity (to measure exposure to the intervention or implementation challenges) or if people returned to the next activity as monitoring only counted the participants. A project member in Nicaragua proposed that there was resistance to monitoring forms (Table 2).
“…Health promoters saw monitoring forms as something bad, as a way to control them, so they were filled as an obligation…”
Our findings related to Impact Evaluation (IE) were also organized into three key elements, according to the RBM framework [21, 26]: 1) identification of success and failure factors to produce outcomes, 2) outcomes being positive, negative, anticipated or unanticipated, 3) cost-effectiveness, specifically the evaluation of the administrative and financial resources necessary to carry out the interventions.
Identification of success and failure factors to produce outcomes: The CERCA Consortium held monthly meetings designed to report out successful activities and key challenges. These meetings led to the addition or removal of some activities without corresponding adjustments to the monitoring plan or consideration of project framework. These changes varied by country site, but a comprehensive record of why changes were made was not maintained (e.g. Table 1 and Appendix 2). Many difficulties related to the project were revealed from the qualitative evaluation after the project had ended. In the qualitative evaluation, it was noted that qualitative findings were not integrated into monthly meetings or the final program evaluation. Rather, the qualitative research was conducted separately, preventing responsive changes in the intervention based on its findings.
Outcomes assessed as positive, negative, anticipated or unanticipated. Not all outcomes were assessed within an adequate time frame to allow for change. For example, reducing the pregnancy rate in adolescents was established as a main outcome (Fig. 2). However, based on existing frameworks of pregnancy prevention interventions, 20 months may be too short of time frame to measure impact . Some positive outcomes were not anticipated, such as a reduction of taboo in a conservative setting through concerted application of multi-faceted strategies, greater sensitization of community to an enabling environment . This also promoted the expansion of SRH networks in Ecuador and Bolivia . CERCA tested and evaluated SMS, phone, and social media to improve communication between health provider and adolescents and within networks of professionals and authorities by using social media, newsletters and websites [3, 23]. These were found to be effective strategies to foster open communication about SRH despite very conservative cultural norms . Among negative outcomes were that parents did not participate in activities in as high numbers as expected or intervention emails had less than 10% opening rate by participants. Some outcomes were only evaluated at the end of the project using data from a self-administered survey. Survey results revealed challenges in the collection and interpretation of the data (attrition, type of questions, cohort coding); and in the implementation of some interventions [(low rates of follow up, high rotation of personnel (in the case of the friends of youth (FOY)] that may have led to unexpected results
Cost-effectiveness: evaluation of the administrative and financial resources required to provide the interventions. There was no plan to assess cost-effectiveness in every intervention or as one multi-component intervention. FOY was considered one of the most heavily promoted interventions of CERCA in Nicaragua but there is no cost-effectiveness evaluation to support this. Personnel of FOY worked for minimal reimbursement in the project, but there is no evaluation if this action would be feasible to scale-up at national level. Further Nicaragua experienced high turnover of personnel (6 months in average) suggesting that staff would likely not stay permanently in a volunteer position, which would hinder its scalability. The aspect that was evaluated for cost-effectiveness was the SMS intervention. The published evaluation reported cost per message, but did not include the cost of the personnel involved or the infrastructure with the software needed (internet and software, type of phone, hours of the personnel) . This limited analysis creates difficulty assessing its feasibility for continuity or scalability at national level.
Based on the results of CERCA, a summary of the success and failure factors are summarized in Table 3.
Discussion and lessons learned
The CERCA programme relied upon three methodological frameworks: action research, community based participatory research and intervention mapping but did not adopt any project management framework. When the project ended, CERCA had not met all of its goals. Using the RBM framework, we expand the CERCA insights to demonstrate the importance of appropriate timeframes and cause-effect analyses while planning activities that can originate both from the community and from review of effective, evidenced-based interventions. Testing the activities for feasibility and acceptability in a pilot can enhance a project’s chances of success. Designing a clear monitoring plan for each activity, reporting failures during the process, and having constant feedback meetings on the advancement process against the ToC is paramount. Evaluation should assess cost-effectiveness of every intervention, success and failure factors to produce outputs and outcomes, and a breakdown analysis to measure the progress at the end of the project so that future initiatives can benefit of this information. This analysis conveys in a summary of lessons learned (Table 4).
Multi-component, community-integrated health intervention packages are necessary to address complex social problems such as adolescent pregnancy . This analysis applied an RBM framework retrospectively in order to learn from a multi-country project to improve adolescent sexual and reproductive health. The analysis using the RBM framework highlights important findings of the project. In Planning, the proposed Theory of Change (ToC) included a range of stakeholders who did not consistently plan activities based on evidence, situation analysis, timeframe or a sequential path of change. Intervention packages changed according to different circumstances and did not always take into account guidelines or quality of delivery, leading to different results across sites which were difficult to compare. Some planned activities did not take into account the possibility to be continued, reproduced or scaled-up. We identified missed opportunities to select evidence-based interventions with appropriate linkages to a ToC that would ensure the intended outcomes were reached. In Monitoring, we also highlighted gaps due to time, resources, and variability across sites. The project oversight was mainly based on administrative and financial requirements rather than monitoring fidelity or challenges to carry out the intervention model regarding content, behavior change, quality of delivery and community perception. In Impact Evaluation, the project did not plan a comprehensive evaluation, did not dedicate sufficient time and budget. Limited data was available due to inadequate record keeping; the original evaluation did not identify success and failure factors related to the outcomes, and the cost-effectiveness analysis was not inclusive of all activities and costs (e.g. analysis did not account for training time and costs, challenges to monitoring their activities and their impact on every adolescent from the intervention group). Examining the project within an RBM framework generates valuable insight for future adolescent health initiatives.
Research is growing in the area of project accountability and assessing validity of evidence from complex interventions Our analysis with RBM is an effective accountability mechanism that controls at all levels and can strengthen management of large-scale projects that include cross-sectoral strategies. The use of RBM from the start to the end of the project encourages managers to make decisions with the results in mind and to plan all aspects of the project in advance with detail, essential for complex interventions . Over the years, many NGOs and other institutions implementing health projects in the field of SRH have focused more on their respective donors’ needs and not on how the project contributes to a broader social change . With the implementation of RBM in international institutions, donors are encouraged to support implementation as well as data collection, evaluation and sharing of information gathered. In the case of CERCA, project leaders had the opportunity to evaluate various factors of success and failure under the supervision of an advisor from the WHO-HRP who oriented these talks and helped understand the lessons learned (meeting in Ghent, 2014); but not all projects have this opportunity as most do not even have budget and time planned for this.
The RBM framework analysis has some limitations. Firstly, this is a retrospective assessment and CERCA did not use it in any phase. Secondly, our analysis relied upon the availability and quality of records kept during planning, implementation and evaluation (Appendix 1). Third, the authors of this paper were the project managers or project implementers in every country; therefore, this may introduce bias into the analysis. Finally, this analysis was conducted retrospectively after the qualitative evaluation and may be influenced by these findings.
This analysis demonstrates the delicate balance between planning comparable multi-country interventions and allowing each country to develop its own approach per their specific needs. A thoughtful project design incorporates a situation analysis of the unique needs of each site together with scientific evidence on what interventions have been shown to work and how they might work in each context. More research is needed in this area to evaluate the feasibility and effectiveness of complex intervention packages.
This analysis showed that multi-country projects with cross-sectoral strategies are complex, entail risks in execution and require rigorous project management. Large-scale, multi-country programmes have the opportunity to create broad, population-based change, but without the use of evidence-based approaches and appropriate management of interventions, their impact may fall short. RBM can be an effective accountability mechanism to ensure a systematic approach and appropriate use of resources at different levels within a multi-country setting. RBM can offer country implementers and managers an avenue to apply systematic management at all levels, improve accountability, promote the development of evidence base and strengthen commitment to adolescent health in future initiatives.
Community-based participatory research
Community-embedded reproductive health care for adolescents
Information and communications technology
Results Based Management
Sexual and reproductive health
Sexually transmitted infections / Human Immunodeficiency Virus / Acquired Immunodeficiency Syndrome
Theory of Change
The World Health Organization’s Department of Reproductive Health and Research
Bester A. Results-based Management in the United Nations Development System: progress and challenges. In: A report prepared for the United Nations Department of economic and social affairs, for the quadrennial comprehensive policy review. New York: United Nations; 2012.
Busjeet G. Planning, monitoring, and evaluation : methods and tools for poverty and inequality reduction programs. Washington, DC: World Bank; 2013.
Cordova, K. & Hagens, A., 2013. Information and Communications Technology in Sexual and reproductive health care for Adolescents - A Bolivian Case Study. GSTF, pp.112-117. ISSN: 2251-3833. https://doi.org/10.5176/2251-3833_GHC13.59.
Cordova-Pozo K, et al. Improving adolescent sexual and reproductive health in Latin America: reflections from an international congress. Reprod Health. 2015;12(12):11.
Crescendo Stat-Gent. The CERCA-project: community embedded reproductive health interventions for adolescents in Latin America. Ghent: ICRH, Ghent University; 2014.
de Meyer S, et al. A cross-sectional study on attitudes toward gender equality, sexual behavior, positive sexual experiences, and communication about sex among sexually active and non-sexually active adolescents in Bolivia and Ecuador. Glob Health Action. 2014;7:24089. https://doi.org/10.3402/gha.v7.24089.
Decat, P. et al., 2015. Sexual onset and contraceptive use among adolescents from poor neighbourhoods in Managua, Nicaragua. Eur J Contracept Reprod Health Care., Mar 4; 20(2)(doi: https://doi.org/10.3109/13625187.2014.955846), p. 88–100.
Decat P, et al. Community embedded reproductive health interventions for adolescents in Latin America: development and evaluation of a complex multi-centre intervention. BMC Public Health, Issue 2013 13:31. 2013;13:31.
Denno, D., Hoopes, A. & Chandra-Mouli, V., 2014. Effective strategies to provide adolescent sexual and reproductive health services and to increase demand and community support.
Echeverri, M. What we should know about Community-Based Participatory Research. 2010. [Accessed 2017 Abr 10]. Available from: https://lacats.pbrc.edu/documents/About%20CBPR%2010-08-13.pdf.
Hainsworth G, Simon D, Rahimtoola M, Ghiron L. Scale-up of adolescent contraceptive services: lessons from a 5-country comparative analysis. J Acquir Immune Defic Syndr. 2014;66(Supplement 2):S200.
HIVOS, 2012. How can ToC thinking help in monitoring and evaluation?. [Online] Available at: http://www.hivos.net/Hivos-Knowledge-Programme/Themes/Theory-of-Change/Resources/4.-How-can-ToC-thinking-help-in-monitoring-and-evaluation [Accessed 21 July 2015].
Ivanova O, et al. Lessons learnt from the CERCA project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation. Eval Program Plann. 2016;58:98–105. (DOI: 10.1016/j.evalprogplan.2016.06.007), pp. 98-105
Jaruseviciene L, et al. Primary healthcare providers’ views on improving sexual and reproductive healthcare for adolescents in Bolivia, Ecuador, and Nicaragua. Glob Health Action. 2013;6:20444. https://doi.org/10.3402/gha.v6i0.20444.
Leeuw, F., & Donaldson S., 2015. Theory in evaluation: Reducing confusion and encouraging debate. Sage Journals. Vol. 21, issue 4, pp. 467-480. https://doi.org/10.1177/1356389015607712.
Michielsen K, et al. Effectiveness of complex interventions: post hoc examination of the design, implementation and evaluation of the CERCA project to understand the results it achieved. Ghent: ICRH, Ghent University; 2015.
Nelson E, et al. The unintended consequences of sex education: an ethnography of a development intervention in Latin America. Anthropology & Medicine. 2014;21(2):189–201. https://doi.org/10.1080/13648470.2014.918932.
Njeuhmeli, E., Hatzold, K. & Gold, E., et al., 2014. Lessons Learned from Scale-Up of Voluntary Medical Male Circumcision Focusing on Adolescents: Benefits, Challenges, and Potential Opportunities for Linkages with Adolescent HIV, Sexual, and Reproductive Health Services. J Acquir Immune Defic Syndr. 2014 Jul 1;Vol. 66 Suppl 2. Pp.193-199. https://doi.org/10.1097/QAI.0000000000000179.
Oringanje, C., Meremikwu, M., Eko, H. & Esu, E., 2016. Interventions for preventing unintended pregnancy among adolescents. Cochrane Fertility Regulation Group, Feb 3;2:CD005215. https://doi.org/10.1002/14651858.CD005215.pub3.
Rahmani, R., 2012. Donors, beneficiaries, or NGOs: whose needs come first? A dilemma in Afghanistan. Journal Development in Practice, Vol. 22, Iss. 3,2012. Pp.295-304. https://doi.org/10.1080/09614524.2012.664622.
Rogers P. Overview of impact evaluation, methodological briefs: impact evaluation 1. Florence: UNICEF Office of Research; 2014.
Schunter, B., Cheng, W., Kendall, M. & Marais, H., 2014. Lessons Learned From a Review of Interventions for Adolescent and Young Key Populations in Asia Pacific and Opportunities for Programming. JAIDS Journal of Acquired Immune Deficiency Syndromes: July 1st, 2014. Vol.66, Issue 2, pp.186–192. https://doi.org/10.1097/QAI.0000000000000185.
South Group. Assessment report on internet for awareness raising activities. Ghent: CERCA Project; 2014.
Swiss Agency for Development and Cooperation. Planning and monitoring in results-based Management of Projects and Programmes. Zurich: Nadel & ETH; 2011.
Tsu, V., Cernuschi, T. & LaMontagne, D., 2014. Lessons learned from HPV vaccine delivery in low-resource settings and opportunities for HIV prevention, treatment, and care among adolescents. J Acquir Immune Defic Syndr. 2014 Jul 1;66 Suppl 2. pp.209-216. https://doi.org/10.1097/QAI.0000000000000175.
UN Women. How to manage gender-responsive evaluation: evaluation handbook. New York: UN-Women; 2015.
UNDP. Handbook on planning, monitoring and evaluating for development results. New York: UNDP; 2009.
UNFPA. Multilateral organisation performance assessment network (MOPAN-2014) - technical report -volume I: results by micro-indicator and by country. New York: UNFPA; 2014.
UNITAR. Monitoring and evaluation policy framework. Geneva: United Nations Institute for Training and Research; 2012.
United Nations Development Group. Results-based management handbook - harmonizing RBM concepts and approaches for improved development results at country level. Nueva York: United Nations Development Group; 2011.
United Nations, 2014. The Millennium Development Goals Report, New York: United Nations, DPI/2590 E.
UN-Women, 2015. Programming Essentials, Monitoring & Evaluation. 2015. [Accessed 3 Aug 2015]. Available at: http://www.endvawnow.org/en/articles/341-health.html.
This study is supported by the Department of Reproductive Health and Research from the World Health Organization. The authors would like to thank the CERCA Project and all its members in the partner institutions in ICRH- Ghent University (Belgium), South Group (Bolivia), Universidad de Cuenca (Ecuador), Kaunas Medical University (Lithuania), CIES-UNAN and ICAS (Nicaragua) and the University of Amsterdam (The Netherlands) for providing all information needed for this research. Authors would like to thank the special contributions that Olena Ivanova and Kristien Michielsen from ICRH-Ghent University did to the elaboration of this paper.
This paper emanated from the post hoc evaluation, which was supported by WHO-HRP.
Availability of data and materials
The datasets used and analyzed during the current study available from the corresponding author on reasonable request.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Cordova-Pozo, K., Hoopes, A.J., Cordova, F. et al. Applying the results based management framework to the CERCA multi-component project in adolescent sexual and reproductive health: a retrospective analysis. Reprod Health 15, 24 (2018) doi:10.1186/s12978-018-0461-3
- Results based management
- Community based interventions
- Latin America
- Sexual and reproductive health
- Theory of change
- Implementation research