Data quality in budget support: weaknesses in some indicators and in the verification of the payment for variable tranches

Executive summary

Budget support is a form of EU aid which involves transferring money to the national treasury of a partner country, subject to that country’s compliance with agreed conditions for payment. With an average of €1.69 billion in annual payments, the EU is the largest provider of budget support. Budget support payments are made in the form of either fixed or variable tranches. Approximately 44 % of EU budget support payments relate to variable tranches. This proportion is as high as 90 % in some EU neighbourhood countries. The amounts paid from variable tranches depend on the performance achieved by partner countries, which is measured by pre-defined performance indicators.

We examined whether the Commission used relevant and reliable performance data for disbursing budget support variable tranches. We conclude that for one third of the performance indicators reviewed their relevance was weakened by their design, and they do not allow results to be measured objectively, thereby undermining their relevance. Furthermore, the Commission’s assessment of whether variable tranche indicators had been met was not always reliable, leading to some insufficiently justified payments.

While the variable tranche performance indicators were aligned to the partner countries’ sector development strategies, most of them were focused on short-term actions rather than longer-term results, including progress over time towards the sustainable development goals. Furthermore, more than one third of the indicators were vaguely defined or had incorrect baselines, or none at all. This allowed for different interpretations as to whether targets had been achieved, making the analysis of the disbursement requests more complex and less objective.

Most variable tranches reviewed provided the intended effect of incentivising partner countries to advance in their reform agendas by having sufficiently ambitious targets. The few exceptions we found concerned indicators with targets which were easily achievable, or which had been achieved via the work carried out by other donors or by external experts paid by the EU. The number of indicators used in each variable tranche was too high in 6 out of the 24 contracts, which further complicated the disbursement process.

The disbursement requests for variable tranches contain an analysis on the fulfilment of agreed conditions and performance indicators. These requests are prepared by the partner countries and thus, the reliability of the underlying performance data depends on the capacity of these countries to produce it. We found that in only 5 of the 24 selected contracts did the Commission draw explicit conclusions as to whether the performance data needed to calculate the variable tranche indicators was reliable. When analysing disbursement requests, the EU Delegations carried out a variety of procedures to verify the reliability of this data. Some of these procedures do not ensure that variable tranche payments are based on reliable data and thus, are not fully justified.

Based on our re-performance of the Commission’s assessments regarding the achievement of indicators and the recalculation of the variable tranche payments for a total of €234 million, we found discrepancies for €16.7 million. From this amount, €13.3 million were insufficiently justified, or not in line with contract provisions. An amount of €3.4 million was paid without actual progress. Additionally, €26.3 million for three variable tranches were paid to Moldova without sufficiently documenting the reasons supporting these payments.

We make six recommendations to the Commission in order to:

• Increase the use of outcome indicators in variable tranches;
• Improve the formulation of performance indicators;
• Safeguard the incentive effect of variable tranches;
• Simplify the disbursement process for variable tranches;
• Improve the assessments of the countries’ capacity to provide performance data used in variable tranches;
• Improve the verification of the performance data used to disburse variable tranches.

Introduction

The concept of budget support

Budget support is a form of EU aid which involves transferring money to the national treasury of a partner country, subject to that country’s compliance with agreed conditions for payment. The funding thus received enters the partner country’s budget, and can be used as the partner country sees fit. The Commission describes budget support as a means of delivering effective aid and durable results in support of EU partners’ reform efforts and the sustainable development goals (SDGs).

In addition to the transfer of financial resources, budget support involves: (i) a dialogue with the partner country on reforms or development results, which budget support can assist, (ii) an assessment of progress achieved, and (iii) capacity development support. Budget support represents a shift away from the traditional focus on activities (e.g. projects) towards results-oriented aid.

The EU (both through the EU budget and the European Development Funds) is the largest provider of budget support globally. During the 2014-2017 period, the EU committed around 11 % (see Annex I of its bilateral development aid budget on budget support: an annual average of around €2.13 billion). In 2017, it provided budget support to 90 countries and territories, which received a total of €1.8 billion. Across all ongoing budget support contracts, the total committed amount is €12.7 billion. Figure 1 shows a breakdown of these figures by region both for the EU general budget and for the European Development Funds.

(in million euros)

Source: ECA, based on EC Budget Support Trends & Results 2018.

EU budget support requires that relevant and credible policies are in place and implemented effectively by the recipient country. According to the Financial Regulation, a country may be considered eligible for budget support when¹:

• The partner country’s management of public finances is sufficiently transparent, reliable and effective;
• The partner country has put in place sufficiently credible and relevant sectoral or national policies;
• The partner country has put in place stability-oriented macroeconomic policies;
• The partner country has put in place sufficient and timely access to comprehensive and sound budgetary information.

The European Commission uses three types of budget support contracts.

1. The Sustainable Development Goals contract (SDGC), which supports national policies and strategies in achieving progress towards the sustainable development goals.
2. The Sector Reform Performance contract (SRPC), which supports specific sectoral reforms.
3. The State and Resilience Building contract (SRBC), which is provided for countries in fragile situations to ensure vital state functions or to support processes towards democratic governance.

The vast majority of the European Commission’s budget support programmes are based on SRPCs (74 % of all ongoing budget support commitments in 2017). In terms of funding, the four largest sectors supported using SRPCs are: education, agriculture and rural development, health, and energy.

Variable tranches as an incentive to achieve results

Prior to each disbursement, the Commission analyses the fulfilment of general conditions attached to the budget support contract. These conditions are in most cases those related to the eligibility criteria for receiving budget support (see paragraph 04). Budget support payments are made in the form of either fixed or variable tranches. Fixed tranches are paid either in full (if all conditions are met) or not at all (if one or more conditions are not met). Variable tranches are used to create incentives for partner countries to improve policy delivery, and are paid based on performance achieved in relation to specified performance indicators and targets, if the general conditions are all met. They can be paid either in full or in part. The performance indicators used for variable tranches can be selected from among the monitoring systems already used by the partner country, or through a common performance assessment framework agreed with the partner country and other donors.

Different types of performance indicators can be used for public policies (see Box 1). In general, the Commission recommends the use of outcome indicators, but other types of indicators can also be appropriate depending on the specific context of the partner country or sector.

Each performance indicator has an associated financial value. One way the Commission calculates the amounts disbursed in the variable tranche is by adding up the amount associated with each performance indicator met by the country. This means that the more indicators a partner country meets, the higher the proportion of the variable tranche paid.

In 2017, 44 % of the Commission’s payments for budget support contracts related to variable tranches². This proportion is as high as 90 % in some EU neighbourhood countries. Figure 2 shows a breakdown by country of the variable tranches paid for SRPCs in 2017.

Source: ECA, based on EC data.

The disbursement of variable tranches

The process for disbursing a variable tranche starts with a disbursement request by the partner country. This request includes an analysis of the extent to which the related performance indicators have been met. The EU Delegations analyse the requests and prepare an evaluation note. Based on this note, and after the approval of the budget support steering committee, the Commission decides the amount of the variable tranche that should be paid (see Figure 3).

The partner country’s performance analysis accompanying a disbursement request is based on data from its own monitoring and evaluation (M&E) systems. These systems are therefore the primary source of information for the disbursement of variable tranches.

Source: ECA, based on EC Budget Support guidelines.

Audit scope and approach

The EU Consensus on Development adopted in 2017 recognises the central place of budget support in fostering partner countries’ efforts towards achieving the sustainable development goals (SDGs). Considering the importance of variable tranches in budget support payments and the fact that the performance data on which those payments are based originate from the recipient countries, incorrect or unreliable data would imply that budget support payments do not reward actual results as intended. This would seriously weaken the very purpose of budget support. The focus of this audit was therefore to scrutinise the quantity and quality (namely relevance and reliability) of the performance indicators used in variable tranches and the Commission’s analysis of payment requests.

The main audit question was: “Did the Commission use relevant and reliable performance data for disbursing budget support variable tranches?” In order to answer this question, we broke it down into the following sub-questions.

1. Did the budget support contracts include indicators that allow relevant results to be monitored effectively in the supported sectors?
2. Did the Commission effectively verify the reliability of performance data in the requests for disbursements of variable tranches?

The audit covered a sample of 24 sector reform performance contracts (see Annex II to the eight partner countries with the largest disbursements of variable tranches paid in 2017³. These countries received 43.29 % of the total variable tranche payments for that year. The variable tranches audited for the selected contracts involved a total of 248 performance indicators. Annex III details the payments made in 2017 and the sectors audited.

For the selected contracts, we examined the Commission’s assessment of the capacity of the countries’ systems to provide reliable performance data. We also checked the design of the budget support contract provisions, in particular the quality of the performance indicators used in the variable tranches. Finally, we examined the Commission’s verifications of the data in the variable tranche disbursement requests. When examining these verifications, we carried out a documentary review of the payment files prepared by the Commission and re-assessed the achievement of targets and the calculation of the variable tranche payments. We then compared our conclusions with those of the Commission.

The audit included visits to three countries: Jordan, Georgia and Bolivia. During these visits, carried out in February and March 2019, we interviewed Commission staff, representatives of national authorities and other donors/stakeholders. In addition to the procedures carried out for the rest of the countries selected, in these three countries we re-performed the Commission’s verifications of the variable tranche disbursement requests and the related payments and compared our results with the results of the analysis done by the Commission. Furthermore, we cross-checked the performance data declared by the partner countries with other sources of evidence from external experts and other donors to assess its reliability.

Observations

Relevance of variable tranche indicators weakened by their design

We examined whether: (i) the performance indicators are relevant for achieving the budget support contracts’ objectives and are aligned with country policies; and (ii) they provide a sound basis for monitoring meaningful results.

Indicators are consistent with country sector strategies, but mostly focus on short-term actions rather than longer-term results

Budget support is regarded by the Commission as a results-based means of delivering aid. Whereas project-based aid is disbursed for eligible expenditure, budget support variable tranches are disbursed when partner countries meet the general conditions and achieve previously agreed results, measured by selected performance indicators. Therefore, the relevance of variable tranches depends on whether these performance indicators measure meaningful results.

As presented in Box 1, there are five types of indicators. Input, process and output indicators are most relevant for the day-to-day management of public spending programmes. Outcome and impact indicators measure longer-term effects, such as the progress of reforms and programmes towards set objectives; for example, the SDGs. Figure 4 includes an example of different types of indicators used for one of the budget support programmes in Bolivia.

Source: OECD Development Assistance Committee result chain, used in EU external actions.

The budget support guidelines allow the use of any of the above type of indicator. According to the Commission, though, preference should be given to outcome indicators, because they encourage evidence-based policy making, protect political space for partner countries to choose their own policies and strategies for achieving them, and stimulate a demand for high-quality statistical information. The greater the confidence in the partner government’s ability to produce reliable data, the more emphasis should be placed on outcome indicators. We also consider impact indicators useful for measuring results, in particular the achievement of the SDGs.

Our analysis of 248 indicators in the audited contracts shows that they are aligned with the sector strategies of the partner countries. However, variable tranche disbursements are mainly based on input, process and output indicators, which together account for 87 % of the total number of indicators. Only 33 indicators (13 %) measured outcome and impact – see Figure 5.

Source: ECA.

We found that most outcome and impact indicators were used in countries with lower and lower-middle income economies, while in upper-middle income countries, such as Jordan and Georgia, which had a better capacity to produce performance data, only four outcome indicators and no impact indicators were used. This shows that indicator selection is not necessarily linked to countries’ development status.

The input indicators used in the audited variable tranches related mainly to the procurement of goods (18 cases) and increases in budgetary allocations (9 cases). Increases in budgets or the procurement of equipment might have the potential to trigger changes in a particular area of interest, but they do not automatically mean that meaningful progress will be achieved.

The process indicators may play a useful role, in particular when the policy is aimed at achieving changes in the regulatory framework. However, as indicated in the Guidelines, these should not focus only on processes, but should also measure qualitative aspects, e.g. what an entity to be set up is expected to cover, etc.⁴ However, this was not always the case. Box 2 presents some examples of indicators, which did not specify any minimum quality requirements regarding the content or the structure of the information to be submitted. This makes it difficult to ensure that variable tranches reward good-quality processes.

One third of indicators do not allow results to be measured objectively

The indicators used to measure progress in the implementation of a public policy need to be specific, have quantified and measurable targets, and when applicable, a reliable baseline. Otherwise, it is not possible to objectively measure progress achieved. Indicators, baselines and targets must be agreed when budget support contracts are formulated and specified in the financing agreement. Although the Guidelines state that changes to the indicators should be avoided, targets might need to be amended during contract implementation to reflect new circumstances, or to correct errors in the definition of baselines or the calculation of indicators.

Our analysis shows that 72 indicators (i.e. 29 % of our sample) were not sufficiently specific. Non-specific indicators are especially problematic during the process of analysing disbursement requests: there is a risk of conflicting judgements as to whether targets have been achieved. This can in turn lead to a variety of different results being arrived at when calculating the variable tranche amount to be paid.

29 % of the indicators were vaguely formulated with no quantified targets, using words such as ”improve”, ”increase the focus”, and ”provide evidence”. We observed that contracts in the agriculture sector included the highest number of specific indicators (86.5 %), whereas in the Public Financial Management (PFM), only 60 % of indicators were specific. Box 3 gives examples of non-specific indicators and the implications for the calculation of variable tranche disbursements.

We have also found good examples of where the Commission had made efforts to measure progress for indicators which were not easy to analyse. In the case illustrated in Box 4, the Commission had led the way for the entire donor community in the area of education.

No baselines or incorrect ones on 41 % of progress indicators

Specifically and unambiguously defined indicators are not the only prerequisite for the effective measurement of results. Progress is only measurable if the targets set can be compared with the situation before the intervention: in other words, with a baseline. Amongst the 85 indicators analysed for which baselines were necessary (i.e. for those indicators which analysed progress⁵), we found problems in 35 of them related to: (i) a lack of necessary baseline values (15), and (ii) incorrect or outdated baselines (20).

There were 15 indicators with no identified baselines. These were spread across four (contracts 6, 7, 8 and 17) of the 24 selected budget support contracts. The main reason for the lack of baselines was that the partner countries had not monitored the situation before the start of the budget support contract. However, we found examples where, in a similar situation, the Commission had used the first variable tranche of the budget support contract to ask the partner country to calculate the baseline. In other cases, the Commission had carried out the necessary work (i.e. surveys) to calculate the missing baselines. We consider these to be good practices. The example of necessary work carried out by the Commission to accurately assess the progress achieved for the relevant indicator in subsequent tranches is described in Box 5.

In addition, 20 indicators had incorrect or outdated baselines. In 11 of these cases, new data relevant for the correct calculation of the baseline became available after the budget support contract had been signed. Although it is possible to amend the related financing agreement in order to reflect the correct baseline, the Commission had not done so in these cases. The use of incorrect or outdated baselines also resulted in indicators having lower targets than the actual baselines. Box 6 contains some examples.

Indicators generally provide the intended incentive effect, but there are too many

A primary objective of budget support is to incentivise partner countries to follow the path of agreed reforms. To this end, the indicators used for variable tranches should require a meaningful effort from the recipient country. The associated targets should also strike the right balance between ambition and ease of achievement⁶.

We found that most variable tranches reviewed had the effect of incentivising partner countries to implement certain aspects of their development strategies. However, the targets used for 11 of the selected indicators were very easy to achieve. This number includes four indicators whose targets were set very low due to the use of incorrect baselines (see paragraphs 31-32). Some examples of indicators with a limited incentive effect are presented in Box 7. Furthermore, for 12 additional indicators involving three contracts in Moldova (contract 16), Bolivia (contract 10) and Pakistan (contract 23), the targets were mostly achieved with support provided by technical assistance paid by the EU or other donors. In our view, such indicators have limited ownership and their incentive effect is weak, because they do not require any significant involvement by the partner countries.

The Guidelines suggest that the number of variable tranche indicators should generally range from 3 to 10. Having too many indicators dilutes the incentive effect and makes monitoring more complicated. The audited contracts had between 4 and 34 indicators per tranche⁷. These indicators were further broken down into sub-indicators, often with several targets. For example, the contract to provide support to implement a visa liberalisation action plan in Moldova (contract 15) included 95 independent targets. Such a high number of indicators and targets is not conducive to a focus on the main policy objectives of the budget support contracts. Although problems linked to the high number of indicators were acknowledged in internal Commission documents, it did not influence the formulation of contracts (see Box 8).

The quality of the Commission's verification of indicator fulfilment varied, leading to some payments being insufficiently justified

The disbursement requests submitted by the recipient countries contain an analysis of progress in the sectors supported by their budget support contracts and information on the fulfilment of agreed conditions and performance indicators. The dossier usually includes a report on progress in a given field (sector), separate reports on fulfilment of every condition and fiches concerning each indicator, with supporting evidence (e.g. letters from a statistical office, reports on surveys etc.). See Figure 3 describing the budget support disbursement process.

Partner countries’ requests are based on data from their own monitoring and evaluation (M&E) systems. These systems are therefore a primary source of information for disbursement decisions. However, countries have varying capacities and systems in place for collecting, storing, analysing and using data. Consequently, before starting budget support operations, the Commission needs to assess the systems in place to produce performance indicators that will be used for subsequent variable tranche payments. In particular, EU Delegations must determine whether weaknesses in statistical systems, availability of data and policy analysis significantly undermine the validity of countries’ disbursement requests.

For the contracts reviewed, we examined whether, when selecting the performance indicators for the variable tranches, the Commission had soundly assessed the reliability of the performance data. We also reviewed whether the Commission had performed a thorough examination of the variable tranche disbursement requests and correctly calculated the amount to be disbursed.

The Commission did not conclude on the countries’ capacity to produce data needed for indicators

The EU Delegations are asked to provide an overview of the partner country’s monitoring and evaluation systems (for the country in general and for the particular sector for SRPC), assessing whether the country’s public policy is in line with the EU objectives and whether institutional capacity is considered sufficient to implement the policy. Furthermore, the Guidelines require the reliability and availability of data to be analysed, and the weaknesses of the statistical systems to be appraised, before budget support contracts are drawn up.

We analysed the Commission’s assessments using criteria established by the European Statistical Office (Eurostat). Eurostat has developed a tool called Snapshot to help EU Delegations assess the strengths and weaknesses of national statistical systems in developing countries. We found that the Commission’s budget support guidelines cover the key requirements of this tool. Nevertheless, Snapshot is more comprehensive and provides detailed explanations of how to measure particular areas⁸, which are relevant for the assessment of statistical systems. This tool is generally unknown to the staff in the Delegations.

Our review of the Commission’s assessment of monitoring and evaluation systems showed that, in practice, the Commission generally describes and assesses the elements mentioned in the Guidelines, but these elements are scattered around several documents. However, even though certain sector-specific weaknesses were mentioned in the majority of contracts (18 out of 24 contracts), only in the five contracts in Jordan did the Commission explicitly draw conclusions as to the reliability of the performance data needed to calculate the variable tranche indicators⁹. Drawing a conclusion on the reliability of data is important for indicator selection, monitoring and disbursement analysis.

The Commission verified the reliability of the data supporting the disbursement requests, but not always thoroughly

The EU Delegations must analyse the disbursement requests presented by partner countries (including the values of the related performance indicators) before disbursing the variable tranche payment. We found a variety of verification procedures, some of which did not provide the necessary assurance to justify the subsequent payments.

Overall, the Commission applied three different types of verification procedures: (i) desk review and arithmetical re-calculation of the indicator values using the data provided by partner countries, (ii) the same method complemented by field visits to verify the reliability of the data provided, and (iii) outsourcing the calculation of performance indicators or the verification of data provided by the partner country to external experts.

The EU Delegation officials carried out field visits to complement the analysis and verify some of the data provided in 14 of the 24 audited contracts (see Box 9), and hired, sometimes in addition to their own field visits, external experts to analyse the fulfilment of the conditions for the variable tranches in 16 of the contracts. In the neighbourhood countries the use of experts was systematic; while in the remaining countries audited, the Delegations used experts in only 5 out of 13 contracts.

The budget support guidelines recommend that experts be called in to assist verification exercises where serious doubts exist about the quality of the data provided. However, without any clear conclusions about the partner countries’ capacity to produce such data (see paragraphs 39-41), it is difficult to decide whether such expert assistance is really needed. For the contracts audited, the average cost of expert verification missions was approximately €110 000.

We found that for 6¹⁰ of the 24 selected contracts, the Commission did not perform additional verification on the data used as basis for the payment of the variable tranches effectively. This was mostly because the EU Delegations relied fully on the accuracy of the performance data provided by partner countries, or on the verification work carried out by external experts on behalf of the Commission, without any further verification of the data provided. In these cases, it is more difficult for the Commission to identify unreliable performance data in the disbursement requests, thereby increasing the risk of unjustified variable tranche payments. Box 10 presents an example of shortcomings in an external review, which were not detected by the Commission.

For some variable tranches, the performance data provided by the partner countries did not justify the payments made

Out of a total of €234 million variable tranche payments reviewed, for five out of eight countries, our assessment of the evidence supporting the fulfilment of variable tranche indicators shows different results from those accepted by the Commission, amounting to €13.3 million, in which one country (Pakistan) represented a 19 % difference to the variable tranche payments. Furthermore, we also found that the Commission disbursed €3.4 million to two countries based on indicators linked to incorrectly set baselines. Although there was a contractual obligation to pay, as the target value had been met, no actual progress in the area measured with the indicator had been achieved. Table 1 summarises these amounts for each country visited.

Table 1 – Discrepancies from the Commission’s assessments

Source: ECA.

The main issues we found related to: (i) targets achieved with delay after the deadlines set; (ii) measurement of results based on incorrectly set baselines; (iii) incorrect or insufficient evidence provided to justify the fulfilment of the indicators; and (iv) targets not achieved at all. Figure 6 shows the types and proportion of discrepancies found from the Commission’s assessments.

Source: ECA.

One of the cases where we considered that incorrect evidence was provided by the partner countries is described in Box 11.

In four contracts we found five cases of indicators¹¹ whose targets had been achieved after the deadline set in the financial agreements. Furthermore, for five additional indicators¹², we consider that partner countries did not meet the targets; however, the Commission deemed them to have been fulfilled. Box 12 provides examples of both cases.

We found three cases where the Commission had made disbursements in accordance with the provisions of the financial agreements, since the partner countries had achieved the agreed targets. However, due to the use of incorrectly set baseline or target values, there was no actual progress in the targeted sectors. Box 13 provides further details on some of these cases.

In addition to the discrepancies quantified in Table 1, we could not confirm the correctness of two variable tranche payments due to the lack of sufficient evidence to support the values reported for five indicators¹³ (see examples Box 14). These indicators represent a disbursed amount of €3.77 million.

Insufficiently documented payments to Moldova

Although not linked directly to the verification of performance indicators we found that the Commission had paid three variable tranches to Moldova, a total amount of €26.3 million, without sufficiently documenting the reasons supporting these payments (see Box 15).

Conclusions and recommendations

We examined whether the Commission used relevant and reliable performance data for disbursing budget support variable tranches. We conclude that for one third of the performance indicators reviewed their relevance was weakened by their design, and they do not allow results to be measured objectively, thereby undermining their relevance. Furthermore, the Commission’s assessments of whether variable tranche indicators had been met was not always reliable, leading to some insufficiently justified payments.

We found that the variable tranche performance indicators were well aligned to the partner countries’ sector development strategies. However, most of them were focused on short-term actions rather than longer-term results, including progress towards the sustainable development goals. Only 13 % of the 248 indicators we reviewed measured outcomes or impacts in the supported sectors (see paragraphs 22 to 25). The use of outcome indicators would enable the Commission to better measure longer-term results in the supported sectors, including progress towards achieving the Sustainable Development Goals.

The Commission should increase the proportion of variable tranches that are paid based on the achievement of outcome indicators.

Timeframe: end of 2021

More than one third of the indicators were vaguely defined or had incorrect baselines, or none at all. This allowed for different interpretations as to whether targets had been achieved, making the analysis of the disbursement requests more complex and less objective (see paragraphs 26 to 32).

The Commission should enhance the quality control arrangements in order to ensure that performance indicators of variable tranches measure the results achieved by partner countries in an objective manner. Particular attention should be paid to:

1. Using performance indicators that are specific and do not allow different interpretations;
2. Using baseline values and targets

Timeframe: end of 2021

An important objective of budget support variable tranches is to produce incentives for partner countries to advance in their reform agendas. Most variable tranches reviewed contained indicators whose targets struck the right balance between being ambitious and achievable, and we consider that they provided the intended incentive effect. The few exceptions we found concerned indicators with targets, which were easily achievable, often due to the use of incorrect baselines, or were achieved via the work carried out by other donors or by external experts paid by the EU (see paragraphs 33 to 34).

The Commission should:

1. update baseline information prior to contract signature or correct the baseline values during contract implementation if necessary, by amending the budget support contract;
2. avoid situations in which the partner country achieves targets exclusively due to EU-funded technical assistance.

Timeframe: end of 2021

The number of indicators used in each variable tranche was often too high, beyond the number recommended by the Commission’s guidelines. This further complicated the disbursement process (see paragraph 35).

The Commission should: refrain from using sub-indicators in order to limit the actual number of indicators to the maximum described by the guidelines.

Timeframe: end of 2021

Variable tranche disbursement requests contain performance data showing the extent to which the agreed conditions and performance indicators have been fulfilled. Since the requests are prepared by the partner countries, the reliability of the underlying performance data depends on the capacity of these countries to produce it. We found that the Commission generally assessed the capacity of partner countries by examining the main elements of their monitoring and evaluation systems, as required by its budget support guidelines. However, only in 5 of the 24 selected contracts did the Commission draw conclusions as to whether the performance data needed to calculate the variable tranche indicators was reliable (see paragraphs 36 to 41). Drawing a conclusion on the reliability of data is important for indicator selection, monitoring and disbursement analysis.

When designing a budget support operation, the Commission should assess the reliability of the performance data, which is to be used as a basis for the disbursement of a variable tranche. The assessment should arrive at an explicit conclusion as to whether or not the systems used to produce these data are sufficiently reliable, and could be based on existing assessments done by other recognised bodies.

Timeframe: end of 2021

We found that the EU Delegations carried out a variety of verification procedures when analysing disbursement requests. In some cases, the EU Delegations’ staff carried out field visits to verify the data provided by partner countries, while in others, they relied fully on this data, or on the external reviews carried out by experts on behalf of the Commission, without further verification work. This does not provide the necessary assurance to justify subsequent variable tranche payments (see paragraphs 42 to 46).

The Commission should:

1. review the underlying evidence supporting the performance data provided by partner countries in the disbursement request, unless it has already explicitly concluded that this data is reliable;
2. when using external reviews, require in the terms of reference the verification of the reliability of key performance data provided by partner countries. Before disbursing the variable tranche, verify that the experts have complied with this requirement.

Timeframe: end of 2021

When we re-performed the Commission’s assessments regarding the achievement of indicators and the recalculation of the variable tranche payments, we found discrepancies from the amounts paid by the Commission. Overall, based on the performance information available, we estimate that from a total of €234 million of variable tranche payments reviewed, we found discrepancies for €16.7 million. From this amount, €13.3 million were insufficiently justified, or not in line with contract provisions. An amount of €3.4 million was paid without actual progress. Additionally, €26.3 million for three variable tranches were paid to Moldova without sufficiently documenting the reasons supporting these payments (see paragraphs 47 to 53).

This Report was adopted by Chamber III, headed by Ms Bettina JAKOBSEN, Member of the Court of Auditors, in Luxembourg at its meeting of 12 November 2019.

For the Court of Auditors

Klaus-Heiner Lehne
President

Annexes

Annex I – Share of budget support in bilateral ODA commitments

Source: ECA, based on https://stats.oecd.org/

Annex II – Audited contracts

Annex III – Disbursements of variable tranches in 2017 for sector reform performance contracts

Source: ECA (In bold countries selected for the audit).

Annex IV – Summary assessment

NOTE: Y: Yes; N: No; P: Partially.

Audit team

The ECA’s special reports set out the results of its audits of EU policies and programmes, or of management-related topics from specific budgetary areas. The ECA selects and designs these audit tasks to be of maximum impact by considering the risks to performance or compliance, the level of income or spending involved, forthcoming developments and political and public interest.

This performance audit was carried out by Audit Chamber III External action/Security and justice, headed by ECA Member Hannu Takkula, supported by Turo Hentila, Head of Private Office and Helka Nykaenen, Private Office Attaché; Alejandro Ballester Gallardo, Principal Manager; Piotr Zych, Head of Task; Eva Coria Paramas, Roberto Ruiz Ruiz, Erika Söveges and Nita Tennila, Auditors. Richard Moore provided linguistic support.

From left to right: Turo Hentila, Helka Nykaenen, Hannu Takkula, Alejandro Ballester Gallardo, Nita Tennila, Piotr Zych, Erika Söveges, Roberto Ruiz Ruiz.

Endnotes

¹ Article 236 (1) of Financial Regulation applicable to the general budget of the Union; July 2018.

² EC Budget Support – Trends and Results, 2018, p. 61.

³ With the exception of Morocco, is the subject of an ECA Special Report to be published in 2020.

⁴ Budget Support guidelines 2017, p. 138.

⁵ In our sample there are 163 indicators with targets which do not require a baseline, because they do not measure the development of a given variable (e.g. ‘a law approved’ or ‘number of meetings held’). These indicators are not taken into consideration for this assessment.

⁶ Budget support guidelines, 2017, p. 139.

⁷ This number does not include indicators from previous years’ tranches.

⁸ Snapshot provides analysis methods for the following key areas necessary for assessing statistical systems: (i) The legal, institutional and strategic frameworks supporting the production of statistics and monitoring at national and sector level; (ii) The adequacy of resources (i.e. quantity and quality of human resources, equipment, financing); (iii) The determinants of data quality (i.e. quality commitment, professional independence, impartiality, objectivity, methodology and appropriate procedures); and (iv) Relations with users (i.e. relevance, accessibility).

⁹ Contracts 1, 2, 3, 4 and 5.

¹⁰ Contracts number 2, 5, 13, 14, 23 and 24.

¹¹ Indicator 2 of Contract 20 , indicator 1.2 of Contract 6, Indicator 5 of Contract 1 and Indicator 2.1 and 2.2 of Contract 16.

¹² Indicators 5, 6 and 8 of Contract 23, Indicator 6.2 of Contract 24 and Indicator 1.7 of Contract 6.

¹³ Indicators 3 and 6 of Contract 22 and indicators 1.2, and 3 and of Contract 5.

Timeline

Contact

EUROPEAN COURT OF AUDITORS
12, rue Alcide De Gasperi
1615 Luxembourg
LUXEMBOURG

Tel. +352 4398-1
Enquiries: eca.europa.eu/en/Pages/ContactForm.aspx
Website: eca.europa.eu
Twitter: @EUAuditors

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2019

© European Union, 2019.

Reuse is authorised provided the source is acknowledged.
For any use or reproduction of photos or other material that is not under the copyright of the European Union, permission must be sought directly from the copyright holders.

GETTING IN TOUCH WITH THE EU

In person
All over the European Union there are hundreds of Europe Direct Information Centres. You can find the address of the centre nearest you at: https://europa.eu/european-union/contact_en

On the phone or by e-mail
Europe Direct is a service that your questions about the European Union. You can contact this service

• by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls),
• at the following standard number: +32 22999696 or
• by electronic mail via: https://europa.eu/european-union/contact_en

FINDING INFORMATION ABOUT THE EU

Online
Information about the European Union in all the official languages of the EU is available on the Europa website at: https://europa.eu/european-union/index_en

EU Publications
You can download or order free and priced EU publications at: https://op.europa.eu/en/web/general-publications/publications. Multiple copies of free publications may be obtained by contacting Europe Direct or your local information centre (see https://europa.eu/european-union/contact_en)

EU law and related documents
For access to legal information from the EU, including all EU law since 1952 in all the official language versions, go to EUR-Lex at: http://eur-lex.europa.eu/homepage.html?locale=en

Open data from the EU
The EU Open Data Portal (http://data.europa.eu/euodp/en/data) provides access to datasets from the EU. Data can be downloaded and reused for free, both for commercial and non-commercial purposes.